From my new article Personality Traits of the Best Software Developers:
“I come from the world of corporate software development. It may not be the most glamorous side of software (it’s nowhere near as interesting as shrinkwrapstartups or those fancy-dancy Web 2.0 companies that show up in your browser every time you mistype a domain name), but it’s stable, pays well, and has its own set of challenges that other types of software development know nothing about.
For example, when was the last time someone working on the next version of Halo spent three weeks trying to gather people from accounting, marketing, product management, and their call center in order to nail down requirements that would likely change in 2 months once they’ve delivered the software?
Or when was the last time someone at 37Signals sat through back to back weeks of JAD sessions?
In this world of corporate development I’ve known a few phenomenal developers. I’m talking about those A+ people whom you would quit your job for to go start a company. And the the more I looked at what makes them so good, the more I realized they all share a handful of personality traits.”
I come from the world of corporate software development. It may not be the most glamorous side of software (it’s nowhere near as interesting as shrinkwrapstartups or those fancy-dancy Web 2.0 companies that show up in your browser every time you mistype a domain name), but it’s stable, pays well, and has its own set of challenges that other types of software development know nothing about.
For example, when was the last time someone working on the next version of Halo spent three weeks trying to gather people from accounting, marketing, product management, and their call center in order to nail down requirements that would likely change in 2 months once they’ve delivered the software?
Or when was the last time someone at 37Signals sat through back to back weeks of JAD sessions?
In this world of corporate development I’ve known a few phenomenal developers. I’m talking about those A+ people whom you would quit your job for to go start a company. And the the more I looked at what makes them so good, the more I realized they all share a handful of personality traits. Well, not exactly a handful, more like four.
Pessimistic
Admiral Jim Stockdale was the highest ranking US military officer imprisoned in Vietnam. He was held in the “Hanoi Hilton” and repeatedly tortured over 8 years. Stockdale told Jim Collins, author of Good to Great, “You must never confuse faith that you will prevail in the end, which you can never afford to lose, with the discipline to confront the most brutal facts of your current reality, whatever they might be.”
After his release, Stockdale became the first three-star officer in the history of the navy to wear both aviator wings and the Congressional Medal of Honor.
Stockdale was a pessimist in the short-term because he faced the brutal facts of his reality, but was an optimist in the long-term because of his confidence that he would prevail in the end.
No one anticipates a catastrophic system failure by looking on the bright side. The best developers I know are experts at finding points of failure. You’ll often hear them quipping “What could possibly go wrong?” after someone makes a suggestion to handle a critical data transfer via nightly FTP over a dial-up connection. The best developers anticipate headaches that other developers never think of, and do everything within their power to avoid them.
On the flip side, great developers are optimistic, even downright confident, about their overall success. They know that by being a pessimist in the short-term, their long-term success is ensured. Just like Jim Stockdale, they realize that by confronting the brutal facts of their current reality they will prevail in the end.
Angered By Sloppy Code
Paul Graham nailed it when he said “…people who are great at something are not so much convinced of their own greatness as mystified at why everyone else seems so incompetent.”
The worst nightmare for a great developer is to see someone else’s software gasping for air while bringing the rest of the system to its knees. It’s downright infuriating. And this isn’t limited to code; it can be bad installation packages, sloppy deployments, or a misspelled column name.
Due to the life and death nature of their products, NASA designs zero-defect software systems using a process that has nearly eliminated the possibility for human error. They’ve added layer after layer of checks and balances that have resulted from years of finding mistakes and figuring out the best way to eliminate them. NASA is the poster child for discovering the source of a mistake and modifying their process to eliminate the possibility of that mistake ever happening again. And it works. A quote from this Fast Company article on NASA’s development process says
“What makes it remarkable is how well the software works. This software never crashes. It never needs to be re-booted. This software is bug-free. It is perfect, as perfect as human beings have achieved. Consider these stats: the last three versions of the program — each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors.”
I’m not saying we have to develop to this standard, but NASA knows how to find and fix bugs, and the way they do it is to find the source of every problem.
Someone who fixes a problem but doesn’t take the time to find out what caused it is doomed to never become an expert in their field. Experience is not years on the job, it’s learning to recognize a problem before it occurs, which can only be done by knowing what causes it in the first place.
Developers who don’t take the time to find the source often create sloppy solutions. For hundreds of examples of sloppy solutions visit The Daily WTF. Here are a few I’ve seen in my career:
An assembly is deleted from a server each time it’s rebooted. You could create a custom script to re-copy that assembly to the server after each reboot, or find out why the assembly is being deleted in the first place.
An image-manipulation script is hogging processor power for minutes at a time when it should run in under 10 seconds. You could make the script run at 2am when no one will notice, or you can take the time to step through the code and figure out where the real problem is.
A shared XML file is being locked by a process, causing other processes to fail when they try to open it. You could make several copies of the XML file so each process has its own, or you could troubleshoot the file access code to find out why it’s locking the file.
And on and on…
Long Term Life Planners
This one was a little puzzling for the longest time, but I think I’ve finally put it together.
People who think many years down the road in their personal life have the gift to think down the road during development. Being able to see the impacts of present-day decisions is paramount to building great software. The best developers I know have stable family lives, save for retirement, own their own home, and eat an apple a day (ok, maybe not that last one). People who have spastic home-lives and live paycheck to paycheck can certainly be good developers, but what they lack in life they tend to lack in the office: the ability to be disciplined, and to develop and adhere to a long-term plan.
Attention to Detail
I’ve known smart developers who don’t pay attention to detail. The result is misspelled database columns, uncommented code, projects that aren’t checked into source control, software that’s not unit tested, unimplemented features, and so on. All of these can be easily dealt with if you’re building a Google mash-up or a five page website. But in corporate development each of these screw-ups is a death knell.
So I’ll say it very loud, but I promise I’ll only say it once:
I have never, ever, ever seen a great software developer who does not have amazing attention to detail.
I worked with a programmer back in school who forced anyone working with him to indent using two spaces instead of tabs. If you gave him code that didn’t use two spaces he would go through it line-by-line and replace your tabs with his spaces. While the value of tabs is not even a question, (I’ve long-chided him for this anal behavior) his attention to such a small detail has served him well in his many years designing chips at Intel.
So There You Have It
The next time you’re interviewing a potential developer, determine if she has the four personality traits I’ve listed above. Here are a few methods I’ve found useful:
Ask if they’re an optimist or a pessimist
Ask about a time when they found the source of a problem
Find out if they save for retirement (you can work this in during discussions of your company’s retirement plan)
Make an obvious misspelling in a short code sample and ask if they see anything wrong
We know from Facts and Fallacies of Software Engineering that the best programmers are up to 28 times better than the worst programmers, making them the best bargains in software. Take these four traits and go find a bargain (or better yet, make yourself into one).
I received a ton of comments and emails about my most recent article Timeline and Risk, which is my most popular article to date. Check out the comments if you haven’t already - a very interesting pattern emerged of people suggesting that I quit. I’ll come back to that point in a future post because I don’t have time to write about it right now, but let’s just say I don’t consider that a feasible solution.
I had a good conversation with one of our senior managers about the article. He agreed with my thoughts on the subject, but interpreted the other manager differently than I had. It was a good chat and I appreciated that he took the time to discuss it with me.
On another note, I apologize for not posting much during the past few weeks. I’ve been working crazy hours trying to wrap up a major project that goes to QA in just over a week.
As counterpoint to that, I’m leaving for Maui in the middle of the week, so I’ll be incommunicado for a few days. I’ll be sure to do some snorkeling for you.
From my new article Timeline and Risk: How to Piss Off Your Software Developers:
“And that’s why telling someone to build higher quality software in the same amount of time is insulting, because you’re essentially telling them they’re not giving it their best effort. That they’re screwing off.
Today we’re going to focus on a sentence that will make almost any developer, even a mild-mannered one, want to box up their red Swingline and head to Tahiti for early retirement.
I believe that software developers strive to do the best work they can. Most programmers will fight tooth and nail, work evenings and weekends, and scale Mt. Everest on their hands to make sure they don’t let their team down. I am, of course, speaking of teams that don’t have morale problems. While I acknowledge that they exist, they’re not the focus of this article.
In the context of a high-functioning team, some developers have the innate gift of self-motivation and strive for their best at all times. For others it’s the team dynamic that keeps their hands on the keyboard. As a developer learns to rely on his teammates and trust relationships form, they make it difficult to coast for fear of letting the group down and the thought of being dubbed the weakest link. This amalgam of competition and camaraderie is what helped companies like Google, Apple and Microsoft dominate their markets and convince their employees to work 90 hours a week and love it.
But getting back to the point: what’s the one sentence that makes software developers angry?
You might be thinking it’s the insidious:
“This bug is only reproducible on Opera v0.3 for the Commodore 64, but fix it anyway,” or the always favorite
“We don’t have a design yet, but start building something because our deadline isn’t moving”.
Nope on both counts.
This sentence I speak of is the lesser known:
“Build higher quality software in the same amount of time.”
That’s it…only ten words, fewer if you overlook some grammar rules. So why is this sentence so potent?
On its own merits, the phrase build higher quality software in the same amount of time will not incite your software developers to riot. But let’s examine the underlying message. The Triangle
Quality and speed are two of the three forces in the famous three-sided triangle of quality, speed and cost. It’s a fairly well-known business principle that you can choose two of the three sides of this triangle; anyone who tells you otherwise is trying to sell you something. As an aside, if you buy what they’re selling, send me an email ‘cuz I have some Beanie Babies I’m looking to get rid of.
There are many definitions of “quality,” but in this article I’m speaking of software that is well-designed and well-written so as to ensure high availability (high uptime) and make it easy to maintain and extend.
With that in mind, let’s look at what happens if you keep cost the same, meaning you don’t hire more developers, but want to develop higher quality software.
You could ask your developers to work more hours. This would likely work temporarily, at least, since more time usually translates into increased productivity. It’s debatable whether someone performing work as concentration-intensive as programming is able to function well after a solid 8+ hours, but I’ll give you the benefit of the doubt on this one. Even so, studies have shown that “continuing scheduled overtime has a strong negative effect on productivity.” So for the short term we can bring out the whip and expect a slight jump in productivity, but assuming we want something that lasts longer than a few months and don’t want to experience a McDonalds-esque churn rate (because that would inevitably increase our cost and decrease the quality of our software), let’s throw this option out the window.
Another option would be to hire better people. This has a decent probability of increasing quality over time, but it’s not a solution one can implement overnight. With a team of 20 developers you’re probably looking at nine to twelve months to clean out less-productive developers and fill it with solid talent, assuming you have a few less-productive members on your team. However, if your development team is already a group of “coding machines,” or if your goal is to implement something immediately then another avenue must be explored, even if you decide to pursue this one for the long-term. Training and/or mentoring is a variation of this option with similar advantages, but a similar time-horizon.
Another alternative would be to simply make your people work harder.
Hmmm…this is an interesting one. If you buy what I said at the start of this article, that most developers are already working at or close to their best due to personal or social reasons, then this hypothesis doesn’t quite pan out. How can someone who’s already working at a high level of productivity all of a sudden work harder? Making someone work harder simply by asking them to is…well…hard to understand.
And that’s why telling someone to build higher quality software in the same amount of time is insulting, because you’re essentially telling them they’re not giving it their best effort. That they’re screwing off.
And that makes people mad.
How do I know this? Because someone said it to me the other day. Not just me, in fact, our entire development team.
Better Software
A non-technical manager told us that we have to build better software in the same amount of time, and even included the phrase “No excuses.”
Needless to say morale has seen a slight decline in the days following as developers express their feelings of defensiveness. People who’ve been giving it their best suddenly feel taken for granted and frustrated, especially the ones who’ve been working extra hours, solving project issues in their spare time, and running performance tests on the weekend. When you’re already working hard it’s difficult to hear someone tell you to work harder.
The talk was prompted by a recent issue where some high-availability systems were down for the better part of a day. These systems were written over the past 4+ years under perilous death-march schedules, with little QA because we were in a hurry to get them out the door. When they went down people panicked, and rightfully so, since many of our customers were foaming at the mouth.
As a result management expressed their concern with the situation, indicating that something had to be done as we move forward. As I was listening I understood “Make better software; reduce risk of failure.” As most developers would agree, this is a great approach, and one that several members of our team have been advocating for for quite some time. I became excited at the thought of ensuring we have ample QA and including more safeguards in our code. But my parade soon ended in a blizzard when I heard “I don’t want to hear excuses about how things are going to take longer. We can get things done fast AND better.” And that’s when I realized I needed to write this article.
But wait, there’s a little more.
A comparison was made between our software and the software of a large financial organization (think Wells Fargo or Washington Mutual). If you’re thinking “I can’t remember the last time one of those guys was down for an extended period of time,” you’re right; they typically aren’t down, and when they are it’s not for very long. But an interesting thing I’ve learned in the past few years is that the software development cycle for these guys is loooooong, to the tune of 2-3 times what our timelines run.
In other words, they’ve decided to take the “time” side of their pyramid and make it really big in exchange for a really big “quality” side. And good for them, because they’ve obviously made a wise choice, as indicated by their availability.
Now that we’re faced with the same dilemma it’s time to pay the piper and admit we need to lower our risk while at the same time admitting to ourselves the sacrifice that it will ultimately require: time.
The Moral
There are at least two morals to this story
The first is that you will eventually reap what you sow, so don’t compromise your design. Fight tooth and nail and push out timelines if you have to. This is not going to be easy, so good luck. Let me know if you discover any secrets because I would love to hear them.
The other is that reducing risk takes time. At this point I should add that it’s our responsibility as technical folk to educate those who don’t know this. A concept that appears so clear to us may not be apparent to a person outside if the IT department. We’re the only people that truly understand the complexity of the systems we build and the ramifications of spending too little time on design or QA.
Reducing risk is a lot like having insurance - if you elect to have it you have to pay for it.
Send this article to a manager or co-worker who’s tried to convince you otherwise.
I don’t want this to be an indictment of all recruiters - only the bad ones.
There has to be a better way.
In the past 9 months I’ve had more awful experiences with technical recruiters than with any other category of professionals in my entire working life. Being badgered on the phone, given the hard-sell, and being provided with un-screened candidates I could have found for free on Craigslist or Dice is not my idea of good customer service - nor my idea of a sustainable business model.
In fact, it’s gotten so bad that recruiters have become a joke around our office. Whenever someone receives a call from a pushy recruiter our ears perk up to hear how our comrade will be badgered, abused, and finally hang up the phone angrily because the person on the other end won’t stop talking long enough to listen to what we have to say. You’d think we were buying timeshares.
With commissions hovering around $20,000 a pop, it’s only a matter of time before someone comes along who’s respectful, listens to our needs, and does a reasonable job of screening candidates. When they do they’ll quickly wind up with more business than they can handle.
Here are the absolute basics of technical recruiting:
Don’t Have 19-year olds Making Cold-Calls. It wastes my time and gives your firm a bad name. The only thing worse than cold-calls is cold-calls by someone who doesn’t listen, interrupts you, and has no idea what they’re talking about.
No Means No. Let’s face it; the hard-sell went out with junk bonds and hula hoops. I hate it, you hate it, so why do we have to deal with it? Stop reading crappy sales manuals from 1988 and get with the times. If I say “no” and you don’t treat me with respect you’re going to be on the receiving end of a dial-tone.
Learn the Technology. You don’t have to become an expert, but be able to discuss it intelligently. Know the difference between Java and JavaScript, ASP and ASP.NET, UML and XML.
Listen. Has anyone else encountered the Micro-Machines guy? I’ve talked to several recruiters who rattle things off like auctioneers. I realize you’re busy…hey, we’ve all got things to do. But slow down, act like you care, and realize that just because you talk fast doesn’t mean that you shouldn’t take the time to listen.
Screen Your Candidates. Does this really need to be on a list? I’ve received resumes with grammar so poor I could barely read them. I’ve done phone interviews with people who couldn’t tell SOAP from shampoo . Time and time again we’ve been asked for screening questions for common programmer skill sets. We’re paying upwards of $20k per placement…shouldn’t this be the recruiter’s job? If you have to hire a developer on an hourly basis to perform resume or phone screenings for you, do it. Just please stop passing the buck to us.
We’ve only found two recruiters in Los Angeles that are consistently respectful, never give us the hard-sell, and actually review resumes. Even though they don’t perform thorough technical screenings we give them our business because they are the best we’ve found.
How long before someone comes along and really starts servicing this big-money niche?
From my new article Software Training Sucks: Why We Need to Roll it Back 1,000 Years:
“This article is about training, but not the typical ‘pay $1500 to sit in a stuffy hotel conference room for five days while some guy who hasn’t written software since Elvis reads his PowerPoint slides to you in an effort to keep you awake’ training.
We’re going to talk about in-your-face, do-as-I-do, on-the-job, get-them-writing-production-code-four-weeks-out-of-school training. It’s the type that makes people clamor to work at your company, and requires a whole heck of a lot more effort than the fifteen hundred bucks to send someone to a week in a hotel conference room.”
This article is about training, but not the typical “pay $1500 to sit in a stuffy hotel conference room for five days while some guy who hasn’t written software since Elvis reads his PowerPoint slides to you in an effort to keep you awake” training.
We’re going to talk about in-your-face, do-as-I-do, on-the-job, get-them-writing-production-code-four-weeks-out-of-school training. It’s the type that makes people clamor to work at your company, and requires a whole heck of a lot more effort than the fifteen hundred bucks to send someone to a week in a hotel conference room.
Every company that writes software needs to make training a top priority. Say that with me: every company that writes software needs to make training a top priority. Why? If word gets around that your company has a killer training program you’ll be beating candidates away with sticks (at least until the first lawsuit), it boosts your bottom line by making programmers better equipped to do their jobs, and gives people a sense that the company cares about their long term success, making them less likely to leave. As a nice side effect, learning new skills makes people more satisfied with their jobs.
So to review, well-executed training attracts developers, retains developers, and makes them happy. Say it one more time: every company that writes software needs to make training a top priority. Developer Traits
Before we delve into the structure of an effective training program, let’s talk about the specific areas we can hope to improve. The two fundamental traits of a developer are: knowledge of theory and experience.
Theory
Theory is what we learn in school: little-o and big-O notation, finite state machines, 3rd normal form, etc… Theory is easy to discuss, but difficult to experience since it’s like a wispy cloud floating around in someone’s mind.
Most developers learn theory in school on the road to obtaining their degree in Computer Science or a related field. Many learn it from books, magazines, and developer websites. Theory is things like object-oriented concepts, database design and normalization, and many aspects of software architecture.
Someone who knows a lot of theory and has a little experience tends to sound more knowledgeable than they are. Their mind is mature in the ways of software, but their code is juvenile by comparison. It tends to be over-designed, overly complex, brittle, and lacking in practical areas such as error handling.
Until you’ve debugged an application that lacks error handling it doesn’t occur to most people that storing this information might be helpful. That’s where experience comes in.
Experience
Experience is about making mistakes. When you study theory there are no mistakes: code is bug-free, APIs are perfect, and operating systems never, ever crash. That’s why there’s no “blue screen” shape in UML.
For a programmer, the more mistakes you make the better you are at writing code. I’ve made mistakes on every project I’ve ever developed, and each mistake added one more notch to my belt. Experience can only be gained through sitting down and writing code; there are no exceptions.
Someone with a lot of experience and a little theory tends to write code that runs well but is difficult to maintain and extend. Since the point of software design (which relies heavily on theory) is to organize a complicated system into something extensible and easily-understood, a solution lacking in design tends to want in these two areas. The larger the system, the worse the problem becomes.
Apprenticeship
It’s a sad day when I cringe at the word “apprentice.” Let me say this up front: I’m not referring to Donald Trump’s TV show. Before the show became popular, “apprentice” described someone who works under the oversight of a mentor. Apprenticeship is about learning while doing.
Apprenticeship was born in the 14th century, and with it came a revolution. Suddenly young men and women could trade their most abundant asset, time, for a truly scarce resource, training. Skilled tradesman, including blacksmiths, cobblers and silk weavers, mentored young men and women with the promise that they might one day take over the business, while in exchange the tradesman lowered their labor costs by having low-cost apprentices on staff.
Apprentices were young men or women, typically 14 to 21. They didn’t make much money, but they learned their trades on the job. A blacksmith apprentice, for instance, learned about building super-hot fires, combining different metals to create mixtures with varying melting points, and how much work it takes to make a rake, a knife, or a plow. Most of it was not something they could learn from in school; they needed years of first-hand experience in order to master the trade.
Fast-forward 700 years and apprenticeship has disappeared. No, wait a minute…it hasn’t disappeared. This ancient tradition still appears everywhere you look, from construction to medical school to priesthood. So why is it that this model, which has certainly passed the test of time, is rarely present in the world of software development?
Is it because you learn software development better from books rather than from doing? Nope.
Is it because programming is a solitary trade, while running electrical wire requires a team of people? Wrong again.
How about the ol’ “physical trade versus thinking trade” argument? Not that one, either.
The only reason I can come up with is that computer programming is a new trade, only coming of age in the last 40 or 50 years, while the others: doctors, priests, construction workers, have all been around for thousands of years. And even though they’ve adapted over time, they remain steeped in the rich tradition of apprenticeship; a tradition that computer programming, a wee lad by comparison, has yet to accept.
Some would argue that apprenticeship is present in software development, and I have to agree to some point. I’ve learned a ton from the experienced developers and architects I’ve worked with (thanks Rick, Jeremy, and Calvin), but it’s always been on an informal basis, through osmosis, and only because they were willing to teach and I was willing to learn. From management’s perspective, my mentoring has typically consisted of: “You know .NET? Here’s a spec; build it.” That’s what needs to change.
There is a better way. Training Done Right
You can probably guess where this is headed by now, so let me cut to the chase: the better way to train is through software apprenticeship. All you need are a senior and a junior developer. Ask if they’d like to participate in an apprenticeship relationship. The junior gets training, the senior gets to pass along his knowledge (most people are flattered if you ask them to be a mentor). The key here is to make a deliberate choice to pair two developers together and get their buy-in. No buy-in, no apprenticeship.
What should their interaction consist of?
Use the time-tested approach of trades that have been doing it for years. Let’s take an electrical apprenticeship as an example: in the United States today, the International Brotherhood of Electrical Workers (I.B.E.W.) trains thousands of electricians every year. They learn through two distinct experiences:
Attending night school during the week to learn the theory of electricity.
Working days on a construction site where they’re able to gain experience applying the theory to the hands-on construction of a building
His first day on the job an apprentice is paired up with a journeyman (an experienced electrician), who shows him the ropes. The journeyman typically talks the apprentice through a task, demonstrates the task, has the apprentice perform the task, then gives feedback. Listen, watch, do, review.
With software it looks like this: the mentor evaluates the task at hand, be it writing data access code or building a web-based user interface, and holds a white-board discussion with the apprentice (listen). Next, the mentor might write sample code demonstrating a particularly difficult or confusing concept (watch). At this point the mentor sends the mentee off to gain their own experience writing code (do). And finally, the mentor should review the code, providing positive and negative feedback and suggesting improvements (review). Listen, watch, do, review.
A final note: the key to any type of apprenticeship is the “do” step. Most software training gives you the listen and watch, but the “do and review” is what inspires growth and advances skills. The beauty of apprenticeship is that it tackles theory and experience in one fell swoop. And it’s easier than you think.
The Bottom Line
Training is critical to any company that writes software, and apprenticeship is the best way to bring new developers on board, make them feel at home, improve their skills, and keep them happy and growing. You’ll keep experienced developers in touch with new approaches, compliment them by asking them to share their wealth of knowledge, and hopefully create a few friendships along the way.
One developer was fired for eating pizza, another for using the word “Stupid” in an error message. Read the complete list of preposterous firings (and see who won a Caribbean cruise for the best story) at Simply Fired.
