Daily Abstraction

If you ever find yourself deliberately using a vague subject for an email because you feel the real subject is sensitive information that you may not want to be seen by casual overlookers, then it may be that you're doing something ethically questionable. Especially if you're the boss. Perhaps you should rethink the entire plan, rather than just trying to keep it a secret.

Readers sensitive to irony will realize that this posting is itself deliberately vague. I'm protecting the guilty.

I just discovered that there's a name for my role at StreamBase: Program Manager. Here's a couple descriptions of what program management is like at Microsoft, in case you aren't familiar with the title. Choice quote:

one way to describe PMs is that they not only "pick up and run with the ball, they go find the ball". That really defines the difference between "knowing what to do and doing it", and "not knowing what to do, but using your own wits to decide what to do, then doing it". That means as a PM you are constantly strategizing and rethinking what is going on to find out if there is something you are missing, or the team is missing. You’re also constantly deciding what is important to do, and whether action needs to be taken. The number of such decisions is staggering. I might make 50 a day, sometimes more than 100 or even 200.

Here's a beautiful simile, courtesy of oditogre:

The programmers in a software company are like the trees in an orchard. Not only do they produce the product that you sell, but they can only be replaced by the same or a similar class. You can't hire just any ol' plant off the street, you have to get a certain kind, or you have to invest lots of time and money in splicing (training). Also, they're finnicky. They need just so much nourishment, so much space, etc. And here's one really important point: If you screw up and lose a bunch of them, they -can- be replaced, but it's going to take a while to get the replacements to the point where they are as productive as the original, and in the meantime not only will you not be making the money you otherwise would have made off the original, but you will be losing extra money getting the new one up to speed.

How much should a really top-notch programmer get paid? There are many ways to approach this question.

First, note that the question is not "how much is a really top-notch programmer worth?" The best programmers are worth at least twice as much as a good programmer, and a good programmer is worth at least twice as much as a mediocre programmer. As for people worse than mediocre, in my opinion they probably don't deserve a salary at all. So, the best programmers are worth at least four times as much as the mediocre programmers. If you assume a mediocre programmer makes at least $60-80k, the best programmers should be averaging much more than a quarter million dollars a year.

I've heard that some programmers do get paid in that range, but only if they work for Wall Street. Why? Does Wall Street have so much extra cash floating around that it's easier to overpay their employees than have to deal with salary negotiations? Or is it because Wall Street, being extremely profit focused, simply realizes that it's worth it to spend that much money? Even though I have absolutely no experience working for a hedge fund, I'm willing to bet it's the latter.

The biggest mistake I've seen people make when taking a new software development job is to stay quiet when they really should be asking a ton of questions. This applies to people fresh out of college just as much as it does to people with decades of experience.

Many companies like to give their new developers some kind of maintenance task -- perhaps cleaning up some code, or adding a relatively small feature to an existing code base. This gives the new person the chance to get familiar with the existing code and development practices, without risking any of the more important projects.

When this happens, some people (especially people with very little experience) think that their goal should be to produce bug-free code while asking as for as little help as possible. I think this is a bad habit that many people develop while in school. Maybe you think that by not asking questions, you're giving the impression that you're already competent at this. Maybe you think that the people with more experience than you are working on more important tasks, and you don't want to bother them or waste their time.

That's not the way it works in industry. We expect people to work together to get stuff done quickly. In school if you tried to help someone, it was usually called "cheating". In industry, it's called being a good member of the team.

Here's my definition of someone who's good at what they do. I think of this as a minimal definition, the least that I'm willing to accept.

You can give them a month-long project, and trust them to do a better than average job, in a shorter than average amount of time, without requiring their work to be reviewed by someone who's better than they are.

This rules out people who are still learning, as well as people who should still be learning even though they think they're long past that stage. It also rules out people who solve a problem exactly the way they were told to, even though the real challenge could have been addressed in a simpler, or more general way. Finally, it rules out people who are sloppy, and who need someone to inspect the end result just to make sure they didn't leave something unfinished.

Now, nobody is going to be perfect, and everyone can benefit from hearing someone else's perspective, and all projects will end up with aspects that could have been done better but weren't thought of until afterwards. I'm not saying that good software developers don't need qa, or that good sales engineers won't occasionally misqualify a lead, or that good project managers never end up with projects that get out of control. What I'm saying is that a good person is someone you can trust to do a good job, and maintain an above-average standard. Someone who doesn't need babysitting.

It is critical to try as hard as you can to only hire good people. If you don't, you'll waste a ton of money. First you'll need to get someone better than them to review their plan before the even begin. Second you'll need someone better than them to review the final result, and point out what needs to be redone. Third you'll need someone to go back and actually make those changes. Fourth, you'll need to make sure the changes are communicated to everyone else who was involved in the project (management, qa, doc, dev, etc), and make sure any knock-on changes are taken care of. In total, you'll probably end up spending at least twice as much effort on a project as you would have spent if you'd started with someone who hadn't needed so much supervision and redirection.

But there's more.

Since it takes twice as much work to get stuff done, you'll need twice as many people. But having twice as many people brings on problems of its own, especially when multiple people are trying to make changes to a shared codebase. These changes must be kept consistent, which requires communication between the developers. With twice as many developers, each one needs to spend twice as much time communicating with the others. If they used to spend 25% of their time communicating, now they have to spend 50%. This means that the time they spend actually being productive has dropped from 75% to only 50%. So now you need another 50% more developers in order to make up the difference. You had better hope they didn't start out spending 50% of their time on communication -- either there will be no time left for anything useful, or people will start making mistakes simply out of ignorance.

And still that's not all.

Eventually your good people will get tired of doing nothing but babysitting, and they'll start leaving. So now you should be reviewing people's work, but there's nobody who's really qualified to do it. And everybody who's left is secretly relieved, because now they don't have to keep making all those annoying changes at the end of a project that kept them from being "productive" in the first place. And thus begins the inexorable process of the product turning into a pile of crap.

If you're a growing software company and you don't have enough manpower to crank out features as fast as you want, the worst thing you can do is lower your hiring standards so that you can build up your team quickly. The second worst thing you can do is decide that, since you have five open reqs, it's okay to hire someone who doesn't quite make the bar, because surely you'll find better people for the next four, and otherwise you'll never manage to fill all those positions.

I've been interviewing a lot of people for the QA position at StreamBase recently. Here's a question that I've found to be very useful during a phone screen:

Suppose you have a simple command-line calculator program called "calc". The program takes an expression on the command line, and prints out the result. For example, if you type "calc 1+2", it will print "3", and then exit. Tell me how you would design a test harness for this program so that it's easy to add tests for new calculator features, such as support for division, or trigonometry functions.

I encourage you to think about how you would solve this yourself, before you continue reading.

Imagine a corporate software development group where the entire group decided on its own way of working, rather than having a boss who was responsible for telling each individual what to do. Features would be prioritized by the group, and individuals would volunteer for the projects they wanted to work on. Salary information would be common knowledge and raises would be decided on by concensus. If someone was underperforming, the group would decide, as a whole, how to deal with it. If the rest of the company felt that they needed a "boss" to serve as a primary interface to the rest of the team, then I'm sure the group could find someone to serve that role. The only remaining roles for a boss would to serve as a kind of mediator, or as a source of wisdom about best practices.

I'm not sure how this group would interact with product management. Perhaps the PM role would eventually change into something more like "customer advocate". Or perhaps the PM would somehow try to enforce constraints on the development group -- such as release dates, flagship features that have to be included, etc. It's hard for me to imagine a situation where a group of developers would refuse to at least try to honor those kinds of constraints, unless the constraints truly were wrong-headed.

I'm sure this has been tried before somewhere, but I'm not aware of it. If you know of such an experiment, please let me know.

Here's another thought: it seems like programmers are primarily motivated by making things that perform useful tasks. So why is it that in corporate software development, programmers are almost never encouraged to show off their latest creations? Lots of development shops have some kind of weekly meeting -- why not make use of that time to have developers give a demo of the feature they're working on?

I bet that if you did this, it would increase motivation, improve communication between people working on different parts of the product, elicit constructive suggestions from the other developers, and reinforce the feeling that everybody is part of a team working towards the same goal.

This weekend I was talking with someone in biotech, and the subject came up of the balance between making sure you understand every aspect of an experiment, versus getting it to a point where you're able to get results even though there are still some aspects that you don't quite understand. For example, maybe an experiment works at 72 degrees F, but not at 68 degrees F. Depending on the experiment, that difference may or may not be important. But the fact that you don't understand why it doesn't work at 68 degrees F decreases general confidence in your results, even if only to some small degree.

The same thing happens in programming. For example, I had a problem a while ago where I had a program that worked fine when run by hand on Linux or Solaris or Windows, but if I tried to run it from within a python script, it failed exclusively on Fedora Core 2. Furthermore, the failure mode was really strange: the program would work fine until after the first TCP/IP connection had been serviced, and then it would disappear. It wouldn't core dump, or print an error message, it would just disappear. Now one could say that it didn't really matter, because this program was not meant to be run from within a python script. But even though the problem only happened in specific circumstances, it reduced our confidence in the robustness of the program in all circumstances.

This is similar to the "broken window theory" -- that if you tolerate trivial problems, big problems become harder to control.

The really interesting thing here is that different people have different tolerances for small problems like this, and it can be very difficult not to look down on people whose tolerance level is different from your own. For example, if someone spends two weeks hunting down some small problem that seems mostly irrelevant to you, then it can seem like that person is being inefficient and wasting their time. But on the other hand, if they blithely ignore what seem to you to be worrying indications of instability, then it can seem like they're being sloppy and unprofessional.

I think the thing to realize is that when someone has a different tolerance level for problems like these, you need to realize that it really is just a matter of taste. If you're their manager, then you should encourage them to change their tolerance level in order to be more in line with the rest of the team, but you have to be careful not to just dismiss them as being a slob, or an idler.

Here's an essay/paper by Alistair Cockburn: Characterizing People as Non-Linear, First-Order Components in Software Development. Yes it's a long, self-important sounding title, but the paper itself is very interesting. It's a discussion about the various ways that people make software development teams productive or unproductive.

The paper has a lot of insights into human nature, all motivated by the core belief that methodologies that work against the grain of humans' natural tendencies are destined to fail.

I don't have enough experience to know whether Alistair's insights are accurate or not, but they are certainly intriguing. For example, a couple weeks ago I had decided that my company needed to have more communication between the various people involved on a project. So I had decided to try to use wiki-based project plans to help keep track of things and make sure everybody kept in the loop. However, Alistair claims that paper is one of the least effective means of communication (something I knew intuitively, but hadn't managed to apply in this situation). So after reading the paper, I'm wondering whether it might not be a better idea to have people change desks whenever they switch projects, so that people working on related projects can share an office. Maybe then they'd talk to each other more.

In a way, having people change offices seems like an obvious solution. And yet it's very non-traditional. Nearly every company I know of would try to solve a communication problem by instituting a more formal process based upon writing things down. I've worked at several companies that did that. What ends up happening is that someone labors over a document that takes too long to write, and then they mail it out and call a meeting to discuss it. When the meeting rolls around it turns out that most of the attendees didn't bother to read the document ahead of time, or they only skimmed it for 5 minutes immediately before the meeting. That sounds to me like a sign of a really poor methodology.

Anyway, read Alistair's paper. There's lots more there.

A co-worker pointed me at a paper describing how to make procrastination into a productivity tool:

The observant reader may feel at this point that structured procrastination requires a certain amount of self-deception, since one is in effect constantly perpetrating a pyramid scheme on oneself. Exactly.

Just thought I'd share the word. I posted a job opening to craigslist, and the response I got was very good. I once worked at a place that posted a software developer job opening to the Boston Globe (back before boston.com), and they got responses from all kinds of people, from janitors to gardeners. So I was expecting the worst. But of the twenty-odd responses I got, all of them were relevant, even though 50-75% of them were either from people with not quite enough experience for the position, or from offshore contracting firms.

In short: I'd use craigslist again.

If you or anybody you know is looking for a QA job, my company is hiring. I'm the QA lead, so you'd be working directly with me. I'm looking for people who are comfortable doing both manual as well as automated testing, of programs written in C++ and Java. GUI testing experience is good, software development experience is good, etc. Read the posting if you want to know more.

I just wrote my first employee rejection letter.

For the last week or two, I've been interviewing potential summer interns. Here's a couple lessons on what to do and what not to do when applying for a job.

First, don't say you're looking forward to working at a leading pharmaceutical firm, when the company you're applying to is actually a small software startup.

Second, put your name in the subject of the email you send (after all, you are the subject). It will help your interviewer pick out your resume from the sea of other resumes all titled "Summer Internship".

Third, if you receive a response from the company, and they say they'd like to speak to you any time except for Thursday from noon to 2pm, don't suggest Thursday at noon as being a good time for you.

Fourth, if they say they'd like to speak to you on the phone, don't reply asking where you should meet them. And especially don't suggest that they meet you near your house.

Fifth, if you schedule a time for a phone screen, don't let your phone run out of batteries, and don't wander into an area where you don't get reception.

Sixth, show up within ten minutes of your scheduled interview time, preferably on the early side. If you happen to arrive a whole hour early, just wait quietly in your car for 50 minutes, or go get some coffee and come back.

Seventh, if you're late, pretend it was because the place was hard to find.

Eighth, don't ask your interviewer if the company has somewhere that you'll be able to stay, because you've been living in a dorm the entire year. In the corporate world, it's customary for employees to make their own housing arrangements.

Ninth, don't use your communication skills merely to communicate your belief that you possess communication skills. Instead, communicate something useful, like what particular things you could do that you think might be useful to the company.

I'm sure I'll have more to add to this list in the future...

I've now been a mini-manager for about a month. One of the things I've been surprised to learn is that, even though there's only one other person in my group, I still have to do a lot of organization, scheduling, prioritization, following up on issues with other people, and so on.

I knew that these things were a big part of management, but I didn't realize how big a percentage of my time they would consume even with such a small group. I'm starting to suspect that these things are kind of like hard drive prices -- you pay $60 for the first 20G, and then each additional 20G is only $20. Either that or I'm just not very good at it yet ;)

So far, I enjoy all these organizational activities. It'll be interesting to see whether I still enjoy them once the newness starts getting old.

I got a call from a headhunter yesterday. I'm used to having headhunters call me and ask whether I'm interested in looking for a new job, however that's not what this one asked. This one wanted to know whether I was interested in hiring. I suppose this only makes sense, since I'm now in more of a management position, but the change is unexpected and somewhat unsettling. In the eyes of the headhunter, my role has changed from product to consumer.