Programmers aren't programs

Yesterday I mentioned one reason why project estimates are often wrong, but I skimmed over why software development in particular has trouble with estimation. Even with a genuine effort to predict, rather than control, the amount of time the project will take, software is genuinely kind of unpredictable. I believe this isn't just a coincidence, but something that is fundamental to the nature of software development. It's because programmers aren't programs.

We used to pay people to machine things, now we mostly get robots to do it. Partly, this is because the actions involved were easy to mechanise, but also because the decisions involved were repetitive enough to turn into a computer program. Driving has much more difficult mechanics, so the sensors and actuators have to be more complex, but the decisions of human drivers are still repetitive enough, predictable enough to be turned into a program. The service industry is more complex still, but I expect it is next.

Many fields are part creative and part uncreative. Architects and engineers design a bridge, builders build it. Building is repetitive and predictable, so it is the easiest to automate. As for engineers, increasingly advanced modeling and simulation systems reduce their work, but will never completely eliminate it. We can make machines solve problems, but we can't tell them which problems to solve. If someone thinks up a new kind of bridge, that work stops being repetitive until such time as we know how to do it reliably, repetitively, and predictably. Then it will be automated away again.

Software isn't an exception to this, quite the opposite; software is the abstraction over all of these examples. It's not that software can't be automated, it's that software is always being automated. Programmers automate away their own jobs every day, then make new jobs for themselves, then automate those away too. They ride the wave of whatever parts of the system have no recognisable pattern, but as soon as that pattern appears, it becomes part of the program too and the programmer moves on to something else.

This is why the field has moved so quickly. Programmers spent a lot of time in the early years figuring out how to make lines move on a screen, connect one computer to another, or store stuff and read it back later. But now we take all of that for granted; it's boring, repetitive, predictable. In other words, those problems have already been written into programs, and most programming work is now in using those programs to solve other problems that still don't have a predictable pattern.

Of course, that's not always true. Despite these lofty ideals, programming still has a lot of busywork. But although that's the case on a small scale, the industry as a whole tends to be on a constant mission to make itself redundant; today's programming becomes tomorrow's programs. And, while some programmers who really just write the same code over and over again may be automated out of a job, for many programmers the day their current problem is automated away will just be a regular day.

This is why programming is hard to predict, because prediction requries a pattern. You can predict how long something takes to build if you've made a hundred things just like it. But anything that is easy to predict no longer needs a programmer, it just needs a program. Once it's predictable enough to give way to estimation, it's predictable enough to stop doing. Programming is whatever part is still too complex to predict. And that's why software estimation is hard.