Why Does Software Rot?

Almost a year ago computer scientist Daniel Lemire wrote a post critical of a hypothesis I’ve favored, one I’ve used in Age of Em. On the “better late than never” principle, I’ll finally respond now. The hypothesis:

Systems that adapt to contexts tend to get more fragile and harder to readapt to new contexts.

In a 2012 post I said we see this tendency in human brains, in animal brains, in software, in product design, in species, and in individual cells. There is a related academic literature on design feature entrenchment (e.g., here, here, here, here).

Lemire’s 2015 response:

I am arguing back that the open source framework running the Internet, and serving as a foundation for companies like Google and Apple, is a counterexample. Apache, the most important web server software today, is an old piece of technology whose name is a play on words (“a patched server”) indicating that it has been massively patched. The Linux kernel itself runs much of the Internet, and has served as the basis for the Android kernel. It has been heavily updated… Linus Torvalds wrote the original Linux kernel as a tool to run Unix on 386 PCs… Modern-day Linux is thousands of times more flexible.

So we have evolved from writing everything from scratch (in the seventies) to massively reusing and updated pre-existing software. And yet, the software industry is the most flexible, fast-growing industry on the planet. .. If every start-up had to build its own database engine, its own web server… it would still cost millions of dollars to do anything. And that is exactly what would happen if old software grew inflexible: to apply Apache or MySQL to the need of your start-up, you would need to rewrite them first… a costly endeavour. ..

Oracle was not built from the ground up to run on thousands of servers in a cloud environment. So some companies are replacing Oracle with more recent alternatives. But they are not doing so because Oracle has gotten worse, or that Oracle engineers cannot keep up. When I program in Java, I use an API that dates back to 1998 if not earlier. It has been repeatedly updated and it has become more flexible as a result…

Newer programming languages are often interesting, but they are typically less flexible at first than older languages. Everything else being equal, older languages perform better and are faster. They improve over time. .. Just like writers of non-fiction still manage to write large volumes without ending with an incoherent mass, software programmers have learned to cope with very large and very complex endeavours. ..

Programmers, especially young programmers, often prefer to start from scratch. .. In part because it is much more fun to write code than to read code, while both are equally hard. That taste for fresh code is not an indication that starting from scratch is a good habit. Quite the opposite! ..
“Technical debt” .. is a scenario whereas the programmers have quickly adapted to new circumstances, but without solid testing, documentation and design. The software is known to be flawed and difficult, but it is not updated because it “works”. Brains do experience this same effect.

I have long relied on a distinction between architecture and content (see here, here, here, here, here). Content is the part of a system that it is easy to add to or change without changing the rest of the system; architecture is the other part. (Yes, there is a spectrum.) The more content that is fitted to an architecture, and the bigger is that architecture, the harder it becomes to change the architecture.

Lemire’s examples seem to be of systems which grow long and large because they don’t change their core architecture. When an architecture is well enough matched to a stable problem, systems build on it can last long, and grow large, because it is too much trouble to start a competing system from scratch. But when different approaches or environments need different architectures, then after a system grows large enough, one is mostly forced to start over from scratch to use a different enough approach, or to function in a different enough environment.

This is probably why “Some companies are replacing Oracle with more recent alternatives.” Oracle’s architecture isn’t well enough matched. I just can’t buy Lemire’s suggestion that the only reason people ever start new software systems from scratch today is the arrogance and self-indulgence of youth. It happens way far too often to explain that way.