If you had a new way of manufacturing a device, you’d put it through a series of tests and see how it compared to the previous approach.
We’re currently running this experiment in software by replacing human-generated code with machine-generated code. How do we know whether the new manufacturing process is better?
We can measure some things like test coverage, complexity, static-analysis warnings and so on. These are comforting (woo, 90% coverage!) but they are indirect measures (at best).
The only thing that really matters is that the software does what users expect, and that’s really hard to measure.
There’s an anti-measure though. How often does the software fail to do what users expect?
Most folks would call these bugs (a previous place I worked called them feature requests1). Either way, they are evidence of a gap between the software that was built and what users thought they were getting.
Like the cat in a box, bugs are only discovered when we exercise the code.
For simple, high-volume consumer products this happens quickly. Millions of users hammer the same paths and problems are rapidly found. For complicated enterprise software, it might take years. Some obscure configuration, workflow or environment2 might be exercised once in a blue moon.
Insurance has a concept for this, IBNR (Incurred But Not Reported). Insurers know that some claims are out there but haven’t been reported yet - a car crash over Christmas that doesn’t get reported until January, for example.
Software has IBNR too.
There are latent bugs sitting in every codebase that no-one has encountered yet. You’ve probably got a feel for how long each release takes before you consider it stable.
That works while history is a reasonable guide to the future, but AI may destroy that assumption.
Firstly, I’m confident that AI will cause us to produce much more code. Secondly, I’m much less confident that users will use proportionally more software.
If code volume grows materially faster than usage, then each piece of code gets less real-world exposure.
That means bugs take longer to surface.
The tail gets longer.
Looking at the early results of the great AI experiment and saying “bug rates are comparable” could be dangerously misleading.
The bugs might not be absent. They might simply not have been discovered yet.
Worse, the historical data we’d normally use to estimate that latent liability comes from a world where humans wrote much less code.
AI isn’t just changing how we manufacture software, it’s also invalidating many of the assumptions we had about software.
This place (not where I currently work) also described QA as the greatest myth in software development.
Just wait until you have customers in regions with non-Gregorian calendars.
Well, if you bring this to its obvious conclusion, you should just use AI to simulate user usage.