Why are IT projects so damn hard?
What’s more risky - building a nuclear power plant or writing an ERP application?
In “The Uniqueness of IT Cost Risk: A Cross-Group Comparison of 23 Project Types”, the authors analyzed a huge number of projects. What they found was that IT projects are the only project type with “infinite mean and variance” for cost overruns. To put another way, conventional forecasting for IT projects simply doesn’t work.
While only 41% of IT projects go over budget (better than most project types), when they do blow up, they blow up spectacularly. The average IT project that goes into cost overrun territory ends up 453% over budget (!). Compare that to roads (101% average overrun), wind power (97%), or even nuclear power plants (204%).
IT projects are in their own unique category- but why?
Why IT Projects Are Uniquely Cursed
The research identifies four key reasons why IT projects are different:
Immaturity: IT as a discipline is decades old, not centuries like construction engineering. We're still figuring this stuff out while building trillion-dollar systems.
Intangibility: You can't see software progress like you can see a building going up. This makes it nearly impossible to spot problems early or communicate status accurately to stakeholders. (Jack Reeves explores this here, where the argument is that writing code is inseparable from design).
Goal Ambiguity: IT projects are "soft projects" where requirements shift constantly. When your building materials are infinitely malleable (unlike actual buildings), scope creep becomes inevitable(software is soft!).
Stakeholder Resistance: IT projects fundamentally change how organizations work, threatening power structures and job security in ways that building a new road simply doesn't.
But the research suggests two additional culprits that particularly caught my attention.
The Bespoke Problem
Look at where IT sits in the risk rankings alongside nuclear storage, defense projects, and the Olympics. What do these have in common? They're all completely bespoke, treating every project as a unique snowflake.
Now look at the low-risk projects: solar, wind, energy transmission. What's different? Modularity.
The researchers point to Apple's App Store as an example. Every app follows mandatory modular patterns using Apple's modular development tools. It's "double modularity," and it's generated hundreds of billions in revenue with minimal project risk.
Wind power used to be high-risk when turbines were built one-off on construction sites, like buildings. When the industry shifted to modular manufacturing with on-site assembly, costs plummeted, and reliability soared.
The "Think Fast" Trap
The other cause? IT projects are rushed. The average IT project duration is 3.2 years compared to 6.9 years for other project types. You'd think shorter projects would be less risky, but the opposite is true for IT.
This suggests IT projects suffer from what Daniel Kahneman calls "fast thinking" - decisions made quickly without proper deliberation. When you're racing to market or responding to urgent business needs, proper planning gets sacrificed for speed.
What this (might) mean for your next project
If you're planning an IT project (and let's face it, these days every project is partly an IT project because it’ll surely involve AI), here's what you need to know:
Conventional estimation techniques will fail you. The study shows most IT risk management assumes normal distributions, which grossly underestimate tail risk (not sure if it’s still fashionable to do #noestimates).
Be boring. Fight the urge to build everything from scratch. Use established platforms, frameworks, and patterns (Choose Boring Technology)
Slow down the decision-making. Yes, business moves fast, but "think slow" approaches to project planning and architecture decisions pay dividends.
Plan for the fat tail. Traditional risk management might suggest adding 20% contingency. For IT projects, you need to prepare for the possibility of 300-400% overruns.
The research doesn't prove modularity and "think slow" approaches solve the IT risk problem, but the patterns are compelling. When we see successful examples like the App Store demonstrating how modular approaches can tame IT risk at massive scale, it's worth paying attention.
As a brief aside, I found the “slow down the decision-making” line interesting with respect to agile development. The original manifesto says X over Y, but this is often interpreted as X instead of Y. I wonder how much this is to blame?
The Bottom Line
Next time someone tells you software is "just" configuration or asks why IT projects take so long and cost so much, you can point them to this research. IT projects aren't just another type of project - they're fundamentally different beasts with unique risk profiles that require different approaches. I mean, it won’t help, but at least you can tell them 🙂