A number on a dashboard is wrong. A delay that blindsided every forecast, a cost overrun outside every budget, a performance gap that defies explanation. Three teams have spent a week inside the data and still cannot say where it started. The organization has more visibility than any operator in history, and less idea what it is looking at.
The instinct is to assume the problem is missing data. It almost never is. The problem is that abstraction — the very thing that let the company grow — quietly severed the link between the symptom and its cause. And AI did not repair that link. It made it shorter-lived. The more abstract the system, the harder it is to fix; AI just shortens the time you have to try.
I work with these systems every day, and the pattern I keep noticing is that the failure always feels like a data shortage right up until you notice you are drowning in data. That is the tell. When more dashboards make the answer harder to find, you are not short on information. You are short on a way down.
Why Every Scaling Decision Adds a Layer You Cannot See Through
As organizations scale, they lean harder on abstraction. Metrics replace direct observation. Standardized processes replace individual judgment. Dashboards replace frontline presence. This is how a few hundred people coordinate work none of them can see in full, and the efficiency is real.
It is also necessary — but it comes with a bill that arrives later. The further a system moves from direct observation, the harder it becomes to know where problems originate. Issues surface as symptoms — delays, cost overruns, performance gaps — while their origins sit several layers down, invisible to anyone working at the surface.
Bret Victor’s ladder of abstraction names the mechanism precisely: the most powerful way to gain insight into a system is by moving between levels of abstraction [1]. Insight lives in the transitions — up to spot the pattern, down to explain it. Most organizations get stuck on one rung and wonder why the same problems keep returning.
They keep returning because the fix lands on the wrong level. Donella Meadows showed that most interventions target low-leverage parameters — the numbers, the surface metrics — while the behavior-generating structure sits higher up, untouched [10]. So the most consequential management decision a company ever makes, the one that lets it coordinate hundreds of people, is also the one that will eventually make its problems hardest to find. You buy coordination now and pay for it in diagnosis later. The invoice does not arrive until something breaks.
What AI Actually Changes Is the Clock
In a slow system, you can live with all of this. Processes are understood, variability is bounded, and a problem that originates in one layer has time to surface before it propagates. The cycle speed itself is a buffer — the lag between cause and symptom is annoying, but it is also forgiving.
AI removes the buffer. By accelerating parts of the system, it speeds up how fast problems propagate; an issue that a slower cycle would have contained now crosses functions before anyone reads the first signal. Charles Perrow’s work on normal accidents explains the architecture: in systems with interactive complexity and tight coupling, failures cascade faster than they can be observed or traced [2]. AI tightens the coupling. It does not invent the structure — it reveals its consequences at speed.
The specific fragility is that AI now lives inside tooling no single person designed, where small shocks propagate into outsized consequences and the distance between action and consequence shortens while traceability does not [7, 9]. KPMG’s analysis of AI and organizational resilience names the precise blind spot: existing risk analyses may underestimate AI impact because the failure modes inside these abstracted, automated systems are still poorly understood [9]. We are embedding AI into systems whose failure signatures we cannot yet read.
No vendor pitches an AI system as “make our problems propagate more efficiently.” In an untraced system, that is exactly what it does.
Forty-Five Minutes, and a Company Was Gone
On August 1, 2012, Knight Capital handled roughly 10% of US equity volume. That morning it pushed updated code to its trading servers — a routine deploy ahead of a new NYSE program. One of eight servers missed the update, leaving a decade-old dormant function, “Power Peg,” reactivated by a repurposed flag. SMARS, the firm’s automated routing layer, began executing at machine speed.
The system sent 97 alert emails before the market opened. The operators read past them. They were formatted as information, not as an action trigger — one more layer of abstraction between the people watching and the cause. For 45 minutes, engineers changed configurations while SMARS kept trading, ultimately executing over 4 million trades across 154 stocks. The damage reached $460 million, more than the firm’s entire equity capital. A 17-year-old company was gone in the time it takes to watch a movie.
What killed Knight was not execution speed. It was diagnosis speed. The system could have been stopped in minutes had the root cause been visible. When automation runs across abstraction layers, the speed of failure routinely outruns the speed of diagnosis. SMARS was algorithmic automation, not a large language model — but the mechanism it demonstrates is the same one modern AI deployments accelerate. The technology changed. The architecture of the failure did not.
More Dashboards, Less Understanding
This is the paradox at the center of the whole thing: organizations have more visibility in terms of data and less clarity in terms of understanding. Dashboards expand, metrics multiply, and the connection between those metrics and the underlying reality gets harder to read, not easier. The numbers say it cleanly.
86% of technology leaders consider it impossible to cost-effectively capture and analyze all the monitoring data their systems generate [3]. Signal volume has outrun the capacity to read it. Mean-time-to-resolution exceeded one hour for 82% of engineering teams in 2024, up from 74% the year before [4] — more monitoring investment, longer diagnosis. And the biggest observability challenge, cited by 48% of organizations, is now lack of knowledge among teams, up from 30% in 2023 [5]. Human understanding is falling behind, not catching up.
The cure has become its own disease. Organizations now run an average of 10 monitoring tools across 12 platforms, and 85% of leaders say the proliferation adds complexity rather than reducing it [6]. We bought more instruments to see clearly and ended up with more to interpret. Most telling: 78% of companies fail to convert their data into decisions, and HBR attributes 93% of analytics gaps to people and process, not technology [12]. The bottleneck was never the instruments. It is understanding — and understanding sits in people, not dashboards.
Why Fixing the Metric Just Rebuilds the Same System, Faster
Fixing a system requires understanding it at a level where cause and effect are visible. When that level is obscured, interventions land on symptoms, the underlying issue persists, and it reappears wearing a different costume next quarter.
Meadows’ iceberg explains the failure: events sit at the surface, patterns one level down, structures below that, mental models at the base [10]. Adjust the event-layer metric and the generating structure stays intact, so the same event recurs — at AI speed, weekly instead of quarterly. It is adjusting the temperature gauge while the engine overheats.
The evidence on AI itself follows the same shape. Only about a third of organizations successfully scale AI, and McKinsey points to measurement immaturity as a primary barrier: they deploy widely but cannot show whether it works, because they lack process-level visibility [8]. Applied without structural understanding, AI makes the system more efficient at producing outcomes — and more sensitive to the hidden flaws underneath them.
Won’t AIOps and Better Tooling Already Close This Gap?
Here is the fairest version of the objection. AIOps platforms — Dynatrace Davis AI, Datadog Watchdog, Splunk — exist precisely to close this gap. They correlate telemetry across abstracted layers automatically and surface root causes in seconds. In greenfield, observability-first stacks, the trace is built in from day one. If AI closes the diagnostic gap, the causality runs backward: AI is the solution, not the aggravator.
The adoption data suggests that buying AIOps does not, by itself, settle the diagnosis problem. 72% of organizations had adopted AIOps by 2024 [6], yet 97% of leaders still need manual analysis and MTTR rose anyway [4]; the tooling added complexity rather than removing it [6]. The distinction matters when someone has to decide what to change after an alert. AIOps improves signal detection — where something deviated — but not structural understanding — why the process was designed to break under load.
A second objection identifies a condition under which abstraction helps: clear contracts and correlation IDs can make it a diagnostic scaffold. But those conditions are rare outside elite engineering orgs, and this thesis is mostly about business systems — spreadsheet handoffs, manual reconciliation, undocumented process. Those have no correlation IDs, and even good abstractions drift as AI violates their contracts one small step at a time.
A third reframing blames incentives, not architecture — also real, and it makes things worse. But you can reward understanding all you like; it has nowhere to anchor if the system retains no trace of what it did [12]. None of these objections are wrong. They describe one set of conditions worth building toward: useful traces, clear contracts and incentives to understand the process.
The Person Everyone Calls When the System Breaks
Perrow, Meadows, and Victor named the architecture of this failure across five decades of systems thinking. What the 2024 data adds is the compression: AI has shortened the interval between a latent flaw and a cascade to below the diagnosis time of most teams, while the gap between signal volume and human understanding keeps widening. Cross-level capability used to be a nice organizational asset. It is now a prerequisite for accelerating anything safely.
The countermeasure is not to eliminate abstraction — it is to reconnect it to reality. Identify where value is actually created, how work flows, and where the constraints sit; then move from high-level metrics to detailed process understanding and back again. The individuals who can do this become critical. They are the person everyone calls when the system breaks — working at exactly the transitions where Victor located insight [1]. Every team has one. Most have not noticed how much they now depend on them.
Boeing’s 737 MAX shows the stakes of the opposite bet. MCAS was architected as an invisible abstraction layer — autonomous, single-sensor, deliberately left out of the pilot manuals because, as a Boeing vice president put it, the goal was not to overload crews with information considered unnecessary. After 346 deaths and a 20-month grounding, the core fix was reconnection: disclose the system in training, require two sensors to agree before it acts. Hidden automation removes the diagnostic window. Reconnecting the operators is what restores it.
The discipline can be built, and it is not exotic. Map the abstraction layers of each critical process before you try to move between them. Profile how fast a failure at each layer surfaces — if AI compresses that below your response time, the automation is a trap, not a tool.
From there it gets practical. Trace a deviating metric down one level at a time; every level you skip is a candidate root cause you chose not to look at. Map who can translate each layer to the ones above and below it; every gap there is a fragility waiting for a bad morning. And install recurring checkpoints where an aggregate number must be explained by the layer beneath it, not merely reviewed on a slide.
What to Do Before You Accelerate
The moves are concrete, and you can start this week. Map where cause and effect are most obscured before you deploy AI to accelerate anything — acceleration does not create those gaps, it exposes the ones you had been tolerating. Name the person who can trace a dashboard anomaly back to the process that produced it; if you cannot name one, that is your first constraint, not your monitoring stack. Measure root-cause traceability separately from MTTR, which arrives too late to act on. Stop adding tools to systems you have not documented. And make pre-deployment comprehension a condition of acceleration, not a post-mortem you write after it fails.
The more abstract the system, the harder it is to fix — unless you can see through it.
References
- [1] Victor, Bret. Up and Down the Ladder of Abstraction. https://worrydream.com/LadderOfAbstraction/
- [2] Perrow, Charles. Normal Accidents: Living with High-Risk Technologies. https://psychsafety.com/normal-accidents/
- [3] Dynatrace. The State of Observability in 2024. https://www.dynatrace.com/news/blog/the-state-of-observability-in-2024/
- [4] Logz.io. Observability Pulse 2024. https://logz.io/observability-pulse-2024/
- [5] Logz.io. Observability Pulse 2024. https://logz.io/observability-pulse-2024/
- [6] Dynatrace. The State of Observability in 2024. https://www.dynatrace.com/news/blog/the-state-of-observability-in-2024/
- [7] KPMG. Managing the AI Impact on Organizational Resilience. https://kpmg.com/xx/en/our-insights/risk-and-regulation/managing-the-ai-impact-on-organizational-resilience.html
- [8] McKinsey & Company. The State of AI: Agents, Innovation, and Transformation. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai
- [9] KPMG. Managing the AI Impact on Organizational Resilience. https://kpmg.com/xx/en/our-insights/risk-and-regulation/managing-the-ai-impact-on-organizational-resilience.html
- [10] Meadows, Donella. Thinking in Systems: A Primer (and Leverage Points: Places to Intervene in a System). https://donellameadows.org/wp-content/userfiles/Leverage_Points.pdf
- [11] Sapphire Ventures. Observability in 2024: Understanding the State of Play and Future Trends. https://sapphireventures.com/blog/observability-in-2024-understanding-the-state-of-play-and-future-trends/
- [12] TI Inside. The Paradox of the Data Age: Much Information and Little Vision. https://tiinside.com.br/en/04/02/2026/o-paradoxo-da-era-dos-dados-muita-informacao-e-pouca-visao/
Case material:
- Knight Capital Group, August 1, 2012 trading incident — SEC administrative proceeding and contemporaneous reporting on the SMARS / “Power Peg” deployment failure.
- Boeing 737 MAX / MCAS — congressional and FAA reviews of the MCAS design, the two crashes (Lion Air 610, Ethiopian Airlines 302), and the subsequent grounding and re-certification.