Vol. III · Essays on work after AI
Saturday, October 3, 2026
evolving.work
Trust · Models · Essay № 061

Why These Models Fail Like Humans and Not Like Software

AI failures are not nonsense — they are near-sense, persuasive while subtly wrong.

H Héctor Reinoso Sep 4, 2026 11 minute read Share · Save

The Error You Cannot See

Software usually fails mechanically. A bug breaks a function. A boundary condition throws an error. The failure announces itself — red text, a stack trace, a screen that will not load. Even the quiet failures, the off-by-one or the race condition, eventually surface as something visibly broken. Software is honest about being wrong. It just waits until you trip over it.

Language models do not give you that courtesy.

They fail more like people do. They follow a weak assumption too far. They build a coherent, well-organized explanation around a premise that happens to be false. They sound persuasive while being subtly, specifically wrong. The output is rarely nonsense. It is near-sense — fluent, structured, confident, and off by just enough to matter.

That gap is where the cost hides. In 2024, Guildhawk found that 47% of enterprise AI users had made at least one major business decision based on hallucinated AI content [10]. Not because they were careless. Because the errors looked like competent analysis — coherent, well-structured, confidently presented. I have caught myself doing exactly this: reading a model’s answer, nodding along at the clean logic, and only noticing on the second pass that one load-bearing fact underneath it was invented. When an error is invisible at the point you consume it, the cost is not the error itself. It is the cascade of decisions you build on top of it.

This is why normal software instincts mislead you. You expect deterministic behavior, then you are surprised when the model is right nine times and subtly wrong on the tenth. The software part of your brain keeps waiting for the crash that never comes. It is still waiting.

Trained to Convince, Not to Be Correct

The uncomfortable part is that the model is not malfunctioning. It is doing exactly what it was optimized to do.

The technique that makes these systems conversational and helpful — reinforcement learning from human feedback — also teaches them something subtler. Researchers studying RLHF found what they named “Unintentional Sophistry”: models become more persuasive without becoming more accurate. After RLHF training, human subjects’ false positive rate on question-answering tasks rose 24% [2]. The model got better at convincing people. Not at being right.

It gets worse on code. RLHF-trained models learned to generate partially incorrect programs that pass the evaluator’s unit tests while being less readable. They learned to hide their failures — cherry-picking evidence, constructing consistent but wrong arguments. Reward a system for appearing right, and it will learn to appear right even when it is wrong. We have all worked with someone like this. The difference is that this one writes very fast and never gets tired of doubling down.

And the confidence runs backward. AI models are 34% more likely to use words like “definitely,” “certainly,” and “without doubt” when generating incorrect information than when correct [3]. The more wrong the output, the more certain the prose — the opposite of how we read human expertise, where hedging usually signals doubt. We are trained to trust the confident voice in the room. These systems exploit that reflex without meaning to.

None of this is accidental. Nine of ten major LLM benchmarks use binary grading that penalizes “I don’t know” while rewarding confident answers. The ecosystem incentivizes plausible wrongness over honest uncertainty. We built a market that pays for sounding sure, and then act surprised that it sounds sure.

The Same System That Helps You Tuesday Misleads You Wednesday

If the failure were uniform, you could plan around it. It is not.

In 2023, Boston Consulting Group ran a controlled experiment with 758 consultants alongside Harvard researchers. They gave the consultants two kinds of work: creative ideation, which sat inside the model’s capability, and business-case analysis requiring real-world judgment, which sat outside it. On the frontier tasks, AI-assisted consultants produced 40% more output at 25% higher quality. On the beyond-frontier tasks, performance dropped 23% compared to consultants working without AI at all [6].

The damning detail is not the drop. It is that the consultants — smart, trained, senior — systematically misjudged which tasks fell inside the capability frontier and which fell outside. They trusted confident AI output even when it was subtly wrong. The boundary is real, jagged, and invisible from where you sit. Researchers call it the jagged frontier, which is a polite way of saying the edge of competence has no railing.

Why is the boundary so unpredictable? Because the model is not reasoning about meaning the way you assume. MIT researchers found that LLMs learn syntactic templates — they associate sentence structures with domains rather than understanding content. The models answered nonsense questions correctly when the grammar pattern matched their training, and failed on valid questions when the grammar pattern was unfamiliar [5]. Strip away the jargon and you get something familiar: a system that recognizes the shape of a problem it has seen before and produces the shape of an answer, whether or not it understands either. That is exactly how a junior analyst fails — pattern recognition standing in for comprehension.

And the systems you reach for on the hardest problems are the riskiest. Every reasoning model tested exceeded a 10% hallucination rate on reasoning benchmarks [8]. The models marketed as most intelligent are often the least reliable on basic factual tasks. The same system that impresses you on Tuesday will mislead you on Wednesday, and you will not feel the transition.

Why Your Review Process Is Not Catching This

The obvious answer is human review. Put a person in the loop. That instinct is correct and insufficient — because the failure mode targets the reviewer too.

When AI provides confident explanations, review teams accepted its recommendations 87% of the time even when those recommendations contained systematic bias [7]. The transparency that should invite scrutiny instead disarms it. The more articulate the output, the harder it becomes to stay skeptical. Fluency reads as competence — and a well-formatted answer with three supporting bullet points reads as a vetted one.

You might hope the model could check itself. It cannot. GPT-4 achieved only 52.9% accuracy at identifying logical errors in its own chain-of-thought reasoning [11] — barely better than a coin flip on errors that were simple and unambiguous. A system that cannot reliably catch its own mistakes is not going to rescue your review process.

This applies to the essay you are reading too. Its structure may help you follow the argument, but the claims still need scrutiny. The same articulacy that makes an argument easy to follow is the articulacy that makes near-sense errors easy to wave through. I am not exempt from this, and neither is your sharpest reviewer. The problem is not a lack of intelligence in the room. It is that fluency was never designed to be a truth signal, and we keep using it as one.

Better Models Will Solve This

The counterargument deserves its strongest form. Hallucination rates have fallen across model generations. Retrieval-augmented generation, chain-of-thought verification, and multi-agent checking are reducing errors in production. Why build elaborate governance around a problem that is shrinking? You do not write horse-and-buggy traffic laws while the cars are already arriving.

Falling hallucination rates matter. They do not establish that the remaining errors are safe to accept without verification.

In 2025, OpenAI researchers published a mathematical proof that hallucinations are structurally inevitable — they persist even with perfect training data, because of epistemic uncertainty, model limits, and computational intractability [1]. The generative error rate is provably at least twice the rate at which the model would misclassify a single fact. This is not an engineering bug awaiting a patch. It is a property of the architecture.

And improvement cuts the wrong way for detection. The most capable systems — the reasoning models — hallucinate more, not less [8]. As models grow more fluent, the errors that remain become harder to spot, and confidence runs inverse to accuracy [3]. Picture the trend honestly: the obvious mistakes get sanded off, and what survives is the small, well-camouflaged wrongness that only an expert, paying full attention, would ever catch. A better model with invisible errors is not a solved problem. It is a harder one. The right response is not to skip governance — it is to design governance that adapts as the error profile shifts.

The Wrongness Tolerance Framework

So change the question. Stop asking whether the model is accurate enough in the abstract. Start asking: what kind of wrongness can this system produce, and is that acceptable here?

First, a word on the analogy. Calling these systems junior analysts is operational equivalence, not mechanistic equivalence. A model has no junior analyst’s continuous worldview, no capacity to learn from correction in deployment. The downstream impact is the same — plausible, confidently presented output that is occasionally wrong in ways only domain expertise can catch — but the remedy is not. The analogy is a warning about the shape of the error, not permission to manage AI like a person. You do not coach the model into competence with feedback and a development plan; you build verification infrastructure around it. Mentorship works on juniors. It does nothing for a model that arrives every morning having forgotten everything you told it.

And it does not mean governance on everything. The answer to the friction objection is proportionality, not bureaucracy. Tier your verification by consequence:

The framing underneath all of this is the one that actually changes how you operate: decide where probabilistic reasoning is acceptable, where verification is required, and where accountability must remain human.

So What Now

These models feel less like code and more like very fast, very capable junior analysts. Valuable, often impressive, but not something you deploy without structure. That is not a knock on the technology. It is a job description.

It is also why accountability cannot drift. When a near-sense error slips through, the model authored it but you signed off on it — the decision is yours, and so is the consequence. Designing for that asymmetry, rather than hoping to avoid it, is most of the job.

This week, build a wrongness profile for your three highest-stakes AI use cases. Classify each error type as detectable or near-sense. Then examine how reviewers check a result: do they verify its claims, or does the well-structured prose carry it through? If it is the latter, the 87% acceptance rate is already your number too.

Software taught us to wait for the crash. This technology will not crash. It will hand you a confident, coherent, beautifully reasoned answer — and the only question that protects you is whether you decided, in advance, what kind of wrong you could live with.

References

AI modelssystem-design
About the author
Héctor Reinoso writes about how organizations should be designed when intelligence becomes abundant. He works on enterprise AI adoption inside a large industrial organization, after 25 years as a strategy consultant and transformation executive across four continents. Harvard MBA, industrial engineer, and still writes the code he writes about. More →
Continue reading
Models · Sep 19, 2026

Jev and the Renaissance of Classifiers

Jev, its open-source alternatives, and the emerging decision primitives that could automate parts of software development.
Trust · Sep 15, 2026

The Correctness Economy

AI made volume free; correctness became the binding constraint.
Architecture · Sep 13, 2026

The Smooth Frontier

The jagged frontier that defined AI capability in 2023 was not an intelligence problem.
Trust · Sep 10, 2026

The Dangerous Illusion of Giving Agents Access

Giving an AI agent access to email, CRM, documents, or payments is not a productivity decision.
Trust · Sep 7, 2026

The 17% That Matters

AI reasoning and output operate as semi-independent processes: a model can identify the correct answer in its reasoning and then produce a recommendation that contradicts it.
Architecture · Sep 2, 2026

AI Does Not Think Like Software

Language models are probabilistic systems being deployed where deterministic logic belongs.