Why your AI harness matters more than your AI model
Take the same model and give it different memory, tools, and rules. One arrangement passes a test; another fails. That is roughly what the benchmarks show. The “jar” around the model is its harness, and choosing it also means choosing how your team will work with the agent.
The Same Brain, Two Different Scores
On SWE-bench Verified — the controlled benchmark the industry uses to rank coding agents — changing only the scaffold moves GPT-5 by up to 11 points and Kimi K2 Thinking by up to 15. Three teams running the same Opus 4.5 landed between 49.8% and 51.8%, and Anthropic reports its custom harness adds ten points over the standardized framework. The same model produced different scores under different harnesses.
One developer analysis pushes it further: the same model swinging from 42% to 78% on coding benchmarks based solely on the surrounding harness. That figure is a secondary source, not primary benchmark data — it should be treated separately from the primary benchmark results. The only thing that changed was the scaffolding: execution environment, memory, tools, feedback loops. The brain was identical. The jar was not.
There is an attribution problem: benchmarks are not production, and SWE-bench measures the harness and the model together, so clean attribution is genuinely hard.
Meanwhile, the capabilities available from different models were converging, making the harness a more consequential variable. The US-China gap on MMLU fell from 17.5 percentage points in 2023 to 0.3 in 2024, and the margins atop Chatbot Arena now sit in measurement-error territory. The brains are converging.
And the cost of renting one is in freefall — OpenAI’s CEO puts it at roughly 10x cheaper every twelve months, an interested party’s estimate that still tracks the price curves practitioners have watched for two years. When capability is a commodity dropping in price, your advantage stops living in the thing you buy and starts living in the thing you build.
Which means the model was never the real decision. Choosing a harness is choosing how humans and machines work together — a philosophy of work — and most organizations are running one they never sat down and chose.
A Harness Encodes a Theory of Work
A harness answers one practical question: how does an agent work in your codebase? It defines what context it receives, which actions it can take, and how its results are checked. Martin Fowler names three pillars — context engineering, architectural constraints, maintenance cycles. Each encodes a decision about how the system should work.
OpenAI’s Codex team shows how far this goes. A small group built a production application north of one million lines in five months, with zero human-authored source code. The team reported that all source code was generated by AI. The developers stopped writing code. They designed environments, specified intent, and fed structured corrections back in — and when an agent struggled, they fixed the harness, never the model.
The developers’ work shifted from writing code to designing and supervising the system that produced it.
One analysis estimates the harness delivers roughly twice the output-quality impact the raw model does: the model is the engine, the harness is the steering and the brakes. Get the engine for free, as convergence increasingly lets you, and steering becomes the whole game.
Notice what OpenAI’s harness chose, though. Fowler points out that it enforces structural correctness — linters, type rules, dependency layering — while skipping any check of whether the software behaves the way a user needs. Structure over behavior. That is a philosophy baked into a million lines, and reversing it later would mean retrofitting tests into a system that was never designed to want them.
Which raises a sharper question: if the harness encodes your assumptions, what happens the day you outgrow them?
You Are Already Locked In
Lock-in does not announce itself. It accumulates across five layers — scaffolding code, team skills, process integration, evaluation criteria, vendor coupling — each making the one beneath it harder to peel off. The code is the cheaper part to change. Retraining people and changing established ways of working cost more.
The teams I talk to can all name their model. Ask them to describe their harness and you get a pause.
Look at where organizations actually sit. Eighty percent of leaders report mature basic automation, but only 28% claim maturity with agents — and just 12% expect agent ROI within three years, against 45% for automation. Frontier models are everywhere. Almost no one has a working harness. The gap is not access to a better brain; it is the scaffolding most teams have not built yet.
We have seen this film before. Cloud lock-in was never really about the virtual machines — it was the operational knowledge, team habits, and proprietary APIs that piled up around one provider’s assumptions. Zynga learned this migrating off AWS onto its own Z-Cloud and partway back, while IBM and Oracle turned multi-year contracts into gravity wells. Switching cost grew with every quarter of accumulated know-how.
The harness is running the same playbook on a shorter clock. “Switching means rebuilding everything” sounds like hyperbole until you price out retraining the team, redrawing the processes, and redefining what good means.
The Protocol Wars Are the Harness Wars
The same choice arises in how agents connect to tools and to one another. Three protocols are fighting for that layer, each from a different anchor: Google’s A2A, Anthropic’s MCP, and Cisco’s AGNTCY. They are not interchangeable plumbing. MCP assumes a model-centric world where tools serve the model; A2A assumes peers collaborating as equals; AGNTCY assumes centralized enterprise governance. Pick one and you have picked an architecture — and an architecture is a philosophy with a wire format.
The money makes the stakes plain. The agent orchestration market runs about $8.5B in 2026, is projected at $35-45B by 2030, and Deloitte expects more than 40% of agentic projects to be cancelled by 2027 — with the field converging to two or three winners. Commit early to the protocol that wins and your investment compounds. Commit to the one that loses and it becomes a migration bill. Either way you are placing the bet now; the only question is whether you know it.
Just Swap the Tooling
A counterargument is that harness code is relatively easy to replace.
The harness, they argue, is thin — config files, prompt templates, workflow scripts, not data gravity or network topology. Teams migrate from Jenkins to GitHub Actions, from Eclipse to VS Code, without existential drama. A competent team rewrites scaffolding in a couple of sprints. Calling that the cloud wars is theater.
They are right about the code. An abstraction layer can de-risk model switching, though it caps how deep you can optimize — you pay for portability in harness quality.
They are wrong about what travels with the code. The code is portable; the philosophy is not. When eighteen months of hiring criteria, review rituals, sprint shapes, and quality metrics have all been bent around one set of assumptions about autonomy and quality, switching is not a refactor — it is reorganizing the engineering function. Fowler’s line is the whole argument: different harnesses embed different assumptions about what quality means. Changing that definition of quality means helping people change how they build and review work, alongside rewriting configuration.
A second objection runs deeper. Model convergence is temporary, the argument goes — the next breakthrough will make today’s harnesses look like the feature-engineering pipelines that deep learning erased, and the teams who invested least will have the least to unwind. The history is real, and for organizations still treating AI as exploratory tooling — pilots, internal experiments, nothing in production — the concern is genuinely legitimate. When harness choices have not compounded into production workflows, the calculus changes.
But for teams where AI now generates the actual software — the Codex team’s million-line codebase being the live instance — we have crossed a threshold the analogy misses. The harness is the production methodology now. A model that scores 95% does not retire it; it raises the bar for safety, auditability, and scale, which means more scaffolding, not less.
And the benchmark objection — that the 42-to-78% gap is misleading and harness-versus-model is a false dichotomy — bends straight back on itself. If a harness must be tuned per model, switching models forces a harness rebuild. Entanglement is not an escape hatch from lock-in; it is the mechanism of it. The 2x quality lift is precisely the coupling that deepens every time you optimize.
Audit the Philosophy You Did Not Know You Chose
Your existing setup already makes choices about autonomy, quality, and feedback. Describing those choices gives the team a chance to decide whether they still fit. Start with the following questions.
Map it. Place your setup on three axes: autonomy (do agents ship, or suggest and wait?), quality (do you enforce structure or behavior?), and feedback (when an agent fails, do you fix the harness or swap the model?). Most teams have never said these out loud — the engineers who built the rig made philosophical calls they would never have called philosophical.
Measure the depth. Score five layers from 1 to 5 — scaffolding code, team skills, process integration, evaluation criteria, vendor coupling — and sum them. Roughly: 5-10 is shallow (switching is a sprint), 11-18 moderate (a quarter), 19-25 deep enough to be a reorganization. The value is in naming the five layers, not in the precise number.
Then two faster checks. Fit: does your philosophy match where your industry’s regulation, error tolerance, and competition are heading — not where they sit today? Insurance: put model calls behind an abstraction layer, keep a behavioral test suite as a portability baseline even when your harness prefers structure, and invest hardest in the skills that transfer. The aim is a survivable migration, not a painless one — over-abstract and you hand back the 2x.
What to Do Before the Cost Compounds
A limit to this argument: no published study tracks harness switching costs yet. The field is too young, so this is a reading of the pattern, not a measured law — and I would hold it loosely if the cloud wars had not already run the experiment.
A model comparison can leave the more lasting decision unexamined: the workflows and skills your team is building around the harness. Run the map and depth score this quarter, while those choices are still easier to change.
Evaluate the harness alongside the model, including the cost of changing the workflows and skills built around it.
References
- Earezki, Harness Engineering: Why Scaffolding Outperforms AI Models in 2026 (Dev|Journal, 2026; 42%-78% same-model swing attributed to Nate B Jones). earezki.com
- VALS.ai / SWE-bench, SWE-bench Verified Agent Scaffolding Performance Data (2026; GPT-5 up to 11pt and Kimi K2 Thinking up to 15pt scaffold-only swings; Opus 4.5 spanning 49.8%-51.8% across three teams; Anthropic custom-harness +10pt disclosure). vals.ai/benchmarks/swebench
- Towards AI, The Great LLM Convergence (2025; MMLU US-China gap 17.5pt → 0.3pt and Chatbot Arena top-margin convergence, drawing on the Stanford 2025 AI Index). towardsai.net
- Towards AI, The Great LLM Convergence (2025; ~10x annual inference-cost decline attributed to Sam Altman). towardsai.net
- Deloitte, Unlocking Exponential Value with AI Agent Orchestration (TMT Predictions 2026; orchestration market ~$8.5B in 2026 → $35-45B by 2030, >40% of agentic projects cancellable by 2027, convergence to two or three leaders). deloitte.com
- Deloitte, Unlocking Exponential Value with AI Agent Orchestration (TMT Predictions 2026; Google A2A / Anthropic MCP / Cisco AGNTCY protocol competition). deloitte.com
- Ryan Lopopolo, Harness Engineering: Leveraging Codex in an Agent-First World (OpenAI, 2026; 1M+ lines, zero human-authored code, in five months). openai.com
- InfoQ, coverage of Ryan Lopopolo, Harness Engineering: Leveraging Codex in an Agent-First World (2026). infoq.com
- Deloitte, Unlocking Exponential Value with AI Agent Orchestration (TMT Predictions 2026; 80% mature on basic automation vs 28% on agents; 45% vs 12% on three-year ROI). deloitte.com
- Martin Fowler, Harness Engineering (martinfowler.com, 2026; three pillars — context engineering, architectural constraints, maintenance cycles — the structure-over-behavior critique, and the ~2x harness-vs-model quality-impact estimate via Dev|Journal). martinfowler.com
- MasterBorn, Cloud Vendor Lock-in: 4 Real-Life Scenarios (IBM/Oracle multi-year contracts, Zynga AWS-to-Z-Cloud migration and partial return). masterborn.com
- AWS Enterprise Strategy, Switching Costs and Lock-In. aws.amazon.com