Vol. III · Essays on work after AI
Saturday, October 3, 2026
evolving.work
Strategy · Architecture · Essay № 011

The Cheap Experiment

The cost of experimentation just collapsed to near-zero — and the organizations that recognize this will run thousands of experiments while competitors are still planning their first.

H Héctor Reinoso Apr 2, 2026 12 minute read Share · Save

In March 2026, Andrej Karpathy pointed a 630-line Python script at a single GPU and went to bed. By morning it had run 700 experiments, found 20 optimizations that transferred to larger models, and cut the time to train GPT-2 by 11%. No research team. No compute cluster. No months of grant proposals and careful hypothesis formation. Karpathy described it on X with characteristic understatement: “All LLM frontier labs will do this. It’s the final boss battle.”

Just an AI agent, a modifiable file, a metric, and a feedback loop that ran twelve times an hour.

I keep turning this over. Not because autonomous ML research is new — people have run hyperparameter sweeps for a decade. What strikes me is how clearly it crystallized a pattern that extends far beyond machine learning. Analyst Janakiram MSV called it “The Karpathy Loop”: an agent with access to a modifiable system, a single objectively testable metric, and a fixed time limit per experiment. When those three preconditions are met — and they are met in far more domains than most people realize — the throughput is roughly 100 experiments overnight at a cost measured in dollars of compute, not months of researcher salaries.

The pattern reproduces. Within months, Red Hat’s developer team ran the same playbook on OpenShift AI and logged 198 experiments with zero human intervention — surfacing counterintuitive findings the human team hadn’t tried, like smaller batch sizes with no gradient accumulation consistently beating larger-model approaches under a fixed time budget. That is what makes this different from a one-off stunt. The loop generalizes the moment you hand it a sandbox and a metric.

We have spent decades treating experimentation as something to be rationed. Plan carefully. Staff appropriately. Budget conservatively. Run the experiment you can afford, not the hundred that would actually tell you something. The cost of that rationing shows up in the survey data: a KPMG study cited by Strategyzer found 80% of CEOs believed their business models were at risk and only 6% of executives were satisfied with their innovation performance. Everyone knows they should run more experiments. Almost no one does.

Ideas were not the scarce part. Paying to test them was. With cheaper tests, we can learn from more of the ideas we would previously have left untried.

The CEO Who Ran the Loop

Within days of Karpathy’s release, Shopify CEO Tobi Lutke ran autoresearch overnight on a machine learning model. Thirty-seven experiments. A 19% performance improvement. A 0.8B parameter model that outperformed the 1.6B model it was designed to replace — on a single GPU, with no ML team.

Interesting, but you could squint and call that a fancy hyperparameter sweep. What happened next is what caught my attention.

Lutke pointed the same pattern at Shopify’s Liquid template engine — a 20-year-old codebase powering every storefront on the platform, with 974 unit tests and two decades of human optimization baked in. The kind of codebase where performance gains are measured in single-digit percentages, if they are found at all. The agent ran roughly 120 experiments overnight, each time modifying source code, running all 974 tests, benchmarking parse-and-render performance, and keeping or discarding changes based on the metric. The result: 53% faster combined parse-and-render time, 61% fewer memory allocations, zero test regressions.

These were not parameter adjustments. The agent replaced the StringScanner tokenizer with a faster byte-index approach. It pre-computed frozen strings for integers 0-999 to avoid hundreds of repeated allocations per render. Structural changes experienced engineers had not attempted in twenty years — not because the engineers were bad, but because when you can only run a few experiments per week, you optimize conservatively. You tune what you know. You do not rip out the tokenizer on a hunch.

Karpathy himself drew the distinction: neural architecture search “is such a weak version of this that it’s in its own category of totally useless by comparison.” AutoML asks “which configuration from this predefined search space?” Autoresearch asks “what if we tried an entirely different approach?” One searches. The other hypothesizes. A hyperparameter sweep is a tourist visiting every restaurant on the same street; autoresearch wanders into neighborhoods the tourist did not know existed.

The Liquid case also answers the Goodhart’s Law worry — that optimizing a single metric at scale degrades everything else. The failure mode is not hypothetical. One Karpathy autoresearch run was widely discussed after the agent “veered off” from its assigned memory-efficiency objective and burned roughly 12 hours of compute exploring the minimal number of model weights needed to preserve performance. Interesting result, not the one anyone asked for.

That is exactly the problem the architecture has to solve. Lutke did not optimize speed in isolation — those 974 unit tests ran alongside every experiment, enforcing correctness as a hard constraint while the agent optimized speed aggressively within that boundary. The pattern is not “optimize blindly.” It is “optimize aggressively on one metric, verify against a broader quality gate before accepting results.”

The Economics of Near-Zero

The experiments become easier to afford because the cost of the AI calls underneath them has fallen.

AI inference costs dropped 280-fold in 18 months — from $20.00 to $0.07 per million tokens at GPT-3.5 performance level, per the 2025 Stanford AI Index. At those prices, one dollar buys 10.77 million words of AI processing. The marginal cost of running an AI-driven experiment is approaching the marginal cost of having the idea for one.

And the collapse is accelerating: Epoch AI’s tracking shows the median rate of inference price decline jumped from 50x per year to 200x per year post-January 2024. This is not a plateau. If your mental model of AI costs is a year old, it’s already an order of magnitude too expensive.

Someone will reasonably note that Karpathy’s autoresearch ran on an H100 GPU that costs $30,000. Fair — but the pattern generalizes beyond GPU-bound training. Optimizing code with an LLM agent costs API tokens, not GPU hours. The Shopify Liquid case ran its entire feedback loop in software: agent-driven code changes evaluated by an existing test suite. At current inference prices, running hundreds of agent-driven experiments on code, marketing copy, or process optimization costs dollars. Not thousands. Dollars. The methodology existed in principle before; the economics did not.

Beyond Software: When Molecules Run the Loop

If cheap experimentation only worked for code, it would be useful but narrow. What’s genuinely striking is how the same pattern shows up in domains where experiments have historically been measured in years and billions.

Insilico Medicine needed a drug candidate for idiopathic pulmonary fibrosis — a progressive, fatal lung disease with limited treatment options. Traditional drug discovery takes 10-15 years and costs $1-2 billion per approved drug, and most of that goes to the discovery phase: iterative wet-lab experimentation to identify targets, design molecules, and optimize candidates.

Insilico’s AI platform identified a novel biological target (TNIK) that no human team had pursued for this disease, then generated and evaluated thousands of candidate molecules computationally. The discovery-to-preclinical phase took 18 months at a cost of $2.6 million, versus the industry average of 4-6 years and roughly $430 million for equivalent scope.

The resulting drug, Rentosertib, became the first in history where both the biological target and the therapeutic compound were discovered by AI. It is now in Phase IIa clinical trials, with results published in Nature Medicine. Clinical trials still take years — the regulatory pathway has its own logic. But the discovery phase, fundamentally an experimentation problem, compressed by orders of magnitude. A systematic review of 173 AI drug discovery studies in Pharmaceuticals found that every single one demonstrated timeline compression. Not most. All of them.

At NC State University, self-driving laboratories are restructuring what a chemistry experiment even is. The traditional approach: set up a synthesis, wait an hour for the reaction, collect one data point, adjust, repeat — five to ten experiments per day. NC State’s dynamic flow experiments capture data every 0.5 seconds, producing 20 data points per experiment instead of one. One day’s output equals three years of traditional lab work.

The pattern I keep noticing is not just “run experiments faster.” It is “redesign the experiment to maximize information per cycle.” NC State did not speed up the old method — they invented one that generates 20x more signal per run. When you collapse the cost of a cycle, you can afford to rethink what a cycle even looks like. The bottleneck moves from “how do we afford the next experiment” to “how do we extract more from each one.”

Where the Loop Breaks

Autonomous experimentation depends on conditions that many worthwhile problems do not meet.

The Karpathy Loop has three hard requirements: a modifiable system, a measurable metric, and a fast feedback loop. Remove any one and the pattern collapses. Organizational culture changes take quarters to evaluate. Brand perception resists quantification. Clinical drug trials — even with Insilico’s compression of discovery — still take years of human oversight before approval. The methodology does not solve slow, irreversible, or fundamentally subjective problems.

It also does not solve the cost problem as cleanly as the headlines suggest. Cerebras’ analysis of autoresearch runs noted that “when each experiment costs real compute, proposal quality dominates total cost,” and showed the agent burning through 2 hours of GPU time on proposals that were ultimately rejected. A 2026 arXiv paper went further: a classical Tree-structured Parzen Estimator with expert-designed hyperparameters could match or beat Karpathy’s agent on the same budget. Autoresearch is powerful, not magic — on well-structured problems, mature optimization methods are still competitive.

The deepest objection is about understanding. Karpathy’s agent found 20 optimizations in 700 experiments, but it does not know why they work. It modified code, measured improvement, kept the change — hill-climbing, not science. The scientific method is hypothesis-test-explain-iterate. An agent that discovers a better learning rate schedule produces a better model but no transferable insight.

I take this objection seriously. Autonomous experimentation can narrow the search; human understanding explains what the useful results mean. The agent narrows a search space from infinite to tractable; humans focus explanatory effort on the 20 winners rather than the 680 that failed. Karpathy’s framing is instructive — “the goal is not to emulate a single PhD student, it’s to emulate a research community of them.” The community handles the volume. The synthesis still requires human minds.

And most organizational experimentation is not frontier science. It is applied optimization where “what works” matters more than “why.” A 53% speedup on Shopify’s Liquid engine is valuable whether or not anyone writes a paper explaining the mechanism. The engineers who review the PR will understand the changes; they don’t need the agent to have understood them first.

The boundary is not static, either. Self-driving chemistry labs scored poorly on the readiness test five years ago. Then researchers deliberately re-engineered their feedback loops from hours to sub-seconds, and the pattern became viable. The honest framing is not “this works for everything.” It is “this works for a large and growing class of problems, and the boundary expands as we learn to digitize feedback loops.”

Your Move: Scoring What You Can Automate

To find a suitable problem in your own domain, I would start with the conditions that made these loops work.

Score your problem on three axes. Modifiable system: fully digital and agent-accessible (3), requires integration work (2), or physical and needs human translation (1)? Testable metric: a single clear metric with automated evaluation (3), a metric that needs a proxy or partial subjectivity (2), or no clear metric (1)? Fast feedback: cycle time under 10 minutes (3), under 1 hour with redesign potential (2), or days or longer (1)?

Score 7-9: proceed — your problem is ready for autonomous experimentation today. Score 5-6: invest in infrastructure — digitize the feedback loop, automate the metric, build the integration layer. The investment is now the bottleneck, not the economics. Score 3-4: not yet viable, but that’s a roadmap, not a verdict. Monitor for enabling technology, or find a narrower sub-problem that scores higher.

Then write your own version of Karpathy’s program.md — the constraint document that turns autonomous iteration from reckless to powerful. Specify what the agent must optimize, what it must not break (your equivalent of Shopify’s 974 unit tests), what approaches are excluded, and when to stop. The constraint document is what separates autonomous experimentation from unsupervised chaos.

The cost of experimentation just fell through the floor. Software optimization, marketing copy, pricing strategy, financial modeling, logistics routing, manufacturing parameters — any domain where you can measure the outcome and close the feedback loop fast enough is a candidate. The organizations that thrive in the next decade will not be the ones with the best single experiment. They will be the ones who realized, before everyone else, that the cost of trying just became negligible — and started running.

References

AI strategyAI transformation
About the author
Héctor Reinoso writes about how organizations should be designed when intelligence becomes abundant. He works on enterprise AI adoption inside a large industrial organization, after 25 years as a strategy consultant and transformation executive across four continents. Harvard MBA, industrial engineer, and still writes the code he writes about. More →
Continue reading
Strategy · Sep 29, 2026

Ten Agents, 250x the Tokens

Forecasts of 24x token-consumption growth by 2030 are not aggressive, they are too low, because they model usage as if it grows linearly per user.
Architecture · Jul 7, 2026

Frontier Models Are Becoming Operating Models

Frontier models are becoming operating models for cognition, not just tools — and that changes the strategic question
Strategy · Oct 3, 2026

Twelve Agents, €200, and a Consultant With a Name

When agents make research cheaper, consulting must earn its fee through the structure of the analysis, the quality of judgment, and clear accountability.
Org Design · Oct 2, 2026

When Coding Gets Cheaper, the IT Handoff Gets Harder to Justify

If your supplier uses coding agents while you still specify and verify the work, it is time to reconsider what you are outsourcing.
Architecture · Oct 1, 2026

From Writing Code to Managing Agents

Coding agents let developers work with larger units than individual lines. The skill is knowing how to delegate, check, and investigate.
Org Design · Sep 26, 2026

The Myth of the 50-Agent Founder

The fantasy of a solo founder running a company with 50 agents misunderstands what agents do to cognitive load.