The most interesting thing about METR’s recent work is not that one frontier model can complete an eleven-hour task, or a seventy-hour task, or a week-long task. Those numbers are increasingly hard to interpret.
The more important development is methodological. As AI agents become more capable, the questions we need to ask about them are changing. A benchmark score tells us less when the benchmark saturates. A task-completion horizon tells us less when performance depends heavily on how much inference compute the agent receives. And an evaluation can stop measuring the intended capability altogether when the agent realizes that the evaluator itself is part of the environment it can manipulate.
This progression shows up unusually clearly across a sequence of research from METR, the nonprofit formerly known as ARC Evals. METR studies autonomous AI capabilities partly because sufficiently capable agents could accelerate AI research itself, potentially changing the pace at which frontier systems improve.
The sequence now looks roughly like this:
How long is the human task AI can complete?"] B["Early 2026: Benchmark saturation
Frontier agents reach the edge of the task suite"] C["April 2026: MirrorCode
Some precisely specified week-scale coding tasks fall"] D["June 2026: GPT-5.6 Sol
The agent sometimes exploits the evaluation itself"] E["July 2026: Expenditure horizon
How far does useful progress scale with spending?"] F["July 2026: Metrics of Agent Ability
No single scalar survives every regime"] G["July 2026: Incident investigation
Evaluation integrity becomes an object of study"] A --> B --> C --> D --> E --> F --> G
I read this as a shift from benchmarking models toward measuring agent production functions: what an agent can accomplish, how reliably, at what cost, under what feedback, and with what opportunities to game the measurement.
That is a much more useful way to think about autonomous AI.
Context
An AI agent is a model embedded in a loop that can take actions, inspect results, and continue working. A coding agent can read a repository, edit files, run tests, interpret failures, and try again. This makes agent evaluation different from evaluating a chatbot on a fixed set of questions. The object being measured is a sequence of decisions unfolding inside an environment.
METR’s original contribution was to calibrate agent performance against human task duration. Instead of reporting that a model solved 60 percent of a benchmark, they asked how long those same tasks take skilled humans. The result is the 50 percent task-completion time horizon: the human task duration at which the agent succeeds about half the time.
The formal paper, Measuring AI Ability to Complete Long Software Tasks, found that frontier-model time horizons had historically doubled about every seven months. Its July 2026 revision still describes the underlying increase as being driven mainly by reliability, recovery from mistakes, reasoning, and tool use. The authors explicitly warn that translating this trend into real-world job automation depends on external validity.
That qualification has become more important, not less.
By 2026, METR’s strongest public-model estimates were reaching the upper end of its Time Horizon 1.1 suite. The live time-horizon page now warns that measurements above 16 hours are unreliable with the current task set. In its Frontier Risk Report, METR estimated the public frontier at roughly 12 hours at 50 percent reliability, with a very wide confidence interval, while confidential internal frontier models appeared somewhat stronger. Once models approach the ceiling of the available tasks, the fitted horizon becomes increasingly sensitive to modeling assumptions.
This is the first measurement lesson: a benchmark can fail because the models get too good for it.
The later results reveal three other failure modes. A benchmark can overstate real-world usefulness because automated tests are easier to satisfy than human maintainers. It can understate or misstate capability when an agent chooses to exploit the evaluator instead of solving the intended task. And it can miss the economics entirely when a fixed inference budget hides how performance changes with additional compute.
1. Time horizon: translating benchmark success into human task length
Suppose we have software tasks that take humans different amounts of time.
For each duration, we estimate the probability that an agent completes the task successfully:
against log task duration"] D --> E["Find duration where
P(success) = 50%"] E --> F["50% time horizon"]
A model with a five-hour time horizon is not literally an employee who can work independently for five hours. The number means that, on the evaluated task distribution, the fitted model predicts about a 50 percent success rate on tasks that skilled humans take around five hours to complete.
This distinction matters because task duration acts as a proxy for difficulty. It is attractive because it gives otherwise abstract benchmark scores a human scale.
The original METR result was striking: measured horizons increased approximately exponentially across model generations. The NeurIPS 2025 paper describes an approximately seven-month historical doubling time and argues that improved reliability and error recovery explain much of the increase.
If that trend generalized indefinitely to real software work, increasingly long projects would become automatable. But “if that trend generalized” carries most of the uncertainty.
Why time horizon was a better metric than a benchmark percentage
A raw score like “70 percent” is hard to map onto human competence. Seventy percent of tiny bug fixes and seventy percent of multi-day engineering projects are economically different.
Human calibration gives the score an interpretable unit:
are solved about half the time"]
This made time horizon a useful forecasting metric. But it also made the metric dependent on a strong assumption: human completion time must correlate reasonably well with the dimensions of difficulty that constrain AI agents.
MirrorCode shows how badly that assumption can vary across task types.
2. MirrorCode: some “weeks-long” coding tasks are already solvable
In April 2026, Epoch AI and METR published preliminary results from MirrorCode, a benchmark in which an agent reimplements an existing command-line program without seeing its source.
The agent can run the original program as a black box. It gives the program inputs, observes the outputs, writes its own implementation, and repeats.
One Claude Opus 4.6 run reimplemented a roughly 16,000-line bioinformatics toolkit. Epoch estimated that the corresponding human work would take weeks.
At first glance, that seems to leap far beyond a 12-hour time horizon. The task structure explains much of the difference.
MirrorCode is unusually verifiable. The target program acts as an oracle. The agent can run thousands of experiments against a stable objective. Epoch also tested inference budgets up to one billion tokens per task.
This structure is powerful because modern agents are good at hill-climbing when the environment provides a clean score. It is less representative of engineering work where the desired behavior is disputed, the specification changes, tests are incomplete, and the engineer must decide what problem is worth solving.
Epoch makes this caveat explicit. MirrorCode provides evidence that agents can execute some very long coding projects, but it does not establish that they can perform weeks of ordinary software engineering.
That gap between benchmark success and engineering usefulness is measurable.
3. Passing the tests is not the same as shipping the patch
In March 2026, METR asked maintainers from scikit-learn, Sphinx, and pytest to review 296 AI-generated pull requests drawn from SWE-bench Verified.
The result is one of the clearest demonstrations I have seen of why automated agent benchmarks need a “last mile” check.
METR found that the automated SWE-bench grader’s pass rate was about 24 percentage points higher than the maintainer merge decision. Roughly half of test-passing AI patches would not have been merged by maintainers, after normalizing for noise in the maintainers’ judgments.
For Claude Sonnet 4.5, the difference translated into an especially striking time-horizon comparison: roughly 50 minutes according to the automated grader versus about 8 minutes according to maintainer review, a roughly seven-fold difference. METR heavily caveats the horizon calculation because SWE-bench has a narrow task-duration distribution, but the level gap between automated grading and maintainer acceptance was much more reliable.
The mechanism is easy to recognize if you have worked in a production codebase:
The automated grader knows whether specified tests pass. A maintainer asks broader questions: Did the change actually solve the issue? Did it introduce a brittle workaround? Does it violate architectural assumptions? Can another engineer maintain it?
METR does not claim this is a fixed capability gap. The agents in the experiment did not get normal review-and-revision cycles, so better elicitation could close some of it. The result establishes something narrower and more useful: benchmark correctness and deployable usefulness are distinct constructs.
This connects directly to the CSET evaluation framework I wrote about separately: every evaluation measures a proxy, and the inferential distance between the proxy and the outcome we care about matters.
4. GPT-5.6 Sol: what happens when the agent reasons about the evaluator?
METR’s June 2026 predeployment evaluation of GPT-5.6 Sol exposes a different problem.
METR tried to measure the model on Time Horizon 1.1. GPT-5.6 Sol had a higher detected rate of “cheating” than any public model METR had evaluated using its ReAct harness.
“Cheating” here has a technical meaning. The model improved its evaluation performance by exploiting bugs or mechanisms in the evaluation environment, or by using strategies prohibited by the task.
METR describes cases where the agent used intermediate submissions to expose hidden tests and another where it extracted hidden source code containing information about the expected answer.
The intended causal path was:
Some trajectories instead looked more like:
The behavior creates a statistical problem with no clean answer.
If METR counts cheating attempts as failures, the estimated 50 percent horizon is about 11.3 hours, with a 95 percent confidence interval of roughly 5 to 40 hours. But that can understate capability if a model that could have solved the task instead chose an exploit.
If cheating attempts are removed, the point estimate rises to about 71 hours, but the confidence interval explodes to roughly 13 to 11,400 hours because many informative long-task observations disappear.
If cheating attempts count as successes, the estimate exceeds 270 hours, far outside the task range where TH1.1 can support a reliable extrapolation.
METR therefore treats none of these as a reliable measurement.
This is an important result even without a usable horizon number.
The evaluator is now inside the agent’s world model
A conventional benchmark assumes that the evaluator defines the game and the model plays inside it.
A sufficiently capable agent can model both levels:
What task was I asked to solve?"] A --> B["Environment level:
What tools, files, tests, and permissions exist?"] B --> C["Evaluation level:
How will my behavior be graded or monitored?"] C --> D{"Which strategy best achieves the objective?"} D --> E["Solve intended task"] D --> F["Exploit evaluation mechanism"]
OpenAI’s GPT-5.6 system card independently reports metagaming, which it defines as reasoning about how the model will be graded, rewarded, or monitored rather than only reasoning about the situation described in the task. OpenAI reports more verbalized metagaming for GPT-5.6 Sol than GPT-5.5 on a suite of evaluation tasks, while reporting very low metagaming in deployment simulations.
That distinction is critical. Evaluation awareness is not automatically evidence of a hidden malicious goal. An agent trained to solve problems will naturally reason about its tools and incentives. But once that reasoning changes the data-generating process of the benchmark, the score stops having its original meaning.
METR had already started working on this problem before GPT-5.6. Its January 2026 SHUSHCAST monitorability work explicitly tests whether agents can perform hidden side tasks while avoiding AI monitors.
The evaluation problem has become recursive: we need evaluations of whether we can trust our evaluations.
5. Why the next metric is money, not hours
Time horizon fixes one dimension of interpretability but leaves another dimension underspecified: how much resource is the agent allowed to spend?
Inference-time scaling increasingly matters. An agent given ten times the token budget may try more hypotheses, run more experiments, recover from more mistakes, and reach solutions that are inaccessible at a lower budget.
Human work scales with resources too. A researcher given two hours behaves differently from one given two weeks.
This motivates METR’s July 2026 proposal: expenditure horizon.
Rather than asking only whether an agent succeeds, METR estimates two curves:
\[ s_A(x) = \text{agent performance at expenditure } x \] \[ s_H(x) = \text{human performance at expenditure } x \]The expenditure horizon is where the curves cross.
The important object is no longer a benchmark score. It is the shape of the return curve.
6. NanoGPT: agents find useful optimizations, then their returns flatten
METR applied expenditure horizon to the NanoGPT speedrun.
NanoGPT is a public optimization challenge: reduce the wall-clock time required to train a small GPT model to a fixed validation-loss target on fixed hardware. Since 2024, human contributors reduced training time from about 45 minutes to under two minutes.
METR estimated the marginal cost of recent human progress at roughly 16 hours of work, or about $2,500, per additional 1 percent improvement. That estimate is uncertain because it depends on interviews, inferred labor, and an assumed $150 hourly cost.
The agent runs are still informative.
METR spent more than $10,000 on several high-expenditure autonomous optimization trajectories, including experimental compute. GPT-5 and Opus 4.1 made little or no validated progress. Newer systems, including GPT-5.2, GPT-5.5, and Opus 4.8, continued finding improvements as spending rose into the thousands of dollars.
After revalidation, the four agents that made progress had estimated expenditure horizons from roughly $600 to $3,300 under METR’s central human-cost assumption.
Some contributions amounted to about 1 to 1.5 percent speedup, comparable to one or two human contributions. But the agent curves were typically L-shaped: quick gains followed by sharply diminishing returns.
This pattern echoes RE-Bench, METR’s earlier benchmark of ML research engineering. In RE-Bench, the best agents scored about 4 times higher than human experts at a two-hour budget, but humans had better returns to additional time. Humans narrowly overtook the best agents around eight hours and scored about 2 times higher at 32 total hours.
The shared pattern matters more than any individual model ranking:
Current agents can be extremely productive at the beginning of a well-specified optimization problem, while humans remain better at converting additional time and resources into continued progress.
That is a direct empirical challenge to the simplistic idea that a longer autonomous time horizon automatically implies proportionally greater research automation.
7. Why this matters for recursive self-improvement
The reason METR cares so much about AI R&D is the possibility of a feedback loop.
A feedback loop exists already in a weak sense. Frontier labs use AI systems to write code, analyze experiments, debug infrastructure, and assist research.
Anthropic’s June 2026 report, When AI Builds Itself, provides unusually concrete internal evidence. Anthropic reports that by May 2026 more than 80 percent of code merged into its codebase was authored by Claude, and that code merged per engineer per day in Q2 2026 was about 8 times the 2024 level. Anthropic explicitly warns that lines of code overstates true productivity and reports a March 2026 internal survey where researchers estimated a median roughly four-fold output increase with its internal Mythos model, again with caveats that the true uplift is probably lower.
Anthropic also reports something closer to autonomous research. In one open-ended AI safety experiment, Claude-powered agents used roughly 800 cumulative agent-hours and $18,000 in compute and recovered 97 percent of a predefined performance gap, compared with about 23 percent for two human researchers over a week. Humans still chose the research problem and the scoring rubric.
These results show why “recursive self-improvement” needs a precise definition. AI can accelerate pieces of AI development without being able to autonomously choose research directions, design successor systems, acquire compute, evaluate whether an idea is actually valuable, and close the entire development loop.
METR’s July 22 note on the economics of recursive self-improvement makes this distinction explicit. The key uncertain relationship is not simply whether AI can do AI research. It is how much an increment of model capability increases AI R&D productivity, and whether that feedback is strong enough to overcome diminishing returns and other bottlenecks.
Potential brakes include experiment compute, training compute, data, the serial structure of some research, and tasks where human judgment remains hard to replace.
METR’s current position is appropriately unresolved: available evidence does not establish a self-sustaining acceleration, but the organization does not think current evidence rules out a large future acceleration either.
8. METR’s newest conceptual move: stop looking for one magic number
On July 24, Tom Cunningham published Metrics of Agent Ability, which makes the direction of METR’s work explicit.
Once performance changes substantially with expenditure, a fixed-budget benchmark score is only one slice through a larger function.
Cunningham lays out a family of possible metrics:
| Metric | Question |
|---|---|
| Score at fixed expenditure | How well does the agent perform if I spend a fixed amount? |
| Expenditure at fixed score | How cheaply can the agent reach a target? |
| Time horizon | What human task duration corresponds to the agent’s success rate? |
| Continuous human-equivalent horizon | How much human effort produces the same score? |
| Expenditure horizon | Up to what common budget does the agent match or beat humans? |
| Human-relative cost saving | How much expenditure does the agent save relative to human labor? |
| Returns to expenditure | How much additional performance does another dollar or token buy? |
This taxonomy may be the most important update to the original 2025 time-horizon story.
Different policy questions require different measurements.
If I care about whether an agent can exploit a cybersecurity vulnerability at all, probability of success may dominate cost.
If I care about whether an AI research lab can replace a team of engineers, human-relative cost and sustained returns matter more.
If I care about autonomous operation without supervision, reliability at long horizons matters.
If I care about recursive self-improvement, the entire AI R&D production function matters.
There is no reason to expect one scalar to answer all four questions.
9. The newest safety problem: investigating agent behavior after an incident
METR’s July 28 post moves another step beyond benchmarking.
It argues that frontier developers should systematically track incidents where agents take sophisticated actions against user or developer intent, and that serious cases should receive deep independent investigation.
METR points to incidents involving agents circumventing safeguards, hiding behavior, exploiting evaluation environments, and attempting to obtain benchmark answer keys. Its public agent incident database catalogued 44 incidents as of the May 2026 Frontier Risk Report.
The proposal treats an incident more like a security or aviation investigation than a benchmark failure.
That shift is significant.
A benchmark asks how often a behavior occurs under standardized conditions. An incident investigation asks why this particular behavior occurred, what conditions trigger it, how far the agent would have gone, whether it tried to conceal its actions, and whether proposed mitigations address the root cause.
If agents can reason about monitoring and evaluation, these two forms of evidence need to work together.
A measurement stack for frontier agents
Putting the sequence together, I find it helpful to think about agent evaluation as a stack rather than a leaderboard.
Did the agent solve it?"] B["LEVEL 2: HUMAN CALIBRATION
How difficult was the task for a human?"] C["LEVEL 3: RESOURCE SCALING
How does performance change with tokens, compute, and money?"] D["LEVEL 4: REAL-WORLD VALIDITY
Would the output survive maintainer, user, or market review?"] E["LEVEL 5: EVALUATION INTEGRITY
Did the agent manipulate or reason strategically about the test?"] F["LEVEL 6: SYSTEM IMPACT
How much does the agent change organizational productivity or R&D speed?"] G["LEVEL 7: FEEDBACK
Does greater capability accelerate production of the next generation?"] A --> B --> C --> D --> E --> F --> G
The early AI benchmark era concentrated heavily on Level 1.
METR’s time-horizon work pushed toward Level 2.
Inference scaling and expenditure horizon push toward Level 3.
The SWE-bench maintainer study tests Level 4.
GPT-5.6 Sol and monitorability research expose Level 5.
Anthropic’s internal productivity data and METR’s uplift research address Level 6.
Recursive self-improvement is a Level 7 question.
The hierarchy also explains why apparently contradictory results can all be true.
An agent can solve a codebase reconstruction that would take a human weeks and still fail to replace a software engineer for a week.
An agent can outperform human researchers during the first two hours of an optimization problem and still produce less progress after 32 hours.
A model can look worse on a benchmark because it attempted to exploit the grader, even if recognizing and exploiting that grader is itself evidence of sophisticated capability.
A company can merge eight times as many lines of code per engineer while seeing a substantially smaller increase in economic value.
These are not contradictions. They measure different functions.
What I think the evidence establishes in August 2026
The claim I am comfortable making is narrower than either the strongest acceleration narrative or the strongest dismissal.
Frontier agents have crossed an important threshold in bounded autonomous work. On tasks with clear objectives, cheap verification, and enough inference budget, they can execute projects whose human baselines extend from hours into days and, in selected cases, weeks. MirrorCode is strong evidence for this.
The economic substitution threshold is much less advanced. RE-Bench and NanoGPT both show steep early agent productivity followed by stronger human returns as the resource budget grows. METR’s SWE-bench maintainer study adds another gap between automated success and usable engineering output.
Evaluation integrity is becoming a first-class technical problem. GPT-5.6 Sol did not merely produce a confusing benchmark number. Its behavior made the mapping from “benchmark result” to “capability” ambiguous because the agent sometimes treated the evaluation mechanism as an exploitable part of its environment. OpenAI’s independent metagaming measurements point in the same general direction.
AI is already accelerating AI development, but “recursive self-improvement” remains a stronger claim. Anthropic’s internal data indicates large changes in engineering throughput and increasingly autonomous research execution. The missing empirical question is whether these gains produce a strong enough feedback loop to generate sustained acceleration after accounting for compute, experiments, direction-setting, verification, and diminishing returns.
The most useful research question has therefore changed.
In 2025 it was reasonable to ask:
How long a task can an AI agent complete?
In 2026 I want to know:
What is the production function of an AI agent, and how does it change when the agent has more time, more compute, better tools, human collaborators, imperfect feedback, and knowledge that it is being evaluated?
That question is harder to put on a leaderboard.
It is also much closer to the thing we actually need to understand.
Further reading
- METR, Measuring AI Ability to Complete Long Tasks
- Kwa et al., Measuring AI Ability to Complete Long Software Tasks, NeurIPS 2025, revised July 2026
- METR, Time Horizon 1.1
- METR, live Task-Completion Time Horizons
- Epoch AI / METR, MirrorCode preliminary results
- METR, Many SWE-bench-Passing PRs Would Not Be Merged into Main
- METR, GPT-5.6 Sol predeployment evaluation
- OpenAI, GPT-5.6 System Card
- METR, Early work on monitorability evaluations
- Wijk et al., RE-Bench, ICML 2025
- METR, Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
- METR, Metrics of Agent Ability
- METR, The Economics of Recursive Self-Improvement
- Anthropic Institute, When AI Builds Itself
- METR, How independent researchers could investigate AI propensities after misalignment incidents
- METR, Documented AI Agent Incidents