Uneven Yardstick: A Statistical Look at the Validity of METR's AI Time Horizons

METR's "50% time horizon" has become one of the most quoted yardsticks of AI capability in the last two years. The idea is simple to state. Label each software task with the time a human expert needs to finish it. Then ask: at what task length does an AI succeed half the time? The larger the number, the longer the stretch of work the AI can carry alone. Because the unit is minutes and hours rather than a percentage score, the metric is easy for the public, for investors and for regulators to grasp. That is also why it is so often drawn as a curve that climbs exponentially over time, the plot known simply as the METR plot. An easy number is not automatically a reliable number. A paper by Drew T. Nguyen and William Fithian, posted to arXiv in October 2026, looks at that curve through a statistical lens. The authors use data on 228 tasks and 26 AI models and recompute the time horizons from scratch. The standard approach carries a hidden assumption: that the difficulty a task poses to an AI is linear in the logarithm of the human completion time. The authors do not take this on trust. They fit splines and use item-response theory, the family of models that psychometricians use to analyze exam questions, and they let the data decide what shape the relationship should have.

The result is striking. The fitted spline can be read as a conversion function. It takes human time as input and returns AI difficulty as output. Between roughly 2 and 30 minutes, this function is nearly flat. Outside that window it is close to linear. In plain terms, across the middle band a task can become many times longer for a person without becoming much harder for the machine. Once tasks leave the band, each doubling of human time again raises the difficulty in a clear way. A direct consequence follows. Two jumps of the same size, ten times, do not mean the same thing. Moving a time horizon from 3 minutes to 30 minutes sits inside the flat stretch, so it is comparatively easy. Moving from 30 minutes to 5 hours means climbing a far steeper slope. If a reader looks only at the multiplier on the original chart, the two achievements look equal. They are not, and the reader may then misjudge how fast AI capability is really growing. This does not mean the original curve is wrong. It means the vertical axis behaves like a ruler whose marks are packed tightly in one place and spread widely in another.

Beyond this structural finding, the paper offers two practical contributions. First, the authors provide time-horizon point estimates that score better under a cross-validated suite of proper scoring rules. A proper scoring rule rewards honest, well-calibrated probability forecasts, so it gives a fair way to compare competing fits. Second, they supply diagnostic plots for judging the construct validity of a time horizon, that is, whether the number really measures what it claims to measure. With these plots, a user can see directly where the linearity assumption holds and where it fails. For the industry, the message of the paper is restrained and specific. Time horizons should be read together with the diagnostic plots. This matters most when new benchmarks built on time horizons are being proposed, or when existing ones grow to include longer tasks. Long tasks are scarce in any sample. At that end of the scale, the estimate leans harder on the assumed functional form, and the uncertainty is larger. Extrapolating a neat trend line to tasks that last many hours or days therefore calls for extra care. The work has limits, of course. It rests on an existing set of tasks and an existing set of models, and whether the conclusions carry over to other task distributions remains to be tested. Still, it sets a good example for the evaluation community. It does not settle for one eye-catching number. It states its assumptions, checks them, and hands others the tools to check them too. For anyone who plans research, judges products or debates policy using this kind of metric, the paper is a methodological reference worth returning to.

There is a practical lesson for benchmark builders as well. When a new benchmark reports a time horizon, ask how many tasks lie in each length band, and ask whether the shape of the difficulty curve was tested or merely assumed. A flat region means that many tasks of different human durations look alike to the model, so extra tasks in that region add little information about progress. Tasks at the long end, by contrast, carry the most information about the frontier, yet they are the hardest and most costly to collect. Good benchmark design must balance these two facts, and the diagnostic plots in this paper give designers a concrete way to see the balance. Finally, the paper reminds us that a unit of time is a convenient label, not a law of nature. Human duration is a proxy for difficulty, and proxies can bend. Treating the proxy as exact invites confident but fragile forecasts. Treating it as a hypothesis that data can reshape leads to better science. That is the quiet, useful point at the heart of this study.