There is a gap between what an agent can do once and what it can do dependably, and almost every disappointing AI deployment I have seen lives inside that gap. The model is not the problem. The demo worked. What failed was the loop around the model — how many turns it could run before it lost the thread, what it did when a tool returned garbage, whether anything checked the output before it reached a customer.
I have come to think of that loop as the actual engineering surface. Not the prompt. Not the model choice. The loop: what goes into context, what comes back out, what gets verified, and what happens on failure. This is the work.
The metric that made the gap legible
For a long time the industry had no honest way to describe agent progress. Benchmark scores went up while day-to-day usefulness stayed stubbornly patchy, and nobody could say why. METR's time-horizon work resolved that contradiction with a single reframe: measure capability in units of human time — the length of task, as timed on human professionals, that a model completes at a given success rate.1
The resulting picture is stark. Frontier models succeed at close to 100% on tasks that take a skilled human under about four minutes, and under 10% on tasks that take more than about four hours.2 Human task duration turns out to be strongly predictive of whether a model will succeed at all1 — which is another way of saying the binding constraint is not knowledge or skill at any single step. It is holding a long sequence together.
Reliability is a different curve entirely
The headline doubling figure — task length at 50% reliability doubling about every seven months1 — is the number everyone quotes. It is also the number most likely to mislead an operator, because 50% is not a shipping threshold. Nobody deploys a workflow that fails half the time.
Raise the bar and the horizon shrinks hard. For Claude 3.7 Sonnet, METR reported a 50% horizon of 59 minutes against an 80% horizon of roughly 15 minutes — the 80% horizon lands near a quarter of the 50% figure.3 Encouragingly, both thresholds improve at nearly the same rate, with doubling times of 213 and 212 days respectively,3 so the reliable horizon is not falling behind — it is just running about two doublings, roughly fourteen months, behind the headline.3
Even 80% understates the difficulty, because a single-attempt score hides why a run failed. Recent work on agent reliability argues that pass@1 cannot distinguish a model that consistently muddles through a two-hour task from one that either nails it or fails catastrophically, run to run — the aggregate is identical, the operational risk is not.4 If you are staking a client deliverable on an agent, variance across repeated runs matters more than the mean.
Capability tells you what is possible. Reliability tells you what is shippable. They are separated by about fourteen months of model progress — or by a loop you build yourself, now.
Prompting, then context, then harnesses
The field's own framing has moved three times in five years. A recent roadmap of long-horizon agent research splits it cleanly: prompt engineering from roughly 2020 to 2023, context engineering from 2023 to 2025, and runtime harnesses from 2025 onward.5 Each stage moved the locus of engineering further out of the model and further into the system wrapped around it.
Practitioners felt the last shift as a step change rather than a gradient. Harrison Chase of LangChain has described a distinct vibe shift around late 2025, when models became good enough that the field "moved from scaffolds to harnesses" — and noted that the durable idea underneath is almost embarrassingly simple: run the model in a loop and let it choose what to pull into context.6 The simplicity is the point. The generality of the loop is what makes it worth engineering carefully.
Why append-only loops fail
Most agents in production are still descendants of ReAct: an iterative reason–act–observe cycle that appends every step to a growing transcript.7 That design is elegant and it does not survive length. Because the context is append-only, long-horizon tasks hit context saturation — critical signals get buried in accumulated noise and reasoning degrades.7
The failure is not simply running out of window. Studies of long-context web agents report success rates of 40–50% on short-horizon variants of a task falling below 10% once the same task is embedded in a longer interaction history — even when the relevant information is still technically inside the context window.2 The information was present and the agent could not use it. That distinction is the whole argument for managing context as an explicit engineering concern rather than trusting a larger window to fix it.
The research response is a family of techniques that treat the transcript as something to be actively sculpted. Context-folding lets an agent manage its own working context rather than accumulate it.8 MEM1 trains agents to operate with effectively constant memory across long multi-turn tasks.9 Anthropic now publishes context engineering as a first-class engineering practice rather than a prompt-writing tip.10
The loop is the product
The clearest evidence that this is engineering rather than prompt craft: optimising how context is assembled beats hand-tuning it, by a lot. On SWE-bench Verified, Meta Context Engineering reports 89.1% against 70.7% for hand-engineered baselines.11 Same models. Same benchmark. Eighteen points from the assembly layer.
That is the number I would put in front of anyone still treating the loop as glue code. If a systematic approach to what enters context outperforms an expert's manual assembly by that margin, the loop is not plumbing around the intelligence. It is where a large share of the realised intelligence actually comes from.
How this changes what I build
Across the Gigaverse ventures the pattern that has held up is unglamorous: decompose long work into segments short enough to sit inside the reliable horizon, then make the seams explicit. Every segment gets a defined input state, a verification step that does not trust the agent's own account of its work, and a written hand-off. When something fails, it fails inside one segment with a recoverable state, instead of somewhere in hour three of an unrecoverable transcript.
Concretely, that means four things I now treat as non-negotiable. Bound the horizon — if a job needs two hours of sustained agency, it needs to become several jobs. Manage context deliberately, folding or re-stating rather than appending. Verify outside the loop, because a step that grades its own homework is not a control. And measure variance, not just averages, because the distribution across runs is what determines whether you can promise an outcome to a client.
None of this is exotic. It is the same instinct that produced CI pipelines and staged rollouts, applied to a component that is probabilistic instead of deterministic.
Planning against the trend
The last question an operator has to answer is how long to wait. If the reliable horizon is fourteen months behind the headline, the temptation is to sit still and let the models arrive.
The trend does not obviously reward waiting. The seven-month doubling has held across six years and is robust to substantial measurement error — METR notes that being off by a factor of ten shifts arrival dates by only about two years, because the curve is so steep.1 More recently the slope may have steepened: fitted to 2024–2025 data alone, horizons doubled roughly every four months.12 Under the seven-month rate, a full working month of human effort — about 167 hours — arrives at 50% reliability around late 2029; under the faster fit, meaningful probability lands in 2027.13
The uncomfortable implication of the trend is that the loop you build now is temporary. Some of today's careful scaffolding will be redundant in two years — Chase expects some manual scaffolding to fall away as models improve.6 That is fine, and it is not an argument for waiting. The segmentation, the verification gates, the instrumentation of variance: those are properties of a system that ships reliably, and they will outlast whichever model is underneath.
What I would watch over the next year is not the headline horizon. It is whether the 80% curve keeps pace with the 50% curve, and whether variance across runs narrows. Those two are what convert a capable agent into a business you can make promises about.