The Remote Labor Index (RLI) is a benchmark that aims to measure how much real remote work AI can do. Forecasting it well could provide insight into the impact of AI on the economy in the near future. We run through the general science of forecasting benchmarks and what this could mean for the RLI.
glVersion ="20260715-131900"// bumped by scripts/deploy-post.sh on every deploygl =import(`/assets/gl-components/gl.js?v=${glVersion}`)glsat =import(`/assets/gl-components/glsat.js?v=${glVersion}`)mc =import(`/assets/gl-components/model-colors.js?v=${glVersion}`)
For all their many flaws, benchmarks are one of the best methods we have for tracking the trajectory of AI capabilities. Beyond tracking the pace of progress to date, benchmarks are also valuable for calibrating our expectations about the future. For instance, the METR time horizons work established a long-running historical trend in the length of software engineering tasks that models could complete 50% of the time, and by projecting that trend forward, predicted the seismic advances that followed.
Our first approach is naive: fit a sigmoid to the points then project it out. From this we derive 50% and 90% crossing points of January and September 2027, respectively.
This approach is limited, however, as it relies only on frontier points, of which there are very few. A better approach involves finding some way to follow the frontier trend while incorporating all observations. An existing paper proposes one such approach by turning the fit into a two-step procedure.
The first step is to go to, instead of straight from date to score, some intermediary general capability metric. This aims to capture the relation between date and the highest available capability. Crucially, as the metric is general, this can be inferred from many benchmarks and, thereby, a greater number of datapoints.
The second step then goes from the metric to actual scores. Again, this can be fit on all points, rather than just the frontier. Given that the capability metric is a more continuous measure than time, it also goes some of the way in suppressing the instability which emerges from model release cycles.
In their original work1, the authors use Arena Elo as their capability metric. We find Arena Elo to no longer work well in this role. The two-step fit requires the metric to have some overall constant trend we can expect to continue into the future; while this was the case for Arena Elo when the paper was published, it ceased to be so several months ago as scores have plateaued. For this reason, we instead opt for the Epoch Capabilities Index as it has been consistently increasing at a constant rate.
Implementing the two-step gives us a second forecast for scores on the RLI. As it explicitly contains some measure of the pace of progress, it is easy to explore how the forecast changes with the expected pace of progress.
rliPimpale =FileAttachment("data/rli_pimpale.json").json()overlayEciTS = ({name:"two-step (ECI)",color:"#7c4dbe",// the rate eases from the fitted clock to the reader's rate via a// smoothstep centred near today; the trajectory is the integral of that// rate, so it is smooth everywhere and the past never movespts: (mult) => {const BASE =13.0, TARGET =13.0* mult;const T0 =+newDate("2026-07-09"), YR =365.25*24*3600e3, W = YR/4;const sm = u => u <=0?0: u >=1?1: u*u*(3-2*u);const rate = t => BASE + (TARGET - BASE) *sm((t - (T0 - W)) / (2*W));const a =+newDate("2025-02-01"), b =+newDate("2028-07-01");const N =480, dt = (b - a)/(N -1);let acc =0;const es = [0];for (let i =1; i < N; i++) { acc +=rate(a + dt*(i -0.5)) * (dt/YR); es.push(acc); }// anchor where the blend BEGINS, so the past never moves with the sliderconst iA =Math.min(N -1,Math.max(0,Math.round((T0 - W - a)/dt)));const off = (161.0- BASE*(W/YR)) - es[iA];const {D, alpha} = rliPimpale[0];return es.map((e, i) => ({t: a + dt*i,v:1/(1+Math.exp(-alpha*(e + off - D)))})); },})glsat.glRliScatter({Plot}, rli, {x:"date",y:"score",label:"pretty",group:"provider",title:"RLI scores",fits: ["sigmoid"],rateInput: viewof rateMult,overlays: [overlayEciTS],width:1120,height:700,displayW:800,yDomain: [0,1],xDomain: [newDate("2025-02-01"),newDate("2028-07-01")],colorDomain:Object.keys(mc.providerColors).filter(k => ["Anthropic","OpenAI","Google","Other"].includes(k)),colorRange: ["Anthropic","OpenAI","Google","Other"].map(k => mc.providerColors[k]),yFmt: v => (100*v).toFixed(0) +"%",})
Enjoyable as it may be to draw these lines, it is not clear whether we should expect them to perform at all well. Intuitively, it seems like such simple approaches should be extremely weak. This leads us to a new question: how well should we expect them to perform? One way to answer this question is to take benchmarks which have since saturated and see how the approaches would have performed on them. We gather 30 benchmarks from Epoch AI and Artificial Analysis, as well as METR’s Time Horizons work, and TerminalBench 2.0 and find frontier trends for each. Every benchmark is shown individually below.
We can model each frontier trend as a sigmoid. The sigmoid produced by modelling the entire trend is the gold standard for our analysis and the basis of our evaluation. This allows us to reframe our task more concretely: we’re trying to find the methodology which lets us get as close as possible to the gold standard, while having seen as few of the earliest points as possible.
In order to determine how much to trust each methodology, we evaluate how they would have performed on prior benchmarks. As the task becomes easier with more of the benchmarks curve being revealed, we consider performance when restricting data in the fit to being within a certain percentage of the earliest point. We then vary that percentage and record the mean absolute error of future points compared to the fit.
We find the two methodologies to be broadly similar, both trending downwards as more of the curve is revealed. This makes intuitive sense – it’s easier to predict how a trend will continue the more you’ve seen of it. Nevertheless, these numbers suggest that in their current form our methods are unlikely to help us much, with >20% expected error based on historic performance. If we want greater confidence in our predictions, our next goal should be identifying the shortcomings in the methods and resolving them, so as to shift those curves down.
The first shortcoming we address is that every new fit goes in blind; it has no prior assumptions about what a saturation curve should look like. This is particularly damaging in the data-scarce setup as with very few points the fit is liable to assume implausible parameters to fit the few points well. This results in extremely brittle fits which do a very poor job forecasting the rest of the curve. We can resolve this by providing some prior to the fit about how trends have panned out in the past, weighting it towards more historically plausible curves than those which fit the immediate data. Applying this fix yields a significant improvement in the low data regime which we care about.
glsat.glMaturityDist(null, btMaturity, {keep: zooKeep,delta:0.2,rliDelta:0.136,stages: [["mid","with the steepness prior","raw","dashed: no prior"]]})
The second problem is specific to the two-step fit and a reliance on a metric which compresses the general notion of capability into a scalar. To understand why it is a problem, we need to step back and remind ourselves what it is we are forecasting here: the frontier scores of benchmarks. With the two-step fit, we aim first to find the leading ECI at a given date, but then second to find only the expected score at that ECI. In this way, we are susceptible to underestimating the frontier which is defined by the outlier models, models which perform better than ECI-expected on a given benchmark. This is evident when we consider the direction of the error of our fits.
We account for this with our next fit by providing a shift parameter that pushes estimates towards the highest score we would see for a given ECI, rather than the mean. We make this parameter benchmark-specific, as it is a function of how well that specific benchmark’s scores are modelled by the ECI.
glsat.glMaturityDist(null, btMaturity, {keep: zooKeep,delta:0.2,rliDelta:0.136,stages: [["full","with the frontier offset","mid","dashed: no offset"]]})
With these improvements made, we can begin to apply our two improved approaches to some individual benchmarks, so as to get an intuition for how they perform.
The graph below is kept up to date with new results which came out after this piece’s publication. In this way, the quality of our forecast will reveal itself with the passage of time.
At the beginning of this piece, we showed two methods for forecasting benchmark scores. Subsequently, we backtested both on since-saturated benchmarks and found weak performance in the low-data regime. We then identified the reasons for this weakness, and implemented fixes. These were verified to robustly improve performance.
We now apply these to arrive at our best guess about how the frontier of scores will progress. We include a shaded band showing the historical spread of forecast error at this maturity. Taken together, our fixed two-step places frontier scores on the RLI at 50% around May 2027 and 90% around May 2028. This is with the implicit assumption that general trends in capability continue as they have. This may not be the case. Explore how the forecast changes if it factors in a speedup or slowdown, with the slider below.
rliFinal =FileAttachment("data/rli_final.json").json()// horizon-resolved error cone from the backtest slice nearest the RLI's own// maturity. Backtest residuals (pred-actual) at this maturity, binned by// forecast horizon (days past the cut-off), are smoothed into monotone// half-widths that grow as c*sqrt(days) - a random-walk envelope, fit separately// up/down, anchored at zero today. So the band pinches to zero at the forecast// line, opens outward, always straddles the curve, and keeps the upside skew// (up >> down: early fits underestimate). Calibrated LOO to ~65% coverage;// NOT a 95% interval (heavier tails, see footnote).rliCone = {const T0 =+newDate("2026-07-09"), DAY =86400e3;const mk = (c) => (H) => H <=0? {up:0,down:0}: {up: c.up*Math.sqrt(H),down: c.down*Math.sqrt(H)};return {eci:mk(rliFinal.cone_eci),time:mk(rliFinal.cone_time), T0, DAY};}// band upper = curve + up, lower = curve - down (both half-widths >= 0, so the// curve is always inside; up is the toward-higher-score side).// bands are functions of the pace multiplier so they track the central line as// the slider moves it (the cone half-width stays a function of calendar horizon;// only the curve it wraps shifts with the pace)rliBand = (mult) => {const clip = v =>Math.max(0,Math.min(1, v)), {eci, T0, DAY} = rliCone;returnfinalCurves(mult).map(p => {const w =eci((+newDate(p.t) - T0) / DAY);return {t: p.t,q16:clip(p.twostep- w.down),q84:clip(p.twostep+ w.up)}; });}rliBandTime = (mult) => {const clip = v =>Math.max(0,Math.min(1, v)), {time, T0, DAY} = rliCone;returnfinalCurves(mult).map(p => {const w =time((+newDate(p.t) - T0) / DAY);return {t: p.t,q16:clip(p.time- w.down),q84:clip(p.time+ w.up)}; });}// the fixed methods, as functions of the reader's pace multiplier. Time is// pace-independent (pure trajectory); the two-step rides a clock whose rate// eases from today's 13 ECI/yr to mult*13 via a smoothstep near today (so the// past never moves); the ensemble averages the two.finalCurves = (mult) => {const EPOCH0 =+newDate("2023-01-01"), DAY =86400e3, YR =365.25*24*3600e3, W = YR/4;const T0 =+newDate("2026-07-09");const sig = v =>1/(1+Math.exp(-Math.max(-500,Math.min(500, v))));const F = rliFinal;const sm = u => u <=0?0: u >=1?1: u*u*(3-2*u);const rate = t => F.rate_base+ (F.rate_base*mult - F.rate_base)*sm((t - (T0 - W))/(2*W));const a =+newDate("2025-02-01"), b =+newDate("2028-07-01"), N =400, dt = (b - a)/(N -1);let acc =0;const es = [0];for (let i =1; i < N; i++) { acc +=rate(a + dt*(i -0.5))*(dt/YR); es.push(acc); }const iA =Math.min(N -1,Math.max(0,Math.round((T0 - W - a)/dt)));const off = (F.eci_today- F.rate_base*(W/YR)) - es[iA];return [...Array(N)].map((_, i) => {const t = a + dt*i;const twostep =sig(F.alpha*((es[i] + off) - F.D_eff));const time =sig(F.time_k*((t - EPOCH0)/DAY - F.time_m_days));return {t, time, twostep,ens: (time + twostep)/2}; });}glsat.glRliScatter({Plot}, rli, {x:"date",y:"score",label:"pretty",group:"provider",title:"RLI scores",bands: rliBand,bandMedian:false,bandColor:"#7c4dbe",bands2: rliBandTime,band2Color:"#2a9d8f",forecastDate:"2026-07-09",rateInput: viewof rateMult2,overlays: [ {name:"time fit",color:"#2a9d8f",pts: (m) =>finalCurves(m).map(p => ({t: p.t,v: p.time}))}, {name:"two-step (ECI)",color:"#7c4dbe",pts: (m) =>finalCurves(m).map(p => ({t: p.t,v: p.twostep}))}, ],width:1120,height:700,displayW:800,yDomain: [0,1],xDomain: [newDate("2025-02-01"),newDate("2028-07-01")],colorDomain:Object.keys(mc.providerColors).filter(k => ["Anthropic","OpenAI","Google","Other"].includes(k)),colorRange: ["Anthropic","OpenAI","Google","Other"].map(k => mc.providerColors[k]),yFmt: v => (100*v).toFixed(0) +"%",})