Elastic Latent Recurrence (ELR): One Checkpoint That Matches a Same-Size Standard Model at One Loop and Buys More Accuracy With Each Extra Loop

The idea of a "latent recurrent language model" is direct. Instead of stacking dozens of separate layers, you reuse one small block of layers several times, and each extra loop costs one more unit of inference compute. One checkpoint can then trade speed for accuracy: fewer loops are faster, more loops are more accurate. There has been an awkward problem, though. At one loop such a model usually falls clearly behind a normal non-recurrent model of the same size. Paper 2610.12582 calls this gap the "one-loop tax", and notes that when latency matters, people have to choose a separate non-recurrent model, which defeats the point of "one checkpoint, many uses". The Elastic Latent Recurrence method, ELR, aims for this: at one loop the model costs the same to serve as a non-recurrent model of the same size and, trained on the same data, reaches similar accuracy, and each additional loop then buys more accuracy for more inference compute.

The authors credit three choices. First, a gated state update with a learned step size, so each pass makes a controlled change to an existing state. Second, loop counts mixed within each optimizer update: for each microbatch a loop count is sampled uniformly and the loss is backpropagated through all the applications of the block. To keep the one-loop accuracy, a rotating fraction of the data-parallel ranks is forced to use one loop, so that one loop has a probability of about one third, and the one-loop and multi-loop gradients are averaged in the same update. Third, each pass is conditioned on its progress in the loop and on the total budget, so the same block can do different things on different passes. Ablations show the common alternatives do not keep the one-loop accuracy. The first two choices preserve it, and the third lowers the loss at every additional loop.

The paper also looks at cost. First, mixed-depth training can be introduced late, in the final quarter of pretraining, for an estimated 16% more training FLOPs than the baseline, and with zero-initialized conditioning added at the switch, this late schedule reaches lower loss than mixed-depth training from scratch. Second, in MoE models, reusing first-loop expert assignments on later loops cuts mixed-depth update time by 11.7% with no measurable change in accuracy. Third, standard inference keeps a separate attention KV cache for each loop, so memory grows with depth. In the post-trained Koi-4B, three of the six attention layers sit in the recurrent block, so four loops need 2.5 times the one-loop KV memory, or 7.5 GiB against 3.0 GiB per 128K-token request in BF16. Keeping only the last loop's KV cache needs no retraining and holds attention KV memory at the one-loop size at every loop count, with score reductions at four loops of 0 to 3% against per-loop caching.

The main results are these. On 4B dense and 30B-A3B MoE hybrid models, ELR at one loop is 0.52 points below and 1.04 points above its matched baseline on a ten-task average, and four loops add 7.25 and 4.75 points. On the 4B model the biggest per-task gains are GSM8K, up 11.8 points, and MATH-500, up 11.3; on the 30B-A3B model they are MMLU-Pro, up 7.5, and GSM8K, up 6.5. The cost shows in latency. A one-loop pass runs each unique layer exactly once, so time to first token and per-token decode latency match the non-recurrent baseline. Each added loop repeats only the recurrent block, and on the 4B model four loops reach 2.33 times the baseline time to first token and 2.06 times the decode latency. Three loops capture 96% of the gain from one to four loops at 1.87 times. Carried through post-training, the recipe gives one checkpoint, Koi-4B, which scores an estimated 18 at one loop, matching Qwen3.5-4B, and 29 at four loops, using the weighting of the Artificial Analysis Intelligence Index v4.1.

How should we read this? First, every number comes from the paper's own experiments. The authors state that the scores for Koi-4B and the reference models come from their own harness, not from Artificial Analysis, and may differ from published figures, and Koi-4B has no matched non-recurrent baseline. Second, "more loops for more accuracy" is not free. Latency, training complexity and deployment support all matter, and four loops more than double the time to first token. Third, the ablations are mostly on smaller models, so scaling up needs later checks. Still, if a product needs both a low-latency quick answer and several rounds of thought on hard questions, the "one checkpoint, adjustable depth" idea is far more attractive than keeping two models.