Ai2 Releases StateBench, a Four-Level Long-Context State Tracking Benchmark: Frontier Models Drop Sharply on Self-Referential and General Problems as Inputs Grow, Directly Trained Hybrid and GDN Models Generalize Farther, but All Tasks Are Synthetic

On November 3, 2026, the Allen Institute for AI (Ai2) published a post on the Hugging Face blog introducing StateBench, a benchmark for long-context state tracking, meaning how well language models keep track of information about entities in long documents as those entities change. The technical report is on arXiv (2611.03094, submitted November 2, by Jackson Petty, William Merrill and Ashish Sabharwal), the data is on Hugging Face, and the code is on GitHub. One point comes first: the tasks in this benchmark are synthetic sequences of variable assignments, not real documents, so the results describe behavior on controlled tasks and are not a failure rate for real use.

StateBench has four difficulty levels, which map onto variable assignment in programming. R-trivial problems use immutable assignment: a value never changes, so the model only has to recover what it was first set to, like a birth date. Aperiodic problems allow reassignment by value, so the model needs the most recent value, like a thermostat setting. Soluble problems add self-referential updates, where each update builds on the current value, like a game score. General finite problems add assignment by reference, where one variable takes another variable's current value, as in swapping two variables, like a magician hiding a ball under one of five cups and swapping them repeatedly. That is the S5 problem, the hardest level. The levels also reflect theoretically motivated distinctions from Krohn–Rhodes theory. For a fair comparison, the researchers start from one general finite sequence and derive three easier versions with the same number of variables, the same distinct values and the same assignment positions, so that only the type of assignment changes. The same underlying problem is also posed in several formats, such as RULER-style prompts and spreadsheet edits.

The first test used open pretrained models: two 7-billion-parameter models, Olmo 3 (a transformer) and Olmo Hybrid (which adds Gated DeltaNet, or GDN, layers), plus smaller models with 275 million and 2.7 billion parameters. All were pretrained on the same data, none had seen contexts longer than 64,000 tokens in training, and most were tested well beyond that. Almost all models did well on r-trivial problems, lost accuracy on aperiodic ones as inputs grew, and stayed at chance on general finite problems. Format mattered: the 275-million-parameter hybrid and GDN models scored near the ceiling on RULER-style r-trivial problems but near chance on the same problems framed as spreadsheet edits. That gap is one reason StateBench poses the same problem in several formats.

The second test used four frontier models, Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna, which were asked to read each sequence of variable assignments and write the final value of every variable. All four did well on r-trivial and aperiodic problems. On soluble and general finite problems, accuracy fell sharply as inputs grew, to 13% or lower on the longest general finite problems. The post names one likely factor, sensitivity: at the harder levels many more steps can change the final answer, so missing one update is more likely to ruin the result. The post gives no reasoning settings, number of repeats or tool use for these runs, and those details would affect any comparison.

The third test was direct training. The researchers trained several 275-million-parameter models from scratch: two transformer variants, two hybrids and two GDN variants with different memory-update constraints. Training sequences had up to 256 assignments, and tests went up to 64,000 assignments, which is 250 times the maximum training length. As sequences grew the architectures diverged. GDN and hybrid models generalized almost perfectly on r-trivial, aperiodic and soluble problems out to the longest sequences tested, tens of thousands of assignments. Transformers learned the easier levels but degraded on longer sequences, and they never learned general finite problems. On the hardest level, the GDN variant with a more flexible memory update generalized the farthest, staying near perfect out to roughly 10 times the longest training sequence. The post concludes that for GDN and hybrid models the gap points to training rather than architecture, since trained directly they handled levels their pretrained counterparts could not, while transformers never learned the hardest level even when trained directly.

The post also lists open questions. A new proof shows that GDN with a more limited memory update, with non-negative eigenvalues, can solve all aperiodic tasks and certain periodic tasks, such as telling whether a repeated input has occurred an even or odd number of times, under the paper's fixed-precision arithmetic with rounding. Note that the post says non-negative eigenvalues while the arXiv abstract says positive eigenvalues; the two wordings differ and are shown here as written. Its exact place in the hierarchy remains open, including whether it can solve all soluble tasks and how numerical precision or combining it with attention layers matters. StateBench also assumes a fixed number of variables, while real tasks often introduce new entities, as in a long conversation where new people and objects appear, and the current framework does not cover that.