Ai2's StateBench finds frontier models lose track of changing values in long documents

Ai2 released StateBench on 3 November, a benchmark for how well language models keep track of values that change over a long document. It has four levels: values set once and never changed; values reassigned, where only the latest counts; updates that build on the current value, such as x = x + 5; and assignments by reference, such as x = y or swaps, the hardest.

Ai2 reports that the frontier models it tested, Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna, did well on the first two levels, but their accuracy on the last two fell sharply as inputs grew, reaching 13% or lower on the longest by-reference problems. Small 275M-parameter Gated DeltaNet and hybrid models trained directly on the task generalised almost perfectly on the first three levels to tens of thousands of assignments; transformers never learned the hardest level. The data, code and technical report are public.