CPU-ARC: Caching LLM Reasoning to Run on CPUs

Background and Context

Large language models tend to blur two separate jobs. The first job is solving a task: finding the method that works. The second job is executing that method on a concrete input. When a model faces a hard math or science problem, it does both in one long stream of generated text. It searches for a solution and then carries out each step, token by token. The cost of that stream is paid again for every new instance, even when thousands of earlier instances belong to the same task type.

A new arXiv paper, titled CPU-ARC: Caching LLM Reasoning to Run on CPUs, starts from this observation. The authors argue that the blur between solving and executing is a main reason inference stays expensive. Their central claim is simple. Once a task type has been reasoned through, applying its solution to a new instance is efficient. It needs almost no language model.

From this claim the authors build a method called agentic reasoning caching. The method takes the reasoning traces of an LLM and turns them into an agentic reasoning cache, or ARC, that runs on a CPU. In plain terms, the large model thinks once. The cache then serves many later requests. This turns reasoning from a one-time, costly computation into an asset that can be stored and reused.

Deep Analysis

What is agentic reasoning caching, and how does a CPU-executable ARC differ from a normal LLM call? Agentic reasoning caching is a method that extracts a reusable solution from an LLM reasoning trace and stores it as an agentic reasoning cache that a CPU can run directly. A normal LLM call regenerates the reasoning from scratch each time. An ARC keeps the solution in an explicit, runnable form and brings in a language model only when it is truly needed. The hard engineering question is how to turn a long, trial-and-error trace into something stable and executable. The source summary does not disclose the cache representation, the extraction algorithm or the training details. This article therefore describes the method only at the level the authors state publicly. Readers who need implementation detail should consult the full paper. The stated properties still carry clear engineering meaning. First, the ARC is fully interpretable. The solution is an explicit artifact, not a pattern buried in model weights, so a person can inspect why it produced a given answer. Second, the ARC can keep training. New reasoning traces can extend or correct the cache, without retraining the whole source model. Third, the ARC stays robust to input perturbations. This suggests the cached solution captures the structure of the task, not the surface form of one example.

That third property answers a common worry about caches: that they only remember questions they have already seen. Here the cache stores a method, not an answer. A changed input can still be handled by the same method, which is why robustness to perturbation matters for the whole design.

Industry Impact

The reported results focus on a set of hard math and science tasks. According to the authors, ARC performs on par with frontier LLMs on these tasks. This matters because many cost-cutting techniques trade accuracy for savings. The authors present ARC as holding performance level while moving the work off the large model. The cost figure is the headline number. Calls to an ARC cost 16 to 291 times less than calls to the source model. The range is wide, which shows that savings vary by task and setting. Even at the low end, the reduction is more than an order of magnitude. The summary does not name the datasets, give accuracy values or report ablations, so this article draws no conclusions beyond the stated range.

What does it mean for deployment? A CPU is far cheaper to run than a GPU serving a frontier model, and it is easier to provision. Workloads with a narrow set of task types and a very large request volume stand to gain most. Examples in the same family as the paper's tests include batch math solving, scientific computation and structured question answering. In such workloads, the same solution would otherwise be regenerated again and again. Interpretability also changes how teams can audit results. A cached solution is an explicit object. An engineer can read it, test it and fix it. A pure black-box model gives no such handle. For regulated or high-stakes settings, this is a practical advantage, though the paper summary does not test those settings directly. For researchers, the ability to keep training suggests a different pattern of work. Each new reasoning trace can add to a growing library of solutions. The library improves with use, and the expensive model is called less often as coverage grows. This offers one concrete route toward agent systems that learn over long periods.

Outlook

The approach has a clear boundary. It works when a task can be abstracted into a reusable solution. Math and science problems with stable structure fit that condition well. Open-ended problems with no stable structure may not. The summary offers no evidence on such cases, so their results remain untested.

Several open questions follow from what the authors report. How does the cache handle a task type it has never seen? When does it fall back to the language model, and how is that decision made? How large does the cache grow as coverage widens, and how are conflicting solutions resolved? The summary does not answer these. They will decide how far the method travels outside the benchmark set.

Still, the direction is clear and practical. Instead of only compressing or speeding up the model, CPU-ARC saves reasoning that has already been paid for and runs it cheaply. The reported 16 to 291 times cost reduction, combined with parity against frontier LLMs on hard math and science tasks, makes the idea worth close study. Teams that run repetitive reasoning workloads should read the full paper, check the benchmark setup and test whether their own task types fit the cache model.