OpenVLHarness: Multimodal Memory for Visual Agents

Background and Context

Multimodal LLMs have improved quickly at visual reasoning. Yet a share of visual tasks still defeats them. The OpenVLHarness paper argues that these tasks lack evidence, not thinking time. Some need precise perceptual grounding. Some need outside knowledge. Some need numbers that only a tool can measure. A model cannot derive such evidence by reasoning internally.

The authors tested this claim in a controlled study on 22 benchmarks. They raised the reasoning budget of the base model and watched the average score. The gain was limited. More internal reasoning did not close the gap. The model had to fetch evidence from outside itself.

From this result the authors built OpenVLHarness, a reasoning harness for multimodal models. The contribution is not a new model. It is a runtime framework built around a capability layer and persistent multimodal memory. Within one query, the model can acquire evidence, inspect it and combine it.

Deep Analysis

The framework has three parts that work together. The first is the capability layer. It pairs task-level tool interfaces with evidence rendering. The interfaces let the model call heterogeneous backends in one uniform way. The rendering step turns each backend's raw output into an observation the model can read. This matters because tools return very different things: images, structured data and web content. Passed on raw, such outputs are hard to read and can swamp the context. The second part is persistent multimodal memory. Images, structured data and web references from tool calls are stored as addressable artifacts. They stay available across the whole query. A later step can refer back to a stored image or table by its address. The model does not need to reload it into the context, and earlier evidence does not fall away as the dialogue grows. The third part is a code interface. The model can generate programs, and those programs run over the data held in memory. A task such as comparing or aggregating numbers from several tools can then go to code. The model does not have to compute it in its head. The loop is simple: gather evidence, check it, then assemble the answer with code. How much does OpenVLHarness improve over a no-tool baseline? The paper evaluates eight open and closed backbones on 22 benchmarks. OpenVLHarness beats the matching no-tool baseline by 9.7 to 12.6 points on average. Qwen3-VL-8B gains 11.2 points, Kimi K3 gains 9.7 and GPT-6 Astra gains 9.9. A small open model and a strong closed model both benefit, so the gain is not tied to one model family.

The cost comparison is the most telling result. GPT-6 Luna at medium reasoning effort beats three rivals: the base model at maximum effort, Codex and Visual Sketchpad. The lead is 7.7 to 8.5 points. Its model token cost is at most 37% of theirs. Spending budget on evidence and its composition pays better than spending it on longer internal reasoning. That matches the controlled study at the start of the paper.

Industry Impact

For builders of agent systems, the paper points to a practical shift. The engineering effort moves from stretching the model's reasoning time to three design questions. How are tools connected? How are results presented? How is evidence stored? A well-designed harness around a mid-effort model can give better results at lower token cost. That is attractive for teams with tight inference budgets.

The design also helps modularity. A uniform tool interface and addressable memory artifacts make backends easier to swap or extend. The name and positioning suggest an open-source audience. A team could replace one perception or retrieval backend without rewriting the reasoning loop.

The comparison with Codex and Visual Sketchpad is also useful for tool-using agents in general. Both are existing approaches that give the model more than plain reasoning. OpenVLHarness still leads them, and it uses fewer tokens. This suggests that how outputs are rendered and kept matters, not only whether tools exist.

Outlook

Several questions remain open. The source summary reports average scores and cost. It gives no per-component ablation numbers. How much the capability layer, the memory and the code interface each contribute is therefore unknown here, and this article draws no conclusion on it. Failure cases are also not described in the material provided.

Persistent multimodal memory raises follow-up research problems. How should a long workflow organize and retrieve its image and data artifacts? How can code and perception tools cooperate more closely? How can the quality of evidence be measured on real-world tasks? Answers to these would show whether the 9.7 to 12.6 point gains hold beyond the 22 benchmarks.

For practitioners, the immediate lesson is concrete. Before paying for maximum reasoning effort, test whether the task lacks evidence. If it does, a harness that fetches, stores and computes over that evidence may beat a larger reasoning budget, as the GPT-6 Luna result at 37% of the token cost suggests.