SCoRE Visual RAG: Explicit Context Selection and Evidence Integration via Agent

Background and Context

Visual Retrieval-Augmented Generation, or VRAG, enables models to browse image-heavy documents, retrieve relevant page images as visual evidence, and reason over their content to answer user queries. The paradigm holds strong potential for document question answering and chart understanding. However, the authors identify two structural challenges that have long constrained effective evidence use. First, evidence genuinely relevant to an answer is often extremely sparse: it may be concentrated in a small local region of a single page, or scattered across several pages, making retrieval and localization difficult. Second, existing agent approaches tend to answer from raw exploration trajectories or compressed textual memory rather than from an explicitly organized, traceable set of supporting images. This leaves answers vulnerable to exploration noise and obscures the evidence-based reasoning chain that should remain clear.

The authors argue that the real bottleneck lies not only in discovering evidence but in preserving and organizing it before answer generation. This reframing shifts the research focus from pure retrieval recall toward retention, structuring, and traceable utilization of evidence. It motivates a framework that addresses both sparsity and the absence of organized evidence in a single mechanism.

Deep Analysis

The central contribution is SCoRE, a unified agent loop that couples explicit evidence selection with integration. During exploration, the model browses and searches documents, but it does not feed all raw observations into final reasoning. Instead, it retains only query-relevant observations together with source pointers, writing them into a continuously maintained textual ledger. This preserves key early findings while bounding the visual context, preventing the context explosion that arises when too many pages are retained.

When exploration terminates and the answering phase begins, the system reloads the cited original images according to the ledger pointers and integrates them into a logically ordered evidence set before generating the final answer. This design achieves two decouplings. Final reasoning is separated from the trial-and-error of exploration, shielding answers from direct exploration noise. Meanwhile, indexed argument-to-image links allow every conclusion to be traced back to its source image, ensuring strict visual grounding.

Training follows an end-to-end optimization of the unified roll-out, combining filtered cold-start trajectory distillation with evidence-aware reinforcement learning. The cold-start phase supplies base trajectories of reasonable quality, then filters out inefficient or incorrect samples to provide a reliable starting point. The reinforcement-learning stage uses a specially designed reward function that simultaneously drives three objectives: sufficient evidence coverage, compactness of the integrated evidence, and final answer correctness, balancing exploration against integration.

Industry Impact

The work offers a reusable design pattern for VRAG. By extending the research focus from whether evidence can be retrieved to whether it can be preserved, organized, and traced, it carries practical significance for large, image-rich real-world documents. The combination of a textual ledger with indexed argument-to-image links controls context cost while guaranteeing the interpretability and verifiability of conclusions, making it well suited to trust-sensitive deployments such as enterprise document intelligence, contract review, and technical manual question answering.

The combined training paradigm of distillation and evidence-aware reinforcement learning also provides a reference path for end-to-end optimization of agent-style models, potentially lowering training difficulty and improving stability. For the open-source community, the unified agent loop and evidence-integration approach offer a new reference baseline and room for improvement in visual reasoning, multimodal agents, and long-document understanding.

Outlook

Experiments were conducted on the core task of visual question answering, evaluating overall performance and the effectiveness of each design across datasets and benchmarks. Metrics span both answer-level accuracy and evidence-level coverage and integration quality, measuring whether the model actually finds relevant evidence, organizes it compactly, and produces reliable answers. Ablation studies isolate the role of key components: whether the textual ledger preserves critical evidence while bounding visual context, whether evidence reloading and integration improve visual grounding, and how much each reward objective contributes independently.

The results indicate that with explicit selection and integration, the model maintains stable answer quality even under sparse evidence, that exploration noise significantly lessens its interference with final conclusions, and that the evidence-backed reasoning chain becomes clearer and more traceable. These findings validate the framework's dual advantages in evidence-usage efficiency and reasoning reliability, pointing toward more trustworthy multimodal document understanding in high-stakes settings.