Silent Loss in Document Ingestion for AI Agents

Background and Context

Enterprise agents rarely read a document directly. Before an agent can follow a policy, answer a compliance question or file a ticket, an ingestion layer must turn a Word or PDF file into tokens. A new arXiv paper argues that this step is where reliability is quietly lost. Authors of policy documents often carry meaning through appearance. They highlight a key requirement, or they set it in a striking color. Mainstream converters tend to drop those signals.

The loss is silent. The agent receives no warning that anything is missing, so it does not look incomplete. It answers with full confidence, and the answer is wrong. This is the central risk the paper names: confident error, instead of visibly answering less.

The authors built their task on 76 real policy documents. The agent must recover the requirements that the author specifically marked. Ground truth comes from the files themselves, through their OOXML properties, which record highlights, colors, bold, italics, monospace and strikethrough. A deterministic matcher scores the answers. No LLM judge is used, so the result does not depend on a second model's opinion.

Deep Analysis

What happens to an agent when its document converter discards highlights and colors? It guesses, and it guesses with confidence. The paper measures this directly. With formatting kept, micro F1 is 0.690. With docling, it falls to 0.150. With plain text extraction, it is 0.075. The gap reproduces across six answering models: five Claude models spanning the Opus, Sonnet and Haiku tiers, plus OpenAI's GPT-5.6. The weakness therefore sits in the input, not in one model. The false-alarm figure explains the confident errors. On requirements the author did not mark, docling's false-positive rate is four times that of the authors' approach. An agent that cannot see the cues cannot tell marked text from unmarked text, so it invents emphasis where none exists. To find which cues matter, the authors ran a span-level leave-one-out test. They erased one cue type at a time and measured the score drop. Holm correction handled the multiple comparisons. Two cues carry almost all the loss: highlights at +0.302 and colors at +0.262, both with p below 1e-4. Bold, italic, monospace and strikethrough each sit within 0.003 of zero. Contrary to common intuition, bold text is not where the damage occurs.

The authors also checked their own labels. They rendered pages as images and verified the labels through an independent visual modality, reaching kappa 0.72. They tested whether wording alone could predict which sentences were marked, and reached an AUC of 0.664. That figure bounds how much a model can infer from text content without seeing the formatting. The same asymmetry between cue types also appeared in PDFs. A preregistered experiment compared two remedies. One only preserves the cue. The other labels the cue by type, so the agent is told that a span is highlighted or colored. Labeling by type adds a further +0.085, with a Holm-corrected p of 0.049. That result only just clears the significance line, so it deserves careful reading and replication.

Industry Impact

The finding matters because the affected cues are common. Across 6,460 documents, 32.8% carry deliberate color or highlighting. In 10.3% of documents, those marks fall on obligation clauses. These are the sentences that compliance agents are asked about most often. Current parsing benchmarks do not score color or highlight preservation, and the mainstream converters do not keep them. This is a blind spot that existing evaluation hides. It may explain a familiar pattern: an agent looks fine in a demo, then errs quietly once deployed against real policy files that people marked up by hand.

For teams building enterprise agents, the practical lesson is to inspect what the document becomes before it reaches the model. Swapping in a newer model will not recover information that never arrived. The repair is also cheap and measurable. Keep colors and highlights at the ingestion layer first. Then label them with their type, which the preregistered test shows adds a further gain. For the open-source community, the evaluation method has value in itself. Labels come from OOXML properties, and a deterministic matcher does the scoring. This removes judge noise and lets others reproduce the numbers. It also makes it straightforward to test new converters or new document types against the same protocol.

Outlook

The paper leaves several open directions. The first is to extend the same test to more file formats beyond the Word and PDF documents studied here. PDF showed the same pattern, but the label source differs there, and the visual cross-check at kappa 0.72 is evidence rather than proof of perfect labels.

The second is converter design. If typed cues help, converters need a standard way to emit them, for example marking a span as highlighted or colored in their output. The +0.085 gain suggests that how a cue is expressed matters, not only whether it survives. The borderline p of 0.049 means this result should be confirmed on other corpora before it hardens into a rule.

The third is benchmarking. If parsing benchmarks begin to score color and highlight retention, converter vendors will have a reason to fix the gap. Until then, teams should audit their own ingestion step, measure how many of their documents carry deliberate marks, and test whether their agents answer differently once those marks are preserved. The paper's core claim is simple: the reliability of an enterprise agent is largely decided at ingestion, before any reasoning begins.