Inject, Align, Recover: A Phased Post-Training Approach for Retrieval-Free Document Knowledge Internalization

Background and Context

Large language models frequently fail to answer questions about a fixed set of documents unless they retrieve the source texts at inference time. A new arXiv paper addresses this limitation by formally defining the scenario as document knowledge internalization, the process of converting a fixed corpus into parametric knowledge that models can use for retrieval-free question answering. Unlike approaches that depend on external retrieval systems, this internalization path lets the model itself store document content, thereby avoiding the latency, retrieval failures, and index-maintenance overhead that accompany retrieval-augmented pipelines.

The authors propose IAR (Inject, Align, Recover), a three-stage post-training framework whose core idea is to split the traditionally entangled process into three single-purpose stages: structured document-knowledge injection, question-answering behavior alignment, and general-capability recovery. They argue that conventional continued pre-training often combines injection and behavior adjustment under a single objective, which tends to let domain knowledge overwrite general capabilities. The phased, decoupled design is intended precisely to mitigate that conflict.

The paper's main contributions are this framework design and its systematic validation across multiple model families and datasets, together with comparisons against several baseline methods. The authors emphasize that separating injection, alignment, and recovery lets each stage perform its own role while complementing the others rather than interfering with them, which distinguishes IAR from one-shot continued pre-training or simple supervised fine-tuning.

Deep Analysis

The first stage, Inject, converts source documents into knowledge the model can learn. Rather than relying on a single continued-pre-training objective, the authors design three distinct objectives. A continuation objective has the model complete document content after a given context. A rewriting objective has the model paraphrase documents using different phrasing. An instruction-conditioned reconstruction objective has the model rebuild document information under instruction guidance. Together, these objectives help the model digest structured document knowledge from multiple perspectives.

The second stage, Align, aligns the injected model's question-answering behavior using answer-only supervision. Its defining feature is that only standard answers serve as the learning target, rather than forcing the model to generate long reasoning chains. This focus more directly shapes retrieval-free answering behavior. The third stage, Recover, restores general capabilities weakened by domain adaptation by merging the domain-adapted model with a base instruction-following model, preserving domain knowledge while recovering the model's original general performance.

Experimentally, the work evaluates on two datasets, Common Corpus (CC) and CCI, covering four mainstream model families: Llama, Phi, Qwen, and SmolLM. Compared with vanilla SFT, IAR improved all four metrics in 7 of 8 dataset-model settings. Domain QA accuracy rose by an average of 3.6 percentage points, while average general performance on IFEval, MMLU, and MSBench improved by 12.1 percentage points, indicating that strengthening domain knowledge did not come at the expense of general ability.

Industry Impact

The paper's practical value lies in offering a viable retrieval-free path for converting private or limited documents into a model's intrinsic knowledge. For application scenarios that wish to avoid external retrieval systems, index maintenance, or inference-time latency, internalization lets the model directly acquire document knowledge, simplifying deployment architecture. The decoupled design also carries methodological implications: splitting injection, alignment, and recovery into independently optimizable and tunable stages provides a framework for finer-grained knowledge management.

The consistency of results across multiple model families and datasets strengthens the method's credibility for the open-source community and industrial adoption. However, the authors note that internalization is constrained by the fixed corpus size and suits clearly bounded document collections rather than infinitely expandable knowledge bases, making its applicable scenarios worth careful weighing in practice. An ablation-style observation on CC further supports this balance: while methods like LoRA and FAPM can lead on individual general metrics, IAR maintains strong general performance among those methods that also reach leading or near-leading domain internalization levels.

Outlook

Overall, the paper delivers solid evidence across framework design, experimental coverage, and the general-domain balance, offering a useful reference for the document knowledge internalization direction.

The three-stage decoupling suggests a promising template for future work seeking to manage specialized knowledge without the operational costs of retrieval. As model families continue to diversify, frameworks like IAR may prove especially relevant for deploying capable models on bounded, high-value document sets where latency, privacy, and deployment simplicity matter more than open-ended knowledge coverage.