Multi-Granularity Context-Enhanced Multimodal Knowledge Graph RAG Method
Retrieval-Augmented Generation (RAG) is widely used to mitigate hallucination in large language models and multimodal LLMs (MLLMs), with knowledge-graph-based RAG supplying high-quality external information through structured knowledge. Recent work explores multimodal knowledge graphs (MMKGs) as the knowledge base for GraphRAG to integrate knowledge across modalities. However, existing methods generally follow a common pipeline that processes each modality independently before fusion, limiting the use of textual context during visual information extraction and knowledge fusion and leaving a semantic gap between images and text, which constrains multimodal GraphRAG performance. We propose a new framework that constructs a context-enhanced MMKG (CEMMKG), supplementing each image with complementary textual context at both local and global scopes. The local context goes beyond surrounding text to include sentences semantically related to the image, while the global context provides a paragraph-level summary; we further introduce a multi-granularity design for the local context to capture information at different detail levels. Extensive experiments on vision-centric datasets validate the effectiveness of CCEMMKG and demonstrate its broad applicability across various MMKG-based RAG methods.
Background and Context
Retrieval-Augmented Generation has become the mainstream approach for reducing hallucinations in both large language models and multimodal large language models, and knowledge-graph-based RAG strengthens this by injecting structured external knowledge that is more reliable than raw retrieval. As multimodal knowledge graphs are increasingly adopted as the knowledge base for GraphRAG, these systems can integrate information across modalities, extending the benefits of structured retrieval into vision-language tasks.
However, the authors observe that most existing multimodal knowledge-graph RAG methods follow a common pipeline in which each modality is processed independently before fusion. This sequential design means textual context is only sparingly used during visual information extraction and knowledge fusion, leaving a semantic gap between images and text that constrains overall multimodal GraphRAG performance.
The paper addresses this core limitation by proposing a new framework to construct a context-enhanced multimodal knowledge graph. Rather than treating modalities in isolation, the framework actively injects rich contextual information during graph construction, linking each image more tightly to its related textual semantics and narrowing the semantic gap at its source.
Deep Analysis
The framework augments every image in the knowledge graph with complementary textual context at two distinct scopes. The local context extends beyond the immediately surrounding text to include sentences semantically related to the image, yielding richer and more precise information than proximity alone. The global context instead supplies a paragraph-level summary that helps the model grasp the broader semantic narrative.
To make the local context adaptable to varied needs, the design introduces multi-granularity, allowing the system to capture semantically relevant information at different detail levels. This preserves key specifics while retaining overall summarization, embedding each image within a multi-layered semantic network rather than leaving it as an isolated visual element.
During construction, the method emphasizes full utilization of textual semantics in the extraction and fusion stages, organizing local and global information into the graph in a structured way. Extensive experiments on vision-centric datasets confirm that CCEMMKG significantly improves multimodal knowledge-graph RAG performance, validating the value of closing the image-text semantic gap.
Industry Impact
Hallucination has long been a key bottleneck preventing large language models and multimodal models from deployment in high-risk domains such as healthcare, law, and finance, and retrieval-augmented generation has become an important route for mitigating it. By bridging the semantic gap during knowledge construction, the proposed approach offers a practical technical path toward more reliable multimodal systems.
Its combination of multi-granularity local context with global summaries provides the open-source community with reusable, scalable knowledge-representation ideas that can advance GraphRAG on more complex multimodal tasks. Because the framework demonstrates broad applicability across various multimodal knowledge-graph RAG methods, industry can integrate this enhancement without substantially altering existing architectures.
Outlook
The framework's method-agnostic design suggests context enhancement can serve as a general-purpose augmentation rather than a fix tied to one specific system. This opens avenues for combining it with different retrieval and fusion strategies as multimodal knowledge graphs grow in scope.
Future work may extend the multi-granularity mechanism to finer detail tiers or adapt the local-global balance to domain-specific retrieval patterns. As multimodal models move toward more trustworthy and practical deployment, structured context enhancement during graph construction is likely to become a standard building block for reliable multimodal generation.