ConceptGuard: A Context-Sensitive Concept Forgetting Benchmark for Large Language Models
This paper studies the ability of large language models to selectively forget harmful or sensitive knowledge—an ability known as «forgetting»—and argues that existing methods and benchmarks suffer from a fundamental flaw: their forgetting and retention sets are mutually independent and are evaluated only through straightforward factual recall, failing to capture the core goal of forgetting, namely removing harmful behavior while preserving benign knowledge. The authors argue that forgetting should operate at the conceptual level, completely removing unsafe applications while preserving their correct and useful modes of use, thereby achieving meaningful complete forgetting. To this end they introduce the notion of «dual-use concepts», i.e. concepts that can be used in both harmful and benign contexts, and build the ConceptGuard benchmark accordingly, making the forgetting and retention sets explicitly complementary in terms of concept use. The benchmark enables forgetting to be explored at the conceptual rather than the sparse-fact level, with an intent-aware evaluation aimed at maximizing contextual separation. Experiments show that existing methods perform poorly under this setting, exhibiting weak contextual separation and low ROUGE and concept-level scores, exposing problems such as a strong forgetting-utility trade-off and inconsistent concept control.
Background and Context
Large language models are now deployed across high-stakes domains, which has made the ability to selectively remove harmful or sensitive knowledge an urgent safety requirement. This capability is known as "forgetting," and the process of evaluating it has long lacked a realistic assessment framework. A new paper introduces ConceptGuard, a context-sensitive concept forgetting benchmark for large language models, arguing that existing methods and benchmarks share a fundamental flaw that has gone unaddressed.
The core problem identified by the authors is that current evaluation setups rely on forgetting sets and retention sets that are mutually independent. These sets are composed of isolated facts, and success is measured only through straightforward factual recall. The authors argue this framework cannot capture the central goal of forgetting: removing harmful behavior while preserving benign knowledge. In effect, the field has been evaluating forgetting at the level of sparse facts rather than at the conceptual level where real safety concerns actually reside.
To address this gap, the paper reframes forgetting as something that should operate at the conceptual level. True forgetting, the authors argue, means completely removing the unsafe applications of a concept while preserving its correct and useful modes of use. This yields a meaningful, complete forgetting rather than a partial one that damages legitimate knowledge. The work deliberately avoids complex network modifications, instead reconstructing the problem from the standpoint of evaluation design.
Deep Analysis
The central technical contribution of the paper is the introduction of the notion of "dual-use concepts"—concepts that can be used in both harmful and benign contexts. Examples include knowledge that can explain dangerous principles while also describing safe applications. Based on this definition, the authors build the ConceptGuard benchmark so that the forgetting set and retention set are explicitly complementary in terms of concept use, rather than being disjoint from one another.
This design means forgetting is no longer a matter of blanking out isolated facts. Instead, it requires a fine-grained distinction at the conceptual level: harmful applications are removed while benign applications are retained. The benchmark is therefore intent-aware, with the primary goal of maximizing contextual separation. This asks the model to suppress relevant behavior in harmful contexts while maintaining normal output in benign contexts, promoting safer overall behavior.
The authors emphasize that this contextual-separation-oriented evaluation differs sharply from the coarse-grained metrics that merely check whether facts have been forgotten. By enabling forgetting to be explored and measured at the conceptual level rather than on sparse facts, the benchmark more closely matches the complex safety demands of real-world deployment.
Industry Impact
The authors evaluated multiple existing forgetting methods on the ConceptGuard benchmark. The results were uniformly poor: the methods exhibited weak contextual separation and scored badly on both ROUGE, a text-similarity measure, and the concept-level metrics. This exposes several deeper problems within the field. There is a strong forgetting-utility trade-off, meaning that over-forgetting damages the model's ability in benign scenarios. Different methods yield only limited gains in contextual sensitivity, and concept-level control is inconsistent, so a single method performs unevenly across different concepts.
Notably, the authors did not fabricate specific figures; instead they presented their conclusions through a combination of qualitative and quantitative analysis using ROUGE and concept-level metrics, keeping the argument rigorous and restrained. The dataset has been made public, providing the community with a reproducible evaluation platform.
For the open-source community, the released dataset and benchmark lower the barrier to evaluating forgetting capabilities, allowing researchers to compare methods under a unified standard. For industrial deployment in medical, legal, and financial sectors, ConceptGuard offers a practical evaluation tool for the delicate task of removing sensitive knowledge without undermining utility. For subsequent research, the revealed trade-offs point toward designing forgetting methods that think in terms of concepts rather than isolated facts.
Outlook
The paper redefines the evaluation standard for large language model forgetting, and its value extends beyond the introduction of a single new benchmark. By shifting the field away from fact-level evaluation toward conceptual, intent-aware assessment, it establishes a framework better aligned with real safety requirements. The poor performance of existing methods under this setting signals that the current generation of forgetting techniques remains insufficient for the nuanced demands of production environments.
The consistent findings—weak contextual separation, low ROUGE and concept-level scores, and inconsistent control—suggest that future work must prioritize concept-level manipulation over blunt factual erasure. Achieving meaningful complete forgetting will require methods that can cleanly separate harmful from benign uses of a concept without collapsing utility. As the dataset becomes available to the community, ConceptGuard is positioned to serve as the standard yardstick against which these next-generation approaches will be measured.