When Safety Routing Fails: A Fisher Geometric Interpretation of Alignment Fragility in Large Models under Benign Fine-Tuning
This study investigates the root cause of the severe degradation in safety alignment capabilities of large language models during benign fine-tuning. While traditional views attribute this to gradient conflict, we propose a novel explanatory framework based on Fisher geometry: safety alignment features exhibit low-rank properties, and the fine-tuning process flattens the safety geometric structure while preserving the output routing path. We find that with just 100 benign samples, this path is selectively sharpened in the output-side MLP module, causing a collapse in safety capabilities while general abilities decline only slightly. This asymmetric vulnerability reveals a retention mechanism for internal safety representations, where a small number of safety samples can restore refusal behavior. Although LoRA and ASAM can suppress early sharpening, their protective efficacy diminishes under large-scale fine-tuning. This research redefines safety failure as the disruption of a low-rank output routing mechanism, offering new insights into the stability of large model alignment.
Background and Context
Large language models typically undergo alignment fine-tuning prior to deployment to ensure safety, yet recent findings reveal a paradoxical vulnerability. Even benign fine-tuning tasks designed to enhance general capabilities often lead to the unexpected collapse of safety mechanisms, a phenomenon termed alignment fragility. Traditionally, this failure was attributed to gradient conflict, where safety and general objectives interfere in parameter space.
However, this study introduces a novel explanatory framework based on Fisher geometry, challenging the conventional view. The research posits that safety alignment is not uniformly distributed across all parameters but exhibits a low-rank structure. This geometric property makes safety alignment flatter in the parameter space but crucially preserves a specific output routing path. During benign fine-tuning, this path is not erased but becomes the pivot for behavioral changes, offering a new theoretical basis for understanding why safety capabilities are so prone to collapse.
Deep Analysis
The study employs precise geometric analysis to dissect the dynamic changes in internal representations during fine-tuning. The Fisher information matrix for safety alignment demonstrates significant low-rank characteristics, indicating that safety capabilities are concentrated in few directions. Initially, the overall geometric structure flattens as the model adapts to new data, but the critical output routing path remains intact.
With just 100 benign samples, this path undergoes selective sharpening in the output-side Multi-Layer Perceptron (MLP) module. This sharpening is highly asymmetric: it preferentially disrupts the safety refusal path while having minimal impact on general ability paths. This mechanism explains the rapid collapse of safety while general performance remains stable. Furthermore, because internal safety representations are not entirely erased, a small number of safety examples can reactivate the routing path, restoring refusal behavior and revealing the redundancy and recoverability of internal safety knowledge.
Industry Impact
These findings have profound implications for the safe deployment of large language models. They warn developers that even seemingly harmless benign fine-tuning carries significant safety risks, and general performance improvements cannot be used as a proxy for safety. The proposed low-rank output routing mechanism offers new ideas for designing more robust alignment algorithms, such as specifically protecting this routing path to enhance the persistence of safety.
For the open-source community, understanding this vulnerability aids in developing more effective safety recovery tools that can quickly repair damaged models using minimal safety samples. In industrial applications, enterprises must reevaluate their fine-tuning workflows, incorporating mitigation strategies like LoRA or ASAM and monitoring the impact of fine-tuning scale on safety. This research underscores the importance of assessing safety independently of general capability metrics during model development.
Outlook
Experimental validation across multiple standard benchmarks confirms the theory, showing that attack success rates rise sharply after minimal benign fine-tuning, while general task metrics decline only slightly. Ablation studies further demonstrate that suppressing the sharpening process in the output MLP module significantly slows the decline in safety capabilities. The study also evaluated existing mitigation strategies, including Low-Rank Adaptation (LoRA) and Adaptive Sharpness-Aware Minimization (ASAM).
While both methods effectively suppress early sharpening and delay safety collapse, their protective efficacy diminishes under large-scale fine-tuning, failing to completely prevent the final failure of safety alignment. This highlights the limitations of current mainstream fine-tuning techniques in handling large-scale data and emphasizes the need for more robust safety protection mechanisms. Future research directions include exploring deeper internal representation structures and alignment stability from geometric and routing perspectives, potentially driving the development of safer and more reliable large model technologies.
Sources
FAQ
What is the new Fisher geometry explanation for why LLM safety alignment collapses during fine-tuning?
Safety alignment features have a low-rank structure. Fine-tuning flattens the safety geometry but preserves an output routing path. As few as 100 benign samples selectively sharpen this path in the output-side MLP, preferentially breaking refusal while leaving general abilities mostly intact.
Why does this finding matter for the safety of deployed large language models?
It replaces the gradient-conflict narrative with a structural mechanism: benign fine-tuning can secretly erode safety. Developers must stop relying solely on general-task performance as a safety proxy and instead monitor and protect the low-rank output routing path that underlies refusal behavior.
How effective are LoRA and ASAM at preventing safety collapse, and what are their limits?
Both techniques suppress early output-side MLP sharpening and delay safety degradation. However, their protective effect weakens considerably under large-scale fine-tuning and cannot fully prevent the eventual failure of safety alignment, indicating the need for stronger safeguards.