When Adaptation Causes Harm: Representation Drift and OOD Failure in MedSAM Fine-Tuning

Background and Context

Foundation models like MedSAM have reshaped medical image segmentation by offering strong zero-shot and few-shot generalization across domains. Yet this promise conceals a central tension: actual performance depends heavily on prompt quality and on how the model adapts to new datasets. Researchers have largely measured this through raw performance numbers, but the deeper question of which adaptation strategies genuinely help versus which harm robustness on unseen distributions has remained underexplored.

This paper addresses that gap by systematically comparing six adaptation strategies within a unified framework. The methods span current parameter-efficient fine-tuning and prompt-learning paradigms: full-model fine-tuning, encoder-only LoRA, shallow and deep visual prompt tuning (VPT), and decoder-only fine-tuning. The study grounds its evaluation on the ISIC 2018 skin lesion dataset while probing generalization across distribution tiers.

The evaluation design deliberately separates distribution shift into distinct levels. Models are tested under both clean and progressively noisy prompts on in-distribution data, close-OOD skin dermoscopy data from PH2, and far-OOD data from BUSI (breast ultrasound) and CBIS-DDSM (breast X-ray). This layered setup allows the authors to diagnose where adaptation helps and where it actively degrades performance.

Deep Analysis

The experimental results reveal a counterintuitive pattern that gives the paper its title. Adaptation strategies reliably improve performance on in-distribution and close-OOD data, but they frequently degrade results on far-OOD data. This trade-off, where improving one regime harms another, demonstrates that fine-tuning is not uniformly beneficial and can carry hidden costs on distributions the model has never encountered. Among the strategies compared, full fine-tuning achieves the best overall balance across all settings. However, encoder-only LoRA emerges as the strongest parameter-efficient alternative, significantly outperforming standard LoRA and VPT under far-OOD shift. This matters because parameter-efficient methods are precisely the tools practitioners reach for when scaling to new clinical settings. To explain these results, the authors deploy centered kernel alignment (CKA) as a representation similarity measure, tracking how both encoder and decoder representations drift before and after adaptation. The analysis shows that far-OOD degradation correlates strongly with decoder representation drift, while encoder similarity alone cannot explain robustness. This distinction is crucial: it separates feature preservation in the encoder from path adaptation in the decoder, pinpointing exactly which component loses robustness.

The CKA findings provide a coherent mechanism. Encoder-only LoRA remains more robust precisely because it adapts to shifts in the visual feature distribution while preserving the stability of the decoder path. Meanwhile, applying 0 to 100 pixels of random jitter to prompts trains more robust, higher-performing models, offering a simple data-enhancement route to prompt robustness.

Industry Impact

The theoretical contribution is notable for being the first to explicitly link representation drift to out-of-distribution failure at a causal level. This reframes a common empirical observation that models degrade on unfamiliar data after fine-tuning into something explainable and quantifiable, giving researchers a scientific basis for choosing adaptation strategies rather than relying on trial and error.

For practitioners, the finding that encoder-only LoRA excels under far-OOD shift offers a cost-effective deployment option under resource constraints. This is especially relevant for medical settings that must rapidly adapt to new departments or imaging equipment without access to large labeled datasets. The jitter-based augmentation further lowers the barrier to building robust prompts.

The authors have released all code, which should accelerate community research into how medical foundation models adapt. This open resource provides both an experimental and theoretical reference for designing smarter, self-adaptive fine-tuning methods in the future.

Outlook

The paper establishes that adaptation strategy selection must simultaneously account for prompt noise exposure, distribution shift, and representation preservation rather than merely chasing training-set performance. This three-dimensional view should guide how teams evaluate fine-tuning decisions in production medical imaging pipelines.

Future work could extend the CKA-based diagnostic framework to other foundation models beyond MedSAM, testing whether the decoder-drift hypothesis holds across segmentation architectures and imaging modalities. Understanding whether the observed trade-offs generalize will determine how widely these recommendations apply.

Ultimately, the work shifts the community's focus from raw accuracy toward robustness under distribution shift. As medical foundation models move from research benchmarks into real clinical workflows, the ability to adapt without degrading on unseen data will become a decisive factor in their safe and effective deployment.