On the Vulnerability of Parameter-Level Defenses to Model Merging

The training-free integration of expert models via modelmerging has exposed significant security risks, enabling free-riders tocombine specialized models without authorization. Recent works pro-pose parameter-level defenses that employ linear parameter transforma-tions to neutralize this threat. In this paper, we systematically analyzesuch defenses and reveal that their protected task vectors are inherentlysmall in magnitude. Consequently, the protected weights remain over-whelmingly dominated by the pretrained model. Based on this observa-tion, we designate the pretrained model as a static reference anchor andpropose the Anchor-Guided Attack (AGA) to circumvent existing safe-guards. Specifically, AGA aligns the protected model with this anchorto recover the transformation matrix analytically. Extensive evaluationsvalidate that AGA consistently bypasses both individual and compositedefenses under realistic defense-agnostic scenarios. Furthermore, we pro-vide Anchor-Repulsive Fine-tuning (ARF), a defense method to mitigatethe anchor dominance leveraged by AGA. Empirical results confirm thatARF effectively defeats the proposed attack. Our code is available athttps://github.com/krumpguo/secure-merge-attack.