Learning to Recover Task Experts from a Multi-Task Merged Model

Multi-task model merging aims to consolidate several task-specific experts into a unified model, yet static merging consistently suf-fers from parameter interference. While dynamic merging models aimto bridge this gap, many works rely on the costly storage and loadingof redundant expert components at inference. In this work, from theperspective of task expert, we view parameter interference as parame-ter perturbation introduced to each expert during merging process. Weshow that such parameter perturbations can be modeled as affine trans-formation, which can be approximated as additive offsets. Motivatedby these, we propose Recover Task eXpert (ReTeX), a frameworkthat predicts those offsets, in order to undo parameter interference andrecover task-expert performance from a single merged checkpoint. Torecover the appropriate expert when task identity is unknown, we in-troduce a router-free task identifier based on SVD subspace signaturescomputed offline before inference. At inference, the identifier selects thetask whose subspace yields the smallest projection residual for a giveninput. As a result, ReTeX recovers over 95% of individual-expert per-formance in both vision and NLP domains, while significantly improvinggeneralization to unseen tasks. Crucially, we also show that the param-eter offset prediction leads to emergent adaptive interpolation of expertknowledge for out-of-distribution (OOD) tasks. ReTeX adaptively in-terpolates seen expert knowledge to handle unseen tasks. Our code isavailable at https://github.com/BAIKLAB/ReTeX