Rethinking Garment Conditioning in Diffusion-based Virtual Try-On: Decouple, Don't Denoise

Virtual Try-On (VTON) synthesizes realistic images of aperson wearing a target garment, with broad applications in e-commerceand fashion. Diffusion-based dual-UNet methods achieve strong resultsbut double the parameters by dedicating a separate network to garmentconditioning. Spatial concatenation offers a simpler single-network alter-native, yet both UNet- and DiT-based instantiations report that full fine-tuning is ineffective, and the community has settled for attention-onlytraining. We ask: why does full fine-tuning fail, and can this be resolved?Through what is, to our knowledge, the first visualization study of dual-UNet reference network behavior, we identify a unifying insight: garmentconditioning must be decoupled from the denoising process. Spatial con-catenation violates this by embedding the garment within the denoisingtarget, causing three conflicts: guidance leakage, gradient competition,and train-test discrepancy. We derive three design principles to restorethis decoupling and implement them as a pure recipe atop a standardarchitecture with no modification. The resulting model, DeCo-VTON(860M params), achieves single-network state of the art, matching thedual-UNet state of the art at half the cost while being preferred in humanevaluation.