Conditional Flow Matching for Visually-Guided Acoustic Highlighting

Visually-guided acoustic highlighting seeks to rebalance au-dio in alignment with the accompanying video, creating a coherent au-dio–visual experience. While visual saliency and enhancement have beenwidely studied, acoustic highlighting remains underexplored, often leadingto misalignment between visual and auditory focus. Existing approachesuse discriminative models, which struggle with the inherent ambiguityin audio remixing, where no natural one-to-one mapping exists betweenpoorly-balanced and well-balanced audio mixes. To address this limitation,we reframe this task as a generative problem and introduce a ConditionalFlow Matching (CFM) framework. A key challenge in iterative flow-basedgeneration is that early prediction errors — in selecting the correct sourceto enhance — compound over steps and push trajectories off-manifold. Toaddress this, we introduce a rollout loss that penalizes drift at the finalstep, encouraging self-correcting trajectories and stabilizing long-rangeflow integration. We further propose a conditioning module that fusesaudio and visual cues before vector field regression, enabling explicit cross-modal source selection. Extensive quantitative and qualitative evaluationsshow that our method consistently surpasses the previous state-of-the-artdiscriminative approach, establishing that visually-guided audio remixingis best addressed through generative modeling. Qualitative samples areavailable at the project page.