Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space

Video-to-audio (V2A) generation aims to synthesize realis-tic audio that is both semantically consistent with and temporally syn-chronized to a silent video. Despite recent progress, many methods stillrely on multi-stage training, resulting in high computational costs andlong runtimes, or transform visual input into text to leverage pretrainedtext-to-audio models, sacrificing fine-grained temporal cues. To over-come these limitations, we propose Flowley, an end-to-end, single-stagetraining architecture that produces soundtracks by combining visual fea-tures with textual prompts. Crucially, we introduce Progressive Soft-masked Cross-Attention, which embeds audio-visual synchronization di-rectly within its attention mechanism, adding zero additional computa-tional cost compared to standard attention layers. We further observethat existing V2A benchmarks lack sound-oriented descriptive captions,which can potentially degrade the quality of the synthesized audio. Toremedy this, we propose SoundCap, a plug-and-play pipeline for creatingdetailed, sound-aware captions that guide the model. Remarkably, with-out integrating any pretrained audio-visual alignment modules, Flow-ley achieves state-of-the-art performance on VGGSound across multiplemetrics. Moreover, by incorporating SoundCap, we further exceed theperformance of the strongest existing close-sourced methods in terms ofaudio quality in the zero-shot setting.