HolisticSemGes: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matching

While the field of co-speech gesture generation has seen sig-nificant advances, producing holistic, semantically grounded gestures re-mains a challenge. Existing approaches rely on external semantic re-trieval methods, which limit their generalisation capability due to de-pendency on predefined linguistic rules. Flow-matching-based methodsproduce promising results; however, the network is optimised usingonly semantically congruent samples without exposure to negative ex-amples, leading to learning rhythmic gestures rather than sparse mo-tion, such as iconic and metaphoric gestures. Furthermore, by modellingbody parts in isolation, the majority of methods fail to maintain cross-modal consistency. We introduce a Contrastive Flow Matching-basedco-speech gesture generation model that uses mismatched audio–textconditions as negatives, training the velocity field to follow the cor-rect motion trajectory while repelling semantically incongruent trajec-tories. Our model ensures cross-modal coherence by embedding text,audio, and holistic motion into a composite latent space via cosine andcontrastive objectives. Extensive experiments and a user study demon-strate that our proposed approach outperforms state-of-the-art methodson two datasets, BEAT2 and SHOW. Our project page is available at:https://marcos452.github.io/HoliticSemGes/