Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

Zero-Shot Compositional Action Recognition (ZS-CAR) re-quires recognizing novel verb–object combinations composed of previ-ously observed primitives. In this work, we tackle a key failure mode:models predict verbs via object-driven shortcuts (i.e., relying on the la-beled object class) rather than temporal evidence. We argue that sparsecompositional supervision and verb–object learning asymmetry can pro-mote object-driven shortcut learning. Our analysis with proposed di-agnostic metrics shows that existing methods overfit to training co-occurrence patterns and underuse temporal verb cues, resulting in weakgeneralization to unseen compositions. To address object-driven short-cuts, we propose Robust COmpositional REpresentations (RCORE) withtwo components. Co-occurrence Prior Regularization (CPR) adds ex-plicit supervision for unseen compositions and regularizes the modelagainst frequent co-occurrence priors by treating them as hard nega-tives. Temporal Order Regularization for Composition (TORC) enforcestemporal-order sensitivity to learn temporally grounded verb representa-tions. Across Sth-com and EK100-com, RCORE reduces shortcut diagnos-tics and consequently improves compositional generalization. The codeis available at https://github.com/KHU-VLL/RCORE.