Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learning

Weakly-supervised video scene graph generation (WS-VSGG)aims to parse video content into structured relational triplets withoutbounding box annotations and with only sparse temporal labeling, sig-nificantly reducing annotation costs. Without ground-truth boundingboxes, these methods rely on off-the-shelf detectors to generate objectproposals, yet largely overlook a fundamental discrepancy from fully-supervised pipelines. Fully-supervised detectors implicitly filter out non-interactive objects, while off-the-shelf detectors indiscriminately detectall visible objects, overwhelming relation models with noisy pairs. We ad-dress this by introducing a learnable pair affinity that estimates the like-lihood of interaction between subject–object pairs. Through Pair AffinityLearning and Scoring (PALS), pair affinity is incorporated into inference-time ranking and further integrated into contextual reasoning throughPair Affinity Modulation (PAM), enabling the model to suppress non-interactive pairs and focus on relationally meaningful ones. To providecleaner supervision for pair affinity learning, we further propose Relation-Aware Matching (RAM), which leverages vision-language grounding toresolve class-level ambiguity in pseudo-label generation. Extensive exper-iments on Action Genome demonstrate that our approach consistentlyyields substantial improvements across different baselines and backbones,achieving state-of-the-art WS-VSGG performance.