Learning to Deny: Action Denial in Multimodal Large Language Models

Multimodal large language models (MLLMs) have rapidlyadvanced video understanding, achieving strong zero-shot and few-shotrecognition across standard benchmarks. Yet their ability to deny anaction by recognizing when an activity is not happening despite strongcontextual cues remains largely unexplored. We introduce UCF101-AD, a large-scale benchmark consisting of paired Action-Presence andAction-Denial clips, designed to evaluate this capacity for denial. Eachnegative video in UCF101-AD preserves the same contextual and mo-tion cues (persons, objects, locations) as its positive counterpart, butthe defining action itself is explicitly absent. Evaluating 20 state-of-the-art MLLMs reveals a consistent failure: models that exceed 85% accu-racy on the positive action classes collapse below 50% on its action-denial counterpart, indicating a strong inclination to affirm plausibleactions rather than verify that they truly occur. This exposes a criti-cal blind spot in modern video understanding: the inability to reasoncausally about whether a motion actually happens. To probe this is-sue, we explore a causal graph formulation, CausalAct, which expressesscene structure through natural-language prompts linking context, in-teraction, and motion. Incorporating such causal cues substantially re-duces false positives, demonstrating that denial is a learnable reason-ing skill. UCF101-AD provides a new lens for diagnosing and improv-ing causal reasoning in multimodal models. Dataset and relevant code:https://github.com/raiyaan-abdullah/Learn-to-Deny.