DART: Deformable Adaptive Reasoning with Temporal Queries for Online Skeleton-Based Action Recognition

Online skeleton-based action recognition requires per-frameclassification using only past and current observations. We identify twokey challenges in this setting: (1) Not all frames are equal: discrimina-tive motion concentrates within specific temporal regions whose loca-tion varies across action categories, yet existing temporal modeling relieson fixed sampling patterns and causal attention that treat all positionsequally. (2) Not all futures need to be predicted: prior methods attempt toexplicitly predict future features to compensate for the unobserved por-tion, but the inherent ambiguity of human motion causes such predictionto degenerate into trivial identity mapping of the input. To address thesechallenges, we propose DART (Deformable Adaptive Reasoning withTemporal Queries). DART employs a Deformable Multi-scale TemporalNetwork (DMTN) that introduces input-dependent offsets into causalconvolutions, enabling content-adaptive sampling of the most informa-tive historical moments. The learned sampling positions naturally forma compact temporal memory, upon which a Temporal Query Decoder(TQD) operates: learnable queries interact with the memory to distillhigh-level action evolution patterns, providing complementary temporalreasoning without explicit future forecasting. Extensive experiments onNTU RGB+D 60, NTU RGB+D 120, and NW-UCLA demonstrate theeffectiveness of DART. On NTU RGB+D 60, DART achieves 73.99%AUC, surpassing the SOTA by +3.04%. Code is available.