H2SVC: Head-aware Heterogeneous Streaming Video Cache for Online Video Understanding

Multimodal Large Language Models (MLLMs) suffer fromsevere memory bottlenecks in streaming video understanding due to ac-cumulated Key-Value (KV) cache with the growth of video length. Exist-ing training-free compression methods employ the same strategy acrossall attention heads in one layer, but ignore their diverse functional roles intemporal representation. In this paper, we reveal the inherent temporalheterogeneity of attention heads in MLLMs through systematic Spatio-temporal Attention Profiling (SAP), and discover that heads naturallyspecialize into instantaneous, short-term, and episodic roles agnostic toinput video frames and their queries. Motivated by this finding, we pro-pose the Head-aware Heterogeneous Streaming Video Cache (H2 SVC),a training-free KV cache compression framework for efficient streamingvideo understanding. H2 SVC features an offline routing strategy that ex-ploits the distinct temporal behavior of heads in each layer and adapttheir KV cache to dedicated memory banks tailored for varying temporalreceptive fields. Furthermore, we develop Semantic Trajectory CurvatureEviction (STCE) for the long-term episodic bank to enhance semanticdiversity under strict streaming budget. STCE models the Value statesas a geometric trajectory to continuously evict predictable frames onsmooth paths, while preserving sharp semantic inflection points. Exten-sive experiments on widely adopted streaming benchmarks demonstratethat H2 SVC achieves state-of-the-art performance, and is comparableor superior to full-context offline models with significantly reduced KVcache memory and accelerated inference speed.