Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding

When videos extend from hours to days, directly process-ing them end-to-end becomes impractical for current Multi-modal LargeLanguage Models (MLLMs). This ultra-long setting necessitates a two-stage paradigm: query-agnostic memory construction followed by retrieval-based inference. Prior work invests in complex memory construction topre-model high-level relations in videos, despite not knowing the down-stream query at build time. We instead prioritize high-recall retrievabil-ity during memory building, and defer query-specific, high-level relationcomposition to inference time. To this end, we propose MERIT (Multi-key Episodic Retrieval with Inference-time Temporal expansion), a sim-ple yet effective agentic framework for ultra-long video understanding.First, we formulate an episodic multi-key representation that enablesprecise retrieval of fine-grained memories through a simple key-matchingmechanism. Second, we introduce a neighbor filtering mechanism to cap-ture broader semantic context without the massive computational over-head of global memory construction. This is achieved by expanding thetemporal scope exclusively around the retrieved segments at inferencetime. By leveraging simple key-matching with this on-demand tempo-ral expansion, MERIT achieves state-of-the-art performance across threelong-video benchmarks: EgoLifeQA, LVBench, and Video-MME(Long).