Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning

Dense video captioning aims to generate temporally groundeddescriptions of video events, benefiting both event-level video understand-ing and generation. In this domain, autoregressive video large languagemodels have emerged as a prevalent paradigm due to their strong gen-erative and cross-modal modeling capacity. However, generating densecaptions under the token-by-token paradigm severely limits inference effi-ciency and hinders scalability as video length and event density increase.In this work, we propose a parallelized autoregressive framework thatnot only improves generation efficiency but also enhances temporallygrounded captioning performance. Our key insight is to exploit the weaklocal dependencies across temporally distinct events to restructure thecausal dependency graph, thereby enabling lossless parallel generation.Specifically, tokens with weak cross-event dependencies can be decoded inparallel, while tightly coupled tokens within each event retain sequentialdecoding to preserve local semantic coherence. To realize this insight, weintroduce two key components for lossless parallel decoding: (1) a latentglobal planning mechanism that automatically learns the event-level struc-ture and produces compact tokens encoding global inter-event causalitywhile adaptively aggregating event-level audio-visual semantics, guidingsubsequent dependency restructuring and parallel decoding; and (2) anevent-factorized parallel decoding mechanism that effectively balanceslocal focus with global inter-event awareness. Experiments on variousbenchmarks demonstrate the clear advantage of our approach in bothefficiency and performance in omni-modal event grounding and captioning.Project website: https://github.com/showlab/PadCaptioner.