OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams

Autonomous agents require representations that are gen-eral, causal, and physically structured to operate in real-time stream-ing environments. However, current vision foundation models remainfragmented, specializing narrowly in image semantic perception, offlinetemporal modeling, or spatial geometry. This paper introduces Om-niStream, a unified streaming visual backbone that effectively per-ceives, reconstructs, and acts from diverse visual inputs. By incorporat-ing causal spatiotemporal attention and 3D rotary positional embeddings(3D-RoPE), our model supports efficient, frame-by-frame online process-ing of video streams via a persistent KV-cache. We pre-train OmniStreamusing a synergistic multi-task framework coupling static and temporalrepresentation learning, streaming geometric reconstruction, and vision-language alignment on 29 datasets. Extensive evaluations show that, evenwith a strictly frozen backbone, OmniStream achieves consistently com-petitive performance with specialized experts across image and videoprobing, streaming geometric reconstruction, complex video and spatialreasoning, and robotic manipulation. Rather than pursuing benchmark-specific dominance, our work demonstrates the viability of training a sin-gle, versatile vision backbone that generalizes across semantic, spatial,and temporal reasoning, i.e., a meaningful step towards general-purposevisual understanding for interactive and embodied agents.