AgentVLN: Towards Agentic Vision-and-Language Navigation

Vision-and-Language Navigation (VLN) requires an embod-ied agent to ground complex natural-language instructions into long-horizon navigation in unseen environments. While Vision-Language Mod-els (VLMs) offer strong 2D semantic understanding, current VLN sys-tems remain constrained by limited spatial perception, 2D–3D repre-sentation mismatch, and monocular scale ambiguity. In this paper, wepropose AgentVLN, a novel and efficient embodied navigation frameworkthat can be deployed on edge computing platforms. We formulate VLNas a Partially Observable Semi-Markov Decision Process (POSMDP)and introduce a VLM-as-Brain paradigm that decouples high-levelsemantic reasoning from perception and planning via a plug-and-playskill library. To resolve multi-level representation inconsistency, we de-sign a cross-space representation mapping that projects perception-layer3D topological waypoints into the image plane, yielding pixel-alignedvisual prompts for the VLM. Building on this bridge, we integrate acontext-aware self-correction and active exploration strategy to recoverfrom occlusions and suppress error accumulation over long trajectories.To further address the spatial ambiguity of instructions in unstructuredenvironments, we propose a Query-Driven Perceptual Chain-of-Thought(QD-PCoT) scheme, enabling the agent to actively query and acquiregeometric depth information according to task demands. Finally, we con-struct AgentVLN-Instruct, a large-scale instruction-tuning dataset withdynamic stage routing conditioned on target visibility. Extensive experi-ments show that AgentVLN consistently outperforms prior state-of-the-art (SOTA) methods on long-horizon VLN benchmarks, offering a prac-tical paradigm for lightweight deployment of next-generation embodiednavigation models. Code: https://github.com/Allenxinn/AgentVLN.