SegDiff: Segmented Trajectory Diffusion for Consistent and Adaptive Robot Manipulation

Imitation learning enables robots to acquire manipulationskills from demonstrations by mapping observations to actions. Existingapproaches predict either short-horizon continuous action sequences ordiscrete keyposes. However, continuous prediction methods suffer fromcompounding errors due to short prediction horizons and struggle withmulti-modal action distributions, whereas keypose-based methods ne-cessitate an external planner, constraining real-time applicability. Toaddress these challenges, we introduce SegDiff, a closed-loop visuomo-tor policy that integrates the strengths of both paradigms. SegDiff de-composes demonstrations into motion segments between keyposes andlearns to predict the continuous trajectory from the current state tothe next keypose, enabling long-horizon prediction with real-time refine-ment. Furthermore, we leverage the capability of diffusion models andDDIM inversion to propose a Dynamic Temporal Ensembling mecha-nism, which allows the policy to efficiently respond to dynamic environ-ments and mitigate discontinuities caused by inconsistent multi-modalsampling. SegDiff demonstrates significant performance gains over exist-ing approaches across various simulated and real-world scenarios, indi-cating its strong ability to reason over extended temporal dependencieswhile maintaining real-time adaptability and control stability.