KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding

Evaluating the physical consistency of embodied world mod-els (EWMs) is a critical open challenge. While closed-loop evaluationvia simulator rollouts offers a more faithful assessment of physical plau-sibility than open-loop alternatives, existing frameworks almost exclu-sively rely on Inverse Dynamics Models (IDMs) for action extraction.Due to the intricate mapping from 2D pixel space to 3D kinematicspace, the learned IDMs can be brittle to data outside their trainingdistribution, resulting in unreliable action extraction from the gener-ated videos with novel objects and scenarios. This creates an unavoid-able attribution ambiguity between world model inaccuracies and ex-tractor errors. To reduce this ambiguity, we present KineBench, anIDM-free closed-loop benchmark for EWMs, built upon an explicit kine-matic grounding pipeline. Given a generated video, KineBench employscascaded visual foundation models to directly extract 6D end-effectorposes from individual frames, which are then executed in a physics sim-ulator for closed-loop validation. This explicit grounding directly teststhe physical feasibility rather than visual plausibility, while remainingsensitive to general physical hallucinations such as gripper vanishing orspatial inconsistency. Beyond execution-based task success, KineBenchincorporates two classical 3D kinematic metrics—Spectral Arc Length(SPARC) and the Maruyama Manipulability Index—to characterize tra-jectory smoothness and kinematic feasibility from a robot-centric per-spective. Across the evaluated models and tasks, these metrics exhibittask- and model-dependent associations with physical success rates, sug-gesting that they provide complementary diagnostic signals for assessingembodied generation quality. Built on 20 diverse manipulation tasks inManiSkill3, KineBench evaluates EWMs across four progressive suites:basic execution, task transfer, visual out-of-distribution generalization,and complexity-conditioned scaling. Evaluation across frontier models re-veals task-complexity-bounded nonlinear scaling in embodied video gen-eration, providing empirical guidance for future data-scaling strategies.The code and datasets are available on GitHub at https://github.com/minecraft-zzz/KineBench and on Hugging Face at https://huggingface.co/datasets/Zorkzak/KineBenchDatasets.