Reward Modeling for Computer-Using Agent from Video Execution

Computer-use agents (CUAs) are becoming increasingly ca-pable; however, it remains difficult to scale evaluation of whether a tra-jectory truly fulfills a user instruction. In this work, we study rewardmodeling from execution video: a sequence of keyframes from an agenttrajectory that is independent of the agent’s internal reasoning or ac-tions. Although video-execution modeling is method-agnostic, it presentskey challenges, including highly redundant layouts and subtle, localizedcues that determine success. We introduce Execution Video Reward 53k(ExeVR-53k), a dataset of 53k high-quality video–task–reward triplets.We further propose adversarial instruction translation to synthesize neg-ative samples with step-level annotations. To enable learning from long,high-resolution execution videos, we design spatiotemporal token prun-ing, which removes homogeneous regions and persistent tokens while pre-serving decisive UI changes. Building on these components, we fine-tunean Execution Video Reward Model (ExeVRM) that takes only a user in-struction and a video-execution sequence to predict task success. Our Ex-eVRM 8B achieves 84.7% accuracy and 87.7% recall on video-executionassessment, outperforming strong proprietary models such as GPT-5.2and Gemini-3 Pro across Ubuntu, macOS, Windows, and Android, whileproviding more precise temporal attribution. These results show thatvideo-execution reward modeling can serve as a scalable, model-agnosticevaluator for CUAs.Code: https://github.com/limenlp/ExeVRMModel: https://huggingface.co/lime-nlp/ExeVRM-8BDataset: https://huggingface.co/datasets/lime-nlp/ExeVR-53K