Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations

Vision-Language Action (VLAs) models promise to extendthe remarkable success of vision-language models (VLMs) to robotics. Yet,unlike VLMs in the vision-language domain, VLAs for robotics requirefinetuning to contend with varying physical factors like robot embodiment,environment characteristics, and spatial relationships of each task. Wepropose leveraging few-shot demonstrations of tasks to better capture thisphysical variability. Overfitting to the trained task is yet another problemof existing methods. Inspired by functional specificity, we hypothesizethat finetuning only task-specific sparse model representations is bothmore effective and better retains pretrained model generality. In thiswork, we introduce Robotic Steering, a finetuning approach groundedin mechanistic interpretability that leverages few-shot demonstrations toidentify and selectively finetune task-specific attention heads aligned withthe physical, visual, and linguistic requirements of robotic tasks. Throughcomprehensive on-robot evaluations with a Franka Emika robot arm, wedemonstrate that Robotic Steering outperforms LoRA while achievingsuperior robustness under task variation, reduced computational cost,and enhanced interpretability for adapting VLAs to diverse robotic tasks.