VISOR++ : VISUAL INPUT BASED STEERING FOR LARGE VISION LANGUAGE MODELS

As Vision Language Models (VLM) are deployed across safety-critical applications, understanding and controlling their behavioral pat-terns has become increasingly important. Existing behavioral controlmethods face significant limitations: system prompting is a popular ap-proach but could easily be overridden by user instructions, while ap-plying activation-based steering vectors requires invasive runtime accessto model internals, precluding deployment with API-based services andclosed-source models. Finding steering methods that transfer across mul-tiple VLMs is still an open area of research. To this end, we introduce vi-sual input based steering for output redirection (VISOR++), a novel ap-proach that achieves behavioral control through optimized visual inputsalone. We demonstrate that a single VISOR++ image can be generatedfor two architecturally diverse VLMs that by itself can emulate each oftheir steering vectors. By crafting universal visual inputs that induce tar-get activation patterns for an ensemble of models, VISOR++ eliminatesthe need for runtime model access while remaining deployment-agnostic.This means that when an underlying model supports multimodal capa-bility, model behaviors can be steered by inserting an image input com-pletely replacing runtime steering vector based interventions. We firstdemonstrate the effectiveness of the VISOR++ images on open-accessmodels such as LLaVA-1.5-7B and IDEFICS2-8B along three alignmentdirections: refusal, sycophancy and survival instinct. Both the model-specific steering images and the jointly optimized images achieve perfor-mance parity closely following that of steering vectors for both positiveand negative steering tasks. We also show early promise of VISOR++images in achieving directional behavioral shifts for unseen models thatinclude both open-access and closed-access models. At the same time,VISOR++ images are able to preserve 99.9% performance on 14,000unrelated MMLU evaluation samples highlighting their specificity to in-ducing only behavioral shifts.