UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment

Vision-language models (VLMs) can describe urban scenesin rich detail, yet consistently fail to produce reliable human preferencelabels in domain-specific tasks such as safety assessment and aestheticevaluation. The standard fix, fine-tuning or RLHF, requires large-scaleannotations and model retraining. We ask a different question: can afrozen VLM be aligned with human preferences without modifying anyweights? Our key insight is that VLMs are strong concept extractorsbut poor decision calibrators. We propose a three-stage post-hoc pipelinethat exploits this asymmetry: (i) interpretable evaluation dimensionsare automatically mined from consensus exemplars; (ii) an Observer–Debater–Judge chain extracts robust concept scores from the frozenVLM; and (iii) locally-weighted ridge regression on a hybrid manifoldcalibrates these scores to human ratings. Applied as UrbanAlign onPlace Pulse 2.0, the framework reaches 70.8% accuracy (κ=0.41) acrosssix perception categories, outperforming all baselines by +9.6 pp andzero-shot VLM by +14.1 pp, with full interpretability and zero weightmodification.