Early Estimation of Language to Latent Alignment in Diffusion Models

Conditional diffusion models frequently suffer from language-image misalignments. Due to the ambiguity of intermediate noise cor-rupted latents, assessing prompt adherence currently requires complet-ing the entire sampling trajectory. This late-stage evaluation incurs evenhigher computational costs during test-time scaling strategies, such asBest-of-N (BoN) sampling, as all misaligned trajectories must finish gen-eration before being discarded. To tackle this, we propose NoisyCLIP, anoise-aware twin-tower model that enables early language-to-latent align-ment estimation. By learning a vision encoder on noise-corrupted latents,we allow the model to "see" through the ambiguity of intermediate diffu-sion steps. To facilitate this training, we investigate noise-data augmen-tation sampling strategies and introduce two new benchmark datasets:Noisy-Conceptual-Captions and Noisy-GenAI-Bench. When applied asan early-stopping criterion for BoN, NoisyCLIP at half cost matches orbeats frozen CLIP at full cost. Ultimately, this transforms alignment as-sessment from an expensive final check into a continuous monitoring tool,drastically reducing compute costs without sacrificing semantic fidelity.