LISA: Locality-Informed Speculative Decoding for Accelerating Autoregressive Image Generation

Autoregressive (AR) models deliver high-quality image gen-eration but suffer from severe inference latency due to sequential decod-ing. While speculative decoding (SD) successfully accelerates large lan-guage models, we demonstrate that state-of-the-art methods like EAGLE-3 provide limited gains for visual AR models. We attribute this to theintrinsic nature of visual tokens: unlike text, their probability distribu-tions are flat and non-discriminative, which hinders drafter–target align-ment and causes verification to collapse under sampling. To bridge thisgap, we propose LISA, a Locality-Informed Speculative framework forAutoregressive image generation. LISA overcomes alignment barriers viatwo key components: (i) Locality-Informed Distillation, which alignsthe drafter by prioritizing supervision on structured uncertainty; and(ii) Geometry-Aware Soft Verification, which leverages embeddingproximity and target confidence to enable more permissive yet safe to-ken acceptance. Experiments across T2I and C2I tasks demonstrate thatLISA improves the speed–quality trade-off, achieving up to 4.12× accel-eration and outperforming recent competitive methods.