Scaling Dense Prediction with Latent Decoding

Modern dense predictors largely follow a pyramidal pixel-decoding paradigm: they progressively lift low-resolution representationsto dense, high-resolution outputs. While effective, this design tightlycouples computation with output resolution, making dense predictionincreasingly costly to scale. In this work, we argue that strong denseprediction can be achieved by separating capacity from resolution: per-forming high-capacity reasoning in a compact latent space, and readingout to pixels with a lightweight operator. To instantiate this principle,we introduce Latent Dense Prediction Transformer (L-DPT), a minimalarchitecture that decouples dense decoding from pixel-space rendering.Instead of progressively decoding toward the pixel grid, L-DPT performsattention-based dense decoding entirely in latent space, keeping compu-tation largely stable as resolution increases while enabling more flexiblefeature integration from the encoder. The final output is produced viaa lightweight pixel readout, such as pixel shuffle, so scaling to extremeresolutions does not amplify decoding cost. As a result, L-DPT achievesstronger dense prediction accuracy while enabling inference on profes-sional ultra-high-resolution images up to 100 megapixels. More broadly,we view L-DPT as an instance of a scalable design principle rather thana task-specific decoder for dense prediction.