T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalability

Despite recent progress, vision-language encoders strugglewith two core limitations: (1) weak alignment between language anddense vision features, which hurts tasks like open-vocabulary semanticsegmentation; and (2) high token counts for fine-grained visual repre-sentations, which limits scalability to long videos. This work addressesboth limitations. We propose T-REN (Text-aligned Region Encoder Net-work), an efficient encoder that maps visual data to a compact setof text-aligned region-level representations (or region tokens). T-RENachieves this through a lightweight network added on top of a frozenvision backbone, trained to pool patch-level representations within eachsemantic region into region tokens and align them with region-level textannotations. With only 3.7% additional parameters compared to thevision-language backbone, this design yields substantially stronger densecross-modal understanding while reducing the token count by orders ofmagnitude. Specifically, T-REN delivers +5.9 mIoU on ADE20K open-vocabulary segmentation, +18.4% recall on COCO object-level text-image retrieval, +15.6% recall on Ego4D video object localization, and+17.6% mIoU on VSPW video scene parsing, all while reducing tokencounts by more than 24× for images and 187× for videos compared to thepatch-based vision-language backbone. The code and model are availableat https://github.com/savya08/T-REN.