UniTranslator: A Unified Multi-modal framework for End-to-end In-Image Machine Translation

In-Image Machine Translation (IIMT) aims to translate scenetext in an image and render the translated text back into the original re-gions while preserving the overall visual appearance. Recent unified mul-timodal models provide a promising solution by combining visual-textunderstanding and image generation within a single framework. However,directly adapting such models to IIMT remains challenging. In particu-lar, they often suffer from understanding-generation conflicts, where thetranslation inferred during understanding is inconsistent with the textsupervision used in generation, and spatial position misalignment, wherethe rendered text does not accurately match the target text regions.To address these issues, we present UniTranslator, a unified multimodalframework for IIMT that tightly couples translation understanding andtext editing. Specifically, we introduce an Understand-Generation Align-ment Module (UGAM) to bridge the representation gap between un-derstanding and generation, encouraging semantic consistency betweentranslated content prediction and text rendering. We further propose aSpatial Mask Decoder (SMD) with pixel-level supervision over text re-gions to improve spatial grounding, geometric alignment, and layout con-trollability during generation. Extensive experiments on multiple bench-marks demonstrate that UniTranslator achieves state-of-the-art perfor-mance across diverse language directions and complex real-world layouts.Moreover, our results reveal a strong mutual reinforcement effect betweentranslation understanding and image generation, highlighting the advan-tage of unified translation multimodal learning. Code is available here.