RefDiT: Local Attribute Guidance in Reference-Based Image Generation

Personalization models generate images guided by a few subject references, while style transfer methods aim to produce images aligned with a global style derived from a reference image. Recent approaches perform well when the reference image contains a single object, effectively capturing a global style that encompasses all implicit attributes. However, when applied to complex real-world scenes containing multiple objects with distinct attributes, existing methods struggle to selectively control generation using attributes from specific local regions of the reference image. Due to their primarily global conditioning strategies, these methods cannot reliably guide generation using attributes from specific local elements in the reference image. Moreover, existing methods typically employ a single identifier token to capture all details from the reference, resulting in a lack of individual, attribute-level control. Motivated by these limitations, we propose RefDiT, a novel framework for reference-guided image generation. RefDiT takes as input a reference image, a text prompt, and an optional user-provided guidance control. RefDiT employs local region guidance using the attributes of local elements. It constructs an attribute-aware conditioning signal from the reference image by performing attribute-level decomposition of the identifier tokens to train low-rank adapter (LoRA) blocks of a diffusion transformer (DiT)-based generative model. RefDiT leverages multi-modal joint attention to learn the correspondence between identifier tokens and local regions in the reference image, enabling more effective local guidance. RefDiT achieves a local attribute matching score of 0.88, outperforming state-of-the-art methods such as UnZipLoRA (0.54), K-LoRA (0.58), and B-LoRA (0.42). Compared to commercial models like Gemini-Banana (0.74) and GPT-5 (0.84), user studies indicate that participants find RefDiT’s outputs to be of similar quality.