CRAFT: A Subject-Personalized Diffusion Model Fine-Tuning Framework Without Synthetic Target Images

Background and Context

Subject-driven image personalization is a foundational capability in visual content creation, aiming to preserve the identity of one or more reference subjects within new scenes. Current mainstream approaches rely heavily on generalized strategies that fine-tune pretrained multimodal diffusion Transformers (MMDiT) using hundreds of thousands to millions of reference-target pairs. This paradigm is flawed due to the immense cost of constructing synthetic targets, which requires a complex, multi-stage data curation pipeline. This process includes LLM-based prompt generation, T2I target synthesis, reference subject extraction, VLM-based quality filtering, and correspondence annotation. Such requirements not only inflate data production costs but also tightly couple the method to specific target synthesizers and curation choices, severely limiting generalization capabilities.

To address these challenges, the paper introduces the CRAFT (Constrained Reward via Attention Fine-Tuning) framework. CRAFT is a single-step ReFL framework designed to efficiently fine-tune pretrained reference-aware MMDiT models using compact, reference-only data. By eliminating the need for synthetic target supervision, CRAFT drastically reduces the complexity and cost of data construction. The framework demonstrates that high-performance training is achievable with only 10,000 reference images and their corresponding subject masks, offering a lighter and more flexible solution for personalized generation.

Deep Analysis

The technical core of CRAFT lies in its unique 'gaze point' principle, which guides the model to learn correct subject localization through attention-level reward mechanisms. Specifically, the method utilizes LoRA adapters to fine-tune the pretrained model, focusing on aligning noise attention with phrase token attention. This alignment ensures that the model accurately attends to the correct reference subject during the generation process. By establishing this precise attention routing, CRAFT prevents the common issues of attention misalignment or identity confusion found in traditional methods.

Building on this foundation, CRAFT introduces per-subject attention masks that gate pixel-level identity rewards. This design ensures that the supervision signal in the image space remains consistent with the learned attention routing. The dual constraint of attention and pixel-level rewards allows CRAFT to maintain high-fidelity subject identity in generated images without explicit synthetic target supervision. This fine-grained attention control strategy enhances the model's ability to capture subject features and improves adaptability across different scenes, allowing generated results to naturally integrate into new background environments while preserving identity.

Industry Impact

In terms of experimental setup, CRAFT was applied to the FLUX.2-klein-9B model and comprehensively evaluated on the XVerseBench benchmark. The results show that CRAFT achieves state-of-the-art performance using only 10,000 reference-only samples. In contrast, previous generalized methods typically require 150,000 to over 2 million synthetic target pairs to achieve similar or inferior results. This outcome strongly demonstrates CRAFT's significant advantage in data efficiency. Ablation studies further reveal the importance of the synergistic effect between attention-level rewards and pixel-level identity rewards, as using either mechanism alone fails to achieve optimal performance.

Notably, the CRAFT recipe exhibits strong transferability. When applied to other reference-aware backbone networks, it consistently improves performance. This indicates that the method is not limited to a specific model architecture but provides a universal, attention-constrained personalization fine-tuning paradigm with broad applicability. By decoupling the model from specific target synthesizers, CRAFT enhances flexibility and robustness, facilitating rapid deployment and iteration in various application scenarios.

Outlook

The introduction of CRAFT has profound implications for both the open-source community and industrial implementation. First, it significantly lowers the data barrier for subject personalization models. This allows small and medium-sized enterprises and research institutions to train high-performance personalized models without the massive investment required to build large-scale synthetic datasets. Second, by decoupling the dependency on specific target synthesizers, CRAFT improves the method's flexibility and robustness, making it easier to deploy in diverse applications.

For future research, the attention-level reward mechanism demonstrated by CRAFT provides new insights into controllable generation for diffusion models. It may inspire more fine-tuning strategies based on attention constraints. Furthermore, the method's ability to maintain state-of-the-art performance while drastically reducing data requirements aligns with the current trend of pursuing efficient and sustainable AI development. CRAFT is poised to become a standard paradigm in personalized image generation, driving visual content creation toward higher quality and lower cost.