DARS: A Two-Level Credit Allocation Reinforcement Learning Framework for Instruction-Based Image Editing
This paper addresses instruction-based image editing by introducing DARS, a reinforcement learning framework that tackles the credit allocation problem in a two-stage planner-renderer pipeline. In this pipeline, a vision-language model first generates an editing plan from the instruction, and a diffusion model then executes it. Training on the final image reward alone is inefficient: a single failed edit cannot tell the optimizer whether to focus on the planner or the renderer, and even when the planner is at fault, pinpointing the problem within a free-form reasoning trace is hard. DARS estimates inter-plan and intra-plan reward variance via multi-plan, multi-render rollouts to enable soft module routing, and builds an adaptive curriculum from the average rollout reward. Within the planner, it uses a four-field structured reasoning output and converts result-level feedback into localized supervision through prefix-gated rewards and token-level advantage reweighting. Experiments across five benchmarks show that DARS consistently outperforms the Joint RL baseline under identical backbone, data, reward model, and rollout budget, with the largest gains on reasoning-intensive editing.
Background and Context
Instruction-based image editing has emerged as a direction in computer vision that combines practical utility with research challenge. Tasks in this area typically rely on a two-stage planner-renderer pipeline: a vision-language model first reads a natural-language instruction and converts it into an executable editing plan, after which a diffusion model executes the plan to produce the final image. This split assigns clear responsibilities, but it also introduces a fundamental optimization difficulty, namely credit allocation.
The authors argue that training the entire system on the final-image reward alone is highly inefficient. A single failed edit does not tell the optimizer whether to direct more resources at the planner or the renderer. In other words, result-level feedback is too sparse to distinguish whether a problem lies in the planning stage or the execution stage. Even when the planner is at fault, locating the problem within a free-form reasoning trace is difficult.
To address this pain point, the authors propose the DARS framework, whose core contribution is a two-level credit allocation mechanism designed for such two-stage scenarios. It decomposes the otherwise coarse result reward into supervision signals that can flow across modules and across tokens, allowing training signal to reach precisely the parts that need improvement and thereby raising editing quality overall.
Deep Analysis
At the module level, DARS adopts a multi-plan, multi-render rollout strategy. For a single instruction, the system generates multiple candidate editing plans and performs multiple render samples for each plan, yielding a rich set of trajectories. From these trajectories, the authors estimate reward variance between plans and reward variance within plans. Inter-plan variance reflects differences in quality among candidate plans, while intra-plan variance reflects the stability of a single plan across different render outputs.
These two variance estimates enable soft module routing, so the system can dynamically decide how much optimization responsibility the planner and renderer each bear for a given sample, rather than applying a one-size-fits-all split. Meanwhile, the average rollout reward is used as a difficulty estimate to build an adaptive curriculum, letting the model improve gradually from easy to hard cases.
Within the planner itself, the authors introduce a four-field structured reasoning output. This design converts the previously free-form, hard-to-track reasoning process into a clear representation that can be parsed field by field. Based on this structured output, they design a prefix-gated reward mechanism combined with token-level advantage reweighting, refining result-level feedback and localizing it to specific tokens and reasoning fields. The result-level reward thus becomes a localized supervision signal that points to exactly where the planner deviated.
Industry Impact
The experiments select five benchmarks to evaluate DARS systematically, using Joint RL as the primary baseline. Crucially, this comparison is conducted under strict variable control: both methods use the same backbone, the same data, the same reward model, and the same rollout budget, ensuring a fair comparison. The results show that DARS consistently outperforms the Joint RL baseline across all five benchmarks.
The most prominent gains appear on reasoning-intensive editing tasks. This finding aligns closely with the design motivation, because it indicates that when an editing task places higher demands on the planner's reasoning ability, the proposed localized credit allocation mechanism delivers greater value. Ablation findings further confirm the necessity of both the two-level credit allocation and the structured reasoning design.
DARS also offers a transferable optimization idea: for multi-stage generation systems, sparse result rewards should be decomposed into supervision signals that flow at different granularities. This reasoning extends beyond image editing to video generation, video editing, and broader plan-execute systems. By emphasizing gains achieved without extra compute, the authors signal that other researchers can directly borrow the credit allocation and curriculum-learning design to lift existing systems at low cost.
Outlook
In terms of industrial deployment, image editing itself has broad application prospects, and how to stably improve editing quality under a limited budget is precisely the engineering concern most central to productization. The two-level credit allocation framework proposed by DARS provides a viable path that balances effectiveness and efficiency, offering strong reference value for subsequent research.
The structured four-field planner output and the soft module routing suggest a general recipe for diagnosing where failures originate in staged generative pipelines, potentially reducing the trial-and-error cost of optimizing such systems. As planning-intensive editing tasks grow more common, localized supervision that pinpoints planner errors may become increasingly valuable.
Whether the adaptive curriculum and variance-based routing generalize to other modalities remains an open question, but the controlled experimental setup establishes a clear benchmark. Future work may extend the framework to longer-horizon generation tasks where credit allocation is even more difficult, testing whether the same two-level decomposition continues to pay off.