SPADE: Self-Play Learning Where Large Models Play Both Environment Designer and Reasoning Agent
This paper addresses the challenge that language agents require continuously expanding, diverse, and adaptive goals for sustained self-improvement, pointing out that existing training environment pools—whether hand-curated, statically synthesized, or built on frozen validators—keep the goal distribution fixed as the learner scales. To this end, the authors propose SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play reinforcement learning framework in which a single large language model plays two roles simultaneously: an environment designer that writes complete, long-horizon training environments in executable code using an OpenAI-Gym-style reset()/step() interface, and a reasoning agent that learns actions within them. Both are stateful, multi-turn environments, so a single interface covers both reasoning problems and multi-step agent tool calls. The reasoning agent's regret is estimated from the reward gap with and without privileged hints, allowing the environment designer to learn to aim at the edge of the agent's capabilities while keeping tasks feasible. Scaling to a 30B-parameter model, SPADE improves by 5.3 on average across eight math, science, code, and reasoning benchmarks over the strongest fixed-environment baselines, gains 5.7 on multi-turn BFCL-v4 and 13.9 on ACEBench-Agent in tool-calling scenarios, and its lead grows with model size in game settings.
Background and Context
Language agents improve most effectively when their training targets keep expanding, diversifying, and adapting to rising capability. Self-improvement depends on a goal pool that the agent itself helps generate, yet the environments used to train them have not tracked that growth. Existing training environment pools remain fixed in their goal distribution as the learner scales, whether they are hand-curated, statically synthesized, or built on frozen validators. Once a model outgrows such a pool, training loses its challenge and plateaus.
To address this bottleneck, the authors propose SPADE, which stands for Self-Play in Adaptive Synthetic Executable Environments. It is a self-play reinforcement learning framework in which a single large language model plays two roles at once: an environment designer and a reasoning agent, driving both to co-evolve. The designer writes complete, long-horizon training environments in executable code, while the agent learns actions inside them.
The key conceptual shift is treating environment design itself as a learnable component rather than a fixed, hand-built step. This moves open-ended self-improvement from an aspiration toward a concrete mechanism. The paper's contributions include a unified interface covering both reasoning problems and multi-step agent tool calls, the use of a regret signal rather than absolute reward to guide design, and a set of experiments identifying the components that make such design succeed.
Deep Analysis
SPADE deploys one large language model as two cooperating roles. The environment designer builds training environments in executable code following an OpenAI-Gym-style reset() and step() interface, so environments can be initialized and advanced programmatically. Each environment is a stateful, multi-turn setting containing transition rules, a reward function, and verification code. A single interface therefore handles both step-by-step reasoning tasks and multi-step tool-calling agent tasks.
The agent's learning signal comes from regret, estimated as the reward gap when privileged hints are present versus absent. Optimizing against this signal, the designer learns to aim at the edge of the agent's capabilities, keeping tasks challenging yet feasible. Rather than rewarding easy success, the framework targets the zone where the agent struggles, forcing continued progress.
Experiments reveal two components essential to success. Grounding the designer on documents sampled from large pretrained corpora supplies semantic priors and background knowledge for environment construction. An accumulating environment memory lets the designer recall what it has already generated, avoiding repetition and pushing the goal distribution forward. Together these keep the self-play process stable and effective, and ablation studies confirm that removing either one degrades results.
Industry Impact
On eight held-out benchmarks spanning math, science, code, and reasoning, SPADE improves by 5.3 points on average over the strongest fixed-environment baselines. In tool-calling scenarios the gains are larger: 5.7 points on multi-turn BFCL-v4 and 13.9 points on ACEBench-Agent, indicating strong benefit for multi-step agent tasks that combine reasoning with external actions.
Because one unified interface serves both reasoning and tool use, the same framework can support a wide range of applications, from solving math problems to operating agents. For the open-source community, letting a single model handle both environment generation and action learning lowers the barrier to building adaptive training environments and offers a reusable pattern. For industrial deployment, this generality reduces the need for task-specific infrastructure.
The paper's identification of grounding and accumulating memory as critical components gives researchers a clear recipe for maintaining diverse, adaptive goal distributions. The consistent gains across domains suggest that foldable environment design can break through the performance ceiling imposed by static environments.
Outlook
The authors scaled the model to 30 billion parameters and observed performance in game settings, where the margin over the strongest baseline grows as model size increases. This widening lead provides strong evidence that the approach scales, with larger agents benefiting disproportionately from adaptive environments.
These findings point toward a future where training environments evolve alongside the models that use them, rather than being fixed in advance. The learnable-design paradigm, supported by grounding and memory, offers a path to sustained self-improvement without endless manual curation. As capability grows, the environment designer can keep pace, continually regenerating targets at the frontier of what the agent can do.
Whether this pattern generalizes beyond the tested benchmarks remains an open question, but the results so far indicate that adaptive, self-play environments are a promising direction. The growing advantage with scale suggests further expansion is feasible, pushing language agents toward more autonomous and more persistent learning.