SWE Refactor Bench: Can Coding Agents Complete Full-Repository Stack Migrations?

Background and Context

Modern software systems accumulate technical debt over decades of development, making code-stack migrations both expensive and heavily dependent on manual labor. As coding agents grow more capable on tasks like bug fixing, a natural question emerges: can they autonomously complete migrations that span an entire repository? Existing benchmarks cannot answer this because they evaluate only behavioral correctness, never verifying whether a migration actually occurred. This creates what the paper calls "blindness"—an agent can simply copy the original implementation to a new location and pass the tests, fooling the evaluation into thinking a migration happened.

To close this gap, the authors introduce SWE Refactor Bench, a benchmark composed of 20 full-repository migration tasks spanning four categories of technical debt. The central contribution is folding whether a migration truly happened into the evaluation, providing a more authentic measure of an agent's engineering ability. The key innovation is a three-stage evaluation protocol that simultaneously measures migration integrity and behavioral correctness, forcing agents to both change the code correctly and preserve behavior. This design strikes at a soft spot of current coding-agent evaluation and sets a stricter benchmark for future work.

Deep Analysis

The three-stage protocol advances layer by layer. The first stage is migration audit, which verifies whether a migration actually occurred, specifically to intercept agents that skip the migration and pass tests by copying the original implementation. The second stage is behavioral testing, using a fixed test suite to measure correctness after migration, exposing any approach that breaks original behavior. The third stage is agent validation, invoking six independent coding agents to generate targeted tests that uncover hidden regressions the fixed suite cannot reach. This multi-agent cross-validation compensates for the insufficient coverage of any single test set.

The entire evaluation ran across 8 frontier models and 26 model-compute configurations, totaling 520 runs that span different model scales and inference budgets. Training and evaluation involved no new model training; the focus was building a rigorous process that distinguishes real migrations from fake ones, decoupling behavioral correctness from migration integrity for finer diagnosis. In the 520 runs, only 28 (5.4%) passed all three stages, meaning agents that complete a genuine migration while preserving behavior are exceedingly rare. Of the 20 tasks, 13 had no accepted solution at all, and the strongest model, claude-opus-5, scored just 47.0/100.

Industry Impact

Ablation-style observations reveal that migration integrity and behavioral correctness are two independent capabilities. A minority of runs preserved behavior by skipping the migration, getting caught at the audit stage; most attempts performed the migration but broke behavior, getting caught at the behavioral-test stage. Among the 340 runs that passed the migration audit, 58% reached 99% on the fixed checks, but only 26% reached 100%, showing that even when a migration truly happens, achieving full correctness remains difficult. Agents also differ sharply across categories: build toolchain rewrites scored 31.4 versus just 5.6 for language rewrites, reflecting that different technical debts pose very different challenges.

For the open-source community and industry, SWE Refactor Bench offers a strict, realistic test platform. Real-world migrations often involve build tools, language versions, and dependency libraries simultaneously, all while preserving behavior—a frequent industrial pain point. Through migration audit and multi-agent validation, the benchmark curbs opportunistic copying and makes results more credible. It shifts coding-agent evaluation from pure behavioral correctness toward comprehensive engineering ability, and the sharp category differences point researchers toward weaknesses, with language rewriting remaining the hardest.

Outlook

The results make clear that today's frontier coding agents cannot yet deliver flawless full-repository migrations. The benchmark provides a rigorous testing platform for developing reliable migration-capable agents, and the decoupling of migration integrity from behavioral correctness offers a diagnostic framework for targeted improvement.

Future work can use the category-specific scores to prioritize research, particularly on language rewrites, where agents scored only 5.6. By exposing the true level of agents on long-horizon, whole-repository engineering tasks, this work charts a direction for the next generation of coding agents able to shoulder reliable migrations.