SWE-Critix: Post-Training Generalist Verifiers for Coding-Agent Trajectories, Judging Success and Explaining Why, With Accuracy Rising From 57% to 78%

Paper 2610.13356 appeared on arXiv in mid-October 2026 as "SWE-Critix: Post-Training Generalist Verifiers for Assessing Coding-Agent Trajectories". The problem grows sharper as coding agents get more capable. They no longer just write a snippet: they inspect repositories, edit files, run commands and tests, interpret feedback and refine, moving from single-function synthesis to repository-level issue resolution and long-horizon software development. The more autonomous they are, the more it matters to verify their execution trajectories, the step-by-step record of an agent solving a task. In realistic end-to-end tasks, though, there is often no ready oracle defining success. A verifier is needed to decide whether the agent finished the task and, when it fails, to say why. The difficulty is the trajectory itself. One can hold hundreds of interleaved user inputs, environment feedback and model responses, which is much harder to inspect than a one-shot answer. The paper cites an example in which trajectories for tasks in LoLBench span 53 to 532 turns. The authors organize existing work along two dimensions: whether a verifier looks only at the final outcome (such as the submitted patch) or also examines the process for undesirable intermediate behavior, and whether it is built with or without training. Training-free methods use general-purpose LLMs or rule-based judges, while training-based methods learn dedicated critics, which can be split further by whether they output only scalar rewards. By this taxonomy, training-based, process-oriented generalist verification is underexplored. SWE-Critix aims to fill the gap: given a task and its trajectory, the learned verifier predicts whether it was completed and generates a natural-language analysis explaining the judgment.

The training pipeline has several steps. The data come from CoderForge-Preview (written CodeForge-Preview in the abstract), a large open dataset of 258K trajectories (155K pass, 103K fail) spanning 51K issues across 1,655 repositories. They were generated with the OpenHands scaffold and Qwen3-Coder-480B on three task sources that provide executable environments. The authors discard any trajectory that exceeds 64K tokens or contains no submitted patch, for example because the task was incomplete, which removes 16% of the data, then split the rest into SFT, RL and test sets. For SFT they pick one successful and one failed trajectory from each issue that has both; the RL set includes all remaining failed trajectories and samples successful ones round-robin across issues until the classes are balanced, to cover as many distinct issues as possible.

The key to the SFT stage is where the reasoning chains come from. A stronger teacher model (GPT-5.5) compares each successful and failed pair, summarizing the procedures, mistake patterns and key factors behind each outcome, and the chains are post-processed for pointwise judgment and assembled into SFT samples. Pairwise comparison is chosen because earlier findings say LLMs do better at pairwise than individual judgments. They fine-tune from Qwen3-30B-A3B-Thinking-2507 with MindSpeed-LLM on 128 Huawei Ascend 910B NPUs across 16 nodes, three epochs and 975 iterations, about 15 hours. The RL stage uses the remaining 91.3% of data, which has unit-test results but no reasoning chains, with a simple rule-based outcome reward: +1 if the predicted judgment matches the ground truth from unit testing, -1 otherwise. Training uses VeRL and GRPO with group size 8 on 128 Ascend 910C NPUs across 8 nodes, with 3,396 iterations per epoch taking roughly 533 hours.

The evaluation is direct: can the model tell successful from failed trajectories? The test set has over 43K long trajectories, each model is evaluated once per sample using vLLM Ascend on a node with 16 Ascend 910C NPUs, and the authors report accuracy plus precision, recall and F1 for the positive (successful) and negative (failed) classes. The results show a clear progression. The base Qwen3 MoE model has a strong bias toward predicting trajectories as correct, with negative recall near zero across all subsets, and 9.6% of its responses do not follow the required output format, so judgments cannot be reliably extracted. SFT alleviates both problems: every response follows the prompt's format and the positive bias drops considerably. RL then lifts overall accuracy from 66% to 78%. SFT mainly fixes the strong positive bias, while RL strengthens discrimination by raising positive recall from 67% to 85% without substantially sacrificing negative recall.

The authors are candid about the shortfalls. First, performance on R2E-Gym is notably lower than on SWE-Rebench and SWE-Smith, possibly because of its smaller data volume. Second, the model is still weaker at recognizing failed trajectories, with overall negative recall at only 64%, suggesting that failure patterns are more diverse and their signals more subtle. For future work the authors plan to add other sources such as Open-SWE-Traces, synthesize trajectories with other model and harness combinations, and study class ratios such as 2:1, 1:2 and 4:1 beyond the balanced mix. Note that all trajectories come from one harness and one model, so there is no evidence yet about how the verifier performs on other harnesses or models. Our reading is that SWE-Critix shows judgments with reasons can be learned through a fairly simple two-stage recipe, and releasing models and training code lets others reproduce and improve it. Its limits are plain: it is a self-described preliminary evaluation with each sample tested once; 78% accuracy still leaves clear misjudgments; and it is weaker on failed trajectories, where in practice a missed failure is usually more dangerous than a false alarm. A team wanting this kind of verifier in its own coding-agent pipeline should treat it as a supporting signal alongside unit tests, spot checks and independent evaluation environments, and measure its classification on its own trajectories before deciding where to place it.