SafeHarness: Making Robot Coding Agents Avoid Obstacles

What the paper reports

A new arXiv paper (arXiv:2609.20822, category cs.RO, submitted 17 September 2026) by Bingxin Xu (USC), Yuzhang Shang (UCF), Zhen Dong (UCSB) and Emilio Ferrara (USC) asks a question that the authors say earlier work left open. In the "coding agent" approach to robot manipulation, a language model writes the robot controller as a program. Such agents now operate robots without robot-specific training. But is this approach also safe?

To find out, the authors evaluate a coding agent under a safety constraint. Each task pairs a manipulation goal with an obstacle the robot must not touch. The agent pursues the goal, and in most cases it collides with the obstacle. The paper then presents SafeHarness, which adds two "obstacle-aware harnesses" around the agent. With GPT-6 as the backbone, SafeHarness reaches 71.9% task success and 87.5% collision avoidance on the SafeLIBERO benchmark. The paper says this beats the previous state of the art by 6.5% and 27.0%, and beats the same agent without harnesses by 2.3 times and 1.5 times.

Background: why safety was never measured

The paper sketches the line of work it builds on. Code as Policies set the pattern: the model composes perception and control APIs into a program, and the program is the policy. Later agents rewrite their controller code after a failure, keep a library of skills that worked, and spend extra test-time compute for reliability. Harness VLA lets the agent explore a scene, store what worked as a skill memory, and call a frozen vision-language-action (VLA) policy only where the task becomes contact-rich.

The authors argue that this work scores a rollout by whether the goal was reached. What the robot touched on the way is never measured. They also cite prior studies to make the point that success does not imply safety: across a model generation the two can move in opposite directions, and training a policy on ten times as many collision-free demonstrations lowers its collision rate by under three points. In their words, completing the task and touching nothing else are two halves of one requirement.

The diagnosis: the constraint never becomes a priority

The authors first evaluate a coding agent on a robot safety benchmark. It completes tasks but hits the obstacle in most cases. They rule out two easy explanations. Perception is not at fault, because the agent identifies the obstacle correctly and mentions it in its reasoning. Instruction is not at fault either, because the prompt already forbids touching it. The agent restates the constraint and then violates it.

The paper locates the fault in planning, and splits each manipulation phase into two parts. On the route through free space, the model has no notion of a path that clears the obstacle, so it takes the shortest one. It also has no notion of replanning when the chosen route becomes infeasible. At the contact, which is the contact-rich moment at the end of the phase, the model does not recognize that the contact itself carries the same safety constraint. A stronger planner does not close the gap, according to the ablation in Section 4.3.

Section 3.4 offers an explanation. An episode is one long conversation of hundreds of steps. Each tool call and each image adds to the context, which grows to tens of thousands of tokens, while the one sentence that names the obstacle sits near the start. The authors point to research showing that recall of mid-context content, instruction following and system-prompt adherence all degrade as context grows. In their rollouts the constraint appears in the first reasoning steps and then drops out.

How SafeHarness works

SafeHarness cuts a task at its contact events into phases. Both harnesses run once per phase.

Obstacle-aware route planning works in a plan, verify, execute cycle. A segmentation model (SAM-3) grounds the target and the obstacle as bounding boxes. The language model proposes a route as a polyline of waypoints that must end at the target and must not cross the obstacle box. A check outside the language model then tests the route against the box. It covers the gripper and, when an object is held, the object too. A route that intrudes into the box is rejected and the model replans. Replanning also fires during execution: if the arm halts, for example because it is stuck, a new route is planned from where it stopped. The harness also limits how often the route-plan image may be read during execution, because repeated identical images only lengthen the context.

Obstacle-aware contact execution starts with an adjacency test between the obstacle and the contact target. With no adjacent obstacle, any strategy is allowed. With one, the gripper approaches from a direction that keeps clearance, choosing the direction farthest from the obstacle. If no direction works, the gripper rotates so its opening axis runs tangent to the obstacle, and the payload is lowered vertically. The same test runs at the pick and at the place.

Benchmark and results

SafeLIBERO takes four tasks from each of the LIBERO Spatial, Goal, Object and Long suites and adds one obstacle to each: a moka pot, storage box, milk carton, wine bottle, mug or book. The task text never mentions the obstacle. Each task appears at two levels. In Level I the obstacle is close to the target and disturbs grasping or releasing. In Level II it is away from the target but on the transport path. The paper uses 32 tasks with 10 seeds per task. It reuses the Harness VLA coding-agent loop, a frozen π0.5 policy and the original skill memory, all unchanged, so the only difference is the two harnesses. Metrics are task success rate (TSR) and collision-avoidance rate (CAR).

Average results in Table 1, in percent, TSR then CAR: OpenVLA-OFT 26.2 and 5.7; π0.5 57.8 and 17.1; AEGIS, a barrier-function safety layer over π0.5 and the strongest prior method, 67.5 and 68.9; SafeHarness with GPT-5.5 70.0 and 86.0; SafeHarness with GPT-6 71.9 and 87.5. On the Long suite, AEGIS scores 46.3 TSR against 54.3 for the frozen policy it wraps.

The ablation in Table 2 compares three agents. Under GPT-6, the model alone reaches 6.0 TSR and 50.0 CAR. The authors call that CAR vacuous, since an arm that fails early touches nothing. Adding skills and the frozen VLA gives 31.0 and 59.0. Adding the harnesses gives 71.9 and 87.5. Under GPT-5.5, skills without harnesses give 28.0 and 31.0, and SafeHarness gives 70.0 and 86.0. The paper's conclusion is that safety here is a property of the harness rather than of the model it wraps.

Our analysis

Three points stand out. First, the main idea is an old engineering habit applied to a new place. Do not trust a rule stated once in a prompt. Re-check it with a tool at the moment it matters. The paper says its verification tools are stateless with respect to the conversation, and that a tool interface is a more reliable carrier of an invariant than the prompt.

Second, the numbers reward careful reading. The 6.5% and 27.0% gains match relative differences from Table 1 (71.9 against 67.5, and 87.5 against 68.9), not gaps in points. The gaps in points are about 4.4 and 18.6. Both readings show a gain, but the point gap on task success is small. The paper's own comparison against the same agent without harnesses, 40.9 and 28.5 points, is the stronger evidence.

Third, the ablation shows a useful split. A stronger planner improves the local decision of which side to approach, and Level I collision avoidance under skills-only rises from 18.8 to 69.0. It does little for route safety at Level II, where the gain is six points. With harnesses, the planner difference shrinks.

Limits and open questions

The authors list their own limits. The evaluation follows SafeLIBERO, with one obstacle per scene and one axis-aligned box as the obstacle model. The contact-side test is tuned for a parallel-jaw gripper. Grounding quality comes from the segmentation model, and a grounding failure shows up as a rejected route rather than a collision. Each episode costs minutes of planner time. The evaluation is simulation-first.

SafeHarness is also not collision-free. It still displaces the obstacle in about one episode in eight. Task success stays lower at Level II (64.0 against 76.0 with GPT-5.5, and 62.5 against 81.2 with GPT-6). The authors think this is most plausibly because a verified detour lengthens an episode under a fixed step budget. That is their guess, not a tested result.

Our own limits as readers: we saw only the paper text, not the code, and we could not reconcile some table entries with the stated protocol. Results come from one benchmark, and no real-robot test is reported.

Practical takeaways

Teams that build LLM agents for physical or high-stakes systems can take three lessons. Measure side effects, not only goal completion. Put hard constraints in verifiers the agent must pass, not in prompt text alone.

Check plans before execution, and keep a replanning path for what the checker cannot see. The paper expects the same tool-based approach to extend to workspace and force limits, but that is an expectation, not a result. Before any deployment, test with more than one obstacle, other grippers and real hardware.