Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

Dexterous manipulation is one of the hardest problems in embodied AI. A multi-fingered hand has many degrees of freedom and a tangled pattern of contacts, and a single task often splits into several stages: pick up a tool, adjust the grasp, then use the tool on another object. The traditional way to teach a robot such a task is teleoperation, where a human drives the robot and the system records the demonstration. That data is expensive and scarce, and it rarely covers the variety of object positions and grasps a deployed robot will meet. This is why many researchers now look at human video. A recording of a person doing the task costs almost nothing to collect. The real difficulty is turning one video into a policy that a robot can execute and that still works when the situation changes. The common answer so far is to imitate motion: retarget the hand trajectory in the video onto the robot hand and match it as closely as possible. The authors of Dex-One2Many name the weakness of this approach plainly in their abstract. Strict motion matching limits generalization. If the initial object pose changes, if the goal pose changes, or if the task needs a grasp that never appears in the video, a policy tied to one trajectory tends to fail. The other route is pure reinforcement learning, which can generalize broadly but lacks guidance. In a high-dimensional action space and a multi-stage task, blind exploration is very inefficient, and the agent may never get past the first stage. These two problems are coupled. Imitation cannot fix the first, and plain reinforcement learning cannot fix the second.

The central insight of the paper is to change the level of abstraction. Instead of recording the exact path the hand took, the method abstracts the video into a sequence of scene graphs, one per stage. A scene graph describes relations: which object is held, which objects touch, and where one object sits relative to another. Think of it as a task specification that lists conditions and omits coordinates. It says what must be true at each step, and it leaves the policy free to discover how to make it true. Because the graph constrains relations and not exact poses, it naturally admits a large family of concrete configurations. That property is what lets one graph play two roles at the same time. The first role is that of a generative constraint. Given the graphs, the system can sample diverse reset states, and these states cover object poses and grasps well beyond those shown in the video. The second role is that of a reward designer. For every stage, the graph supplies a dense reward that tells the learner how far the current state is from the goal of that stage. Together the two roles address the exploration problem directly. A multi-stage task punishes exploration from scratch, because the policy must get many steps right in a row before any final reward appears. Dex-One2Many avoids this. It initializes each stage from a sampled reset state and guides it with a dense reward, so each stretch of exploration is short and has a direction. Since the reset states come from relational constraints and not from a single demonstration, the poses and grasps the policy sees are diverse from the start.

Training runs entirely in simulation. The phrase real-to-sim-to-real describes the pipeline: build a simulated scene from the real video, learn in simulation, and transfer the policy zero-shot to a real multi-fingered hand. No robot demonstration is used. The video supplies the structure of the task, and reinforcement learning polishes the motion details inside the simulator. This division of labor has a practical benefit. The mismatch between a human hand and a robot hand, which causes retargeting error in imitation methods, does not pass straight into the final policy. The evaluation covers five tool-use and manipulation tasks. In seen configurations, the ones that match the video, the method exceeds the baselines by 6.5%. That margin is modest, and it tells us the baselines are not weak under familiar conditions. The striking number is the one for unseen scenarios. When object poses, goal poses and grasps move away from the demonstration, the gap widens to 71%. This contrast supports the paper's argument. Strict imitation holds up near the demonstration and degrades quickly away from it, while relation-level abstraction keeps generalization intact. The two figures also answer different questions. The first measures the success gap under familiar conditions. The second measures robustness to out-of-distribution change, and that is the question the work sets out to answer.

A careful reader should keep several limits in mind. These numbers come from the abstract of an arXiv preprint. The exact makeup of the baselines, the difficulty spread across the five tasks, and the number of real-robot trials need the full paper to judge, so the results should not be extrapolated far. The abstract also does not say whether the scene graphs are extracted automatically or need human correction, and that choice directly affects how well the method scales. The simulation-to-reality gap still depends on the quality of the scene reconstruction and the physics model. And it remains to be shown whether a stage-based decomposition holds up in longer and more open-ended tasks than the five reported. For the industry, the value of this work lies in a cheaper path around the data bottleneck. One human video and a simulator may be enough to train a policy that runs on a real hand. The paper also points to a more general design principle: between demonstration and exploration, the level of abstraction matters more than the quantity of data. When prior knowledge is expressed as relational constraints, it does not lock the policy in place the way trajectory imitation does, and it does not leave exploration to luck the way raw reinforcement learning does. Expressing a human demonstration through relations instead of trajectories is an idea that teams building dexterous robots should follow closely, and it will be worth watching how well it holds up on longer tasks and on more varied hardware.