GoalDex

Abstract

Dexterous in-hand manipulation is commonly formulated around explicit object pose estimation, yet reliable pose tracking is challenging under severe hand-object occlusions and often requires object-specific supervision. This dependence creates a bottleneck between perception and control that can limit robustness and scalability.

We present an end-to-end visuomotor framework for dexterous in-hand reorientation that operates on RGB observations and proprioception without intermediate pose estimation. Our approach jointly learns visual representations and control policies through behavior distillation from privileged-state teachers and auxiliary reconstruction objectives. Furthermore, we introduce image-goal-conditioned reorientation, allowing desired object configurations to be specified directly through images rather than canonical pose representations.

We evaluate our approach across a diverse set of object geometries and instances in simulation, where it consistently outperforms pose-based methods and policies built on pretrained visual encoders. Finally, we demonstrate direct sim-to-real transfer of a single image-goal-conditioned policy across five real-world objects, showing that accurate dexterous reorientation can be achieved without explicit pose estimation.

Real-world demo

One policy, five objects, no pose estimator.

A single image-goal-conditioned policy, trained only in simulation, reorients a cube, a 3D printed toy, a tablet bottle, a rubber duck and a globe on a real Allegro hand. It sees just an RGB camera image, its own joint states and a picture of the goal.

The image goal and current object-masked observation are stacked on the left; the robot camera is on the right. A thin green border and a one-second pause highlight each success prediction, with a large running success counter. The timeline shows predicted angular distance and the 0.4 rad threshold. Each object video contains 30 seconds of source motion plus the prediction pauses.

Method

From a privileged teacher to a policy that reads goal images.

GoalDex follows a two-stage teacher-student scheme. The student never sees object pose: it learns what it needs about the object directly from pixels.

Privileged teacher

A teacher is trained with PPO in simulation on the full state, jointly across objects of different shapes and sizes, with a one-hot object identity.

Visuomotor student

The deployable student sees a goal image, the current RGB observation and proprioception. A recurrent memory keeps a belief over the object state through occlusions, and the network is trained end-to-end by behavior cloning.

Reconstruction and goal alignment

An auxiliary head reconstructs privileged simulation state. A continuous contrastive objective aligns current and goal image embeddings by their angular distance, so the policy can tell how close it is to the goal and when it has succeeded.

GoalDex training architecture — Training architecture. Goal and current RGB images are encoded into visual representations and fused with proprioceptive inputs. A recurrent policy integrates them over time to predict joint actions, success and privileged state, while the teacher supervises it in simulation only.

Results

What it takes to reorient from pixels.

Zero-shot transfer to five real objects

Each object is evaluated on 20 randomly sampled goal images. The policy succeeds in 80 of 100 trials. Under the same protocol, the strongest pose-based baseline (with class encoding) succeeds in 27. Prediction failures, where the policy declares success while still more than 0.4 rad from the goal, are the most common failure.

30 seconds of deployment footage, with one-second pauses on success predictions. All five objects: 80 / 100 successes (16.0 ± 1.7 per object).

Language-guided reorientation with a VLM

Because goals are images, a vision-language model can choose them. For each object we render 128 candidate goal views. Given an instruction, the VLM picks the image, or ordered sequence of images, that satisfies it, and the policy executes the reorientation. Tasks include finding a product's expiration date, comparing two continents on a globe, and spelling "HERE" on the letter cube.

A VLM selects image goals from 128 candidate views for GoalDex to find an expiration date, compare continents, and spell HERE on a cube.

Scaling object diversity

We trained GoalDex on 25 objects and achieved 41.4 consecutive successes.

Teacher-policy showcase across the 25-object training set, with each target orientation shown beside the hand.

3D asset credits

Citation

BibTeX

@inproceedings{bhardwaj2026goaldex,
  title     = {GoalDex: Learning Image-Goal-Conditioned Policies for Dexterous In-Hand Manipulation},
  author    = {Bhardwaj, Arjun and Mittal, Mayank and Wilder-Smith, Maximum and Patil, Vaishakh and Hutter, Marco},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026}
}

With support from

Acknowledgements

This work was funded by ETH Zurich (research grant no. 22-2 ETH-47), the Swiss National Science Foundation (NCCR Automation grant no. 51NF40 225155), Swiss Federal Railways (SBB) via ETH Mobility Initiative, ETHAR, and grants from NVIDIA, with a donation from Meta.