HOT
Learning Generalizable Hand-Object Tracking
without Human Demonstration
* Equal contributors ✉ Corresponding authors
Accept to NeurIPS 2026
Overview #
We present a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations. Our approach makes two key contributions: (1) HOG, a highly efficient Hand-Object data generator that rapidly synthesizes diverse, large-scale, and uniformly distributed trajectories; and (2) HOT, a Hand-Object Tracker that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning. Notably, we demonstrate that the fast and intentionally noisy data generation process of HOG is crucial for endowing HOT with robust interaction.
Contribution Summary
-
1.
Synthetic data only
-
2.
HOG trajectory synthesis
-
3.
HOT robust tracking
-
4.
Generalization across settings
Method
Explore the framework Click (a), (b), or (c) for details
Framework
(a) HOG synthesizes manipulation trajectories for meta-skills. (b) HOT is trained through a two-stage teacher-student framework using reinforcement learning with a unified HOI imitation reward, enabling robust tracking of the target HOI trajectories. (c) At inference, the system can accept high-level waypoints from language models, generative models, or human data, which HOG converts into trajectories for HOT to track, enabling diverse applications.HOG synthesizes trajectories and HOT learns to track them for downstream manipulation.
HOG synthesizes manipulation trajectories from grasp poses generated by force-closure optimization and refined by RL. Its grammar-based approach supports eight composable meta-skills, offering multi-source parameter control via randomization, LLM/VLM instructions, or from human demonstrations. The system naturally generalizes across diverse hands and objects for scalable data coverage.
Diverse Zero-Shot Tracking Applications
Tracking with VLM ➕ HOG Tracking with Generative Model Tracking with Human Demonstration1. HOG Synthetic Trajectories / HOT Tracking Results #
HOG generates diverse, large-scale, and uniformly distributed trajectories. HOT learns robust hand-object interaction dynamics by tracking these synthesized trajectories.
2. Zero-Shot Tracking from Generated and Human HOI References #
HOT zero-shot tracks challenging hand-object sequences from GRAB, and also accepts text-conditioned interaction goals such as Text2HOI prompts.
3. Tracking Long Sequences #
HOT enables dexterous hands to track long-horizon sequences including rearrangement and multi-stage manipulation.
4. Different Dexterous Hand #
The framework extends beyond a single hand morphology and generalizes to different dexterous hand embodiments.
5. VLM Planning + Our Method #
HOT can take high-level waypoints from upstream planners and track the resulting hand-object interaction sequence.
6. Sim-to-Sim Results #
We validate sim-to-sim transfer in MuJoCo across two dexterous hand embodiments.
7. Real-World Replay #
We replay precomputed trajectories on a physical robot system comprising a Flexiv Rizon 4 arm and a Revo3 hand. These videos demonstrate real-world trajectory replay, not direct deployment of the HOT policy.
Citation #
@article{wang2025learning,
title={Learning generalizable hand-object tracking from synthetic demonstrations},
author={Yu, Runyi and Lin, Xiaoyi and Wang, Yinhuai and Tsui, Hok Wai and Peng, Zhijie and Zhang, Hui and Zhao, Qihan and Fan, Ke and Li, Miao and Song, Jie and Wang, Jingbo and others},
journal={arXiv preprint arXiv:2512.19583/v2},
year={2026}
}