Heterogeneous Body–Hand Pre-training for Humanoid Loco-Manipulation

English · 中文

WB-WAM is a world-action model for whole-body humanoid loco-manipulation. Through pre-training on heterogeneous data, intermediate training with PICO motion transfer, and robot-task post-training, it brings body, root, and dexterous-hand supervision into a unified action space and jointly learns visual dynamics and whole-body actions. During real-robot deployment, SONIC executes the body and root references, while hand targets control the dexterous hands directly. This repository provides training, data collection, Unitree G1 deployment, and HumanoidArena evaluation.

Real-robot hardware support: The released deployment supports only a Unitree G1 equipped with Wuji Hands and the G1 2-DOF camera head module (RealSense D455); other hand or camera configurations are not supported.

Wuji–G1 mounting adapter: download the STL model for the hand-to-robot connector.

Models: pre-train and mid-train · HumanoidArena post-train. Download selected checkpoints with scripts/download_models.py.

Datasets: Pico · Self-Collected.

PICO egocentric data collection and processing: Pico-Ego-Collector provides recording import, episode annotation, G1 retargeting, and LeRobot v3 export.

Want an AI agent to set up environments, download selected assets, or prepare deployment? Give it the agent setup guide. It will follow your selected workflow, ask only the missing questions in short steps, and confirm downloads when needed.

The agent can configure and verify both the PC and robot environments, including installation, downloads, configuration files, Wuji left/right identification, camera and head-servo setup, and dry-run checks. After the intended live operation is authorized and the on-site operator is ready, it can also start, monitor and stop hand/body controllers, policies and data collection. You handle physical connections, wearable devices and the two independent hand-side observations.

Copy the prompt for the environment you need into your AI agent:

Please follow docs/agent_setup.md and help me set up the WB-WAM training environment.

Please follow docs/agent_setup.md and help me set up the WB-WAM real-robot deployment environment.

Please follow docs/agent_setup.md and help me set up the WB-WAM data collection environment.

Please follow docs/agent_setup.md and help me set up the HumanoidArena evaluation environment.

Run the following commands from the repository root. Training and the WB-WAM policy server for HumanoidArena evaluation share one Python 3.10 conda environment. The Isaac simulator still requires a separate environment, as described in the evaluation guide.

conda create -n wbwam python=3.10
conda activate wbwam
python -m pip install --upgrade pip
python -m pip install torch==2.7.1+cu128 torchvision==0.22.1+cu128 \
  --extra-index-url https://download.pytorch.org/whl/cu128
python -m pip install -e ./training

Training also needs the system FFmpeg shared libraries for video decoding. On Ubuntu, install them if needed:

sudo apt-get update
sudo apt-get install -y ffmpeg

Set inference_python in benchmark/humanoidarena/runtime.local.yaml to this environment's bin/python. For datasets, pre-trained weights, and training commands, see the training guide. For simulator setup, assets, and evaluation commands, see the HumanoidArena guide.

The real-robot WB-WAM policy and SONIC collector do not share the conda environment above or each other's environment:

Humanoid-GPT collection also needs a separate controller environment.

Anyone deploying or running this project’s code must have adequate safety awareness and be fully capable of controlling the robot and stopping it promptly in an emergency. The authors and maintainers of this project accept no responsibility for any personal injury or property damage resulting from deployment or use of this code.