showlab/Show-Harness

English | 简体中文

Just a VLM Agent Can Play Robots

Yanzhe Chen^* · Zechen Bai^* · Zhijun Cao^* · Wenzheng Zeng^* · Kevin Qinghong Lin
Yiqi Lin · Guoqiang Liang · Kevin Yuchen Ma · Qiming Huang · Mike Zheng Shou^†

^* equal contribution  ·  ^† corresponding author

Show Lab @ National University of Singapore

📄 Arxiv Paper  |  🤗 Daily Paper  |  🦾 Models  |  📊 Dataset  |  🌐 Project Page  |  💬 X (Twitter)

• 2026.09 Beyond Show-Harness, we release Awesome Multimodal Embodied Agents, our survey of the Agent + Robot landscape from computer-use to robot-use.
• 2026.09 Public release: the harness, GUMI collectors, the plugin suite, and the training pipeline.
• 2026.09 Six LoRA adapters on 🤗 Show-Harness-VLMs and the demonstration corpus on 🤗 Show-Harness-Data.

• 🌟 Overview
• 🚀 Quick Start
  • 1. Environments
  • 2. Collect demonstrations with GUMI
  • 3. Run a real robot
  • 4. Repository layout
• 🤖 Two modes, one interface
• 📦 Released checkpoints and data
• 🧩 Plugins
• 🙏 Acknowledgements
• 📌 Citation

Show-Harness is an embodied harness: a compact semantic interface that lets a vision-language model "play" a robot. The model reasons over discrete, incremental action units; embodiment-specific interpreters ground each unit into motion, deterministically — so the VLM stays directly responsible for every physical decision.

Through the same interface, a closed-source frontier VLM controls a robot zero-shot, and a small open model becomes a capable policy with less than a few H200 GPU-hours of fine-tuning.

• 🤖 Two modes, one interface — a frontier VLM zero-shot, or a fine-tuned small VLM emitting one action token per step.
• 🦾 Embodiment-agnostic — Franka, AgileX Piper (single and dual arm), ManiSkill, and Isaac Lab share one vocabulary and one prompt set.
• 🎮 GUMI — demonstrate a task by playing the robot in a browser; no teleoperation hardware, no post-processing.
• 🧩 Ablation-grade plugins — one directory, one boolean, and byte-identical to no plugin when disabled.

Separate venvs, because their pins conflict. Start with base; add the rest only when you need them. bash scripts/setup.sh with no arguments prints which already exist.

bash scripts/setup.sh base --real adds the Franka/Piper hardware layer (RealSense, ROS shims, teleop window). Zero-shot and sim work do not need it.

Serving is its own process, so the harness talks to any OpenAI-compatible endpoint — a hosted model, or a colleague's server — without .venv-vllm existing at all.

Training is self-contained under train/ and builds its own venvs against upstream LLaMA-Factory (bash train/scripts/setup_llamafactory.sh); nothing in the sections above depends on it.

GUMI maps every action unit to a key or button, so a human — or a GUI-driving agent — demonstrates a task by playing the robot in the browser, and every step is recorded as a training-ready (observation, action) pair. It drives the real rigs below; a synthetic tabletop world (--sim) lets you try the interface before any hardware is set up:

bash scripts/setup.sh base
.venv/bin/python gumi/collect_rollouts_web.py data/rollouts_demo --sim
# open http://localhost:8600 and drive the gripper with WASD / arrow keys

The same servers run against the real Franka/Piper rigs (drop --sim), and the same key bindings power live human takeover during autonomous rollouts. See gumi/README.md for the keyboard UI, the dual-arm UI, and the agent operators.

1. Copy configs/site/franka.yaml.example to configs/site/franka.yaml and fill in your robot address and camera serials (Piper: site/piper_arms.yaml.example).
2. Copy configs/secrets.env.example to configs/secrets.env and add an API key for the backend you use (GEMINI_API_KEY by default), or serve a local VLM with scripts/serve_vlm.sh.
3. Calibrate the safety floor and begin pose for your table — the shipped values are examples, and every autonomous run refuses to descend below the calibrated floor.
4. Preflight — checks the environment, your site config and calibration, the VLM backend (live), and that the robot and cameras answer: python scripts/check_setup.py --robot-config configs/robot_franka.yaml
5. python scripts/run_real.py --robot-config configs/robot_franka.yaml

The full walkthroughs are in docs/franka.md and docs/piper.md; simulators in docs/simulators.md.

Zero-shot — a frontier VLM operates the full plugin harness with no robot-specific training (scripts/run_real.py, scripts/run_real_dual.py).

Fine-tuned — a small VLM fine-tuned on GUMI demonstrations emits one action token per step, planner-free (scripts/run_real_mvtoken.py). The real-robot configs default to vlm_backend: qwen3_5_2b (the qwen3_5_2b_showharness_ft adapter); serve your own checkpoint instead and select it with vlm_backend: finetuned_local. See docs/finetuned.md, including the training contracts that must not drift.

Switching embodiments changes only the interpreter and its configs/primitives_<embodiment>.yaml; the model-facing vocabulary and prompts stay the same.

Five LoRA adapters trained on the real corpus, one per backbone, at showlab/Show-Harness-VLMs: qwen3_5_0_8b, qwen3_5_2b, qwen3_5_4b, qwen3_5_9b, gemma4_e4b; plus qwen3_5_2b_sim, one simulation policy covering both simulators. The demonstrations they were trained on are at showlab/Show-Harness-Data (real Franka/Piper rollouts plus RoboLab and ManiSkill).

Fetch an adapter with the base model it needs, serve it, drive the robot:

# 1) fetch the adapter together with the base model it needs
ADAPTER=qwen3_5_2b WITH_BASE=1 bash scripts/model/download_vlm_model.sh

# 2) serve it — the script activates .venv-vllm itself
MODEL=Qwen/Qwen3.5-2B \
  LORA=qwen3_5_2b_showharness_ft=models/Show-Harness-VLMs/qwen3_5_2b \
  FAMILY=qwen3_5 bash scripts/serve_vlm.sh

# 3) drive the robot
python scripts/run_real_mvtoken.py --robot-config configs/robot_franka_ft.yaml

FAMILY picks a jinja template from models/chat_templates/. Training never reads one — LlamaFactory renders the conversation itself — so these exist only to make vLLM reproduce that rendering at serve time. A base model's own template does not, and the mismatch fails silently — see models/README.md.

To fine-tune your own, train/ takes rollouts (yours or the released set) to a LoRA on any of the three supported families.

Each plugin mounts on one stage of the loop, is toggled by one boolean, and leaves the loop byte-identical when disabled.

plugins/README.md documents the contract for writing your own.

Show-Harness builds on the following open-source work:

• Training — LLaMA-Factory
• Serving — vLLM
• Franka control — Polymetis
• Simulation — ManiSkill, Isaac Lab
• Hardware SDK — AgileX Piper
• Open backbones — Qwen3.5, Gemma 4, and InternVL3.5, which the released adapters are trained on

Thanks to all Show Lab @ NUS members for their support.

If you find Show-Harness useful, please cite:

@article{chen2026show,
  title={Show-harness: Just a vlm agent can play robots},
  author={Chen, Yanzhe and Bai, Zechen and Cao, Zhijun and Zeng, Wenzheng and Lin, Kevin Qinghong and Lin, Yiqi and Liang, Guoqiang and Ma, Kevin Yuchen and Huang, Qiming and Shou, Mike Zheng},
  journal={arXiv preprint arXiv:2609.10522},
  year={2026}
}

If you like the project, please give us a star ⭐ — it is how we hear that it is useful.