PerfectXu88/DriveHierarchy

Four capability ranks connect open-loop understanding to interactive driving evaluation.

DriveHierarchy is a hierarchical benchmark for diagnosing vision-language models (VLMs) in autonomous driving. It connects fine-grained open-loop assessment with closed-loop simulation to reveal capability strengths, weaknesses, and their relationship to driving behavior.

• 🧩 Diagnose across four ranks. Evaluate perceptual grounding, contextual memory, mental reasoning, and closed-loop execution within one capability hierarchy.
• 🎨 Build your own scenarios, no coding required. Use the interactive scenario editor to visually customize ego initialization, surrounding traffic, trajectories, and weather. Design driving interactions around your own research questions without writing scenario scripts.
• 🔗 Connect understanding to action. Pair 14 open-loop tasks with 100 interactive scenarios in a CARLA–SUMO co-simulation platform built on a real-world road layout.
• 🎯 Turn diagnosis into targeted improvement. Experiments on 15 VLMs and a benchmark-guided fine-tuning case study examine how improvements in open-loop capabilities transfer to closed-loop driving.
• ⚙️ Run a unified evaluation workflow. Use model presets, vLLM or Transformers inference, and standardized scoring scripts for reproducible comparisons.

The corpus statistics describe the full benchmark in the paper. Reported open-loop results use a refined 14,000-record evaluation set, with 1,000 records per task, as specified by the active dataset catalog.

Your scenario, your design — no programming required. DriveHierarchy includes a visual, interactive editor for creating custom driving scenarios. Configure the scene and its participants through the interface, from the ego vehicle's starting state to surrounding traffic and their trajectories.

Design custom interactions visually, evaluate them in CARLA–SUMO, and inspect the resulting driving behavior.

Create → Simulate → Inspect. The platform connects scenario design to CARLA's rendering, vehicle physics, and sensors, with SUMO managing background traffic. Simulation logs and replay support failure analysis and further scenario refinement.

The benchmark provides 100 curated scenarios across 10 families on a real-world road layout. The editor lets you create additional scenarios tailored to the behaviors you want to investigate.

Representative scenario types distributed across the test-site road network.

Explore the released scenarios or follow the closed-loop evaluation guide to prepare the simulator and run R4.

The paper studies 15 open-source VLMs, including generalist and driving-specialized models.

• Driving capabilities are related but distinct. R1 and R2 are strongly associated (Spearman ρ = 0.843), while their associations with R3 are weaker. A single aggregate score can hide meaningful differences between capability profiles.
• Open-loop understanding is informative about closed-loop behavior. Correlations with R4 are 0.664 for R1, 0.596 for R2, and 0.418 for R3, motivating evaluation across both settings.
• Diagnosed weaknesses can guide improvement. In the Qwen3-VL-8B-Instruct case study, jointly fine-tuning on the identified weak capability groups improves the average R4 score from 0.902 to 8.021, without using R4 as a supervision target.

These are results under the paper's evaluation protocol; the fine-tuning result is a case study on one base model. See Tables 1–4 and Figure 5 for full results and analysis.

The radar plots show how model strengths vary across individual open-loop tasks. The comparison below highlights capability profiles among representative 8B–12B models.

Task-level profiles reveal strengths and weaknesses that an overall score can obscure.

The figures use R2-1-A, R2-1-B, and R2-1-C for the tasks named R2_1, R2_2_A, and R2_2_B in the repository, respectively. All three plots assess R1–R3; closed-loop R4 is evaluated separately.

Start with open-loop evaluation. For CARLA–SUMO setup, scenario selection, output formats, and advanced options, see the complete evaluation guide.

git clone https://github.com/PerfectXu88/DriveHierarchy.git
cd DriveHierarchy

conda create -n drivehierarchy_openloop python=3.10 -y
conda activate drivehierarchy_openloop
pip install --upgrade pip
pip install -r requirements.txt

Run all commands from the repository root. Inference requires a compatible model runtime and sufficient hardware for the selected checkpoint; model-specific dependencies may also be needed.

Task JSONL files are included in Open_Loop_Evaluation. Prepare the referenced source imagery before inference and make sure the image paths in the records resolve on your machine. Some records use absolute paths such as /data/sets/nuscenes/...; mount the data there or update those references. See data and environment preparation.

Validate the model preset and dataset selection without loading a model:

bash scripts/run_open_loop.sh \
  --backend vllm \
  --config qwen3_vl_8b_instruct_vllm \
  --dry-run

The dry-run checks configuration and dataset resolution; it does not validate image availability or GPU readiness.

bash scripts/run_open_loop.sh \
  --backend vllm \
  --config qwen3_vl_8b_instruct_vllm \
  --score

Predictions are written to result/open_loop/qwen3_vl_8b_instruct_vllm/, with the score summary at evaluation/final_score.json inside that directory.

Next steps: Choose another model · Use Transformers or selected tasks · Run closed-loop evaluation.

Closed-loop assets: CarlaUE4/ and HDMaps/ are distributed separately from Git. Restore both under Close_Loop_Evaluation/Carla_Simulation/ before running R4. See external asset setup.

If DriveHierarchy supports your research, please cite our paper using the following provisional BibTeX entry:

@inproceedings{xu2026drivehierarchy,
  title     = {{DriveHierarchy}: A Benchmark for Diagnosing {VLM} Driving Capabilities from Open-Loop Understanding to Closed-Loop Execution},
  author    = {Xu, Chengkai and Liu, Jiaqi and Guo, Yicheng and Hang, Peng and Sun, Jian},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://github.com/PerfectXu88/DriveHierarchy}
}

DriveHierarchy supports non-commercial research on capability measurement, controlled comparison, error analysis, and simulator-based experimentation. The LICENSE is the authoritative notice:

• Original annotations, scenario descriptions, and documentation: CC BY-NC-SA 4.0.
• DriveHierarchy-authored source code: Apache-2.0, unless otherwise stated.
• Third-party and derived material: subject to the original source terms, including nuScenes/nuPlan, NAVSIM, Wayve LingoQA, HRI DRAMA, CARLA, SUMO, WOMD-Reasoning, and the Waymo Open Motion Dataset.

Back to top ↑