xl-tang3/BaryBind

BaryBind is a multimodal alignment framework that learns a Multimodal Wasserstein Barycenter (MWB) as a shared anchor, then aligns modalities via barycenter-anchored volumetric contrast. Rather than fixing a modality-specific reference, it optimizes the anchor across modality distributions and captures their higher-order geometry.

_{WB optimization learns the shared anchor; volumetric alignment binds modalities around it. Figure 3 from the paper.}

Method · Results · Installation · Pretrained Encoders · Model Zoo · Data · Training · Evaluation · Citation

• A learned shared anchor. Optimize a Wasserstein barycenter across modality distributions to approximate their geometric semantic center.
• Higher-order alignment. Construct a barycenter simplex from the WB embedding and modality-to-WB gaps, and contrast its volume across matched and mismatched samples.
• Scaling Beyond three modalities. Extend alignment to text, video, audio, subtitles, and depth.

BaryBind connects distribution-level alignment with sample-level multimodal geometry in three steps.

The MWB objective is

$$ \mathcal{L}_{\mathrm{MWB}}^* = \inf_{Q\in\mathcal{P}(\mathcal{M}_B)} \sum_{k=1}^{K}\lambda_k W(P_k,Q), \qquad \lambda_k\geq 0,\quad \sum_{k=1}^{K}\lambda_k=1. $$

Here, $P_k$ is the distribution of k-modality features and $Q$ is the WB distribution. The learned map produces $b=T_\theta(m_0)$ from an initializer $m_0$, while its optimization uses all modalities.

With $R=[b,r_1,\ldots,r_K]$, the paper defines the squared barycenter simplex volume as

where $\mathbf G=[\boldsymbol b,\boldsymbol r_1,\ldots,\boldsymbol r_K,]$ collects the modality-to-WB gaps; $\boldsymbol b^\top\mathbf G$ captures barycenter-relative geometry, while $\mathbf G^\top\mathbf G$ encodes gap magnitudes and inter-modality correlations.

The Barycenter-Anchored Volumetric Contrastive (BVC) loss contrasts matched WB–gap pairs against mismatched ones. Data-Anchor Matching (DAM) further supervises whether the anchor and multimodal features match. The combined objective is

$$ \mathcal{L}=\mathcal{L}_{\mathrm{MWB}} +\alpha_1\mathcal{L}_{\mathrm{BVC}} +\alpha_2\mathcal{L}_{\mathrm{DAM}}. $$

_{Multimodal embeddings group around WB anchors while retaining class separation. Original experimental visualization, Figure 4.}

Selected results from the paper. Higher is better for retrieval and classification; lower is better for FID. T / V / A / S / D denote text, video, audio, subtitle, and depth, respectively.

Recall@1 (%) · T-VA configuration · Table 2. Each cell reports T2V / V2T.

Under the five-modality T-VASD configuration, BaryBind reaches 57.8 / 54.6 on MSR-VTT and 86.1 / 85.4 on VATEX (Table 2).

See the paper for complete comparisons, experimental protocols, and ablations.

The reported code environment uses Python 3.9 and CUDA 11.7, with PyTorch and additional dependencies specified in preinstall.sh.

git clone https://github.com/xl-tang3/barybind.git
cd barybind

conda create -n barybind python=3.9
conda activate barybind
sh preinstall.sh

Run the commands below from the repository root. Other environment versions may work, but compatibility has not been established here.

Prepare the visual, audio, and text encoder checkpoints before training.

mkdir -p pretrained_weights/clip

wget -P pretrained_weights/clip/ \
  https://huggingface.co/QuanSun/EVA-CLIP/resolve/main/EVA01_CLIP_g_14_psz14_s11B.pt

Download BEATs_iter3_plus_AS2M.pt from the official BEATs repository and place it in pretrained_weights/beats/.

mkdir -p pretrained_weights/beats

Download and save both the model and tokenizer:

from transformers import BertModel, BertTokenizer

save_dir = "pretrained_weights/bert/bert-base-uncased"

model = BertModel.from_pretrained("bert-base-uncased")
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

model.save_pretrained(save_dir)
tokenizer.save_pretrained(save_dir)

Expected checkpoint locations:

The processed pretrained_weights path should be as follows:

    ├── pretrained_weights
    │   ├── beats
    │   │   └── BEATs_iter3_plus_AS2M.pt
    │   ├── bert
    │   │   └── bert-base-uncased
    │   ├── clip
    │   │   └── EVA01_CLIP_g_14_psz14_s11B.pt

Also, download the vast weights as a start checkpoint in VAST

When downloading a pretrained BaryBind model, retain the entire model directory, including both log/ and ckpt/. For example:

• pretrained_models/barybind_pretrained_4modalities/log/
• pretrained_models/barybind_pretrained_4modalities/ckpt/

Pass the model directory containing these two folders to --pretrain_dir. This is distinct from --checkpoint, which points to an individual .pt file for evaluation.

BaryBind uses a 150K-sample subset of VAST-27M for continued pretraining.

Save the subset annotation file as annotations150k.json. The annotation file does not replace the corresponding media; prepare those following VAST's instructions.

Update the annotation and local data paths in:

• scripts/barybind/finetune_ret.sh
• config/barybind/finetune_cfg/finetune-area.json

For downstream evaluation, configure the relevant dataset annotations and media paths in the selected task configuration.

After preparing the encoder weights, model initialization, and subset data, set the appropriate paths and configuration in the launch script, then run:

sh scripts/barybind/finetune_ret.sh

Select the downstream task configuration and update the checkpoint, dataset, and output paths in scripts/barybind/finetune_ret.sh, then run the same script:

sh scripts/barybind/finetune_ret.sh

For example, the following command fine-tunes a retrieval model with the MSR-VTT configuration. Set the paths before launching. The example uses eight local GPU processes; adjust the process count and training configuration for your hardware.

export PRETRAIN_DIR="/path/to/barybind_pretrained_4modalities"
export OUTPUT_DIR="/path/to/outputs/msrvtt"

python3 -m torch.distributed.launch \
  --nnodes 1 \
  --node_rank 0 \
  --nproc_per_node 8 \
  --master_port 9834 \
  ./run.py \
  --learning_rate 2e-5 \
  --checkpointing true \
  --first_eval true \
  --save_best true \
  --config ./config/barybind/finetune_cfg/retrieval-msrvtt.json \
  --pretrain_dir "$PRETRAIN_DIR" \
  --output_dir "$OUTPUT_DIR"

Use the matching task configuration and specify the saved fine-tuned checkpoint. Add these arguments to the corresponding task command:

--mode testing \
--checkpoint /path/to/saved_checkpoint.pt

export PRETRAIN_DIR="/path/to/barybind_pretrained_4modalities"
export FINETUNED_CHECKPOINT="/path/to/saved_checkpoint.pt"
export EVAL_OUTPUT_DIR="/path/to/outputs/msrvtt-eval"

python3 -m torch.distributed.launch \
  --nnodes 1 \
  --node_rank 0 \
  --nproc_per_node 8 \
  --master_port 9834 \
  ./run.py \
  --config ./config/barybind/finetune_cfg/retrieval-msrvtt.json \
  --pretrain_dir "$PRETRAIN_DIR" \
  --output_dir "$EVAL_OUTPUT_DIR" \
  --mode testing \
  --checkpoint "$FINETUNED_CHECKPOINT"

If you find BaryBind useful in your research, please cite:

@inproceedings{tang2026barybind,
  title     = {Binding Multiple Modalities via Multimodal Wasserstein Barycenter},
  author    = {Tang, Xiaole and Xu, Jiayi and Gu, Xiang and Yang, Yan and Sun, Jian},
  booktitle = {NeurIPS},
  year      = {2026}
}

BaryBind builds on VAST and uses pretrained components from EVA-CLIP, BEATs, and BERT via Hugging Face Transformers. We thank the authors for making their research and code available.