LynnReal-AI/LynnReal-Omni

Video generation, references, motion controls and editing.

Paper^🔥HOT  ·  Introduction  ·  Overview  ·  Models  ·  Quick start  ·  ComfyUI  ·  Tasks  ·  Usage guides  ·  Roadmap  ·  Changelog  ·  Organization  ·  Acknowledgments  ·  Citation

For limited VRAM, we strongly recommend Standard Lite to preserve Standard quality, or Flash Lite for a much smaller memory footprint and very fast generation. Start with the matching Lite checkpoint and *_lite.json ComfyUI workflow when choosing a memory-saving setup.

• Standard Lite · quality first: BF16 Lite is 37.6 GiB; INT8 Lite is 20.4 GiB. At the shipped 4 steps, each preserves its corresponding original checkpoint's tested outputs; the BF16/INT8 switch remains available.
• Flash Lite · memory and speed first: the INT8 DiT is 16.7 GiB, paired with the Light VAE and shipped 3-step workflows. It retains Flash's fast generation and tested quality; Lite reduces the checkpoint footprint rather than promising extra speed over full Flash.

These are DiT file sizes, not total runtime VRAM requirements. See the ComfyUI Lite setup and checkpoints.

LynnReal-Omni: Native multi-modal Video Generation for Agentic Visual Workflows
Technical report, September 2026.

Preview compressed to ~10 MB for this repository; for the full-quality demo, click the YouTube or Bilibili link below.

One model, unified across many video tasks. Built on a 32B shared multimodal diffusion transformer (following the MiniMax H3 architecture), LynnReal-Omni brings text-to-video, image-to-video, human- and hand-pose guided generation, structural control, omni-reference generation, style transfer, video editing, degraded-video restoration (we find that it also repairs videos affected by accumulated error) and streaming long-video generation into a single framework, all at four-step fast generation. It also accepts heterogeneous inputs such as appearance references, editable 3D renders and game recordings, so an Agent can compose visual conditions inside one model.

Flash, built for real-time rendering. We train a 27B LynnReal-Omni-Flash (three-step generation) that lowers inference cost through model and decoding acceleration and a lightweight VAE decoder. On a single H100, warm generation and decoding of a 22-frame 540p video takes 843 ms with the Standard model and 377 ms with Flash, laying the groundwork for real-time streaming video generation.

Data pipeline and MSAVP evaluation. We build a systematic multi-shot and omni-reference data pipeline covering video cleaning, subject association, multimodal annotation and alignment control. From the large corpus of collected videos we select a high-quality multi-shot audio-visual subset and extract a variety of omni-reference condition controls. We also propose MSAVP for evaluating multi-shot audio-visual generation.

Standard · 4 steps   /   Flash · 3 steps   /   Optional lightweight VAE

Standard uses four denoiser forwards. Flash uses three forwards with 42 blocks, spatial token selection, INT8 projection weights and INT8 activations. Step counts refer to each generation call: Standard first-pass generation uses 4 steps; independent frame repair uses 4 steps per source frame. Long streaming can add 2 refinement steps per later chunk; see streaming generation.

The repositories below are the planned Hugging Face upload locations. Weights will become downloadable as their uploads are completed.

Place each downloaded bundle under weight/, preserving its configurations and component subdirectories:

weight/
├── standard/           # Standard bundle: DiT, text encoder, codecs and configs
│   └── transformer/    # The single physical Standard DiT checkpoint
├── flash/              # Flash bundle
├── vae/                # Official VAE
└── light-vae/          # Optional lightweight VAE

Pass the complete weight/standard/ bundle to --weights. Its DiT lives in weight/standard/transformer/; the other required components are loaded from the bundle too. The lightweight VAE does not replace the DiT or text encoder.

Standard DiT INT8 is coming soon. Experimental Standard INT8 launchers are already in the codebase; the downloadable checkpoint has not been released. Flash W8A8 is a separate model variant.

Run the following commands from the repository root, with the required local weights in place.

From Conda base, install this checkout and activate the new environment:

pip install .
conda activate lynnreal

The bootstrap creates or reuses a Python 3.12 environment named lynnreal and checks attention on the visible GPU. Use pip install . -v for detailed progress. See Runtime notes for other environments, offline installation and hardware-specific behavior.

# Standard: five seconds, native 768p, four denoiser forwards
bash script/sample/standard/bf16/t2v.sh --frame 5s --resolution 768p

# Flash: three denoiser forwards
bash script/sample/flash/int8/t2v.sh

# First-frame conditioning
bash script/sample/standard/bf16/ti2v.sh --image test/assets/times_square.png

# Continue a video using the launcher's example input
bash script/sample/standard/bf16/v2v.sh

# Subject references: replace these paths with your own images
bash script/sample/standard/bf16/ref2v.sh --ref_image first.jpg,second.jpg

Each launcher saves the video, prompt, full log and measured DiT/decoder times under output/. --frame 120f requests exactly 120 frames; --frame 5s is five seconds at 24 fps. The default output is 1344 × 768. Standard BF16 and Flash W8A8 have separate entry points. The experimental script/sample/standard/int8/ launchers remain available for development; the downloadable Standard DiT INT8 checkpoint is coming soon.

For a repeatable run, use bash script/sample/standard/bf16/t2v.sh --seed 7 --frame 5s --name my_case. Choose a new output name for each run. Keep prompt, seed, geometry, references and attention backend fixed for comparisons. Changing attention precision can change a four-step result. Standard BF16 defaults to the native attention backend and official VAE.

We also ship a ComfyUI port — it is open, and you are very welcome to try it! 🎉

comfyui/ contains workflows for t2v, i2v, r2v, pose2v and v2v on the Standard four-step checkpoints and for t2v, ti2v and ref2v on the Flash three-step checkpoint, the small custom node pack they need (ComfyUI-LynnReal), the demo assets the workflows load, and the exact model files each task needs. Those files are published in ComfyUI format alongside the release weights on Hugging Face under 🤗 stdstu123/LynnReal-Onmi-beta-0.1 · comfyui/models. Drop the workflows, the node pack and the models into your ComfyUI install and they run — no launcher flags needed. Every task, its workflow and the weights it loads are listed in comfyui/README.md.

The Flash checkpoints run in ComfyUI at their trained three steps: t2v, ti2v (one first frame) and ref2v (reference pictures), W8A8 DiT plus the Light VAE! On a single H100 80 GB at 1344×768, warm — model already loaded, the way a session runs — three measured runs per cell:

A five-second 1344×768 clip with native stereo audio — three denoiser steps and both decoders included — in about eight and a half seconds, on one GPU. Generate is the release's own Generate-wall convention (first denoiser forward to decoded frames); click-to-video is what you actually wait for, prompt encoding and muxing included, and repeat runs agree to ±0.02 s!

Nothing is assumed about your card: the node pack picks the fastest attention it can find and verifies it numerically (FlashAttention-3 → FA2 → cuDNN SDPA → native), falls back to ComfyUI's own block math whenever a fused kernel is unavailable, and tunes the INT8 GEMMs for the GPU it actually runs on!

script/sample.py accepts repeated --reference paths. Image and video prompt labels count separately from one. --aligned-reference indexes the complete input list from zero: an appearance image followed by a pose video uses index 1.

Image editing saves both a selected image and its complete companion clip. See the prompt-writing guide for task-specific instructions and reference conventions.

frame_repair.sh is a general repair entry point. Supply your own video and repair instruction for the scene and artifacts you want to address.

bash script/sample/standard/bf16/frame_repair.sh \
  --video /path/to/input.mp4 \
  --prompt-file /path/to/repair_prompt.txt \
  --output output/my_frame_repair

• Input: 24 fps; width and height must be divisible by 32.
• Process: edit each source frame independently, then assemble a silent video at 24 fps.
• Budget: four DiT forwards per source frame; 120 frames require 480 forwards.
• Preview: add --indices 0,24,48 --keep-clips to inspect a few frames before a full run.
• Limitation: independent edits may introduce brightness or shape fluctuations across frames.

script/sample/standard/bf16/stream.sh generates an image-conditioned first chunk and continues its latent history. Run these commands from the repository root:

# Default: 5 seconds, 768p, 24 fps, seed 7; four DiT forwards per chunk.
bash script/sample/standard/bf16/stream.sh --frame 5s

# 30 seconds: preserve the initial four-step prefix, then use 4+2-step refinement.
bash script/sample/standard/bf16/stream.sh --frame 30s

# 30 seconds with four steps throughout; disable refinement and context refresh.
bash script/sample/standard/bf16/stream.sh --frame 30s --no-refresh-context

# Generate only the native 22-frame first chunk.
bash script/sample/standard/bf16/stream.sh --first-chunk-only

The default five-second run uses 4 steps throughout. Above 136 requested frames, the default enables context refresh: the bootstrap and first seven continuation chunks use 4 Standard BF16 DiT forwards each; continuation chunk eight onward uses 4 + 2 forwards, including the second pass. --no-refresh-context keeps the fixed-context, four-step-only policy at any duration. These modes use the same DiT in weight/standard/transformer/ and the official VAE.

The totals include the bootstrap and every continuation chunk. They describe sampling work, rather than elapsed time or text-encoder/decoder computation. Long clips refresh image-aware text from the preceding chunk after the stable prefix. Local texture and shape drift can still occur.

bash script/sample/standard/bf16/stream.sh \
  --image inputs/first.png \
  --prompt-file inputs/initial.txt \
  --captions inputs/continuations.json \
  --frame 5s --seed 7 \
  --output output/my_stream

Each run saves video.mp4, sample.log and run.json under output/standard/bf16/stream/<timestamp>/, or the directory supplied with --output. The first_chunk/ and continuation/ subdirectories retain their prompts, latents and per-chunk timing records. run.json records the planned step budget; per-chunk metadata records actual DiT forwards.

Long-stream context refresh stages the text encoder through CPU memory on a single GPU, which increases end-to-end latency. An optional second-GPU conditioning worker can be selected with --conditioner-service. See streaming instructions for worker startup, native overlap, appearance constraints and --audit-codec verification. For continuation from an existing video, use script/sample/standard/bf16/v2v.sh --continuation.

python script/setup_env.py --current-env

python script/setup_env.py --pypi-only
conda activate lynnreal

python script/setup_env.py --attention-only
# Strict FA3 acceptance on a supported Hopper GPU:
python script/setup_env.py --attention-only --attention _flash_3

model/   inference, conditioning, codecs and kernels
script/  sampling and video continuation
skill/   prompt-writing guidance
weight/  standard/, flash/, vae/, light-vae/
test/    default prompts, reference media and continuation plans
output/  generated media, complete logs and timing records
tool/    helpers required by streaming and structure editing

All model components are local to weight/, including the text encoder, tokenizer, image processor, audio codec and schedulers. The DiT bundles contain complete inference parameters. No additional adapter is required. Weight paths outside this directory are rejected. Model licenses and upstream attribution are retained in each bundle.

Standard uses one DiT checkpoint in weight/standard/transformer/. Standard clip launchers validate this directory and run four denoiser forwards per generated clip. Streaming uses four per chunk; when context refresh is enabled, continuation chunk eight onward adds two refinement forwards. The internal reference-task component name transformer_ref maps to the same transformer/ directory through modular_model_index.json; no transformer_ref/ weight directory is needed. The sampler loads only the DiT component selected by the task layout. See the model card included with the downloaded Standard weight bundle for details.

This repository will continue to receive updates. The current beta is still imperfect, and we appreciate your patience with its remaining defects.

Our planned releases and improvements include:

• The Standard DiT INT8 checkpoint.
• Improved model checkpoints with better generation quality and consistency.
• Training code to support further experimentation and model development.
• Selected datasets or dataset subsets, subject to their redistribution permissions.
• Continued fixes to installation, hardware compatibility, inference, documentation, and reproducible examples.
• ComfyUI: keep the Light VAE decoder compiled on DynamicVRAM machines — it currently falls back to eager decoding there, which roughly doubles decode time.

These are development plans, not a fixed release schedule. Availability and usage instructions will be updated here as each release is ready. Reproducible bug reports and feedback on failure cases are welcome.

This project is developed by Lynnreal Lab.

Leader: Xiaofeng Mao, Shaohao Rui, Weijie Ma

Core Contributors: Xiaofeng Mao, Peijia Lin, Shaohao Rui, Yibo Zhang, Haibin Wan, Weijie Ma

Welcome students with backgrounds in 3D reconstruction and interactive world models to apply for internships and collaborate! Please send your resume to hr@lynnreal.com.

We especially thank the MiniMax H3 team for the model that forms the foundation of LynnReal-Omni.

We also thank Qwen3-VL for the multimodal encoder, tokenizer and processor implementation used for conditioning.

The following community projects provided useful implementation references and technical discussions during our development and experiments:

We thank the authors of these projects for sharing H3 refinement implementations and workflows that informed our second-pass sampling experiments:

These acknowledgments include development references and experimental comparisons; the released sampling paths and their validated settings are documented separately in this repository. We appreciate the authors' open-source contributions and the community's reproducible bug reports and feedback.

Please see LICENSE, NOTICE, and the respective upstream projects for their license terms and attribution notices.

If you use Yume for your research, please cite our paper:

@misc{mao2026lynnrealomninativemultimodalvideo,
      title={LynnReal-Omni: Native multi-modal Video Generation for Agentic Visual Workflows}, 
      author={Xiaofeng Mao and Peijia Lin and Shaohao Rui and Yibo Zhang and Haibin Wan and Weijie Ma},
      year={2026},
      eprint={2609.15863},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.15863}, 
}

Thank you for trying LynnReal-Omni.
Better models, training code and selected datasets are planned.
Back to overview ↑