_{by Mia's AI Lab}

Serve GLM-5.3-Flash from two NVIDIA DGX Sparks (GB10, 128 GB each, linked by their ConnectX-7 ports) through an OpenAI-compatible API, with 4 concurrent requests, the model's full 1,048,576-token context and image and video input. It runs TensorFold v0.6.0 on both Sparks (one rank on each) in NVIDIA's PyTorch container, plus 53 patches: DFlash2 and copy drafts, 4-bit dense weights, an FP8 KV cache, faster prompt kernels, a one-shot RoCE all-gather between the Sparks, several requests over one shared cache pool, vision, tool calling, /tokenize and /metrics.

• Checkpoint: Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw (EXL3 routed experts at 4 bits a weight, BF16 elsewhere, ~176 GB), a byte-identical mirror of brandonmusic/GLM-5.3-Flash-tr3-4bpw
• Drafter: incoai/GLM-5.3-Flash-DFlash2, or the checkpoint's own MTP head (DRAFTER, see Configuration)
• API model id: GLM-5.3-Flash-EXL3
• Context: 1,048,576 tokens a request; the 4 requests share an FP8 KV pool of about 2.9M tokens (2,922,496 at the measured start)
• Tool calling, structured outputs (xgrammar), /tokenize, and reasoning_effort low / high / max
• One command on the first Spark: ./start.sh sets up both Sparks and starts both ranks; ./stop.sh stops them

Two DGX Sparks at the default configuration (4 streams, 1,048,576-token window, FP8 KV cache, 4-bit dense weights, DFlash2 plus copy drafts, vision on), with the GPU clocks capped at 2,200 MHz. Decode and prefill were measured with sparkDash through the OpenAI API, from another machine on the network.

Decode (aggregate across the concurrent requests, per request, and time to first token)

Replies served 4 at a time are identical to the same requests served one at a time (11 of 11 cases staggered, and 11 of 11 sent in a burst).

Prefill

Prompt reuse (the server resumes from a kept prompt state instead of prefilling, with the same reply)

Quality (FP8 KV cache and 4-bit dense weights, see Checks; measured on a build with identical replies)

• Two DGX Sparks (or two GB10 systems with 128 GB unified memory), with nothing else large on their GPUs: each needs ~110 GiB free memory when the server starts (start.sh warns below that; stop other GPU work).
• A direct ConnectX-7 link: a QSFP cable between the CX7 ports and an IPv4 address on each end in one private subnet (e.g. 192.0.2.1/24 and 192.0.2.2/24; ping must work), with a RoCE v2 GID (start.sh checks). With both ports cabled and addressed, both are used (a prompt chunk's all-gather is ~1.8x faster on two rails).
• Key-based ssh from the first Spark (the head, which runs ./start.sh and the API) to the second (the worker): ssh-copy-id user@<worker> (after ssh-keygen -t ed25519 if you have no key); check with ssh -o BatchMode=yes user@<worker> true.
• Docker with the NVIDIA container runtime, your user in the docker group, and rsync, on both Sparks.
• Disk, on each Spark: ~205 GB: ~176 GB (164 GiB) for the checkpoint and ~2.3 GB for DFlash2 under ~/.cache/huggingface, ~25 GB for the image under Docker's root. prepare.sh wants 180 GB free for the download (MIN_FREE_GB) and 35 GB under Docker's root (IMAGE_FREE_GB; the sum when they share a filesystem), and checks the worker before the copy. With WORKER_WEIGHTS=nfs the worker needs only the image (Worker weights over NFS).
• Optional: the hf CLI on the head (faster download) and a Hugging Face token (~/.cache/huggingface/token or HF_TOKEN).

On the head:

git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold.git
cd GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold
cp scripts/local.sh.example scripts/local.sh     # set WORKER=user@192.0.2.2 in it (and FABRIC_PEER, see below)
./start.sh

WORKER is the worker's ssh target. The ranks talk over the route to it: if it is on another network than the link, set FABRIC_PEER to the worker's CX7 address. The worker needs no copy of this repository.

The first run sets up both Sparks (see below): the image (~25 GB) on each, the checkpoint (~176 GB) and DFlash2 downloaded on the head and copied to the worker, then the CUDA kernels compile once per image (into ~/.cache/tensorfold-glm53/<image hash>). Later starts take 2 to 6 minutes to load ~80 GiB of weights on each Spark. start.sh shows each step and the server's log, runs a smoke test through both ranks, and prints GLM-5.3-Flash-EXL3 is now LIVE! on port 8888 with the endpoint.

Any OpenAI client works with base_url = "http://<head-address>:8888/v1" and the model GLM-5.3-Flash-EXL3. The model thinks before it answers (reasoning_content), so give replies enough max_tokens.

curl -s http://<head-address>:8888/v1/models
curl -s http://<head-address>:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "GLM-5.3-Flash-EXL3",
  "messages": [{"role": "user", "content": "Write a Python fibonacci function."}],
  "max_tokens": 2000
}'

./start.sh restart                                       # restart both ranks, e.g. after changing a setting
./stop.sh                                                # stop both ranks and free their GPU memory
docker logs -f glm53-flash-tf                            # rank 0's log (here)
ssh <worker> docker logs -f glm53-flash-tf               # rank 1's log (why it exited, if it did)
curl -s http://<head-address>:8888/health                # busy flag, streams, free pool tokens

If start.sh stops at a check:

GLM's own vision tower runs on the head (rank 0: 1.05 GiB of bf16 weights and 0.75 GiB of workspace). Send images and videos as OpenAI-style content parts in a user message:

IMG=$(base64 -w0 photo.jpg)
curl -s http://<head-address>:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "GLM-5.3-Flash-EXL3",
  "messages": [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,'"$IMG"'"}},
    {"type": "text", "text": "What is in this picture?"}]}],
  "max_tokens": 2000
}'

A video is a video_url part ({"type": "video_url", "video_url": {"url": "data:video/mp4;base64,..."}}).

A request body can be up to 96 MiB, so data URLs carry about 70 MB of pictures and clips in all. By default only data URLs are accepted; VISION_URLS=1 also lets the server fetch public https:// URLs. VISION=0 serves text only and leaves the tower's ~1.8 GiB on rank 0 to the cache.

./start.sh works in five steps, each shown as it runs:

1. Setup: scripts/prepare.sh, when the setup is not ready on both Sparks (first run, new patches, another model, drafter, revision, image or worker).
2. Checks: the arguments (TensorFold's own parser), the link, the previous server (stopped on restart, or when only one rank is up; a stopped container left from an earlier run is removed), the port, free memory (a warning below ~110 GiB on either Spark).
3. Launch: rank 1 on the worker over ssh, then rank 0 and the API here, with the settings from scripts/config.sh.
4. Loading: rank 0's log as it comes, and every 15 s the elapsed time and how much of the startup estimate is on each Spark's GPU; if a rank stops, both ranks' last log lines and why.
5. Smoke test: one chat completion through both ranks, then the LIVE message and the endpoint.

A running server is left alone; ./start.sh restart stops it only after the setup and argument check pass, so a typo leaves it running (running requests are cut off; stop.sh warns). Extra arguments go to tensorfold serve on both ranks after the defaults, so they win (./start.sh restart --max-tokens 16384); ./start.sh --help lists all. FOREGROUND=1 ./start.sh stays attached to rank 0's log and exits with its exit code (for a systemd unit), without the progress lines, the window retry or the smoke test; when either rank ends, it stops the other one.

scripts/prepare.sh does the one-time setup, and is safe to re-run (each step skips work already done):

1. Preflight on both Sparks: Docker, the GPU, rsync, key-based ssh, the RoCE link, disk space.
2. The image tensorfold-glm53:v0.6.0 on the head: TensorFold v0.6.0 with every patches/*.patch applied, plus PyAV (video decoding) and xgrammar (structured outputs), on NVIDIA's nvcr.io/nvidia/pytorch:26.07-py3. It first pulls the published image ghcr.io/miaai-lab/glm-5.3-flash-exl3-2x-dgx-sparks-tensorfold:v0.6.0-<image hash>, by the digest pinned in scripts/config.sh (IMAGE_TAG / IMAGE_DIGEST) while the patches are this release's (the hash covers the patches and those pip packages); after you change patches/, it pulls that hash's tag if one is published, else (or with PULL=0) it builds.
3. The same image on the worker: pulled, else streamed from the head (docker save | docker load), checked identical.
4. The checkpoint, and DFlash2 with DRAFTER=dflash2, downloaded into ~/.cache/huggingface on the head at their pinned revisions (the checkpoint checked with tensorfold info), then copied to the worker with rsync over ssh and checked file by file. Both resume.

scripts/prepare.sh             # set up both Sparks without starting the server
scripts/prepare.sh --rebuild   # rebuild the image from scratch
PREPARE=1 ./start.sh restart   # force prepare.sh, then restart; PREPARE=0 skips the check

After changing patches/, scripts/publish-image.sh pushes the new image to GitHub Container Registry (latest and v0.6.0-<image hash>, the tag prepare.sh looks for).

GLM-5.3-Flash keeps a compressed latent cache (DSA) and the sparse indexer's pooled keys for every token, plus small recurrent states for its linear-attention (KDA) layers. With PARALLEL above 1, all requests draw their per-token caches from one shared pool:

Any one request can grow to the full window, and the four together share the pool: e.g. one 1M-token conversation next to one more of 1M, or next to three of ~640k. A request the pool cannot place yet waits until others finish (kept prompt states give way first); /health shows pool_tokens, pool_free_tokens and the streams decoding, filling and paused.

TensorFold's budget on each Spark is MemAvailable at start minus a host reserve (MEMORY_RESERVE_GIB, 14.5 GiB here; TensorFold's own default is a tenth of RAM). The server uses about 10 GiB beyond its own estimate at its peak, so the reserve also sets the lowest free memory. On the Spark's unified memory, running out tends to freeze the machine rather than fail an allocation. A setting that does not fit is refused before any weights load, with the largest window that fits; start.sh then restarts once with that window and says so (free memory on both Sparks for the full one). Other settings' windows:

By default the worker keeps its own copy of the checkpoint and DFlash2 (~166 GiB, copied over the link by prepare.sh). With WORKER_WEIGHTS=nfs it keeps none: rank 1 reads the head's Hugging Face cache over NFS, read-only. Loading is as fast as from the worker's own disk (both ranks were live in ~2.2 minutes).

1. On the head, export the cache to the worker once (this needs root; the worker needs nothing installed):

   sudo apt install nfs-kernel-server
   echo "$HOME/.cache/huggingface <worker CX7 address>(ro,no_subtree_check)" | sudo tee -a /etc/exports
   sudo exportfs -ra

2. Put WORKER_WEIGHTS=nfs in scripts/local.sh or .env, and run ./start.sh restart.

prepare.sh then creates a read-only docker NFS volume on the worker (NFS_VOLUME, default glm53-hf, no sudo) and checks that the worker sees every file of both snapshots as the head has them, instead of copying. NFS_PATH is the exported path as the worker mounts it (default: the head's HF_CACHE; / for an NFSv4 export with fsid=0), and NFS_SERVER the head's address (default: its address on the link).

Every setting lives in scripts/config.sh. Set one for a single run from the environment (PARALLEL=2 ./start.sh restart), or keep it in scripts/local.sh (sourced as bash) or in a .env file next to start.sh (plain KEY=value lines, read, never run); both files are yours, not the repository's. The first that sets a value wins: the environment, then scripts/local.sh, then .env, then the default.

Speed settings' measured effects: What the patches change. Any TENSORFOLD_*, TF_GLM_* or TF_ROCE_* variable is passed to both ranks (export it in scripts/local.sh; .env lines are).

Revision pins. The checkpoint and DFlash2 are pinned to the revisions this recipe was measured with (MODEL_REVISION, DFLASH2_REVISION): both ranks serve exactly those from the local cache, and a new upstream commit changes nothing until the pin does. Set one empty to take the Hub's main when first downloaded.

Less common settings are described in scripts/config.sh and scripts/nodes.sh: MODEL_ID, DFLASH2_ID, TF_VERSION, TF_REPO, BASE_IMAGE (the patches are made for TensorFold v0.6.0; after changing any of these run scripts/prepare.sh --rebuild), IMAGE, CONTAINER_NAME, GHCR_IMAGE, HF_CACHE (default $HF_HOME or ~/.cache/huggingface), KERNEL_CACHE, STATE_DIR, MIN_FREE_GB, IMAGE_FREE_GB, NCCL_RAILS (1: one CX7 port even when both are up), NCCL_CHANNELS (4), NCCL_DEBUG, RSYNC_OPTS. start.sh also takes HF_HUB_OFFLINE=0 (let the server reach Hugging Face; by default it serves from the local cache only).

The checkpoint's own sampling defaults apply (temperature 1.0, top_p 0.95). Per request:

• temperature, top_p, top_k, min_p and seed override them (temperature: 0 decodes greedily). Without a seed, the sampler's key comes from the prompt, so the same request gives the same reply.
• reasoning_effort: "low", "high" or "max", at the top level or in chat_template_kwargs ("medium" is "high", "xhigh" is "max"). Without it, GLM's template uses max; "none" (or "chat_template_kwargs": {"enable_thinking": false}) answers without thinking.
• The reasoning comes back in reasoning_content, the answer in content. A request without max_tokens gets 32,768 tokens for both (MAX_TOKENS; cut to what the window has left, never refused). An empty content means the model thought until max_tokens: give more, or use reasoning_effort: "low".

• Endpoints: /v1/chat/completions, /v1/completions, /v1/responses, /v1/models, /tokenize and /detokenize (also under /v1/), /health, and Prometheus /metrics (TensorFold v0.6.0's request counters, latency and time-to-first-token histograms, plus /health's figures as tensorfold_health: metrics). No Anthropic /v1/messages.
• Context limits: a request whose prompt plus max_tokens does not fit the window is refused with HTTP 400 in OpenAI's wording, "code": "context_length_exceeded" and param naming the field (messages or prompt).
• Tool calling: tools / tool_calls, each call streamed whole once it is written (empty deltas every 2 s meanwhile, for clients with an idle timeout; a reply that ends inside a call, at its token limit, ends with length and never sends that call), with arguments typed by their schema (an array as a JSON array, null and const values as such). Calls written inside the think block count when the reply ends on them, and a tool-calling step's reasoning is put back into the next request when an agent client drops it (TF_GLM_KEEP_REASONING=0 turns that off). A past tool call whose arguments are not a JSON object is left out of the prompt with its result (and logged) instead of failing the request.
• Structured outputs: response_format (json_object or json_schema) and the guided_json / guided_regex / guided_choice / guided_grammar / structured_outputs fields, enforced with xgrammar (after the think block).
• /tokenize / /detokenize: vLLM's fields; a prompt or messages (rendered as the chat route does) to token ids with their count and max_model_len, and back.
• Refused with HTTP 400: logprobs: true / top_logprobs (this engine returns no token probabilities) and n other than 1. Accepted but ignored, with no error: logit_bias, and the presence, frequency and repetition penalties. priority: "background" serves a request after the others.
• Prompt reuse: a new prompt that extends a recent one token for token resumes from its kept state, and a shared system prompt's state is reused (SHARED_PREFIX).

scripts/prepare.sh bakes every patches/*.patch into the image (diffs against TensorFold v0.6.0's site-packages, applied with patch -p0 in filename order); start.sh rebuilds or re-pulls the image when the patches change.

Outputs. Drafts only propose: every drafted token is checked against the model's own keyed sample, so drafted replies equal TensorFold's serial, one-token-at-a-time decoding (send "draft": false for that reference). Replies served 4 at a time equal the same requests served alone; COMM=roce, SPLIT=1, TF_GLM_L2PF and TF_GLM_EXL3_LOADS move the same bits. Three defaults are not exact against the checkpoint in bf16, for speed and the 1M window: DENSE=q4 and KV=fp8 are lossy (quality and the 1M needle: Performance), and KDA_CHUNKED=1 is close to the serial kernel but not its bits. DENSE=bf16 KV=bf16 KDA_CHUNKED=0 serves the checkpoint as it is, with a shorter window (KV pool and memory).

The checks in tools/ talk to the running server (API_URL, default http://127.0.0.1:8888; or just PORT), from the head or another machine (API_URL=http://<head-address>:8888 tools/needle.py). Performance is measured with sparkDash (Performance).

start.sh      set up (first run) and start both ranks
stop.sh       stop them
scripts/      config.sh (all settings), local.sh.example (this setup's WORKER), prepare.sh (image + checkpoint on
              both Sparks), nodes.sh (ssh and the RoCE link), publish-image.sh (push the image to GHCR),
              banner.sh (start.sh's banner)
patches/      patches baked into the image
tools/        checks against the running server (needle, tool calls)
CHANGELOG.md  what changed in each release
CREDITS.md    who and what this builds on
LICENSE       Apache License 2.0
NOTICE        third-party notices (TensorFold's MIT and Apache-2.0 notices, b12x, glm53-tensorfold-spark, ShapleyMcg)

Apache License 2.0, see LICENSE. NOTICE carries the third-party notices that go with it: the files in patches/ modify TensorFold v0.6.0, and the TensorFold code they change or quote as context stays under TensorFold's licenses (Apache 2.0 from v0.6.0, and the MIT notice of code written before it, both in NOTICE); parts of patches 0006 (b12x), 0036, 0046 and 0047 (glm53-tensorfold-spark) come from Apache-2.0 projects, credited there and in CREDITS.md. The model files are downloaded from Hugging Face and are not part of this repository:

• The checkpoint is under the ShapleyMcg License 1.0, an attribution-required license; its model card and LICENSE file have the terms. Its attribution notice:

  This work includes or was produced using ShapleyMcg, created by Brandon M. Music (https://github.com/brandonmmusic-max/shapleymcg). ShapleyMcg is licensed under the ShapleyMcg License v1.0, an attribution-required license that grants no rights to the person known as "0xSero." Use of ShapleyMcg without this attribution is unlicensed.

• The base model GLM-5.3-Flash is under the license on its model card.

• The DFlash2 drafter is under CC BY-NC-ND 4.0, non-commercial use only (commercial licensing: contact@inco.ai); DRAFTER=mtp serves without it.

Third-party software in the image. The prebuilt image (and the one scripts/prepare.sh builds) is based on NVIDIA's PyTorch container nvcr.io/nvidia/pytorch:26.07-py3, redistributed as a value-added runtime image. The NVIDIA software in it is governed by the NVIDIA Software License Agreement and the Product-Specific Terms for NVIDIA AI Products, which the container prints at every start; by pulling or running the image you accept them. The image also contains PyAV (BSD) with its FFmpeg libraries (LGPL) and xgrammar (Apache 2.0). The Apache License above covers this repository's own work only.

Built on TensorFold by Ash Hart (ashhart), GLM-5.3-Flash by Z.ai, the EXL3 quantization by Brandon M. Music (ShapleyMcg), the DFlash2 drafter by IncoAI, b12x's RoCE transport by local-inference-lab, and code from glm53-tensorfold-spark by Jay Leaton (tool calling, L2 prefetch, expert loads). The full list, including the runtime stack and licenses, is in CREDITS.md.