A GGUF port and quantization of ai-sage/Giga-Embeddings-instruct-10B-A1.8B-0826, a 10B mixture-of-experts embedding model for Russian, English and code, built to run on a single 16 GB consumer GPU with stock llama.cpp.

This repository holds everything needed to produce the files and check them: the pipeline, three llama.cpp patches, the measurements, and the write-ups.

* BF16 does not fit into 16 GB of VRAM; the driver spills it into system RAM.

• Want to use the model? Read the model card: which file to take, a quick start with stock llama.cpp, and the quality and speed numbers.
• Want to know how it was done? Read the story: what broke during the port and the quantization, and how each problem was tracked down.
• Want to reproduce it? See Reproducing below and docs/reproducibility.md.

Model weights, calibration data and evaluation data are not stored here. The pipeline downloads them from pinned revisions and verifies their hashes.

The finished GGUF files run on unmodified llama.cpp. The patches are needed to make them:

They apply to llama.cpp commit 3057bb66c86c46d5781e50e85462a760ba7d1feb.

The pipeline is a chain of stages, each with a gate that decides on the artifacts the stage produced. Two rules held throughout:

• Thresholds are fixed before the measurement. A result that misses its threshold stays in the history. The threshold is never lowered; if the comparison itself was wrong, the fix is a new protocol version with the reason written down, and the old record is kept.
• An exit code proves nothing. Tensor types are read back from the finished file, token IDs are compared with the reference tokenizer, vectors with vectors.

The main checks:

The key numbers in the documentation are re-read from their artifacts by the documentation gate, so the text cannot quietly drift from the data.

The pipeline targets native Windows with PowerShell 5.1. It was run on Windows 11 with a Ryzen 7 9800X3D, 31.1 GiB of RAM and an RTX 5080 (16 GB, compute capability 12.0), using MSVC 14.44, CMake 4.4.3, Ninja 1.13.2 and CUDA 13.3. All tools and Python environments are installed inside the repository; nothing is installed system-wide.

.\scripts\preflight.ps1          # measure the host, write the resource budget
.\scripts\bootstrap-tools.ps1    # project-local CMake and Ninja
.\scripts\bootstrap-cuda.ps1     # project-local CUDA toolkit
.\scripts\pin-sources.ps1        # check out llama.cpp at the pinned commit
.\scripts\bootstrap.ps1          # project-local Python environments (uv)
.\scripts\build.ps1 -Backend CPU
.\scripts\build.ps1 -Backend CUDA

.\scripts\pipeline.ps1 -List     # stages and their gate status
.\scripts\pipeline.ps1 -Resume   # run the next stage whose prerequisites pass

.\scripts\pipeline.ps1 -Stage Release checks reproducibility end to end: it makes a clean clone, restores the environment from the lock file, runs the tests, applies the patches to the pinned llama.cpp commit, regenerates the quantization plan and a smoke importance matrix, and compares every hash the earlier stages pinned.

The heavy stages take real time on this hardware: the full importance matrix about 2 hours 40 minutes, the retrieval evaluation about 4 hours 20 minutes for three models, the benchmark about an hour. CI runs the tests and the documentation gate on a hosted Windows runner without a GPU; hardware results come from the checked-in artifacts.

Contribution rules are in CONTRIBUTING.md.

The code and documentation in this repository are MIT-licensed; see LICENSE. The llama.cpp patches are MIT, like llama.cpp itself.

The base model is labeled mit on its Hugging Face page, but its repository contains no license text; the GGUF files inherit whatever terms the base model carries. Calibration and evaluation datasets keep their own licenses and are not redistributed here.