Reading the repo
Opening the box…
Reading the repo
Opening the box…
How to install on Windows
Small, opensource LLM, consumer-level, focused benchmarks
Saved setup from 4 h ago
Will it work on Windows?
winget install Git.Git) or WSL2 (wsl --install).docs/setup.mdbwrap), socat, git, curl, rsync and Python 3.11 or newer.hip-halo on the Strix Halo.OPENCODE_BINARY at your own copy if it lives elsewhere.--mtp).| Variable | Used by | Default |
|---|---|---|
| STRATA | run_quality.sh, run_hard.sh | ~/.local/share/strata/components/control/0.1.51/bin/strata |
| OPENCODE_BINARY | all coding canaries | the mise opencode 2.1.4 native binary |
| REPS | run_hard.sh | 3 |
| BASALT_DIR | run_basalt.sh | ~/Projects/local-llm/basalt/src (a built Basalt checkout) |
| BASALT_PY | run_basalt.sh | ~/Projects/local-llm/basalt/venv/bin/python |
| REPS | run_basalt.sh | 3 |
| STRIX_HOST | run_strix.sh, strix_relay.sh | 192.168.0.90 |
| STRIX_PY, STRIX_STRATA | run_strix.sh | the Strata 0.1.45.3 terminal install on the Strix |
All engine state, weights and serving configs live in ~/Projects/local-llm
(see the repository README). The drivers take LOCAL_LLM (default
~/Projects/local-llm) and read configs from $LOCAL_LLM/strata.
Strata server configs are in configs/5090 and configs/strix. The drivers read the 5090 ones from ~/Projects/local-llm/strata/qwen-<version>.json, so copy them there and edit the model and pack paths. That tree holds the models and packs (about 277 GB), so it lives outside this repository.
The key flags:
--vram-elastic --vram-reserve-mib 4096, which leaves room for the desktop;--kv int8 --max-context 131072;--spec 4 with the MTP draft.--resident-experts (or --resident-budget-gib 38 for Q4_K_XL);--kv int8 --max-context 131072;STRATA_RESIDENT_HEADROOM_GIB=8.Basalt is a Strata fork for Blackwell GPUs (sm_120, CUDA 13). It serves single-file .basalt packs instead of Strata's GGUF plus pack. Everything lives in ~/Projects/local-llm/basalt/, outside this repository:
mkdir -p ~/Projects/local-llm/basalt && cd ~/Projects/local-llm/basalt
git clone https://github.com/jesdga95/basalt src
git -C src checkout v0.2.1 # pinned release; build as its README describes:
# cd src && cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
# (needs CMake ≥3.24, a C++20 compiler, CUDA 13, and CMAKE_CUDA_ARCHITECTURES=120 for the 5090)
python3 -m venv venv
venv/bin/pip install -r src/requirements.txt
hf download jesdga/Qwen3.8-Flash-Next-Basalt \
Qwen3.8-FN-GSQ-RCO-Q2_0.basalt Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt \
--local-dir models # about 157 GB for both
Engine version / pack format. Pinned at v0.2.1 (was commit 5554887). v0.2.x reads .basalt
format v2; the packs on Hugging Face and any built before the bump are v1, and a v2 engine
refuses them (.basalt version 1, this build reads version 2). Upgrade each pack in place — byte-for-byte,
no re-quantizing, ~80 s and ~80 GB of free disk per pack — then swap it in:
cd ~/Projects/local-llm/basalt
venv/bin/python src/tools/migrations/basalt_v1_to_v2.py \
models/Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt models/IQ3_S.v2.basalt
mv models/Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt{,.v1.bak} # keep until verified
mv models/IQ3_S.v2.basalt models/Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt
The Q2_0 pack is still v1 — migrate it the same way before serving it on a v0.2.x engine.
The server configs are configs/5090/basalt-q2_0.json and basalt-iq3_s.json. Model paths in them are relative to the config file. The key settings:
kv: int8 and max-context: 131072, the same as the Strata runs;vram-reserve-mib: 4608. Basalt counts its scratch space against this reserve. At 4096, free VRAM came out at about 4.0 GB with the desktop running, so it was raised to keep a margin;basalt serve needs absolute paths for --python; the driver passes them.
The jesdga/Qwen3.8-Flash-Next-Basalt packs already embed the vision projector (vision/mmproj +
vision/text_vocab sections survive the v1→v2 migration), so no external mmproj is needed. Vision is a
server option, not an engine flag, and it is off unless you ask for it. Two settings turn it on:
"vision": "dynamic" — the image encoder runs inside the engine on the main GPU, borrowing expert-cache
slots only while it reads an image (no resident VRAM cost; load is per-image). "cpu" or a GPU number
(sidecar via bin/basalt-vision) are the other modes; booleans are refused."lazy": false — lazy: true turns vision off (it brings up a text-only API and loads on first
request). idle-unload is fine: it re-spawns the engine with the same --vision setting.A minimal vision-enabled serving config (~/.config/basalt/qwen-iq3_s.json on this box, port 8080, also
carrying the refusal-direction control vector):
{ "server": { "model": "…/Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt", "gpu": "0", "port": 8080,
"lazy": false, "idle-unload": 600, "vision": "dynamic" },
"engine": { "max-context": 262144, "kv": "int8", "vram-reserve-mib": 4608 } }
Verify: the startup log should print vision mode=dynamic gpu=0 … load=per-image, and an OpenAI
chat/completions call with an image_url returns a real answer (a solid-red test image → "Red").
This survives idle-unload: after the engine unloads and the next request reloads it, the log shows
weights loaded … images=on and images still work.
For a server you keep around (rather than a session-bound shell), run it under systemd so it is
independent of any terminal/agent session, restarts on failure, and comes back after a reboot. Unit at
~/.config/systemd/user/basalt.service:
[Service]
Type=simple
WorkingDirectory=/mnt/data/Projects/llm-opensource-benchmark/.basalt/src
Environment=PATH=/opt/cuda/bin:/usr/local/bin:/usr/bin:/bin
ExecStart=/mnt/data/Projects/llm-opensource-benchmark/.basalt/src/bin/basalt serve \
--config /home/akitaonrails/.config/basalt/qwen-iq3_s.json \
--python /mnt/data/Projects/llm-opensource-benchmark/.basalt/venv/bin/python
Restart=on-failure
RestartSec=5
[Install]
WantedBy=default.target
systemctl --user enable --now basalt.service
loginctl enable-linger "$USER" # keep it running without an active login / across reboots
CUDA runtime libs resolve via ldconfig (/opt/cuda/lib64), so no LD_LIBRARY_PATH is needed.
Restart=on-failure also covers the brief case where a previous instance has not yet released port
8080 — the unit simply retries until the port is free.
Strata serves each version from a pack built from its GGUF:
| Version | Tools |
|---|---|
| Q2_0 | strata_pack.py build, then pack_index.py and strata_tokenizer.py |
| IQ3_XXS, IQ3_S | iq_pack.py |
| UD-Q4_K_XL (Unsloth) | iq_pack.py --compat-bf16 |
bench/strix_build_pack.sh has the exact commands used on the Strix. It deliberately builds into its own folder. Strata's setup.py --data-dir moves model files from other installs on the same drive, so it was avoided on a shared machine.
bench/run_quality.sh q2_0 iq3_xxs iq3_s # quality canary, one run per version
REPS=3 bench/run_hard.sh q2_0 iq3_s # hard canary, three runs per version
Both drivers:
For Basalt:
REPS=3 bench/run_basalt.sh q2_0 iq3_s # speed canary, then three hard runs per version
It works the same way as the Strata drivers, and writes to results/basalt/<version>-speed and results/basalt/<version>-hard-r<n>.
To run a single canary against a server that is already up:
python3 bench/hard_canary.py --base-url http://127.0.0.1:18082/v1 \
--model qwen3.8-flash-next-q2_0 --label my-run --output results/hard/my-run
To see the long first prompt without running anything, add --print-prompt.
bench/run_strix.sh q2_0 iq3_xxs iq3_s
For each version it:
That Strix's sshd sets AllowTcpForwarding no, so ssh -L won't work. Instead:
socat listener runs bench/strix_relay.sh for each connection;The Strix normally hosts the non-coding benchmark's IQ3_S server on port 18081. It was stopped for these runs and restored afterwards; see results/strix/restore-18081.md.
--file).systemd-oomd kills the whole tmux scope under memory pressure. Run long batches detached with setsid nohup.cd dir && setsid server & over ssh keeps ssh's pipes open, so ssh never returns. Use cd dir; setsid server > log 2>&1 < /dev/null &.pkill -f pattern can match the shell that runs it. Kill by PID or process group instead.lazy: true disables it outright; set "vision": "dynamic" and "lazy": false (see Image input). The projector is already in the pack, so no --vision-mmproj is needed..basalt version 1, this build reads version 2: the engine was upgraded across the format bump (v0.2.x) but the pack is still v1. Migrate it with tools/migrations/basalt_v1_to_v2.py (see Engine version / pack format); it does not re-quantize.You need Linux with bwrap, socat and Python 3.11+, plus Strata or Basalt, opencode 2.1.4 and the model files. Details are in docs/setup.md.
# Copy and adapt a server config (model and pack paths)
mkdir -p .strata && cp configs/5090/qwen-q2_0.json .strata/
# Run the hard coding test three times on that version
REPS=3 bench/run_hard.sh q2_0
# Or the shorter quality canary
bench/run_quality.sh q2_0
# Basalt on the 5090: speed test plus three hard runs per version
REPS=3 bench/run_basalt.sh q2_0 iq3_s
Each run writes a report.json with the score, the result of every hidden test, and timings.
Then run powershell -ExecutionPolicy Bypass -File .\setup.ps1. It stops at the first error and is safe to run again. Servers and tests aren't started; it lists them at the end. Read it first.
Needs Git: winget install Git.Git
git clone https://github.com/akitaonrails/llm-opensource-benchmark.git cd llm-opensource-benchmark
Run these in Git Bash (installed with Git) or WSL — not PowerShell. Do the README instructions above first.
mkdir -p .strata && cp configs/5090/qwen-q2_0.json .strata/ REPS=3 bench/run_hard.sh q2_0 bench/run_quality.sh q2_0 REPS=3 bench/run_basalt.sh q2_0 iq3_s
The steps run shell scripts (.sh). Use Git Bash (winget install Git.Git) or WSL2 (wsl --install).
No reports for Windows yet.
GitManual AI Agent reads the 2 key files and writes a step-by-step guide for Windows. Once written, it's saved for everyone.