Reading the repo
Opening the box…
Reading the repo
Opening the box…
How to install on Linux
Small, opensource LLM, consumer-level, focused benchmarks
Saved setup from 4 h ago
Will it work on Linux?
docs/setup.mdbwrap), socat, git, curl, rsync and Python 3.11 or newer.hip-halo on the Strix Halo.OPENCODE_BINARY at your own copy if it lives elsewhere.--mtp).| Variable | Used by | Default |
|---|---|---|
| STRATA | run_quality.sh, run_hard.sh | ~/.local/share/strata/components/control/0.1.51/bin/strata |
| OPENCODE_BINARY | all coding canaries | the mise opencode 2.1.4 native binary |
| REPS | run_hard.sh | 3 |
| BASALT_DIR | run_basalt.sh | ~/Projects/local-llm/basalt/src (a built Basalt checkout) |
| BASALT_PY | run_basalt.sh | ~/Projects/local-llm/basalt/venv/bin/python |
| REPS | run_basalt.sh | 3 |
| STRIX_HOST | run_strix.sh, strix_relay.sh | 192.168.0.90 |
| STRIX_PY, STRIX_STRATA | run_strix.sh | the Strata 0.1.45.3 terminal install on the Strix |
All engine state, weights and serving configs live in ~/Projects/local-llm
(see the repository README). The drivers take LOCAL_LLM (default
~/Projects/local-llm) and read configs from $LOCAL_LLM/strata.
Strata server configs are in configs/5090 and configs/strix. The drivers read the 5090 ones from ~/Projects/local-llm/strata/qwen-<version>.json, so copy them there and edit the model and pack paths. That tree holds the models and packs (about 277 GB), so it lives outside this repository.
The key flags:
--vram-elastic --vram-reserve-mib 4096, which leaves room for the desktop;--kv int8 --max-context 131072;--spec 4 with the MTP draft.--resident-experts (or --resident-budget-gib 38 for Q4_K_XL);--kv int8 --max-context 131072;STRATA_RESIDENT_HEADROOM_GIB=8.Basalt is a Strata fork for Blackwell GPUs (sm_120, CUDA 13). It serves single-file .basalt packs instead of Strata's GGUF plus pack. Everything lives in ~/Projects/local-llm/basalt/, outside this repository:
mkdir -p ~/Projects/local-llm/basalt && cd ~/Projects/local-llm/basalt
git clone https://github.com/jesdga95/basalt src
git -C src checkout v0.2.1 # pinned release; build as its README describes:
# cd src && cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
# (needs CMake ≥3.24, a C++20 compiler, CUDA 13, and CMAKE_CUDA_ARCHITECTURES=120 for the 5090)
python3 -m venv venv
venv/bin/pip install -r src/requirements.txt
hf download jesdga/Qwen3.8-Flash-Next-Basalt \
Qwen3.8-FN-GSQ-RCO-Q2_0.basalt Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt \
--local-dir models # about 157 GB for both
Engine version / pack format. Pinned at v0.2.1 (was commit 5554887). v0.2.x reads .basalt
format v2; the packs on Hugging Face and any built before the bump are v1, and a v2 engine
refuses them (.basalt version 1, this build reads version 2). Upgrade each pack in place — byte-for-byte,
no re-quantizing, ~80 s and ~80 GB of free disk per pack — then swap it in:
cd ~/Projects/local-llm/basalt
venv/bin/python src/tools/migrations/basalt_v1_to_v2.py \
models/Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt models/IQ3_S.v2.basalt
mv models/Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt{,.v1.bak} # keep until verified
mv models/IQ3_S.v2.basalt models/Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt
The Q2_0 pack is still v1 — migrate it the same way before serving it on a v0.2.x engine.
The server configs are configs/5090/basalt-q2_0.json and basalt-iq3_s.json. Model paths in them are relative to the config file. The key settings:
kv: int8 and max-context: 131072, the same as the Strata runs;vram-reserve-mib: 4608. Basalt counts its scratch space against this reserve. At 4096, free VRAM came out at about 4.0 GB with the desktop running, so it was raised to keep a margin;basalt serve needs absolute paths for --python; the driver passes them.
The jesdga/Qwen3.8-Flash-Next-Basalt packs already embed the vision projector (vision/mmproj +
vision/text_vocab sections survive the v1→v2 migration), so no external mmproj is needed. Vision is a
server option, not an engine flag, and it is off unless you ask for it. Two settings turn it on:
"vision": "dynamic" — the image encoder runs inside the engine on the main GPU, borrowing expert-cache
slots only while it reads an image (no resident VRAM cost; load is per-image). "cpu" or a GPU number
(sidecar via bin/basalt-vision) are the other modes; booleans are refused."lazy": false — lazy: true turns vision off (it brings up a text-only API and loads on first
request). idle-unload is fine: it re-spawns the engine with the same --vision setting.A minimal vision-enabled serving config (~/.config/basalt/qwen-iq3_s.json on this box, port 8080, also
carrying the refusal-direction control vector):
{ "server": { "model": "…/Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt", "gpu": "0", "port": 8080,
"lazy": false, "idle-unload": 600, "vision": "dynamic" },
"engine": { "max-context": 262144, "kv": "int8", "vram-reserve-mib": 4608 } }
Verify: the startup log should print vision mode=dynamic gpu=0 … load=per-image, and an OpenAI
chat/completions call with an image_url returns a real answer (a solid-red test image → "Red").
This survives idle-unload: after the engine unloads and the next request reloads it, the log shows
weights loaded … images=on and images still work.
For a server you keep around (rather than a session-bound shell), run it under systemd so it is
independent of any terminal/agent session, restarts on failure, and comes back after a reboot. Unit at
~/.config/systemd/user/basalt.service:
[Service]
Type=simple
WorkingDirectory=/mnt/data/Projects/llm-opensource-benchmark/.basalt/src
Environment=PATH=/opt/cuda/bin:/usr/local/bin:/usr/bin:/bin
ExecStart=/mnt/data/Projects/llm-opensource-benchmark/.basalt/src/bin/basalt serve \
--config /home/akitaonrails/.config/basalt/qwen-iq3_s.json \
--python /mnt/data/Projects/llm-opensource-benchmark/.basalt/venv/bin/python
Restart=on-failure
RestartSec=5
[Install]
WantedBy=default.target
systemctl --user enable --now basalt.service
loginctl enable-linger "$USER" # keep it running without an active login / across reboots
CUDA runtime libs resolve via ldconfig (/opt/cuda/lib64), so no LD_LIBRARY_PATH is needed.
Restart=on-failure also covers the brief case where a previous instance has not yet released port
8080 — the unit simply retries until the port is free.
Strata serves each version from a pack built from its GGUF:
| Version | Tools |
|---|---|
| Q2_0 | strata_pack.py build, then pack_index.py and strata_tokenizer.py |
| IQ3_XXS, IQ3_S | iq_pack.py |
| UD-Q4_K_XL (Unsloth) | iq_pack.py --compat-bf16 |
bench/strix_build_pack.sh has the exact commands used on the Strix. It deliberately builds into its own folder. Strata's setup.py --data-dir moves model files from other installs on the same drive, so it was avoided on a shared machine.
bench/run_quality.sh q2_0 iq3_xxs iq3_s # quality canary, one run per version
REPS=3 bench/run_hard.sh q2_0 iq3_s # hard canary, three runs per version
Both drivers:
For Basalt:
REPS=3 bench/run_basalt.sh q2_0 iq3_s # speed canary, then three hard runs per version
It works the same way as the Strata drivers, and writes to results/basalt/<version>-speed and results/basalt/<version>-hard-r<n>.
To run a single canary against a server that is already up:
python3 bench/hard_canary.py --base-url http://127.0.0.1:18082/v1 \
--model qwen3.8-flash-next-q2_0 --label my-run --output results/hard/my-run
To see the long first prompt without running anything, add --print-prompt.
bench/run_strix.sh q2_0 iq3_xxs iq3_s
For each version it:
That Strix's sshd sets AllowTcpForwarding no, so ssh -L won't work. Instead:
socat listener runs bench/strix_relay.sh for each connection;The Strix normally hosts the non-coding benchmark's IQ3_S server on port 18081. It was stopped for these runs and restored afterwards; see results/strix/restore-18081.md.
--file).systemd-oomd kills the whole tmux scope under memory pressure. Run long batches detached with setsid nohup.cd dir && setsid server & over ssh keeps ssh's pipes open, so ssh never returns. Use cd dir; setsid server > log 2>&1 < /dev/null &.pkill -f pattern can match the shell that runs it. Kill by PID or process group instead.lazy: true disables it outright; set "vision": "dynamic" and "lazy": false (see Image input). The projector is already in the pack, so no --vision-mmproj is needed..basalt version 1, this build reads version 2: the engine was upgraded across the format bump (v0.2.x) but the pack is still v1. Migrate it with tools/migrations/basalt_v1_to_v2.py (see Engine version / pack format); it does not re-quantize.You need Linux with bwrap, socat and Python 3.11+, plus Strata or Basalt, opencode 2.1.4 and the model files. Details are in docs/setup.md.
# Copy and adapt a server config (model and pack paths)
mkdir -p .strata && cp configs/5090/qwen-q2_0.json .strata/
# Run the hard coding test three times on that version
REPS=3 bench/run_hard.sh q2_0
# Or the shorter quality canary
bench/run_quality.sh q2_0
# Basalt on the 5090: speed test plus three hard runs per version
REPS=3 bench/run_basalt.sh q2_0 iq3_s
Each run writes a report.json with the score, the result of every hidden test, and timings.
Then run bash setup.sh. It stops at the first error and is safe to run again. Servers and tests aren't started; it lists them at the end. Read it first.
git clone https://github.com/akitaonrails/llm-opensource-benchmark.git cd llm-opensource-benchmark
In the maintainers' order. Do the README instructions above first — they cover steps outside the terminal.
mkdir -p .strata && cp configs/5090/qwen-q2_0.json .strata/ REPS=3 bench/run_hard.sh q2_0 bench/run_quality.sh q2_0 REPS=3 bench/run_basalt.sh q2_0 iq3_s
No reports for Linux yet.
GitManual AI Agent reads the 2 key files and writes a step-by-step guide for Linux. Once written, it's saved for everyone.