How the harness works & per-model test matrix →
results.html has the full per-test breakdown and links to the raw game HTML the model wrote — open them in a tab to play. Rebuild any time with python3 build_automated_tests.py.| Model | Composite | Tests | Speed | Context window | Hardware | Thinking | Results |
|---|---|---|---|---|---|---|---|
| qwen3.8-flash-next qwen3.8-flash-next-iq3_xxs #1 | 9.8 | ✓ 8/8 | 56.1 | 131,072 tokens, text+image input | strata · NVIDIA GeForce RTX 3080 Ti 12 GB (serving GPU; the box also has an Intel Arc Battlemage installed on the `xe` driver, but Strata is not using it) | 0 tok ≈0s | qwen3.8-flash-next/results.html → |
| gemma-4-12b-it-qat C:\Users\admin\models_bench\gemma-4-12b-it-qat-q4_0.gguf #2 | 9.1 | ✓ 8/8 | 21.1 | 24K (24576) — resident (16K 23.6 vs 24K 22.3 tok/s, no spill cliff) | erwing · GTX 1080 8 GB · 16 GB RAM | 0 tok ≈0s | gemma-4-12b-it-qat/results.html → |
| gemma-4-26b-a4b-iq2_xxs C:\Users\admin\models_bench\gemma-4-26B-A4B-it-UD-IQ2_XXS.gguf #3 | 9.1 | ✓ 8/8 | 15.4 | 16K (16384) — 9.24 GiB model exceeds VRAM; MoE experts offloaded to system RAM via --cpu-moe (15.9 tok/s vs 4.3 auto-fit) | erwing · GTX 1080 8 GB · 16 GB RAM | 0 tok ≈0s | gemma-4-26b-a4b-iq2_xxs/results.html → |
| qwen3.8-27b qwen3.8-flash-next-iq3_xxs #4 | 9.1 | ✓ 8/8 | 34.1 | not recorded (migrated from old layout) | box strata (retired Oct 2026): Intel Arc B70 (SYCL llama.cpp ) per the swift-1.5-iq2-s env record; Arc VRAM not recorded | 0 tok ≈0s | qwen3.8-27b/results.html → |
| swift-1.5-iq2-s swift-1.5-qwen3.8-27b-gsq-rco@iq2_s #5 | 8.8 | ✓ 8/8 | 27.2 | 29,184 tokens (PAR 1, Q8_0 KV, kv_unified) | box strata (retired Oct 2026): RTX 3080 Ti 12 GB for this run (llama-server via LM Studio ); same box's Intel Arc B70 ran the benchmark reference model on | 69,329 tok ≈2577s | swift-1.5-iq2-s/results.html → |
| gemma-4-12b google/gemma-4-12b-qat #6 | 8.6 | ✓ 8/8 | 73.4 | not recorded in manifest; estimate brackets say ≤32K (32K ≈ 11.8 GiB fits 12 GB; 64K ≈ 16.8 GiB would not) | box strata (retired Oct 2026): RTX 3080 Ti 12 GB — LM Studio | 8,170 tok ≈110s | gemma-4-12b/results.html → |
| qwen3.8-27b-gptq-int4-b70-vllm qwen38 #7 | 8.5 | ✓ 8/8 | 57.4 | 100K (max-model-len 100000), fp8 KV cache, gpu-mem-util 0.88 | strata box, SECOND GPU: Intel Arc Pro B70 32GB (xe driver), vLLM xpu docker — the box's main GPU is the RTX 3080 Ti, not used by this run | 0 tok ≈0s | qwen3.8-27b-gptq-int4-b70-vllm/results.html → |
| qwen3.5-9b qwen3.5-9b #8 | 8.4 | ✓ 8/8 | 27.3 | 131,072 (practical VRAM ceiling; native 262,144 does not fit 8 GB) | erwing (Windows): GTX 1080 8 GB (8,192 MiB; 427–477 MiB desktop baseline), 16 GB system RAM | 0 tok ≈0s | qwen3.5-9b/results.html → |
| frognano-4b-q80 frognano-4b-2609 #9 | 8.3 | ✓ 8/8 | 165.5 | 16,384 tokens | box strata (retired Oct 2026): RTX 3080 Ti 12 GB — LM Studio | 12,249 tok ≈75s | frognano-4b-q80/results.html → |
| mellum2.1-12b-a2.5b-thinking C:\Users\admin\models_bench\Mellum2.1-12B-A2.5B-Thinking-MXFP4_MOE.gguf #10 | 8.3 | ✓ 8/8 | 68.0 | 24K (24576) — fully resident (16K and 24K both ~71 tok/s) | erwing · GTX 1080 8 GB · 16 GB RAM | 0 tok ≈0s | mellum2.1-12b-a2.5b-thinking/results.html → |
| swift-1.5-iq2-s-rx6700xt swift-1.5-qwen3.8-27b-gsq-rco@iq2_s #11 | 7.6 | ✓ 8/8 | 18.0 | 20,480 tokens, Q4_0 KV, kv_unified, kv-offload, PAR 1 (fit-limited on 12 GB card: Q8_0 KV at 20K died with ErrorDeviceLost at ~145s, Q4_0 survives 13K-token stress; stable-but-narrow 12,288/Q8_0 config starved the uncapped thinking games; reference 3080 Ti row ran 29,184/Q8_0) | ryzen3090 (rebuilt box): AMD Ryzen 3 2200G (4C/4T), 16 GB DDR4-2133, AMD Radeon RX 6700 XT 12 GB — llama.cpp Vulkan (llama-server 2.64.0, scheduled task; LM Studio proxy stopped for the run) | 0 tok ≈0s | swift-1.5-iq2-s-rx6700xt/results.html → |
| gemma-4-e4b gemma-4-e4b #12 | 7.1 | ✓ 8/8 | 36.9 | 131,072 (model native max) | erwing (Windows): GTX 1080 8 GB (8,192 MiB; 427–477 MiB desktop baseline), 16 GB system RAM | 0 tok ≈0s | gemma-4-e4b/results.html → |
| frognano-4b-q80-128k frognano-4b-2609 #13 | 6.9 | ✓ 8/8 | 164.1 | 131,072 tokens (measured ceiling ~140–150K; 160K will not load) | box strata (retired Oct 2026): RTX 3080 Ti 12 GB — LM Studio | 12,764 tok ≈80s | frognano-4b-q80-128k/results.html → |
| ornith-1.5-9b-q5_k_m C:\Users\admin\models_bench\Ornith-1.5-9B-Q5_K_M.gguf #14 | 6.3 | ✓ 8/8 | 23.6 | 24K (24576) — fully resident (16K and 24K both 24.9 tok/s, no spill) | erwing · GTX 1080 8 GB · 16 GB RAM | 0 tok ≈0s | ornith-1.5-9b-q5_k_m/results.html → |
| lfm2.5-8b-a1b-q6_k C:\Users\admin\models_bench\LFM2.5-8B-A1B-Q6_K.gguf #15 | 6.1 | ✓ 8/8 | 92.4 | 24K (24576) — largest fully-resident ctx; 32K spilled experts to RAM (14.9 vs 94 tok/s) | erwing · GTX 1080 8 GB · 16 GB RAM | 0 tok ≈0s | lfm2.5-8b-a1b-q6_k/results.html → |
| qwen3.8-flash-next-iq3-s-0.2.4 qwen3.8-flash-next-iq3_s #16 | 10.0 | ◔ 6/8 | 51.9 | 131,072 tokens, text+image input | strata · RTX 3080 Ti 12 GB (serving GPU) | 0 tok ≈0s | qwen3.8-flash-next-iq3-s-0.2.4/results.html → |
env.json — the per-model record of what it ran on, at what context, and the conclusions. 'not recorded' means the run predates this convention.