frognano-4b-2609 · server http://strata · run 2026-10-11 00:13:09 → 2026-10-11 00:15:37
OVERALL
Composite: 6.9 ✓ 8/8 tests
Speed (median): 164.1 tok/s
Thinking cost: 12,764 tokens · ≈80s · starved: none
ENVIRONMENT & LOAD (verified)
FrogNano-4B Q8_0 re-run at 128K context — the highest context window that comfortably fits in a 12 GB 3080 Ti. Model + Q8_0 KV cache = 10,699 / 12,288 MiB (~87%), single slot, fully on GPU, ~164 tok/s. This is the context-ceiling test: same model/quant/scores as the 16K entry, only the context window changed. Every number below was read from the live machine (nvidia-smi / lms ps) at load time.
| Purpose | Answer: what is the highest context window FrogNano-4B Q8_0 comfortably fits in 12 GB VRAM? Bracketed with lms load --estimate-only, then verified with real loads + nvidia-smi. |
| Context ceiling (measured) | 64K=8,267 MiB (67%) · 96K=9,483 MiB (77%) · 128K=10,699 MiB (87%) · ~140K≈11.5 GiB (94%, tight) · 160K=12,400 MiB (estimate, WILL NOT LOAD) · 256K=16,830 MiB (way over). Ceiling is ~140-150K; 128K is the highest round value with headroom. |
| Model file actually loaded | frognano-4b-2609 — bartowski/FrogNano-4B-2609-GGUF Q8_0 (imatrix), 5.29 GB on disk, 4.93 GiB runtime (lms ps). Base: microsoft/FrogNano-4B-2609 (4.66B, Qwen3.5-4B, RL-trained coding agent, MIT). |
| Context | 131,072 tokens (128K) |
| Parallel slots | 1 (PAR 2 would double the KV cache and roughly halve the ceiling) |
| GPU layer split | All layers on GPU (4B — no CPU offload) |
| GPU usage (measured, idle) | 10,699 / 12,288 MiB — ~87% of the card; ~1.6 GiB headroom; 0% util (idle sample) |
| Speed (median of 9 tests) | 164.1 tok/s — unchanged from the 16K run (~165.5); context size has no effect on tok/s for a 4B model that already fits |
| Reasoning model | YES — emits reasoning_content (199-5971 thinking tokens per test). Creative this run: 5,971 thinking + a complete ~100-word story (no starve, cap removed in v3). |
| Concurrent load | Qwen3.8-27B Q4_K_M reference model still running on the Arc B70 at — different GPU, same box, shares CPU/RAM/PCIe. |
| Scoring | Objective: fib/constraints/json all PASS (10). Subjective: reasoning 9, creative 8, breakout 2 (BROKEN this run), delta 6, fractal 5. Composite 6.9 — dragged down by a genuine breakout regression vs the 16K run. |
| Known defect (this run) | breakout: TWO fatal JS bugs — (1) draw() is called on page load (line 505 'Initial Draw') BEFORE initBricks(), so bricks[0][0] throws 'Cannot read properties of undefined (reading 0)'; (2) paddleX is declared const (line 187) but reassigned in resetBall()/movePaddle() -> 'Assignment to constant variable'. Result: canvas stays 0% painted even after clicking Start. Unplayable. A real variance/regression vs the 16K run's polished breakout (9). |
Observations
· [user] Can you test the frognano LLM and find the highest context window that comfortably fits in VRAM? (DONE — bracketed + verified: 128K (10,699/12,288 MiB, 87%) is the highest comfortable; ceiling ~140-150K, 160K won't load.)
· [user] With that context, rerun the tests and add as a new entry in the results page. (DONE — ran all 9 tests at 128K (slug frognano-4b-q80-128k), scored, built the page, verified over the network.)
· [assistant] Same model, quant, and scores as the 16K entry — the ONLY variable is context window. tok/s is identical (~164). The one difference in results is a genuine per-run coding variance: breakout was a polished 9 at 16K but is broken (2) at 128K, and the two game defects flipped (fractal now renders Mandelbrot cleanly, breakout regressed). That variance is exactly what the benchmark surfaces. (CONFIRMED by results.)
Still to verify
· The two FrogNano rows (16K and 128K) differ only in context window; the score delta is run-to-run coding variance, not a context effect.
RAW TEXT / CODE OUTPUTS
speed.txt (140 bytes, full file beside this page)1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
fib.txt (126 bytes, full file beside this page)def fib(n):
if n == 0:
return 0
a, b = 0, 1
for _ in range(2, n + 1):
a, b = b, a + b
return b reasoning.txt (142 bytes, full file beside this page)Reasoning: The phrase "all but 9 die" means that every sheep except for 9 died. Therefore, only the 9 survivors remain alive.
Final number: 9
constraints.txt (249 bytes, full file beside this page)Dark clouds filled the sky as a gentle drizzle began to fall from above. The rain intensified into a steady rhythm that drummed against every surface in sight. I stayed indoors with a warm cup of tea while watching the world blur through the window.
json.txt (87 bytes, full file beside this page){
"name": "Alice Smith",
"age": 30,
"hobbies": ["reading", "hiking", "cooking"]
} creative.txt (578 bytes, full file beside this page)Commander Jax checked his boots for debris before every shift change in deep orbit. He expected fresh blood, but found only dust motes dancing in the harsh light. "Shift's over," said Sarah from the comms chair. Jax froze. She wasn't there. No one else aboard. Just him and the hum of life support systems. He walked to the wall console, checking his ID badge under bright lights. It read 'Admin'. A glitch? Then a message flashed on every screen: Maintenance Complete. Crew Replaced. Jax looked down at his hands. They were cold steel. Not skin. The silence was truly absolute.
