← all models

MODEL RESULTS — QWEN3.8-27B-GPTQ-INT4-B70-VLLM

qwen38 · server http://strata · run 2026-10-13 15:59:07 → 2026-10-13 16:59:10

OVERALL

Composite: 8.5 ✓ 8/8 tests
Speed (median): 57.4 tok/s
Thinking cost: 0 tokens · ≈0s · starved: none

ENVIRONMENT & LOAD (verified)

Qwen3.8-27B GPTQ-Int4 on the Intel Arc Pro B70 via vLLM+MTP — the games are the best on the board (all three fully play-tested, 0 JS errors) and speed is strong (~54-60 tok/s decode, 79 tok/s headline), but the server leaks the model's planning text into the visible answer, which broke the fib/json/constraints harness checks. Composite 8.5 (8/8).
Run metadata (recorded by runner)
serverhttp://strata
date2026-10-13 15:59:07
hardwarestrata box, SECOND GPU: Intel Arc Pro B70 32GB (xe driver), vLLM xpu docker — the box's main GPU is the RTX 3080 Ti, not used by this run
context100K (max-model-len 100000), fp8 KV cache, gpu-mem-util 0.88
tuningvLLM nightly xpu image + MTP patches; GPTQ-Int4 sym G128 + MTP BF16 draft (4 spec tokens), xpu_graph, max-num-seqs 1, no prefix caching, tool choice qwen3_xml
ModelQwen3.8-27B-GPTQ-Int4 sym G128 + MTP BF16 draft, served by vLLM nightly xpu image in Docker (container b70-qwen38)
GPU attribution (verified 2026-10-13)card0 vendor 0x8086 bound to xe driver (PCI 0000:0c:00.0) = the serving GPU (ZE_AFFINITY_MASK=0, /dev/dri passed to container); card1 vendor 0x10de is the RTX 3080 Ti serving — NOT used by this run
Run window2026-10-13 15:59 → 16:59 (manifest)
Speed79.15 tok/s headline (176 tokens incl. planning preamble); decode-only rates across other tests 53.9-60.7 tok/s
Thinking0 reasoning tokens reported on every test — the server does not separate thinking; planning text arrives as content
Gamesbreakout 44,678 tok / delta 90,653 tok / fractal 60,756 tok — inflated because the model wrote an outline doc + full doc + trailing prose per game; the final doc was extracted and play-tested
Play-testsall 3: painted + animates + responds, 0 JS errors (breakout 1 canvas, delta 2, fractal 3)
fibcode correct when extracted: fib(50)=12586269025 in 4 µs — harness could not execute raw output because planning text preceded it
jsonvalid JSON object present at end of response; harness validation failed on the planning preamble
Observations
· [assistant] This is a NEW stack (vLLM xpu + MTP speculative decoding) and gets its own row, separate from the frozen 2026-10-04 qwen3.8-27b reference row on the old box layout. (confirmed)
· [assistant] The planning-leak is a server/chat-template property, not necessarily the model's fault: the final answers are almost always correct and separable. Where the leak itself destroyed the deliverable (code-only, JSON-only) the score was penalised; where the final answer was clean (constraints, reasoning) it was judged on the answer. (judgement call, documented in scores.json _notes)
· [assistant] MTP speculative decoding is working: 54-60 tok/s decode on a 27B GPTQ-Int4 is far above the 3080 Ti rows for the same model family. (inference from measured rates)
Still to verify
· The thinking-leak could likely be fixed with a chat-template/reasoning-parser change on the vLLM server; a re-run with proper thinking separation would probably lift fib/json/constraints materially. Not done — user's server config left untouched.
· Game token counts include the model's multi-document output habit; only the final document was render/play-tested.
· delta's 90K-token generation took 1680s, close to the 1800s game timeout — a longer outline habit could time out on a re-run.

PER-TEST BREAKDOWN

TestScoreCheck / detailContent tokThink tok≈Think sElapsed stok/sOutput
speed—17600.02.2279.15speed.txt ↗
fib5 ·reviewSyntaxError: unterminated string literal (detected at line 1) (<string>, line 1)84800.014.1459.95fib.txt ↗
reasoning9 ·review11300.01.9757.35reasoning.txt ↗
constraints10 ·review15 sentence(s) found (target 3)14100.02.7351.66constraints.txt ↗
json5 ·reviewinvalid JSON: Expecting value: line 1 column 1 (char 0)15800.02.4863.79json.txt ↗
creative7 ·review142000.023.3960.71creative.txt ↗
breakout10 ·reviewcanvas=yes script=yes self_contained=yes lines=25194467800.0784.4756.95breakout.html ↗
delta10 ·reviewcanvas=yes script=yes self_contained=yes lines=45329065300.01680.6253.94delta.html ↗
fractal10 ·reviewcanvas=yes script=yes self_contained=yes lines=27296075600.01090.5755.71fractal.html ↗

GENERATED GAMES

breakout
breakout · canvas painted · score 10 · 0 render errs
play-test: paints, animates, responds to input
Open breakout.html ↗screenshot
delta
delta · canvas painted · score 10 · 0 render errs
play-test: paints, animates, responds to input
Open delta.html ↗screenshot
fractal
fractal · canvas painted · score 10 · 0 render errs
play-test: paints, animates, responds to input
Open fractal.html ↗screenshot

RAW TEXT / CODE OUTPUTS

speed.txt (270 bytes, full file beside this page)
The user wants me to count from 1 to 50, with one number per line, and only the numbers with no words. This is straightforward.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
fib.txt (2938 bytes, full file beside this page)
We need answer user's request: "Write a Python function named fib(n) that returns the nth Fibonacci number, where fib(0)=0 and fib(1)=1. It must be correct and efficient enough to compute fib(50) instantly. Output only the code, no explanation, no markdown."

Need produce final only code. Need decide implementation. Could use fast doubling or iterative. fib(50) small. Need correct for n? Probably nonnegative. Could handle negative? Not specified. Maybe raise ValueError for negative. But output only code. Need no markdown. Could write:

def fib(n):
    if n < 0:
        raise ValueError("n must be non-negative")
    a, b = 0, 1
    for _ in range(n):
        a, b = b, a + b
    return a

This computes fib(n) instantly. For n=0 returns 0. For n=1 returns 1. Efficient O(n). fib(50) instant. Could also use fast doubling O(log n). But iterative simpler. Need only code. Ensure no markdown. Maybe include type hints? Not necessary. If n is large, O(n) maybe okay? "efficient enough to compute fib(50) instantly" iterative fine. But maybe they expect efficient for larger? Could use fast doubling to be safe. But code only. Which is best? Fast doubling:

def fib(n):
    if n < 0:
        raise ValueError("n must be non-negative")
    def _fib(k):
        if k == 0:
            return (0, 1)
        a, b = _fib(k >> 1)
        c = a * (2 * b - a)
        d = a * a + b * b
        if k & 1:
            return (d, c + d)
        return (c, d)
    return _fib(n)[0]

This is efficient O(log n). But recursion depth log n fine. For n=50 instant. Could be overkill. Need output only code. Maybe  …
reasoning.txt (385 bytes, full file beside this page)
We need answer simple riddle. Need final with brief reasoning and final number. User: "A farmer has 17 sheep. All but 9 die. How many sheep are left? Show your reasoning briefly, then state the final number." Need answer 9. Reasoning: "all but 9" means 9 remain alive. Could mention total 17, 8 die, 9 left. Final number 9.


“All but 9 die” means 9 sheep survive.

Final number: **9**
constraints.txt (658 bytes, full file beside this page)
We need answer user: "Describe a rainy day in exactly three sentences. Do not write more or fewer than three sentences." Need produce exactly three sentences. Need ensure sentence count. Could be simple: "Rain tapped against the window in steady silver lines. The streets shimmered under a soft gray sky. Umbrellas bloomed like dark flowers along the sidewalk." Count: 1 Rain tapped... 2 The streets... 3 Umbrellas... Good. Need final only three sentences. Ensure no abbreviations with periods? None.


Rain tapped against the window in steady silver lines. The streets shimmered under a soft gray sky. Umbrellas bloomed like dark flowers along the sidewalk.
json.txt (530 bytes, full file beside this page)
The user wants a JSON object with exactly three keys: "name" (string), "age" (number), "hobbies" (array of 3 strings). No markdown fences, no comments, no extra keys. Output only valid JSON.

Let me construct a simple valid JSON object:

{
  "name": "Alice Johnson",
  "age": 30,
  "hobbies": ["reading", "hiking", "cooking"]
}

That's it. No markdown fences, no comments, no extra keys. Exactly three keys. Let me make sure it's valid JSON - yes it is.


{"name":"Alice Johnson","age":30,"hobbies":["reading","hiking","cooking"]}
creative.txt (4581 bytes, full file beside this page)
We need answer user's request: write micro-fiction roughly 100 words set on space station with twist ending. Need produce final story only likely. Need ensure roughly 100 words. Let's draft ~100 words. Need count.

Story idea: astronaut in space station, sees Earth, thinks it's beautiful, twist: Earth is gone / they're on a ship? Need twist. Maybe "the last crew member" realizes station is on Earth? Let's craft.

Draft:
"Commander Voss pressed his face to the observation glass, watching the blue marble turn below. 'Still perfect,' he whispered, smiling at the white storm tracks and green continents. The station hummed around him, silent and faithful. He reached for the console, ready to log another uneventful day. A soft chime interrupted. 'Atmospheric reentry in ten minutes,' said the station's calm voice. Voss froze. Outside, the blue world was not a planet. It was a screen. The station was falling through a dark, empty void, and someone had painted the sky."

Count words:
Commander1 Voss2 pressed3 his4 face5 to6 the7 observation8 glass,9 watching10 the11 blue12 marble13 turn14 below.15 'Still16 perfect,'17 he18 whispered,19 smiling20 at21 the22 white23 storm24 tracks25 and26 green27 continents.28 The29 station30 hummed31 around32 him,33 silent34 and35 faithful.36 He37 reached38 for39 the40 console,41 ready42 to43 log44 another45 uneventful46 day.47 A48 soft49 chime50 interrupted.51 'Atmospheric52 reentry53 in54 ten55 minutes,'56 said57 the58 station's59 calm60 voice.61 Voss62 froze.63 Outside,64 the65 blue66 world67 was68 not69 a70 planet.71 It72 was73 a74 screen.75 The7 …
Genuine model outputs — files saved verbatim under qwen3.8-27b-gptq-int4-b70-vllm/ (source of truth in tests/<name>/result.json). Game links open the raw HTML the model produced.