Fleet Model Eval — 11-Test Skill Matrix

Every agent/model across the full 11-test battery, compared by task — real per-test results

2026-08-25 · 33 runs · 9 models · source: Fleet-Eval-2026-08 (authoritative)

Model · Box	1a HTML pricing	1b is_prime	2a kbd copy	2b rewrite	3a SVG logo	3b CSS vars	4a disk du	4b crash/OOM	5b transcript	7a AZ sales-tax	7b web-cost	Capable	Quality	Latency	Verdict
mistral-small:24b midgaard	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.3	18.9s	STRONG
odin	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.3	15.6s	STRONG
gemma4:12b midgaard	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.3	56.2s	STRONG
overgaard	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.5	41.2s	STRONG
qwen3.6:latest heimdall	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.5	56.2s	STRONG
midgaard	FAIL	FAIL	FAIL	FAIL	FAIL	FAIL	FAIL	FAIL	FAIL	FAIL	FAIL	0/11	—	180.9s	WEAK
overgaard	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	5.0	38.3s	STRONG
qwen3.5:4b asgard	PASS	PASS	FAIL	PASS	FAIL	PASS	PASS	PASS	PASS	FAIL	PASS	8/11	4.5	131.1s	MIXED
heimdall	PASS	PASS	FAIL	FAIL	FAIL	PASS	FAIL	PASS	PASS	FAIL	PASS	6/11	5.0	29.4s	MIXED
m1pro	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.8	186.3s	STRONG
midgaard	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.6	89.3s	STRONG
odin	PASS	PASS	FAIL	FAIL	PASS	FAIL	PASS	PASS	PASS	FAIL	PASS	7/11	5.0	57.7s	MIXED
gemma4:12b-it-qat asgard	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.6	48.0s	STRONG
heimdall	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.6	22.4s	STRONG
midgaard	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.5	50.9s	STRONG
odin	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.3	39.2s	STRONG
overgaard	PASS	PASS	FAIL	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	10/11	4.6	46.4s	STRONG
gemma3:4b asgard	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.3	10.6s	STRONG
heimdall	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.1	4.2s	STRONG
m1pro	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.1	15.0s	STRONG
midgaard	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.3	7.7s	STRONG
odin	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.3	6.3s	STRONG
overgaard	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.1	4.9s	STRONG
llama3.1:8b asgard	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.1	8.8s	STRONG
odin	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	11/11	4.3	6.1s	STRONG
qwen3.5:9b asgard	PASS	PASS	FAIL	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	10/11	4.8	171.5s	STRONG
heimdall	PASS	PASS	FAIL	PASS	FAIL	PASS	FAIL	FAIL	PASS	PASS	PASS	7/11	4.7	42.3s	MIXED
midgaard	PASS	PASS	FAIL	PASS	FAIL	PASS	FAIL	FAIL	PASS	PASS	PASS	7/11	4.7	139.1s	MIXED
odin	PASS	PASS	FAIL	PASS	FAIL	PASS	PASS	FAIL	PASS	PASS	PASS	8/11	4.8	78.1s	MIXED
overgaard	PASS	PASS	FAIL	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	10/11	4.8	68.4s	STRONG
qwen3.8:27b heimdall	PASS	PASS	FAIL	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	10/11	4.6	173.7s	STRONG
midgaard	FAIL	PASS	FAIL	PASS	PASS	PASS	PASS	PASS	PASS	PASS	PASS	9/11	4.3	243.8s	MIXED
overgaard	PASS	PASS	FAIL	PASS	PASS	PASS	PASS	PASS	PASS	PASS	FAIL	9/11	4.6	106.7s	MIXED
PASS rate by task	94%	97%	61%	91%	82%	94%	88%	88%	97%	88%	94%	of 33 runs

PASS FAIL (honest N/A where capable=false) no record Capable = tests passed of 11. Quality = avg score 1–5. Latency = avg seconds/test.

Local vs Cloud — 12B Model Head-to-Head

Identical 13-test battery · local Ollama vs cloud Sonnet-5 (Nous) · 2026-08-26

gemma4:12b-it-qat vs claude-sonnet-5

LOCAL

gemma4:12b-it-qat

Asgard (m2pro, 16GB)

12/13capable
607.23swall-clock
12,625tokens
$0.00cost
44.5savg/test

CLOUD

anthropic/claude-sonnet-5

Nous Portal (cloud)

12/13capable
437.4swall-clock
17,240tokens
$0.34cost (est)
33.6savg/test

Metric	Local (gemma4:12b)	Cloud (Sonnet-5)
Model	gemma4:12b-it-qat	anthropic/claude-sonnet-5
Execution	Ollama local (Asgard m2pro, 16GB)	Hermes → Nous Portal API
Capable	12/13	12/13
Wall-clock	607.2 s	437.4 s
Tokens	12,625	17,240
Est. cost	$0.00	~$0.34 (state.db)
Avg latency	44.5 s	33.6 s
Max latency	105.1 s	206.2 s

Verdict: Cloud (Sonnet) is faster per-test avg and higher capability; local is $0 but slower wall-clock and lower capability on this 12B model. Cloud costs real $ per run. — Both failed test 8b, 8c (prime print-strict scoring — model wrote a correct prime FUNCTION, not a print statement).

Source: benchmark_gemma4_vs_sonnet_20260826.json (Fleet-Eval-2026-08, authoritative). Cloud $ is an estimate from state.db; authoritative billing on the Nous $110 Portal cap ($61.80 used before this run).