Leaderboard
Each row is the share of 750 hidden Norwegian Bokmål test decisions a model takes on its own, with a Clopper-Pearson bound on its error. Ranked by coverage at the selected target. There is no composite score. Each language is scored separately.
As of 28 October 2026, Mammoth-1 leads Tacit v0.2 in Norwegian Bokmål with 99.3% certified coverage at a 5% error target (36 errors in 745 automated decisions, realised error 4.83%), ahead of basal-omni-1.0-ultra (basal 2.3.3) at 98.8%. 20 models are ranked.
- Benchmark
- v0.2
- Language
- Norwegian Bokmål
- Last updated
- 28 October 2026 (what changed)
- Protocol
- tacit-devfit-1, pooled fit
- Hidden test
- n = 750
- Runs
- by the maintainer, no vendor approval
- Earlier
- v0.1, Polish
Error target
Calibration
Tacit fit, pooled: temperature and threshold are fitted per question type on the public dev splits of all 4 languages together (2,852 items), then frozen and applied once to the Norwegian Bokmål hidden test. This is the ranking column.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 5%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1Mammoth-1API, size not disclosedBaseten Model APIsCertified coverage99.3%Automated745Errors36Realised error, 95% interval4.83% [3.41–6.63]Holds at 5%Accuracy, 95% CI94.9 [93.1–96.3]ECE raw / cal.017 / .017Brier raw / cal.071 / .071Latency p50 / p95260 / 397 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 260 / 397 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.pooled.json (Tacit fit on pooled dev)
2basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage98.8%Automated741Errors25Realised error, 95% interval3.37% [2.20–4.94]Holds at 5%Accuracy, 95% CI96.5 [95.0–97.6]ECE raw / cal.011 / .013Brier raw / cal.052 / .052Latency p50 / p95563 / 807 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
2DREX MaxAPI, size not disclosedOpenRouterCertified coverage98.8%Automated741Errors31Realised error, 95% interval4.18% [2.86–5.89]Holds at 5%Accuracy, 95% CI95.2 [93.4–96.5]ECE raw / cal.016 / .009Brier raw / cal.066 / .065Latency p50 / p952059 / 4900 ms$ per 1k$0.018
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2059 / 4900 ms on the provider endpoint; $0.018 per 1,000 decisions
- Calibration file
- drex-max.pooled.json (Tacit fit on pooled dev)
4Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage96.7%Automated725Errors38Realised error, 95% interval5.24% [3.74–7.12]Misses the 5% targetAccuracy, 95% CI93.2 [91.2–94.8]ECE raw / cal.034 / .025Brier raw / cal.100 / .099Latency p50 / p95336 / 401 ms$ per 1k$0.010
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 336 / 401 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- microsoft-decision-1.pooled.json (Tacit fit on pooled dev)
5Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage96.0%Automated720Errors27Realised error, 95% interval3.75% [2.49–5.41]Holds at 5%Accuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.013 / .012Brier raw / cal.070 / .072Latency p50 / p95272 / 350 ms$ per 1k$0.011
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 272 / 350 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- d1-ultra.pooled.json (Tacit fit on pooled dev)
6pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage95.7%Automated718Errors37Realised error, 95% interval5.15% [3.65–7.03]Misses the 5% targetAccuracy, 95% CI93.5 [91.5–95.0]ECE raw / cal.013 / .014Brier raw / cal.092 / .092Latency p50 / p95300 / 443 ms$ per 1k$0.006
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 300 / 443 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.pooled.json (Tacit fit on pooled dev)
7Jev 1.14API, size not disclosedOpenRouterCertified coverage94.9%Automated712Errors35Realised error, 95% interval4.92% [3.45–6.77]Holds at 5%Accuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.018 / .018Brier raw / cal.094 / .094Latency p50 / p95270 / 326 ms$ per 1k$0.022
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 270 / 326 ms on the provider endpoint; $0.022 per 1,000 decisions
- Calibration file
- jev-1.14.pooled.json (Tacit fit on pooled dev)
8GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage94.0%Automated705Errors39Realised error, 95% interval5.53% [3.96–7.49]Misses the 5% targetAccuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.028 / .030Brier raw / cal.109 / .113Latency p50 / p95295 / 510 ms$ per 1k$0.033
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 295 / 510 ms on the provider endpoint; $0.033 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.pooled.json (Tacit fit on pooled dev)
9Elephant-1API, size not disclosedOpenRouterCertified coverage93.7%Automated703Errors41Realised error, 95% interval5.83% [4.22–7.83]Misses the 5% targetAccuracy, 95% CI91.2 [89.0–93.0]ECE raw / cal.020 / .027Brier raw / cal.119 / .119Latency p50 / p95268 / 359 ms$ per 1k$0.011
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 268 / 359 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- elephant-1.pooled.json (Tacit fit on pooled dev)
10Liquid d1.1API, size not disclosedOpenRouterCertified coverage93.2%Automated699Errors33Realised error, 95% interval4.72% [3.27–6.57]Holds at 5%Accuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.016 / .017Brier raw / cal.106 / .105Latency p50 / p952376 / 5413 ms$ per 1k$0.005
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2376 / 5413 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- d1.1.pooled.json (Tacit fit on pooled dev)
11Mercury DecideAPI, size not disclosedOpenRouterCertified coverage90.5%Automated679Errors24Realised error, 95% interval3.53% [2.28–5.21]Holds at 5%Accuracy, 95% CI91.5 [89.2–93.3]ECE raw / cal.058 / .026Brier raw / cal.132 / .114Latency p50 / p95352 / 556 ms$ per 1k$0.006
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 352 / 556 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- mercury-decide.pooled.json (Tacit fit on pooled dev)
11basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage90.5%Automated679Errors33Realised error, 95% interval4.86% [3.37–6.76]Holds at 5%Accuracy, 95% CI90.9 [88.7–92.8]ECE raw / cal.042 / .037Brier raw / cal.122 / .114Latency p50 / p95231 / 344 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
13Respan Span-02API, size not disclosedOpenRouterCertified coverage90.1%Automated676Errors33Realised error, 95% interval4.88% [3.38–6.79]Holds at 5%Accuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.035 / .029Brier raw / cal.111 / .109Latency p50 / p95311 / 1735 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 311 / 1735 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.pooled.json (Tacit fit on pooled dev)
14basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage89.9%Automated674Errors29Realised error, 95% interval4.30% [2.90–6.12]Holds at 5%Accuracy, 95% CI91.6 [89.4–93.4]ECE raw / cal.042 / .033Brier raw / cal.121 / .113Latency p50 / p95223 / 339 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
15basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage81.1%Automated608Errors25Realised error, 95% interval4.11% [2.68–6.01]Holds at 5%Accuracy, 95% CI89.1 [86.6–91.1]ECE raw / cal.060 / .034Brier raw / cal.161 / .147Latency p50 / p9567 / 87 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
16basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage80.3%Automated602Errors19Realised error, 95% interval3.16% [1.91–4.88]Holds at 5%Accuracy, 95% CI89.6 [87.2–91.6]ECE raw / cal.054 / .024Brier raw / cal.148 / .137Latency p50 / p95318 / 452 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-max.basal233.pooled.json (Tacit fit on pooled dev)
17Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage51.1%Automated383Errors17Realised error, 95% interval4.44% [2.61–7.01]Holds at 5%Accuracy, 95% CI91.2 [89.0–93.0]ECE raw / cal.020 / .023Brier raw / cal.132 / .133Latency p50 / p95349 / 680 ms$ per 1k$0.021
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 349 / 680 ms on the provider endpoint; $0.021 per 1,000 decisions
- Calibration file
- clef.pooled.json (Tacit fit on pooled dev)
18basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage35.9%Automated269Errors13Realised error, 95% interval4.83% [2.60–8.12]Holds at 5%Accuracy, 95% CI83.1 [80.2–85.6]ECE raw / cal.099 / .028Brier raw / cal.274 / .246Latency p50 / p95147 / 198 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-4.5B.basal233.pooled.json (Tacit fit on pooled dev)
19basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage29.6%Automated222Errors13Realised error, 95% interval5.86% [3.15–9.81]Misses the 5% targetAccuracy, 95% CI76.7 [73.5–79.6]ECE raw / cal.116 / .025Brier raw / cal.352 / .323Latency p50 / p95199 / 279 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-omni-pico.basal233.pooled.json (Tacit fit on pooled dev)
20basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage18.5%Automated139Errors6Realised error, 95% interval4.32% [1.60–9.16]Holds at 5%Accuracy, 95% CI73.1 [69.8–76.1]ECE raw / cal.066 / .032Brier raw / cal.385 / .377Latency p50 / p9559 / 76 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-mini.basal233.pooled.json (Tacit fit on pooled dev)
Per-language fit, a secondary view: the same protocol fitted on the Norwegian Bokmål dev split alone (750 items). It shows how far a language-specific calibration differs from the pooled one.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 5%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage98.3%Automated737Errors21Realised error, 95% interval2.85% [1.77–4.32]Holds at 5%Accuracy, 95% CI96.5 [95.0–97.6]ECE raw / cal.011 / .013Brier raw / cal.052 / .052Latency p50 / p95563 / 807 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
2Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage94.9%Automated712Errors23Realised error, 95% interval3.23% [2.06–4.81]Holds at 5%Accuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.013 / .012Brier raw / cal.070 / .072Latency p50 / p95272 / 350 ms$ per 1k$0.011
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 272 / 350 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- d1-ultra.per-language.json (Tacit fit on Norwegian Bokmål dev)
3DREX MaxAPI, size not disclosedOpenRouterCertified coverage94.3%Automated707Errors16Realised error, 95% interval2.26% [1.30–3.65]Holds at 5%Accuracy, 95% CI95.2 [93.4–96.5]ECE raw / cal.016 / .009Brier raw / cal.066 / .065Latency p50 / p952059 / 4900 ms$ per 1k$0.018
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2059 / 4900 ms on the provider endpoint; $0.018 per 1,000 decisions
- Calibration file
- drex-max.per-language.json (Tacit fit on Norwegian Bokmål dev)
4Jev 1.14API, size not disclosedOpenRouterCertified coverage91.5%Automated686Errors27Realised error, 95% interval3.94% [2.61–5.68]Holds at 5%Accuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.018 / .018Brier raw / cal.094 / .094Latency p50 / p95270 / 326 ms$ per 1k$0.022
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 270 / 326 ms on the provider endpoint; $0.022 per 1,000 decisions
- Calibration file
- jev-1.14.per-language.json (Tacit fit on Norwegian Bokmål dev)
5pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage91.1%Automated683Errors23Realised error, 95% interval3.37% [2.15–5.01]Holds at 5%Accuracy, 95% CI93.5 [91.5–95.0]ECE raw / cal.013 / .014Brier raw / cal.092 / .092Latency p50 / p95300 / 443 ms$ per 1k$0.006
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 300 / 443 ms on the provider endpoint; $0.006 per 1,000 decisions
6Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage90.1%Automated676Errors26Realised error, 95% interval3.85% [2.53–5.58]Holds at 5%Accuracy, 95% CI93.2 [91.2–94.8]ECE raw / cal.034 / .025Brier raw / cal.100 / .099Latency p50 / p95336 / 401 ms$ per 1k$0.010
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 336 / 401 ms on the provider endpoint; $0.010 per 1,000 decisions
7Liquid d1.1API, size not disclosedOpenRouterCertified coverage87.7%Automated658Errors23Realised error, 95% interval3.50% [2.23–5.20]Holds at 5%Accuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.016 / .017Brier raw / cal.106 / .105Latency p50 / p952376 / 5413 ms$ per 1k$0.005
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2376 / 5413 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- d1.1.per-language.json (Tacit fit on Norwegian Bokmål dev)
8Mercury DecideAPI, size not disclosedOpenRouterCertified coverage86.1%Automated646Errors15Realised error, 95% interval2.32% [1.31–3.80]Holds at 5%Accuracy, 95% CI91.5 [89.2–93.3]ECE raw / cal.058 / .026Brier raw / cal.132 / .114Latency p50 / p95352 / 556 ms$ per 1k$0.006
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 352 / 556 ms on the provider endpoint; $0.006 per 1,000 decisions
9basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage84.9%Automated637Errors12Realised error, 95% interval1.88% [0.98–3.27]Holds at 5%Accuracy, 95% CI90.9 [88.7–92.8]ECE raw / cal.042 / .037Brier raw / cal.122 / .114Latency p50 / p95231 / 344 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
10Mammoth-1API, size not disclosedBaseten Model APIsCertified coverage84.8%Automated636Errors21Realised error, 95% interval3.30% [2.06–5.00]Holds at 5%Accuracy, 95% CI94.9 [93.1–96.3]ECE raw / cal.017 / .017Brier raw / cal.071 / .071Latency p50 / p95260 / 397 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 260 / 397 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.per-language.json (Tacit fit on Norwegian Bokmål dev)
11basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage84.7%Automated635Errors13Realised error, 95% interval2.05% [1.09–3.48]Holds at 5%Accuracy, 95% CI91.6 [89.4–93.4]ECE raw / cal.042 / .033Brier raw / cal.121 / .113Latency p50 / p95223 / 339 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
12GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage81.2%Automated609Errors16Realised error, 95% interval2.63% [1.51–4.23]Holds at 5%Accuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.028 / .030Brier raw / cal.109 / .113Latency p50 / p95295 / 510 ms$ per 1k$0.033
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 295 / 510 ms on the provider endpoint; $0.033 per 1,000 decisions
13Respan Span-02API, size not disclosedOpenRouterCertified coverage75.7%Automated568Errors11Realised error, 95% interval1.94% [0.97–3.44]Holds at 5%Accuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.035 / .029Brier raw / cal.111 / .109Latency p50 / p95311 / 1735 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 311 / 1735 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.per-language.json (Tacit fit on Norwegian Bokmål dev)
14Elephant-1API, size not disclosedOpenRouterCertified coverage66.9%Automated502Errors12Realised error, 95% interval2.39% [1.24–4.14]Holds at 5%Accuracy, 95% CI91.2 [89.0–93.0]ECE raw / cal.020 / .027Brier raw / cal.119 / .119Latency p50 / p95268 / 359 ms$ per 1k$0.011
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 268 / 359 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- elephant-1.per-language.json (Tacit fit on Norwegian Bokmål dev)
15Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage63.2%Automated474Errors14Realised error, 95% interval2.95% [1.62–4.91]Holds at 5%Accuracy, 95% CI91.2 [89.0–93.0]ECE raw / cal.020 / .023Brier raw / cal.132 / .133Latency p50 / p95349 / 680 ms$ per 1k$0.021
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 349 / 680 ms on the provider endpoint; $0.021 per 1,000 decisions
- Calibration file
- clef.per-language.json (Tacit fit on Norwegian Bokmål dev)
16basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage57.2%Automated429Errors5Realised error, 95% interval1.17% [0.38–2.70]Holds at 5%Accuracy, 95% CI89.1 [86.6–91.1]ECE raw / cal.060 / .034Brier raw / cal.161 / .147Latency p50 / p9567 / 87 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
17basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage32.9%Automated247Errors1Realised error, 95% interval0.40% [0.01–2.23]Holds at 5%Accuracy, 95% CI89.6 [87.2–91.6]ECE raw / cal.054 / .024Brier raw / cal.148 / .137Latency p50 / p95318 / 452 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
18basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage14.4%Automated108Errors5Realised error, 95% interval4.63% [1.52–10.47]Holds at 5%Accuracy, 95% CI73.1 [69.8–76.1]ECE raw / cal.066 / .032Brier raw / cal.385 / .377Latency p50 / p9559 / 76 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
19basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage6.4%Automated48Errors0Realised error, 95% interval0.00% [0.00–7.40]Holds at 5%Accuracy, 95% CI76.7 [73.5–79.6]ECE raw / cal.116 / .025Brier raw / cal.352 / .323Latency p50 / p95199 / 279 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
20basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI83.1 [80.2–85.6]ECE raw / cal.099 / .028Brier raw / cal.274 / .246Latency p50 / p95147 / 198 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
Vendor file: the calibration file the model author ships, applied unchanged. Only 8 of 20 models ship one. A model can reach a coverage and still miss the target on test; that is reported, not hidden.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 5%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage96.5%Automated724Errors19Realised error, 95% interval2.62% [1.59–4.07]Holds at 5%Accuracy, 95% CI96.5 [95.0–97.6]ECE raw / cal.011 / .013Brier raw / cal.052 / .052Latency p50 / p95563 / 807 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
2basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage89.3%Automated670Errors34Realised error, 95% interval5.07% [3.54–7.02]Misses the 5% targetAccuracy, 95% CI91.6 [89.4–93.4]ECE raw / cal.042 / .033Brier raw / cal.121 / .113Latency p50 / p95223 / 339 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
3basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage84.1%Automated631Errors34Realised error, 95% interval5.39% [3.76–7.45]Misses the 5% targetAccuracy, 95% CI89.1 [86.6–91.1]ECE raw / cal.060 / .034Brier raw / cal.161 / .147Latency p50 / p9567 / 87 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
4basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage82.4%Automated618Errors34Realised error, 95% interval5.50% [3.84–7.60]Misses the 5% targetAccuracy, 95% CI90.9 [88.7–92.8]ECE raw / cal.042 / .037Brier raw / cal.122 / .114Latency p50 / p95231 / 344 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
5basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage50.3%Automated377Errors3Realised error, 95% interval0.80% [0.16–2.31]Holds at 5%Accuracy, 95% CI89.6 [87.2–91.6]ECE raw / cal.054 / .024Brier raw / cal.148 / .137Latency p50 / p95318 / 452 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage46.8%Automated351Errors39Realised error, 95% interval11.11% [8.02–14.88]Misses the 5% targetAccuracy, 95% CI76.7 [73.5–79.6]ECE raw / cal.116 / .025Brier raw / cal.352 / .323Latency p50 / p95199 / 279 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
7basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage29.5%Automated221Errors5Realised error, 95% interval2.26% [0.74–5.20]Holds at 5%Accuracy, 95% CI83.1 [80.2–85.6]ECE raw / cal.099 / .028Brier raw / cal.274 / .246Latency p50 / p95147 / 198 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
8basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage2.9%Automated22Errors1Realised error, 95% interval4.55% [0.12–22.84]Holds at 5%Accuracy, 95% CI73.1 [69.8–76.1]ECE raw / cal.066 / .032Brier raw / cal.385 / .377Latency p50 / p9559 / 76 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
9Mammoth-1API, size not disclosedBaseten Model APIsCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.9 [93.1–96.3]ECE raw / cal.017 / .017Brier raw / cal.071 / .071Latency p50 / p95260 / 397 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 260 / 397 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.pooled.json (Tacit fit on pooled dev)
9DREX MaxAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.2 [93.4–96.5]ECE raw / cal.016 / .009Brier raw / cal.066 / .065Latency p50 / p952059 / 4900 ms$ per 1k$0.018
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2059 / 4900 ms on the provider endpoint; $0.018 per 1,000 decisions
- Calibration file
- drex-max.pooled.json (Tacit fit on pooled dev)
9Microsoft-Decision-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.2 [91.2–94.8]ECE raw / cal.034 / .025Brier raw / cal.100 / .099Latency p50 / p95336 / 401 ms$ per 1k$0.010
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 336 / 401 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- microsoft-decision-1.pooled.json (Tacit fit on pooled dev)
9Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.013 / .012Brier raw / cal.070 / .072Latency p50 / p95272 / 350 ms$ per 1k$0.011
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 272 / 350 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- d1-ultra.pooled.json (Tacit fit on pooled dev)
9pplx-decider-v1.2API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.5 [91.5–95.0]ECE raw / cal.013 / .014Brier raw / cal.092 / .092Latency p50 / p95300 / 443 ms$ per 1k$0.006
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 300 / 443 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.pooled.json (Tacit fit on pooled dev)
9Jev 1.14API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.018 / .018Brier raw / cal.094 / .094Latency p50 / p95270 / 326 ms$ per 1k$0.022
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 270 / 326 ms on the provider endpoint; $0.022 per 1,000 decisions
- Calibration file
- jev-1.14.pooled.json (Tacit fit on pooled dev)
9GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.028 / .030Brier raw / cal.109 / .113Latency p50 / p95295 / 510 ms$ per 1k$0.033
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 295 / 510 ms on the provider endpoint; $0.033 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.pooled.json (Tacit fit on pooled dev)
9Elephant-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI91.2 [89.0–93.0]ECE raw / cal.020 / .027Brier raw / cal.119 / .119Latency p50 / p95268 / 359 ms$ per 1k$0.011
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 268 / 359 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- elephant-1.pooled.json (Tacit fit on pooled dev)
9Liquid d1.1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.016 / .017Brier raw / cal.106 / .105Latency p50 / p952376 / 5413 ms$ per 1k$0.005
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2376 / 5413 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- d1.1.pooled.json (Tacit fit on pooled dev)
9Mercury DecideAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI91.5 [89.2–93.3]ECE raw / cal.058 / .026Brier raw / cal.132 / .114Latency p50 / p95352 / 556 ms$ per 1k$0.006
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 352 / 556 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- mercury-decide.pooled.json (Tacit fit on pooled dev)
9Respan Span-02API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.035 / .029Brier raw / cal.111 / .109Latency p50 / p95311 / 1735 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 311 / 1735 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.pooled.json (Tacit fit on pooled dev)
9Cloudflare ClefAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI91.2 [89.0–93.0]ECE raw / cal.020 / .023Brier raw / cal.132 / .133Latency p50 / p95349 / 680 ms$ per 1k$0.021
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 349 / 680 ms on the provider endpoint; $0.021 per 1,000 decisions
- Calibration file
- clef.pooled.json (Tacit fit on pooled dev)
Tacit fit, pooled: temperature and threshold are fitted per question type on the public dev splits of all 4 languages together (2,852 items), then frozen and applied once to the Norwegian Bokmål hidden test. This is the ranking column.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 1%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage85.7%Automated643Errors1Realised error, 95% interval0.16% [0.00–0.86]Holds at 1%Accuracy, 95% CI96.5 [95.0–97.6]ECE raw / cal.011 / .013Brier raw / cal.052 / .052Latency p50 / p95563 / 807 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
2Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage83.6%Automated627Errors3Realised error, 95% interval0.48% [0.10–1.39]Holds at 1%Accuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.013 / .012Brier raw / cal.070 / .072Latency p50 / p95272 / 350 ms$ per 1k$0.011
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 272 / 350 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- d1-ultra.pooled.json (Tacit fit on pooled dev)
3DREX MaxAPI, size not disclosedOpenRouterCertified coverage79.5%Automated596Errors3Realised error, 95% interval0.50% [0.10–1.46]Holds at 1%Accuracy, 95% CI95.2 [93.4–96.5]ECE raw / cal.016 / .009Brier raw / cal.066 / .065Latency p50 / p952059 / 4900 ms$ per 1k$0.018
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2059 / 4900 ms on the provider endpoint; $0.018 per 1,000 decisions
- Calibration file
- drex-max.pooled.json (Tacit fit on pooled dev)
4Jev 1.14API, size not disclosedOpenRouterCertified coverage76.1%Automated571Errors5Realised error, 95% interval0.88% [0.28–2.03]Holds at 1%Accuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.018 / .018Brier raw / cal.094 / .094Latency p50 / p95270 / 326 ms$ per 1k$0.022
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 270 / 326 ms on the provider endpoint; $0.022 per 1,000 decisions
- Calibration file
- jev-1.14.pooled.json (Tacit fit on pooled dev)
5pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage75.9%Automated569Errors3Realised error, 95% interval0.53% [0.11–1.53]Holds at 1%Accuracy, 95% CI93.5 [91.5–95.0]ECE raw / cal.013 / .014Brier raw / cal.092 / .092Latency p50 / p95300 / 443 ms$ per 1k$0.006
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 300 / 443 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.pooled.json (Tacit fit on pooled dev)
6Mercury DecideAPI, size not disclosedOpenRouterCertified coverage73.6%Automated552Errors6Realised error, 95% interval1.09% [0.40–2.35]Misses the 1% targetAccuracy, 95% CI91.5 [89.2–93.3]ECE raw / cal.058 / .026Brier raw / cal.132 / .114Latency p50 / p95352 / 556 ms$ per 1k$0.006
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 352 / 556 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- mercury-decide.pooled.json (Tacit fit on pooled dev)
7basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage72.4%Automated543Errors2Realised error, 95% interval0.37% [0.04–1.32]Holds at 1%Accuracy, 95% CI91.6 [89.4–93.4]ECE raw / cal.042 / .033Brier raw / cal.121 / .113Latency p50 / p95223 / 339 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
8basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage71.9%Automated539Errors4Realised error, 95% interval0.74% [0.20–1.89]Holds at 1%Accuracy, 95% CI90.9 [88.7–92.8]ECE raw / cal.042 / .037Brier raw / cal.122 / .114Latency p50 / p95231 / 344 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
9Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage64.3%Automated482Errors4Realised error, 95% interval0.83% [0.23–2.11]Holds at 1%Accuracy, 95% CI93.2 [91.2–94.8]ECE raw / cal.034 / .025Brier raw / cal.100 / .099Latency p50 / p95336 / 401 ms$ per 1k$0.010
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 336 / 401 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- microsoft-decision-1.pooled.json (Tacit fit on pooled dev)
10Mammoth-1API, size not disclosedBaseten Model APIsCertified coverage62.0%Automated465Errors4Realised error, 95% interval0.86% [0.23–2.19]Holds at 1%Accuracy, 95% CI94.9 [93.1–96.3]ECE raw / cal.017 / .017Brier raw / cal.071 / .071Latency p50 / p95260 / 397 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 260 / 397 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.pooled.json (Tacit fit on pooled dev)
11Elephant-1API, size not disclosedOpenRouterCertified coverage52.3%Automated392Errors3Realised error, 95% interval0.77% [0.16–2.22]Holds at 1%Accuracy, 95% CI91.2 [89.0–93.0]ECE raw / cal.020 / .027Brier raw / cal.119 / .119Latency p50 / p95268 / 359 ms$ per 1k$0.011
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 268 / 359 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- elephant-1.pooled.json (Tacit fit on pooled dev)
12GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage47.1%Automated353Errors0Realised error, 95% interval0.00% [0.00–1.04]Holds at 1%Accuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.028 / .030Brier raw / cal.109 / .113Latency p50 / p95295 / 510 ms$ per 1k$0.033
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 295 / 510 ms on the provider endpoint; $0.033 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.pooled.json (Tacit fit on pooled dev)
13Respan Span-02API, size not disclosedOpenRouterCertified coverage42.3%Automated317Errors1Realised error, 95% interval0.32% [0.01–1.74]Holds at 1%Accuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.035 / .029Brier raw / cal.111 / .109Latency p50 / p95311 / 1735 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 311 / 1735 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.pooled.json (Tacit fit on pooled dev)
14Liquid d1.1API, size not disclosedOpenRouterCertified coverage33.1%Automated248Errors0Realised error, 95% interval0.00% [0.00–1.48]Holds at 1%Accuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.016 / .017Brier raw / cal.106 / .105Latency p50 / p952376 / 5413 ms$ per 1k$0.005
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2376 / 5413 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- d1.1.pooled.json (Tacit fit on pooled dev)
15basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage24.8%Automated186Errors0Realised error, 95% interval0.00% [0.00–1.96]Holds at 1%Accuracy, 95% CI89.1 [86.6–91.1]ECE raw / cal.060 / .034Brier raw / cal.161 / .147Latency p50 / p9567 / 87 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
16basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage24.0%Automated180Errors1Realised error, 95% interval0.56% [0.01–3.06]Holds at 1%Accuracy, 95% CI89.6 [87.2–91.6]ECE raw / cal.054 / .024Brier raw / cal.148 / .137Latency p50 / p95318 / 452 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-max.basal233.pooled.json (Tacit fit on pooled dev)
17Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage20.4%Automated153Errors2Realised error, 95% interval1.31% [0.16–4.64]Misses the 1% targetAccuracy, 95% CI91.2 [89.0–93.0]ECE raw / cal.020 / .023Brier raw / cal.132 / .133Latency p50 / p95349 / 680 ms$ per 1k$0.021
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 349 / 680 ms on the provider endpoint; $0.021 per 1,000 decisions
- Calibration file
- clef.pooled.json (Tacit fit on pooled dev)
18basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage2.7%Automated20Errors0Realised error, 95% interval0.00% [0.00–16.84]Holds at 1%Accuracy, 95% CI73.1 [69.8–76.1]ECE raw / cal.066 / .032Brier raw / cal.385 / .377Latency p50 / p9559 / 76 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-mini.basal233.pooled.json (Tacit fit on pooled dev)
19basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI83.1 [80.2–85.6]ECE raw / cal.099 / .028Brier raw / cal.274 / .246Latency p50 / p95147 / 198 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-4.5B.basal233.pooled.json (Tacit fit on pooled dev)
19basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI76.7 [73.5–79.6]ECE raw / cal.116 / .025Brier raw / cal.352 / .323Latency p50 / p95199 / 279 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-omni-pico.basal233.pooled.json (Tacit fit on pooled dev)
Per-language fit, a secondary view: the same protocol fitted on the Norwegian Bokmål dev split alone (750 items). It shows how far a language-specific calibration differs from the pooled one.
No model is certified at 1% yet. Every decision goes to a person.
The per-language fit sees only the Norwegian Bokmål dev split: 300 yes/no, 294 choice and 156 score items. A 1% bound needs at least 299 accepted dev decisions of one type, all of them correct, so this fit can usually certify 5% only. The table still shows each model's accuracy and calibration.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 1%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1Mammoth-1API, size not disclosedBaseten Model APIsCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.9 [93.1–96.3]ECE raw / cal.017 / .017Brier raw / cal.071 / .071Latency p50 / p95260 / 397 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 260 / 397 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.per-language.json (Tacit fit on Norwegian Bokmål dev)
1basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI96.5 [95.0–97.6]ECE raw / cal.011 / .013Brier raw / cal.052 / .052Latency p50 / p95563 / 807 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1DREX MaxAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI95.2 [93.4–96.5]ECE raw / cal.016 / .009Brier raw / cal.066 / .065Latency p50 / p952059 / 4900 ms$ per 1k$0.018
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2059 / 4900 ms on the provider endpoint; $0.018 per 1,000 decisions
- Calibration file
- drex-max.per-language.json (Tacit fit on Norwegian Bokmål dev)
1Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI93.2 [91.2–94.8]ECE raw / cal.034 / .025Brier raw / cal.100 / .099Latency p50 / p95336 / 401 ms$ per 1k$0.010
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 336 / 401 ms on the provider endpoint; $0.010 per 1,000 decisions
1Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.013 / .012Brier raw / cal.070 / .072Latency p50 / p95272 / 350 ms$ per 1k$0.011
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 272 / 350 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- d1-ultra.per-language.json (Tacit fit on Norwegian Bokmål dev)
1pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI93.5 [91.5–95.0]ECE raw / cal.013 / .014Brier raw / cal.092 / .092Latency p50 / p95300 / 443 ms$ per 1k$0.006
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 300 / 443 ms on the provider endpoint; $0.006 per 1,000 decisions
1Jev 1.14API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.018 / .018Brier raw / cal.094 / .094Latency p50 / p95270 / 326 ms$ per 1k$0.022
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 270 / 326 ms on the provider endpoint; $0.022 per 1,000 decisions
- Calibration file
- jev-1.14.per-language.json (Tacit fit on Norwegian Bokmål dev)
1GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.028 / .030Brier raw / cal.109 / .113Latency p50 / p95295 / 510 ms$ per 1k$0.033
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 295 / 510 ms on the provider endpoint; $0.033 per 1,000 decisions
1Elephant-1API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI91.2 [89.0–93.0]ECE raw / cal.020 / .027Brier raw / cal.119 / .119Latency p50 / p95268 / 359 ms$ per 1k$0.011
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 268 / 359 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- elephant-1.per-language.json (Tacit fit on Norwegian Bokmål dev)
1Liquid d1.1API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.016 / .017Brier raw / cal.106 / .105Latency p50 / p952376 / 5413 ms$ per 1k$0.005
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2376 / 5413 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- d1.1.per-language.json (Tacit fit on Norwegian Bokmål dev)
1Mercury DecideAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI91.5 [89.2–93.3]ECE raw / cal.058 / .026Brier raw / cal.132 / .114Latency p50 / p95352 / 556 ms$ per 1k$0.006
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 352 / 556 ms on the provider endpoint; $0.006 per 1,000 decisions
1basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI90.9 [88.7–92.8]ECE raw / cal.042 / .037Brier raw / cal.122 / .114Latency p50 / p95231 / 344 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1Respan Span-02API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.035 / .029Brier raw / cal.111 / .109Latency p50 / p95311 / 1735 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 311 / 1735 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.per-language.json (Tacit fit on Norwegian Bokmål dev)
1basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI91.6 [89.4–93.4]ECE raw / cal.042 / .033Brier raw / cal.121 / .113Latency p50 / p95223 / 339 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI89.1 [86.6–91.1]ECE raw / cal.060 / .034Brier raw / cal.161 / .147Latency p50 / p9567 / 87 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI89.6 [87.2–91.6]ECE raw / cal.054 / .024Brier raw / cal.148 / .137Latency p50 / p95318 / 452 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI91.2 [89.0–93.0]ECE raw / cal.020 / .023Brier raw / cal.132 / .133Latency p50 / p95349 / 680 ms$ per 1k$0.021
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 349 / 680 ms on the provider endpoint; $0.021 per 1,000 decisions
- Calibration file
- clef.per-language.json (Tacit fit on Norwegian Bokmål dev)
1basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI83.1 [80.2–85.6]ECE raw / cal.099 / .028Brier raw / cal.274 / .246Latency p50 / p95147 / 198 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI76.7 [73.5–79.6]ECE raw / cal.116 / .025Brier raw / cal.352 / .323Latency p50 / p95199 / 279 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI73.1 [69.8–76.1]ECE raw / cal.066 / .032Brier raw / cal.385 / .377Latency p50 / p9559 / 76 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
Vendor file: the calibration file the model author ships, applied unchanged. Only 8 of 20 models ship one. A model can reach a coverage and still miss the target on test; that is reported, not hidden.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 1%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage63.5%Automated476Errors1Realised error, 95% interval0.21% [0.01–1.16]Holds at 1%Accuracy, 95% CI96.5 [95.0–97.6]ECE raw / cal.011 / .013Brier raw / cal.052 / .052Latency p50 / p95563 / 807 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
2basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage54.9%Automated412Errors4Realised error, 95% interval0.97% [0.27–2.47]Holds at 1%Accuracy, 95% CI90.9 [88.7–92.8]ECE raw / cal.042 / .037Brier raw / cal.122 / .114Latency p50 / p95231 / 344 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
3basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage54.4%Automated408Errors3Realised error, 95% interval0.74% [0.15–2.13]Holds at 1%Accuracy, 95% CI91.6 [89.4–93.4]ECE raw / cal.042 / .033Brier raw / cal.121 / .113Latency p50 / p95223 / 339 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
4basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage50.0%Automated375Errors3Realised error, 95% interval0.80% [0.17–2.32]Holds at 1%Accuracy, 95% CI89.1 [86.6–91.1]ECE raw / cal.060 / .034Brier raw / cal.161 / .147Latency p50 / p9567 / 87 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
5basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage17.6%Automated132Errors3Realised error, 95% interval2.27% [0.47–6.50]Misses the 1% targetAccuracy, 95% CI76.7 [73.5–79.6]ECE raw / cal.116 / .025Brier raw / cal.352 / .323Latency p50 / p95199 / 279 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI89.6 [87.2–91.6]ECE raw / cal.054 / .024Brier raw / cal.148 / .137Latency p50 / p95318 / 452 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI83.1 [80.2–85.6]ECE raw / cal.099 / .028Brier raw / cal.274 / .246Latency p50 / p95147 / 198 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI73.1 [69.8–76.1]ECE raw / cal.066 / .032Brier raw / cal.385 / .377Latency p50 / p9559 / 76 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
9Mammoth-1API, size not disclosedBaseten Model APIsCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.9 [93.1–96.3]ECE raw / cal.017 / .017Brier raw / cal.071 / .071Latency p50 / p95260 / 397 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 260 / 397 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.pooled.json (Tacit fit on pooled dev)
9DREX MaxAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.2 [93.4–96.5]ECE raw / cal.016 / .009Brier raw / cal.066 / .065Latency p50 / p952059 / 4900 ms$ per 1k$0.018
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2059 / 4900 ms on the provider endpoint; $0.018 per 1,000 decisions
- Calibration file
- drex-max.pooled.json (Tacit fit on pooled dev)
9Microsoft-Decision-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.2 [91.2–94.8]ECE raw / cal.034 / .025Brier raw / cal.100 / .099Latency p50 / p95336 / 401 ms$ per 1k$0.010
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 336 / 401 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- microsoft-decision-1.pooled.json (Tacit fit on pooled dev)
9Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.013 / .012Brier raw / cal.070 / .072Latency p50 / p95272 / 350 ms$ per 1k$0.011
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 272 / 350 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- d1-ultra.pooled.json (Tacit fit on pooled dev)
9pplx-decider-v1.2API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.5 [91.5–95.0]ECE raw / cal.013 / .014Brier raw / cal.092 / .092Latency p50 / p95300 / 443 ms$ per 1k$0.006
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 300 / 443 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.pooled.json (Tacit fit on pooled dev)
9Jev 1.14API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.018 / .018Brier raw / cal.094 / .094Latency p50 / p95270 / 326 ms$ per 1k$0.022
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 270 / 326 ms on the provider endpoint; $0.022 per 1,000 decisions
- Calibration file
- jev-1.14.pooled.json (Tacit fit on pooled dev)
9GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.028 / .030Brier raw / cal.109 / .113Latency p50 / p95295 / 510 ms$ per 1k$0.033
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 295 / 510 ms on the provider endpoint; $0.033 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.pooled.json (Tacit fit on pooled dev)
9Elephant-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI91.2 [89.0–93.0]ECE raw / cal.020 / .027Brier raw / cal.119 / .119Latency p50 / p95268 / 359 ms$ per 1k$0.011
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 268 / 359 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- elephant-1.pooled.json (Tacit fit on pooled dev)
9Liquid d1.1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.016 / .017Brier raw / cal.106 / .105Latency p50 / p952376 / 5413 ms$ per 1k$0.005
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2376 / 5413 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- d1.1.pooled.json (Tacit fit on pooled dev)
9Mercury DecideAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI91.5 [89.2–93.3]ECE raw / cal.058 / .026Brier raw / cal.132 / .114Latency p50 / p95352 / 556 ms$ per 1k$0.006
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 352 / 556 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- mercury-decide.pooled.json (Tacit fit on pooled dev)
9Respan Span-02API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.035 / .029Brier raw / cal.111 / .109Latency p50 / p95311 / 1735 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 311 / 1735 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.pooled.json (Tacit fit on pooled dev)
9Cloudflare ClefAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI91.2 [89.0–93.0]ECE raw / cal.020 / .023Brier raw / cal.132 / .133Latency p50 / p95349 / 680 ms$ per 1k$0.021
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 349 / 680 ms on the provider endpoint; $0.021 per 1,000 decisions
- Calibration file
- clef.pooled.json (Tacit fit on pooled dev)
Automated shareDeferred to a personRealised error and 95% intervalError target
Accuracy by question type
Accuracy per question type on the Norwegian Bokmål hidden test. Thresholds are fitted per type, so a model can be certified for one type and defer every decision of another.
| Model | Yes/no n=300 | Choice n=294 | Score n=156 |
|---|---|---|---|
| Mammoth-1 | 93.3 | 96.3 | 95.5 |
| basal-omni-1.0-ultra (basal 2.3.3) | 96.3 | 96.3 | 97.4 |
| DREX Max | 93 | 95.9 | 98.1 |
| Microsoft-Decision-1 | 92.7 | 95.2 | 90.4 |
| Liquid d1-ultra | 93.7 | 95.6 | 95.5 |
| pplx-decider-v1.2 | 92.3 | 94.6 | 93.6 |
| Jev 1.14 | 90.3 | 95.6 | 96.8 |
| GPT-6 Luna Decisions | 91 | 93.5 | 93.6 |
| Elephant-1 | 88.3 | 94.2 | 91 |
| Liquid d1.1 | 89.7 | 94.9 | 93.6 |
| Mercury Decide | 87.3 | 94.9 | 92.9 |
| basal-omni-1.0-max (basal 2.3.3) | 86.7 | 92.5 | 96.2 |
| Respan Span-02 | 92.7 | 94.9 | 93.6 |
| basal-omni-1.1-max (basal 2.3.3) | 87.7 | 92.2 | 98.1 |
| basal-omni-1.0-mini (basal 2.3.3) | 85.3 | 91.8 | 91 |
| basal-2.0-max (basal 2.3.3) | 86.3 | 92.5 | 90.4 |
| Cloudflare Clef | 89.7 | 93.2 | 90.4 |
| basal-2.0-4.5B (basal 2.3.3) | 81 | 86.1 | 81.4 |
| basal-omni-pico (basal 2.3.3) | 73.3 | 82.7 | 71.8 |
| basal-2.0-mini (basal 2.3.3) | 72.7 | 78.9 | 62.8 |
Accuracy in %, hidden test.
Accuracy by family
19 families, 30 to 80 test items each. Darker means more accurate. Small families move a lot with one item, so read single cells with care.
FamilyMammoth-1omni-1.0-ultra (basal 2.3.3)DREX MaxMicrosoft-Decision-1Liquid d1-ultrapplx-decider-v1.2Jev 1.14GPT-6 Luna DecisionsElephant-1Liquid d1.1Mercury Decideomni-1.0-max (basal 2.3.3)Respan Span-02omni-1.1-max (basal 2.3.3)omni-1.0-mini (basal 2.3.3)2.0-max (basal 2.3.3)Cloudflare Clef2.0-4.5B (basal 2.3.3)omni-pico (basal 2.3.3)2.0-mini (basal 2.3.3)
public administrationpublic_admin, n=4090.095.095.092.590.087.587.590.087.590.090.085.090.085.085.087.590.085.080.077.5
bank / AMLbank_aml, n=4087.592.592.590.097.597.590.090.092.590.090.095.0100.095.095.095.087.585.075.082.5
cybersecuritycybersecurity, n=8090.095.092.592.596.388.891.387.587.586.387.585.093.887.580.087.585.076.368.870.0
escalationescalation, n=30100.0100.0100.0100.0100.0100.0100.0100.0100.096.7100.0100.0100.0100.093.393.3100.093.393.386.7
HRhr, n=4095.090.092.590.090.087.587.5100.080.087.582.585.095.087.590.085.090.085.070.065.0
customer irritationirritation, n=3093.393.390.093.386.790.090.080.090.080.086.780.086.780.080.086.790.093.380.083.3
IT securityit_security, n=4097.5100.095.090.092.592.597.590.085.092.587.590.090.092.587.585.082.580.072.560.0
law firmlaw_firm, n=4097.592.597.587.590.090.087.592.592.592.582.587.597.587.580.090.092.587.587.565.0
completenesscompleteness, n=3093.3100.086.786.7100.096.7100.096.790.0100.096.790.093.393.386.790.090.083.366.770.0
logisticslogistics, n=4092.595.092.597.592.592.587.590.087.592.587.587.585.087.585.077.587.577.565.080.0
moderationmoderation, n=40100.0100.0100.095.0100.097.5100.097.592.597.597.597.597.597.592.592.595.082.572.577.5
phishingphishing, n=30100.096.7100.0100.096.7100.093.396.793.396.796.793.393.393.393.393.393.386.780.083.3
manufacturingmanufacturing, n=4095.095.097.595.090.097.597.592.597.597.592.597.592.5100.090.090.092.582.572.570.0
clinicclinic_admin, n=4097.597.592.590.095.090.095.082.587.587.587.587.590.087.592.580.092.580.080.075.0
complaintscomplaint, n=30100.0100.0100.096.7100.0100.0100.0100.0100.0100.096.7100.096.7100.0100.0100.0100.0100.073.366.7
routingrouting, n=40100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.090.092.580.0
insuranceinsurance, n=4092.5100.095.095.097.592.595.095.095.092.592.595.0100.095.092.597.597.587.587.575.0
universityuniversity, n=4095.0100.097.592.590.095.092.592.592.595.097.587.587.587.590.090.092.570.082.567.5
housing communityhousing, n=4095.095.095.090.097.590.095.092.590.092.595.092.592.590.090.087.585.070.067.562.5