Leaderboard
Each row is the share of 750 hidden German test decisions a model takes on its own, with a Clopper-Pearson bound on its error. Ranked by coverage at the selected target. There is no composite score. Each language is scored separately.
As of 28 October 2026, DREX Max leads Tacit v0.2 in German with 99.5% certified coverage at a 5% error target (30 errors in 746 automated decisions, realised error 4.02%), ahead of basal-omni-1.0-ultra (basal 2.3.3) at 98.7%. 20 models are ranked.
- Benchmark
- v0.2
- Language
- German
- Last updated
- 28 October 2026 (what changed)
- Protocol
- tacit-devfit-1, pooled fit
- Hidden test
- n = 750
- Runs
- by the maintainer, no vendor approval
- Earlier
- v0.1, Polish
Error target
Calibration
Tacit fit, pooled: temperature and threshold are fitted per question type on the public dev splits of all 4 languages together (2,852 items), then frozen and applied once to the German hidden test. This is the ranking column.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 5%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1DREX MaxAPI, size not disclosedOpenRouterCertified coverage99.5%Automated746Errors30Realised error, 95% interval4.02% [2.73–5.69]Holds at 5%Accuracy, 95% CI95.9 [94.2–97.1]ECE raw / cal.022 / .015Brier raw / cal.064 / .062Latency p50 / p952044 / 4457 ms$ per 1k$0.016
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2044 / 4457 ms on the provider endpoint; $0.016 per 1,000 decisions
- Calibration file
- drex-max.pooled.json (Tacit fit on pooled dev)
2basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage98.7%Automated740Errors23Realised error, 95% interval3.11% [1.98–4.63]Holds at 5%Accuracy, 95% CI96.7 [95.1–97.7]ECE raw / cal.014 / .013Brier raw / cal.056 / .053Latency p50 / p95539 / 817 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
3Mammoth-1API, size not disclosedBaseten Model APIsCertified coverage98.4%Automated738Errors25Realised error, 95% interval3.39% [2.20–4.96]Holds at 5%Accuracy, 95% CI96.0 [94.3–97.2]ECE raw / cal.011 / .013Brier raw / cal.063 / .063Latency p50 / p95251 / 467 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 251 / 467 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.pooled.json (Tacit fit on pooled dev)
4Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage97.5%Automated731Errors35Realised error, 95% interval4.79% [3.36–6.60]Holds at 5%Accuracy, 95% CI94.1 [92.2–95.6]ECE raw / cal.024 / .015Brier raw / cal.090 / .087Latency p50 / p95335 / 395 ms$ per 1k$0.009
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 335 / 395 ms on the provider endpoint; $0.009 per 1,000 decisions
- Calibration file
- microsoft-decision-1.pooled.json (Tacit fit on pooled dev)
5Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage97.3%Automated730Errors20Realised error, 95% interval2.74% [1.68–4.20]Holds at 5%Accuracy, 95% CI95.7 [94.0–97.0]ECE raw / cal.011 / .011Brier raw / cal.061 / .058Latency p50 / p95272 / 330 ms$ per 1k$0.010
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 272 / 330 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- d1-ultra.pooled.json (Tacit fit on pooled dev)
6Jev 1.14API, size not disclosedOpenRouterCertified coverage95.6%Automated717Errors22Realised error, 95% interval3.07% [1.93–4.61]Holds at 5%Accuracy, 95% CI94.7 [92.8–96.1]ECE raw / cal.023 / .011Brier raw / cal.076 / .077Latency p50 / p95265 / 329 ms$ per 1k$0.022
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 265 / 329 ms on the provider endpoint; $0.022 per 1,000 decisions
- Calibration file
- jev-1.14.pooled.json (Tacit fit on pooled dev)
7pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage95.3%Automated715Errors19Realised error, 95% interval2.66% [1.61–4.12]Holds at 5%Accuracy, 95% CI95.1 [93.3–96.4]ECE raw / cal.017 / .006Brier raw / cal.063 / .063Latency p50 / p95298 / 379 ms$ per 1k$0.005
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 298 / 379 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.pooled.json (Tacit fit on pooled dev)
8GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage93.9%Automated704Errors33Realised error, 95% interval4.69% [3.25–6.52]Holds at 5%Accuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.021 / .027Brier raw / cal.107 / .107Latency p50 / p95295 / 466 ms$ per 1k$0.031
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 295 / 466 ms on the provider endpoint; $0.031 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.pooled.json (Tacit fit on pooled dev)
9Liquid d1.1API, size not disclosedOpenRouterCertified coverage93.1%Automated698Errors30Realised error, 95% interval4.30% [2.92–6.08]Holds at 5%Accuracy, 95% CI92.9 [90.9–94.6]ECE raw / cal.012 / .008Brier raw / cal.097 / .097Latency p50 / p951068 / 5038 ms$ per 1k$0.005
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 1068 / 5038 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- d1.1.pooled.json (Tacit fit on pooled dev)
10Elephant-1API, size not disclosedOpenRouterCertified coverage92.9%Automated697Errors25Realised error, 95% interval3.59% [2.33–5.25]Holds at 5%Accuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.022 / .016Brier raw / cal.086 / .084Latency p50 / p95269 / 358 ms$ per 1k$0.010
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 269 / 358 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- elephant-1.pooled.json (Tacit fit on pooled dev)
11basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage91.5%Automated686Errors27Realised error, 95% interval3.94% [2.61–5.68]Holds at 5%Accuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.028 / .010Brier raw / cal.104 / .100Latency p50 / p95208 / 311 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
12basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage91.1%Automated683Errors23Realised error, 95% interval3.37% [2.15–5.01]Holds at 5%Accuracy, 95% CI92.4 [90.3–94.1]ECE raw / cal.027 / .015Brier raw / cal.101 / .098Latency p50 / p95227 / 352 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
13Mercury DecideAPI, size not disclosedOpenRouterCertified coverage90.9%Automated682Errors20Realised error, 95% interval2.93% [1.80–4.49]Holds at 5%Accuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.046 / .025Brier raw / cal.107 / .096Latency p50 / p95349 / 552 ms$ per 1k$0.005
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 349 / 552 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- mercury-decide.pooled.json (Tacit fit on pooled dev)
14Respan Span-02API, size not disclosedOpenRouterCertified coverage90.5%Automated679Errors20Realised error, 95% interval2.95% [1.81–4.51]Holds at 5%Accuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.029 / .023Brier raw / cal.076 / .078Latency p50 / p95302 / 1228 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 302 / 1228 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.pooled.json (Tacit fit on pooled dev)
15basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage83.3%Automated625Errors24Realised error, 95% interval3.84% [2.48–5.66]Holds at 5%Accuracy, 95% CI90.5 [88.2–92.4]ECE raw / cal.049 / .023Brier raw / cal.148 / .135Latency p50 / p95313 / 466 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-max.basal233.pooled.json (Tacit fit on pooled dev)
16basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage83.2%Automated624Errors21Realised error, 95% interval3.37% [2.10–5.10]Holds at 5%Accuracy, 95% CI89.9 [87.5–91.8]ECE raw / cal.051 / .026Brier raw / cal.149 / .137Latency p50 / p9562 / 83 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
17Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage53.1%Automated398Errors14Realised error, 95% interval3.52% [1.94–5.83]Holds at 5%Accuracy, 95% CI92.3 [90.1–94.0]ECE raw / cal.019 / .020Brier raw / cal.114 / .111Latency p50 / p95338 / 658 ms$ per 1k$0.021
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 338 / 658 ms on the provider endpoint; $0.021 per 1,000 decisions
- Calibration file
- clef.pooled.json (Tacit fit on pooled dev)
18basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage40.4%Automated303Errors13Realised error, 95% interval4.29% [2.30–7.23]Holds at 5%Accuracy, 95% CI84.8 [82.1–87.2]ECE raw / cal.096 / .035Brier raw / cal.242 / .216Latency p50 / p95155 / 220 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-4.5B.basal233.pooled.json (Tacit fit on pooled dev)
19basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage38.9%Automated292Errors9Realised error, 95% interval3.08% [1.42–5.77]Holds at 5%Accuracy, 95% CI79.9 [76.8–82.6]ECE raw / cal.103 / .039Brier raw / cal.290 / .264Latency p50 / p95168 / 244 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-omni-pico.basal233.pooled.json (Tacit fit on pooled dev)
20basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage31.5%Automated236Errors7Realised error, 95% interval2.97% [1.20–6.02]Holds at 5%Accuracy, 95% CI78.0 [74.9–80.8]ECE raw / cal.065 / .035Brier raw / cal.295 / .284Latency p50 / p9565 / 91 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-mini.basal233.pooled.json (Tacit fit on pooled dev)
Per-language fit, a secondary view: the same protocol fitted on the German dev split alone (750 items). It shows how far a language-specific calibration differs from the pooled one.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 5%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1DREX MaxAPI, size not disclosedOpenRouterCertified coverage98.9%Automated742Errors29Realised error, 95% interval3.91% [2.63–5.57]Holds at 5%Accuracy, 95% CI95.9 [94.2–97.1]ECE raw / cal.022 / .015Brier raw / cal.064 / .062Latency p50 / p952044 / 4457 ms$ per 1k$0.016
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2044 / 4457 ms on the provider endpoint; $0.016 per 1,000 decisions
- Calibration file
- drex-max.per-language.json (Tacit fit on German dev)
2Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage97.9%Automated734Errors25Realised error, 95% interval3.41% [2.22–4.99]Holds at 5%Accuracy, 95% CI95.7 [94.0–97.0]ECE raw / cal.011 / .011Brier raw / cal.061 / .058Latency p50 / p95272 / 330 ms$ per 1k$0.010
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 272 / 330 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- d1-ultra.per-language.json (Tacit fit on German dev)
3Mammoth-1API, size not disclosedBaseten Model APIsCertified coverage97.2%Automated729Errors23Realised error, 95% interval3.16% [2.01–4.70]Holds at 5%Accuracy, 95% CI96.0 [94.3–97.2]ECE raw / cal.011 / .013Brier raw / cal.063 / .063Latency p50 / p95251 / 467 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 251 / 467 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.per-language.json (Tacit fit on German dev)
4basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage96.9%Automated727Errors18Realised error, 95% interval2.48% [1.47–3.88]Holds at 5%Accuracy, 95% CI96.7 [95.1–97.7]ECE raw / cal.014 / .013Brier raw / cal.056 / .053Latency p50 / p95539 / 817 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
5pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage93.7%Automated703Errors13Realised error, 95% interval1.85% [0.99–3.14]Holds at 5%Accuracy, 95% CI95.1 [93.3–96.4]ECE raw / cal.017 / .006Brier raw / cal.063 / .063Latency p50 / p95298 / 379 ms$ per 1k$0.005
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 298 / 379 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.per-language.json (Tacit fit on German dev)
6Jev 1.14API, size not disclosedOpenRouterCertified coverage92.5%Automated694Errors15Realised error, 95% interval2.16% [1.21–3.54]Holds at 5%Accuracy, 95% CI94.7 [92.8–96.1]ECE raw / cal.023 / .011Brier raw / cal.076 / .077Latency p50 / p95265 / 329 ms$ per 1k$0.022
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 265 / 329 ms on the provider endpoint; $0.022 per 1,000 decisions
- Calibration file
- jev-1.14.per-language.json (Tacit fit on German dev)
7Elephant-1API, size not disclosedOpenRouterCertified coverage90.3%Automated677Errors18Realised error, 95% interval2.66% [1.58–4.17]Holds at 5%Accuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.022 / .016Brier raw / cal.086 / .084Latency p50 / p95269 / 358 ms$ per 1k$0.010
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 269 / 358 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- elephant-1.per-language.json (Tacit fit on German dev)
8GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage90.1%Automated676Errors22Realised error, 95% interval3.25% [2.05–4.89]Holds at 5%Accuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.021 / .027Brier raw / cal.107 / .107Latency p50 / p95295 / 466 ms$ per 1k$0.031
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 295 / 466 ms on the provider endpoint; $0.031 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.per-language.json (Tacit fit on German dev)
9basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage89.7%Automated673Errors19Realised error, 95% interval2.82% [1.71–4.37]Holds at 5%Accuracy, 95% CI92.4 [90.3–94.1]ECE raw / cal.027 / .015Brier raw / cal.101 / .098Latency p50 / p95227 / 352 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
10Mercury DecideAPI, size not disclosedOpenRouterCertified coverage89.5%Automated671Errors19Realised error, 95% interval2.83% [1.71–4.39]Holds at 5%Accuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.046 / .025Brier raw / cal.107 / .096Latency p50 / p95349 / 552 ms$ per 1k$0.005
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 349 / 552 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- mercury-decide.per-language.json (Tacit fit on German dev)
11basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage89.2%Automated669Errors20Realised error, 95% interval2.99% [1.84–4.58]Holds at 5%Accuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.028 / .010Brier raw / cal.104 / .100Latency p50 / p95208 / 311 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
12Liquid d1.1API, size not disclosedOpenRouterCertified coverage88.9%Automated667Errors19Realised error, 95% interval2.85% [1.72–4.41]Holds at 5%Accuracy, 95% CI92.9 [90.9–94.6]ECE raw / cal.012 / .008Brier raw / cal.097 / .097Latency p50 / p951068 / 5038 ms$ per 1k$0.005
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 1068 / 5038 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- d1.1.per-language.json (Tacit fit on German dev)
13Respan Span-02API, size not disclosedOpenRouterCertified coverage88.4%Automated663Errors17Realised error, 95% interval2.56% [1.50–4.07]Holds at 5%Accuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.029 / .023Brier raw / cal.076 / .078Latency p50 / p95302 / 1228 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 302 / 1228 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.per-language.json (Tacit fit on German dev)
14Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage87.2%Automated654Errors21Realised error, 95% interval3.21% [2.00–4.87]Holds at 5%Accuracy, 95% CI94.1 [92.2–95.6]ECE raw / cal.024 / .015Brier raw / cal.090 / .087Latency p50 / p95335 / 395 ms$ per 1k$0.009
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 335 / 395 ms on the provider endpoint; $0.009 per 1,000 decisions
- Calibration file
- microsoft-decision-1.per-language.json (Tacit fit on German dev)
15Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage77.1%Automated578Errors9Realised error, 95% interval1.56% [0.71–2.94]Holds at 5%Accuracy, 95% CI92.3 [90.1–94.0]ECE raw / cal.019 / .020Brier raw / cal.114 / .111Latency p50 / p95338 / 658 ms$ per 1k$0.021
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 338 / 658 ms on the provider endpoint; $0.021 per 1,000 decisions
- Calibration file
- clef.per-language.json (Tacit fit on German dev)
16basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage70.5%Automated529Errors7Realised error, 95% interval1.32% [0.53–2.71]Holds at 5%Accuracy, 95% CI89.9 [87.5–91.8]ECE raw / cal.051 / .026Brier raw / cal.149 / .137Latency p50 / p9562 / 83 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
17basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage58.1%Automated436Errors4Realised error, 95% interval0.92% [0.25–2.33]Holds at 5%Accuracy, 95% CI90.5 [88.2–92.4]ECE raw / cal.049 / .023Brier raw / cal.148 / .135Latency p50 / p95313 / 466 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
18basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage28.1%Automated211Errors5Realised error, 95% interval2.37% [0.77–5.44]Holds at 5%Accuracy, 95% CI84.8 [82.1–87.2]ECE raw / cal.096 / .035Brier raw / cal.242 / .216Latency p50 / p95155 / 220 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
19basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage12.7%Automated95Errors0Realised error, 95% interval0.00% [0.00–3.81]Holds at 5%Accuracy, 95% CI78.0 [74.9–80.8]ECE raw / cal.065 / .035Brier raw / cal.295 / .284Latency p50 / p9565 / 91 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
20basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI79.9 [76.8–82.6]ECE raw / cal.103 / .039Brier raw / cal.290 / .264Latency p50 / p95168 / 244 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
Vendor file: the calibration file the model author ships, applied unchanged. Only 8 of 20 models ship one. A model can reach a coverage and still miss the target on test; that is reported, not hidden.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 5%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage96.8%Automated726Errors22Realised error, 95% interval3.03% [1.91–4.55]Holds at 5%Accuracy, 95% CI96.7 [95.1–97.7]ECE raw / cal.014 / .013Brier raw / cal.056 / .053Latency p50 / p95539 / 817 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
2basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage90.4%Automated678Errors25Realised error, 95% interval3.69% [2.40–5.40]Holds at 5%Accuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.028 / .010Brier raw / cal.104 / .100Latency p50 / p95208 / 311 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
3basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage85.5%Automated641Errors31Realised error, 95% interval4.84% [3.31–6.79]Holds at 5%Accuracy, 95% CI89.9 [87.5–91.8]ECE raw / cal.051 / .026Brier raw / cal.149 / .137Latency p50 / p9562 / 83 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
4basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage82.8%Automated621Errors21Realised error, 95% interval3.38% [2.11–5.12]Holds at 5%Accuracy, 95% CI92.4 [90.3–94.1]ECE raw / cal.027 / .015Brier raw / cal.101 / .098Latency p50 / p95227 / 352 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
5basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage52.0%Automated390Errors4Realised error, 95% interval1.03% [0.28–2.61]Holds at 5%Accuracy, 95% CI90.5 [88.2–92.4]ECE raw / cal.049 / .023Brier raw / cal.148 / .135Latency p50 / p95313 / 466 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage49.6%Automated372Errors23Realised error, 95% interval6.18% [3.96–9.13]Misses the 5% targetAccuracy, 95% CI79.9 [76.8–82.6]ECE raw / cal.103 / .039Brier raw / cal.290 / .264Latency p50 / p95168 / 244 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
7basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage40.4%Automated303Errors8Realised error, 95% interval2.64% [1.15–5.14]Holds at 5%Accuracy, 95% CI84.8 [82.1–87.2]ECE raw / cal.096 / .035Brier raw / cal.242 / .216Latency p50 / p95155 / 220 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
8basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage10.8%Automated81Errors1Realised error, 95% interval1.23% [0.03–6.69]Holds at 5%Accuracy, 95% CI78.0 [74.9–80.8]ECE raw / cal.065 / .035Brier raw / cal.295 / .284Latency p50 / p9565 / 91 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
9DREX MaxAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.9 [94.2–97.1]ECE raw / cal.022 / .015Brier raw / cal.064 / .062Latency p50 / p952044 / 4457 ms$ per 1k$0.016
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2044 / 4457 ms on the provider endpoint; $0.016 per 1,000 decisions
- Calibration file
- drex-max.pooled.json (Tacit fit on pooled dev)
9Mammoth-1API, size not disclosedBaseten Model APIsCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.0 [94.3–97.2]ECE raw / cal.011 / .013Brier raw / cal.063 / .063Latency p50 / p95251 / 467 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 251 / 467 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.pooled.json (Tacit fit on pooled dev)
9Microsoft-Decision-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.1 [92.2–95.6]ECE raw / cal.024 / .015Brier raw / cal.090 / .087Latency p50 / p95335 / 395 ms$ per 1k$0.009
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 335 / 395 ms on the provider endpoint; $0.009 per 1,000 decisions
- Calibration file
- microsoft-decision-1.pooled.json (Tacit fit on pooled dev)
9Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.7 [94.0–97.0]ECE raw / cal.011 / .011Brier raw / cal.061 / .058Latency p50 / p95272 / 330 ms$ per 1k$0.010
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 272 / 330 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- d1-ultra.pooled.json (Tacit fit on pooled dev)
9Jev 1.14API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.7 [92.8–96.1]ECE raw / cal.023 / .011Brier raw / cal.076 / .077Latency p50 / p95265 / 329 ms$ per 1k$0.022
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 265 / 329 ms on the provider endpoint; $0.022 per 1,000 decisions
- Calibration file
- jev-1.14.pooled.json (Tacit fit on pooled dev)
9pplx-decider-v1.2API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.1 [93.3–96.4]ECE raw / cal.017 / .006Brier raw / cal.063 / .063Latency p50 / p95298 / 379 ms$ per 1k$0.005
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 298 / 379 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.pooled.json (Tacit fit on pooled dev)
9GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.021 / .027Brier raw / cal.107 / .107Latency p50 / p95295 / 466 ms$ per 1k$0.031
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 295 / 466 ms on the provider endpoint; $0.031 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.pooled.json (Tacit fit on pooled dev)
9Liquid d1.1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI92.9 [90.9–94.6]ECE raw / cal.012 / .008Brier raw / cal.097 / .097Latency p50 / p951068 / 5038 ms$ per 1k$0.005
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 1068 / 5038 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- d1.1.pooled.json (Tacit fit on pooled dev)
9Elephant-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.022 / .016Brier raw / cal.086 / .084Latency p50 / p95269 / 358 ms$ per 1k$0.010
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 269 / 358 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- elephant-1.pooled.json (Tacit fit on pooled dev)
9Mercury DecideAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.046 / .025Brier raw / cal.107 / .096Latency p50 / p95349 / 552 ms$ per 1k$0.005
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 349 / 552 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- mercury-decide.pooled.json (Tacit fit on pooled dev)
9Respan Span-02API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.029 / .023Brier raw / cal.076 / .078Latency p50 / p95302 / 1228 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 302 / 1228 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.pooled.json (Tacit fit on pooled dev)
9Cloudflare ClefAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI92.3 [90.1–94.0]ECE raw / cal.019 / .020Brier raw / cal.114 / .111Latency p50 / p95338 / 658 ms$ per 1k$0.021
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 338 / 658 ms on the provider endpoint; $0.021 per 1,000 decisions
- Calibration file
- clef.pooled.json (Tacit fit on pooled dev)
Tacit fit, pooled: temperature and threshold are fitted per question type on the public dev splits of all 4 languages together (2,852 items), then frozen and applied once to the German hidden test. This is the ranking column.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 1%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage87.5%Automated656Errors6Realised error, 95% interval0.91% [0.34–1.98]Holds at 1%Accuracy, 95% CI96.7 [95.1–97.7]ECE raw / cal.014 / .013Brier raw / cal.056 / .053Latency p50 / p95539 / 817 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
2Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage85.1%Automated638Errors4Realised error, 95% interval0.63% [0.17–1.60]Holds at 1%Accuracy, 95% CI95.7 [94.0–97.0]ECE raw / cal.011 / .011Brier raw / cal.061 / .058Latency p50 / p95272 / 330 ms$ per 1k$0.010
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 272 / 330 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- d1-ultra.pooled.json (Tacit fit on pooled dev)
3DREX MaxAPI, size not disclosedOpenRouterCertified coverage82.5%Automated619Errors1Realised error, 95% interval0.16% [0.00–0.90]Holds at 1%Accuracy, 95% CI95.9 [94.2–97.1]ECE raw / cal.022 / .015Brier raw / cal.064 / .062Latency p50 / p952044 / 4457 ms$ per 1k$0.016
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2044 / 4457 ms on the provider endpoint; $0.016 per 1,000 decisions
- Calibration file
- drex-max.pooled.json (Tacit fit on pooled dev)
4pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage77.9%Automated584Errors2Realised error, 95% interval0.34% [0.04–1.23]Holds at 1%Accuracy, 95% CI95.1 [93.3–96.4]ECE raw / cal.017 / .006Brier raw / cal.063 / .063Latency p50 / p95298 / 379 ms$ per 1k$0.005
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 298 / 379 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.pooled.json (Tacit fit on pooled dev)
5Jev 1.14API, size not disclosedOpenRouterCertified coverage77.5%Automated581Errors1Realised error, 95% interval0.17% [0.00–0.96]Holds at 1%Accuracy, 95% CI94.7 [92.8–96.1]ECE raw / cal.023 / .011Brier raw / cal.076 / .077Latency p50 / p95265 / 329 ms$ per 1k$0.022
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 265 / 329 ms on the provider endpoint; $0.022 per 1,000 decisions
- Calibration file
- jev-1.14.pooled.json (Tacit fit on pooled dev)
6Mercury DecideAPI, size not disclosedOpenRouterCertified coverage74.1%Automated556Errors2Realised error, 95% interval0.36% [0.04–1.29]Holds at 1%Accuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.046 / .025Brier raw / cal.107 / .096Latency p50 / p95349 / 552 ms$ per 1k$0.005
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 349 / 552 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- mercury-decide.pooled.json (Tacit fit on pooled dev)
7basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage72.0%Automated540Errors3Realised error, 95% interval0.56% [0.11–1.61]Holds at 1%Accuracy, 95% CI92.4 [90.3–94.1]ECE raw / cal.027 / .015Brier raw / cal.101 / .098Latency p50 / p95227 / 352 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
8basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage71.6%Automated537Errors6Realised error, 95% interval1.12% [0.41–2.42]Misses the 1% targetAccuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.028 / .010Brier raw / cal.104 / .100Latency p50 / p95208 / 311 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
9Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage64.0%Automated480Errors1Realised error, 95% interval0.21% [0.01–1.16]Holds at 1%Accuracy, 95% CI94.1 [92.2–95.6]ECE raw / cal.024 / .015Brier raw / cal.090 / .087Latency p50 / p95335 / 395 ms$ per 1k$0.009
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 335 / 395 ms on the provider endpoint; $0.009 per 1,000 decisions
- Calibration file
- microsoft-decision-1.pooled.json (Tacit fit on pooled dev)
10Mammoth-1API, size not disclosedBaseten Model APIsCertified coverage63.1%Automated473Errors2Realised error, 95% interval0.42% [0.05–1.52]Holds at 1%Accuracy, 95% CI96.0 [94.3–97.2]ECE raw / cal.011 / .013Brier raw / cal.063 / .063Latency p50 / p95251 / 467 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 251 / 467 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.pooled.json (Tacit fit on pooled dev)
11Elephant-1API, size not disclosedOpenRouterCertified coverage54.0%Automated405Errors1Realised error, 95% interval0.25% [0.01–1.37]Holds at 1%Accuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.022 / .016Brier raw / cal.086 / .084Latency p50 / p95269 / 358 ms$ per 1k$0.010
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 269 / 358 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- elephant-1.pooled.json (Tacit fit on pooled dev)
12GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage47.5%Automated356Errors3Realised error, 95% interval0.84% [0.17–2.44]Holds at 1%Accuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.021 / .027Brier raw / cal.107 / .107Latency p50 / p95295 / 466 ms$ per 1k$0.031
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 295 / 466 ms on the provider endpoint; $0.031 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.pooled.json (Tacit fit on pooled dev)
13Respan Span-02API, size not disclosedOpenRouterCertified coverage42.8%Automated321Errors1Realised error, 95% interval0.31% [0.01–1.72]Holds at 1%Accuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.029 / .023Brier raw / cal.076 / .078Latency p50 / p95302 / 1228 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 302 / 1228 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.pooled.json (Tacit fit on pooled dev)
14Liquid d1.1API, size not disclosedOpenRouterCertified coverage35.9%Automated269Errors0Realised error, 95% interval0.00% [0.00–1.36]Holds at 1%Accuracy, 95% CI92.9 [90.9–94.6]ECE raw / cal.012 / .008Brier raw / cal.097 / .097Latency p50 / p951068 / 5038 ms$ per 1k$0.005
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 1068 / 5038 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- d1.1.pooled.json (Tacit fit on pooled dev)
15basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage26.3%Automated197Errors0Realised error, 95% interval0.00% [0.00–1.86]Holds at 1%Accuracy, 95% CI90.5 [88.2–92.4]ECE raw / cal.049 / .023Brier raw / cal.148 / .135Latency p50 / p95313 / 466 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-max.basal233.pooled.json (Tacit fit on pooled dev)
16basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage25.9%Automated194Errors0Realised error, 95% interval0.00% [0.00–1.88]Holds at 1%Accuracy, 95% CI89.9 [87.5–91.8]ECE raw / cal.051 / .026Brier raw / cal.149 / .137Latency p50 / p9562 / 83 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
17Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage24.5%Automated184Errors0Realised error, 95% interval0.00% [0.00–1.98]Holds at 1%Accuracy, 95% CI92.3 [90.1–94.0]ECE raw / cal.019 / .020Brier raw / cal.114 / .111Latency p50 / p95338 / 658 ms$ per 1k$0.021
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 338 / 658 ms on the provider endpoint; $0.021 per 1,000 decisions
- Calibration file
- clef.pooled.json (Tacit fit on pooled dev)
18basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage10.7%Automated80Errors0Realised error, 95% interval0.00% [0.00–4.51]Holds at 1%Accuracy, 95% CI78.0 [74.9–80.8]ECE raw / cal.065 / .035Brier raw / cal.295 / .284Latency p50 / p9565 / 91 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-mini.basal233.pooled.json (Tacit fit on pooled dev)
19basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI84.8 [82.1–87.2]ECE raw / cal.096 / .035Brier raw / cal.242 / .216Latency p50 / p95155 / 220 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-4.5B.basal233.pooled.json (Tacit fit on pooled dev)
19basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI79.9 [76.8–82.6]ECE raw / cal.103 / .039Brier raw / cal.290 / .264Latency p50 / p95168 / 244 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-omni-pico.basal233.pooled.json (Tacit fit on pooled dev)
Per-language fit, a secondary view: the same protocol fitted on the German dev split alone (750 items). It shows how far a language-specific calibration differs from the pooled one.
No model is certified at 1% yet. Every decision goes to a person.
The per-language fit sees only the German dev split: 300 yes/no, 294 choice and 156 score items. A 1% bound needs at least 299 accepted dev decisions of one type, all of them correct, so this fit can usually certify 5% only. The table still shows each model's accuracy and calibration.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 1%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1DREX MaxAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI95.9 [94.2–97.1]ECE raw / cal.022 / .015Brier raw / cal.064 / .062Latency p50 / p952044 / 4457 ms$ per 1k$0.016
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2044 / 4457 ms on the provider endpoint; $0.016 per 1,000 decisions
- Calibration file
- drex-max.per-language.json (Tacit fit on German dev)
1basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI96.7 [95.1–97.7]ECE raw / cal.014 / .013Brier raw / cal.056 / .053Latency p50 / p95539 / 817 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1Mammoth-1API, size not disclosedBaseten Model APIsCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI96.0 [94.3–97.2]ECE raw / cal.011 / .013Brier raw / cal.063 / .063Latency p50 / p95251 / 467 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 251 / 467 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.per-language.json (Tacit fit on German dev)
1Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.1 [92.2–95.6]ECE raw / cal.024 / .015Brier raw / cal.090 / .087Latency p50 / p95335 / 395 ms$ per 1k$0.009
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 335 / 395 ms on the provider endpoint; $0.009 per 1,000 decisions
- Calibration file
- microsoft-decision-1.per-language.json (Tacit fit on German dev)
1Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI95.7 [94.0–97.0]ECE raw / cal.011 / .011Brier raw / cal.061 / .058Latency p50 / p95272 / 330 ms$ per 1k$0.010
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 272 / 330 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- d1-ultra.per-language.json (Tacit fit on German dev)
1Jev 1.14API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.7 [92.8–96.1]ECE raw / cal.023 / .011Brier raw / cal.076 / .077Latency p50 / p95265 / 329 ms$ per 1k$0.022
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 265 / 329 ms on the provider endpoint; $0.022 per 1,000 decisions
- Calibration file
- jev-1.14.per-language.json (Tacit fit on German dev)
1pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI95.1 [93.3–96.4]ECE raw / cal.017 / .006Brier raw / cal.063 / .063Latency p50 / p95298 / 379 ms$ per 1k$0.005
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 298 / 379 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.per-language.json (Tacit fit on German dev)
1GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.021 / .027Brier raw / cal.107 / .107Latency p50 / p95295 / 466 ms$ per 1k$0.031
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 295 / 466 ms on the provider endpoint; $0.031 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.per-language.json (Tacit fit on German dev)
1Liquid d1.1API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI92.9 [90.9–94.6]ECE raw / cal.012 / .008Brier raw / cal.097 / .097Latency p50 / p951068 / 5038 ms$ per 1k$0.005
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 1068 / 5038 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- d1.1.per-language.json (Tacit fit on German dev)
1Elephant-1API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.022 / .016Brier raw / cal.086 / .084Latency p50 / p95269 / 358 ms$ per 1k$0.010
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 269 / 358 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- elephant-1.per-language.json (Tacit fit on German dev)
1basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.028 / .010Brier raw / cal.104 / .100Latency p50 / p95208 / 311 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI92.4 [90.3–94.1]ECE raw / cal.027 / .015Brier raw / cal.101 / .098Latency p50 / p95227 / 352 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1Mercury DecideAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.046 / .025Brier raw / cal.107 / .096Latency p50 / p95349 / 552 ms$ per 1k$0.005
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 349 / 552 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- mercury-decide.per-language.json (Tacit fit on German dev)
1Respan Span-02API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.029 / .023Brier raw / cal.076 / .078Latency p50 / p95302 / 1228 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 302 / 1228 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.per-language.json (Tacit fit on German dev)
1basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI90.5 [88.2–92.4]ECE raw / cal.049 / .023Brier raw / cal.148 / .135Latency p50 / p95313 / 466 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI89.9 [87.5–91.8]ECE raw / cal.051 / .026Brier raw / cal.149 / .137Latency p50 / p9562 / 83 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI92.3 [90.1–94.0]ECE raw / cal.019 / .020Brier raw / cal.114 / .111Latency p50 / p95338 / 658 ms$ per 1k$0.021
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 338 / 658 ms on the provider endpoint; $0.021 per 1,000 decisions
- Calibration file
- clef.per-language.json (Tacit fit on German dev)
1basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI84.8 [82.1–87.2]ECE raw / cal.096 / .035Brier raw / cal.242 / .216Latency p50 / p95155 / 220 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI79.9 [76.8–82.6]ECE raw / cal.103 / .039Brier raw / cal.290 / .264Latency p50 / p95168 / 244 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI78.0 [74.9–80.8]ECE raw / cal.065 / .035Brier raw / cal.295 / .284Latency p50 / p9565 / 91 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
Vendor file: the calibration file the model author ships, applied unchanged. Only 8 of 20 models ship one. A model can reach a coverage and still miss the target on test; that is reported, not hidden.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 1%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage66.8%Automated501Errors3Realised error, 95% interval0.60% [0.12–1.74]Holds at 1%Accuracy, 95% CI96.7 [95.1–97.7]ECE raw / cal.014 / .013Brier raw / cal.056 / .053Latency p50 / p95539 / 817 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
2basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage57.1%Automated428Errors3Realised error, 95% interval0.70% [0.14–2.03]Holds at 1%Accuracy, 95% CI92.4 [90.3–94.1]ECE raw / cal.027 / .015Brier raw / cal.101 / .098Latency p50 / p95227 / 352 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
3basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage55.7%Automated418Errors3Realised error, 95% interval0.72% [0.15–2.08]Holds at 1%Accuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.028 / .010Brier raw / cal.104 / .100Latency p50 / p95208 / 311 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
4basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage50.7%Automated380Errors4Realised error, 95% interval1.05% [0.29–2.67]Misses the 1% targetAccuracy, 95% CI89.9 [87.5–91.8]ECE raw / cal.051 / .026Brier raw / cal.149 / .137Latency p50 / p9562 / 83 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
5basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage23.3%Automated175Errors3Realised error, 95% interval1.71% [0.35–4.93]Misses the 1% targetAccuracy, 95% CI79.9 [76.8–82.6]ECE raw / cal.103 / .039Brier raw / cal.290 / .264Latency p50 / p95168 / 244 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI90.5 [88.2–92.4]ECE raw / cal.049 / .023Brier raw / cal.148 / .135Latency p50 / p95313 / 466 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI84.8 [82.1–87.2]ECE raw / cal.096 / .035Brier raw / cal.242 / .216Latency p50 / p95155 / 220 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI78.0 [74.9–80.8]ECE raw / cal.065 / .035Brier raw / cal.295 / .284Latency p50 / p9565 / 91 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
9DREX MaxAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.9 [94.2–97.1]ECE raw / cal.022 / .015Brier raw / cal.064 / .062Latency p50 / p952044 / 4457 ms$ per 1k$0.016
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 2044 / 4457 ms on the provider endpoint; $0.016 per 1,000 decisions
- Calibration file
- drex-max.pooled.json (Tacit fit on pooled dev)
9Mammoth-1API, size not disclosedBaseten Model APIsCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.0 [94.3–97.2]ECE raw / cal.011 / .013Brier raw / cal.063 / .063Latency p50 / p95251 / 467 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 251 / 467 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.pooled.json (Tacit fit on pooled dev)
9Microsoft-Decision-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.1 [92.2–95.6]ECE raw / cal.024 / .015Brier raw / cal.090 / .087Latency p50 / p95335 / 395 ms$ per 1k$0.009
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 335 / 395 ms on the provider endpoint; $0.009 per 1,000 decisions
- Calibration file
- microsoft-decision-1.pooled.json (Tacit fit on pooled dev)
9Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.7 [94.0–97.0]ECE raw / cal.011 / .011Brier raw / cal.061 / .058Latency p50 / p95272 / 330 ms$ per 1k$0.010
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 272 / 330 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- d1-ultra.pooled.json (Tacit fit on pooled dev)
9Jev 1.14API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.7 [92.8–96.1]ECE raw / cal.023 / .011Brier raw / cal.076 / .077Latency p50 / p95265 / 329 ms$ per 1k$0.022
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 265 / 329 ms on the provider endpoint; $0.022 per 1,000 decisions
- Calibration file
- jev-1.14.pooled.json (Tacit fit on pooled dev)
9pplx-decider-v1.2API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.1 [93.3–96.4]ECE raw / cal.017 / .006Brier raw / cal.063 / .063Latency p50 / p95298 / 379 ms$ per 1k$0.005
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 298 / 379 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.pooled.json (Tacit fit on pooled dev)
9GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI92.5 [90.4–94.2]ECE raw / cal.021 / .027Brier raw / cal.107 / .107Latency p50 / p95295 / 466 ms$ per 1k$0.031
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 295 / 466 ms on the provider endpoint; $0.031 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.pooled.json (Tacit fit on pooled dev)
9Liquid d1.1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI92.9 [90.9–94.6]ECE raw / cal.012 / .008Brier raw / cal.097 / .097Latency p50 / p951068 / 5038 ms$ per 1k$0.005
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 1068 / 5038 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- d1.1.pooled.json (Tacit fit on pooled dev)
9Elephant-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.7 [91.8–95.3]ECE raw / cal.022 / .016Brier raw / cal.086 / .084Latency p50 / p95269 / 358 ms$ per 1k$0.010
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 269 / 358 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- elephant-1.pooled.json (Tacit fit on pooled dev)
9Mercury DecideAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.046 / .025Brier raw / cal.107 / .096Latency p50 / p95349 / 552 ms$ per 1k$0.005
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 349 / 552 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- mercury-decide.pooled.json (Tacit fit on pooled dev)
9Respan Span-02API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.029 / .023Brier raw / cal.076 / .078Latency p50 / p95302 / 1228 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 302 / 1228 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.pooled.json (Tacit fit on pooled dev)
9Cloudflare ClefAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI92.3 [90.1–94.0]ECE raw / cal.019 / .020Brier raw / cal.114 / .111Latency p50 / p95338 / 658 ms$ per 1k$0.021
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 338 / 658 ms on the provider endpoint; $0.021 per 1,000 decisions
- Calibration file
- clef.pooled.json (Tacit fit on pooled dev)
Automated shareDeferred to a personRealised error and 95% intervalError target
Accuracy by question type
Accuracy per question type on the German hidden test. Thresholds are fitted per type, so a model can be certified for one type and defer every decision of another.
| Model | Yes/no n=300 | Choice n=294 | Score n=156 |
|---|---|---|---|
| DREX Max | 94 | 97.3 | 96.8 |
| basal-omni-1.0-ultra (basal 2.3.3) | 94 | 98 | 99.4 |
| Mammoth-1 | 94.7 | 97.6 | 95.5 |
| Microsoft-Decision-1 | 92 | 94.6 | 97.4 |
| Liquid d1-ultra | 93 | 97.6 | 97.4 |
| Jev 1.14 | 91.7 | 97.3 | 95.5 |
| pplx-decider-v1.2 | 92.7 | 96.6 | 96.8 |
| GPT-6 Luna Decisions | 90 | 92.9 | 96.8 |
| Liquid d1.1 | 91.7 | 94.2 | 92.9 |
| Elephant-1 | 92 | 95.6 | 93.6 |
| basal-omni-1.1-max (basal 2.3.3) | 90 | 94.9 | 94.2 |
| basal-omni-1.0-max (basal 2.3.3) | 90 | 94.2 | 93.6 |
| Mercury Decide | 90.3 | 93.5 | 96.2 |
| Respan Span-02 | 94 | 97.3 | 98.1 |
| basal-2.0-max (basal 2.3.3) | 86.7 | 93.9 | 91.7 |
| basal-omni-1.0-mini (basal 2.3.3) | 87 | 91.8 | 91.7 |
| Cloudflare Clef | 88.7 | 95.2 | 93.6 |
| basal-2.0-4.5B (basal 2.3.3) | 80.7 | 91.5 | 80.1 |
| basal-omni-pico (basal 2.3.3) | 77.3 | 85.7 | 73.7 |
| basal-2.0-mini (basal 2.3.3) | 74.3 | 85.4 | 71.2 |
Accuracy in %, hidden test.
Accuracy by family
19 families, 30 to 80 test items each. Darker means more accurate. Small families move a lot with one item, so read single cells with care.
FamilyDREX Maxomni-1.0-ultra (basal 2.3.3)Mammoth-1Microsoft-Decision-1Liquid d1-ultraJev 1.14pplx-decider-v1.2GPT-6 Luna DecisionsLiquid d1.1Elephant-1omni-1.1-max (basal 2.3.3)omni-1.0-max (basal 2.3.3)Mercury DecideRespan Span-022.0-max (basal 2.3.3)omni-1.0-mini (basal 2.3.3)Cloudflare Clef2.0-4.5B (basal 2.3.3)omni-pico (basal 2.3.3)2.0-mini (basal 2.3.3)
public administrationpublic_admin, n=4092.592.590.090.092.592.595.090.090.087.592.592.592.597.587.585.092.585.080.072.5
bank / AMLbank_aml, n=4092.590.090.097.592.590.087.590.090.090.092.592.587.595.090.097.585.085.082.575.0
cybersecuritycybersecurity, n=8096.398.893.892.597.596.395.090.092.593.890.091.388.895.091.388.892.587.577.573.8
escalationescalation, n=30100.093.3100.096.7100.093.3100.096.793.396.793.393.3100.093.390.096.796.793.393.386.7
HRhr, n=4097.597.597.595.090.095.097.595.097.592.592.595.095.0100.092.592.592.580.080.082.5
customer irritationirritation, n=30100.0100.0100.096.7100.0100.0100.0100.096.7100.0100.0100.0100.0100.0100.093.3100.0100.093.386.7
IT securityit_security, n=4097.592.590.090.097.587.597.582.582.585.082.577.587.590.080.077.582.570.060.065.0
law firmlaw_firm, n=4092.597.592.592.582.595.090.090.092.595.087.585.092.5100.092.585.087.580.077.580.0
completenesscompleteness, n=3096.7100.0100.096.7100.0100.096.796.7100.0100.096.796.796.796.793.383.3100.080.073.373.3
logisticslogistics, n=4097.597.597.595.095.097.595.095.092.592.592.590.090.092.592.590.095.090.085.072.5
moderationmoderation, n=4090.095.095.090.097.590.090.095.095.090.092.592.592.595.090.087.585.082.572.577.5
phishingphishing, n=30100.0100.0100.0100.0100.093.3100.0100.0100.096.7100.0100.0100.0100.0100.096.793.393.393.386.7
manufacturingmanufacturing, n=40100.097.5100.0100.097.5100.090.090.090.092.592.592.590.097.582.582.592.572.570.072.5
clinicclinic_admin, n=4092.595.092.592.592.587.595.080.092.5100.090.087.587.595.090.090.087.577.575.080.0
complaintscomplaint, n=30100.0100.0100.096.7100.0100.0100.096.796.796.796.796.7100.096.790.093.3100.086.780.083.3
routingrouting, n=4097.5100.0100.097.5100.0100.097.5100.0100.097.5100.0100.0100.0100.097.5100.0100.097.597.5100.0
insuranceinsurance, n=4090.095.092.585.097.595.090.092.585.087.587.587.587.595.087.587.592.580.080.082.5
universityuniversity, n=4097.595.0100.095.092.595.097.597.590.092.592.592.590.092.585.090.092.582.580.075.0
housing communityhousing, n=4095.0100.0100.095.097.592.597.590.095.0100.0100.0100.097.597.592.595.092.592.577.567.5