Leaderboard
Each row is the share of 750 hidden English test decisions a model takes on its own, with a Clopper-Pearson bound on its error. Ranked by coverage at the selected target. There is no composite score. Each language is scored separately.
As of 28 October 2026, DREX Max leads Tacit v0.2 in English with 98.7% certified coverage at a 5% error target (23 errors in 740 automated decisions, realised error 3.11%), ahead of basal-omni-1.0-ultra (basal 2.3.3) at 98.1%. 20 models are ranked.
- Benchmark
- v0.2
- Language
- English
- Last updated
- 28 October 2026 (what changed)
- Protocol
- tacit-devfit-1, pooled fit
- Hidden test
- n = 750
- Runs
- by the maintainer, no vendor approval
- Earlier
- v0.1, Polish
Error target
Calibration
Tacit fit, pooled: temperature and threshold are fitted per question type on the public dev splits of all 4 languages together (2,852 items), then frozen and applied once to the English hidden test. This is the ranking column.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 5%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1DREX MaxAPI, size not disclosedOpenRouterCertified coverage98.7%Automated740Errors23Realised error, 95% interval3.11% [1.98–4.63]Holds at 5%Accuracy, 95% CI96.4 [94.8–97.5]ECE raw / cal.016 / .012Brier raw / cal.057 / .056Latency p50 / p951039 / 4649 ms$ per 1k$0.014
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 1039 / 4649 ms on the provider endpoint; $0.014 per 1,000 decisions
- Calibration file
- drex-max.pooled.json (Tacit fit on pooled dev)
2basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage98.1%Automated736Errors11Realised error, 95% interval1.49% [0.75–2.66]Holds at 5%Accuracy, 95% CI97.9 [96.6–98.7]ECE raw / cal.008 / .014Brier raw / cal.032 / .032Latency p50 / p95504 / 750 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
3Mammoth-1API, size not disclosedBaseten Model APIsCertified coverage98.0%Automated735Errors16Realised error, 95% interval2.18% [1.25–3.51]Holds at 5%Accuracy, 95% CI96.7 [95.1–97.7]ECE raw / cal.022 / .011Brier raw / cal.050 / .047Latency p50 / p95247 / 448 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 247 / 448 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.pooled.json (Tacit fit on pooled dev)
4Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage97.6%Automated732Errors20Realised error, 95% interval2.73% [1.68–4.19]Holds at 5%Accuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.012 / .020Brier raw / cal.060 / .061Latency p50 / p95331 / 392 ms$ per 1k$0.008
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 331 / 392 ms on the provider endpoint; $0.008 per 1,000 decisions
- Calibration file
- microsoft-decision-1.pooled.json (Tacit fit on pooled dev)
5Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage97.5%Automated731Errors24Realised error, 95% interval3.28% [2.11–4.85]Holds at 5%Accuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.013 / .010Brier raw / cal.059 / .060Latency p50 / p95271 / 352 ms$ per 1k$0.008
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 271 / 352 ms on the provider endpoint; $0.008 per 1,000 decisions
- Calibration file
- d1-ultra.pooled.json (Tacit fit on pooled dev)
6pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage96.5%Automated724Errors28Realised error, 95% interval3.87% [2.58–5.54]Holds at 5%Accuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.010 / .009Brier raw / cal.074 / .073Latency p50 / p95294 / 385 ms$ per 1k$0.005
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 294 / 385 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.pooled.json (Tacit fit on pooled dev)
7Jev 1.14API, size not disclosedOpenRouterCertified coverage95.7%Automated718Errors20Realised error, 95% interval2.79% [1.71–4.27]Holds at 5%Accuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.032 / .012Brier raw / cal.060 / .058Latency p50 / p95251 / 308 ms$ per 1k$0.019
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 251 / 308 ms on the provider endpoint; $0.019 per 1,000 decisions
- Calibration file
- jev-1.14.pooled.json (Tacit fit on pooled dev)
8GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage95.6%Automated717Errors20Realised error, 95% interval2.79% [1.71–4.28]Holds at 5%Accuracy, 95% CI95.7 [94.0–97.0]ECE raw / cal.015 / .030Brier raw / cal.061 / .068Latency p50 / p95288 / 475 ms$ per 1k$0.028
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 288 / 475 ms on the provider endpoint; $0.028 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.pooled.json (Tacit fit on pooled dev)
9Elephant-1API, size not disclosedOpenRouterCertified coverage94.4%Automated708Errors24Realised error, 95% interval3.39% [2.18–5.00]Holds at 5%Accuracy, 95% CI94.0 [92.1–95.5]ECE raw / cal.032 / .020Brier raw / cal.074 / .073Latency p50 / p95269 / 363 ms$ per 1k$0.009
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 269 / 363 ms on the provider endpoint; $0.009 per 1,000 decisions
- Calibration file
- elephant-1.pooled.json (Tacit fit on pooled dev)
10basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage93.1%Automated698Errors18Realised error, 95% interval2.58% [1.54–4.05]Holds at 5%Accuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.012 / .017Brier raw / cal.075 / .072Latency p50 / p95201 / 292 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
11basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage92.3%Automated692Errors20Realised error, 95% interval2.89% [1.77–4.43]Holds at 5%Accuracy, 95% CI94.5 [92.7–95.9]ECE raw / cal.016 / .028Brier raw / cal.078 / .076Latency p50 / p95182 / 268 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
12Liquid d1.1API, size not disclosedOpenRouterCertified coverage92.1%Automated691Errors20Realised error, 95% interval2.89% [1.78–4.43]Holds at 5%Accuracy, 95% CI94.7 [92.8–96.1]ECE raw / cal.014 / .011Brier raw / cal.077 / .077Latency p50 / p95609 / 4770 ms$ per 1k$0.004
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 609 / 4770 ms on the provider endpoint; $0.004 per 1,000 decisions
- Calibration file
- d1.1.pooled.json (Tacit fit on pooled dev)
13Mercury DecideAPI, size not disclosedOpenRouterCertified coverage91.5%Automated686Errors18Realised error, 95% interval2.62% [1.56–4.12]Holds at 5%Accuracy, 95% CI94.3 [92.4–95.7]ECE raw / cal.036 / .012Brier raw / cal.093 / .085Latency p50 / p95343 / 568 ms$ per 1k$0.004
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 343 / 568 ms on the provider endpoint; $0.004 per 1,000 decisions
- Calibration file
- mercury-decide.pooled.json (Tacit fit on pooled dev)
14Respan Span-02API, size not disclosedOpenRouterCertified coverage90.0%Automated675Errors15Realised error, 95% interval2.22% [1.25–3.64]Holds at 5%Accuracy, 95% CI96.4 [94.8–97.5]ECE raw / cal.023 / .021Brier raw / cal.071 / .072Latency p50 / p95300 / 1264 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 300 / 1264 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.pooled.json (Tacit fit on pooled dev)
15basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage85.6%Automated642Errors17Realised error, 95% interval2.65% [1.55–4.21]Holds at 5%Accuracy, 95% CI91.5 [89.2–93.3]ECE raw / cal.045 / .022Brier raw / cal.128 / .116Latency p50 / p9557 / 74 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
16basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage84.7%Automated635Errors15Realised error, 95% interval2.36% [1.33–3.87]Holds at 5%Accuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.036 / .017Brier raw / cal.112 / .102Latency p50 / p95255 / 376 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-max.basal233.pooled.json (Tacit fit on pooled dev)
17Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage59.2%Automated444Errors10Realised error, 95% interval2.25% [1.09–4.10]Holds at 5%Accuracy, 95% CI92.9 [90.9–94.6]ECE raw / cal.019 / .025Brier raw / cal.104 / .102Latency p50 / p95338 / 628 ms$ per 1k$0.017
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 338 / 628 ms on the provider endpoint; $0.017 per 1,000 decisions
- Calibration file
- clef.pooled.json (Tacit fit on pooled dev)
18basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage45.5%Automated341Errors3Realised error, 95% interval0.88% [0.18–2.55]Holds at 5%Accuracy, 95% CI89.6 [87.2–91.6]ECE raw / cal.062 / .024Brier raw / cal.168 / .150Latency p50 / p95143 / 188 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-4.5B.basal233.pooled.json (Tacit fit on pooled dev)
19basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage44.8%Automated336Errors12Realised error, 95% interval3.57% [1.86–6.16]Holds at 5%Accuracy, 95% CI83.3 [80.5–85.8]ECE raw / cal.093 / .033Brier raw / cal.252 / .231Latency p50 / p95165 / 223 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-omni-pico.basal233.pooled.json (Tacit fit on pooled dev)
20basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage39.5%Automated296Errors10Realised error, 95% interval3.38% [1.63–6.13]Holds at 5%Accuracy, 95% CI84.9 [82.2–87.3]ECE raw / cal.052 / .027Brier raw / cal.229 / .222Latency p50 / p9553 / 70 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-mini.basal233.pooled.json (Tacit fit on pooled dev)
Per-language fit, a secondary view: the same protocol fitted on the English dev split alone (750 items). It shows how far a language-specific calibration differs from the pooled one.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 5%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage96.3%Automated722Errors19Realised error, 95% interval2.63% [1.59–4.08]Holds at 5%Accuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.012 / .020Brier raw / cal.060 / .061Latency p50 / p95331 / 392 ms$ per 1k$0.008
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 331 / 392 ms on the provider endpoint; $0.008 per 1,000 decisions
2basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage95.7%Automated718Errors7Realised error, 95% interval0.97% [0.39–2.00]Holds at 5%Accuracy, 95% CI97.9 [96.6–98.7]ECE raw / cal.008 / .014Brier raw / cal.032 / .032Latency p50 / p95504 / 750 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
3DREX MaxAPI, size not disclosedOpenRouterCertified coverage94.9%Automated712Errors14Realised error, 95% interval1.97% [1.08–3.28]Holds at 5%Accuracy, 95% CI96.4 [94.8–97.5]ECE raw / cal.016 / .012Brier raw / cal.057 / .056Latency p50 / p951039 / 4649 ms$ per 1k$0.014
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 1039 / 4649 ms on the provider endpoint; $0.014 per 1,000 decisions
- Calibration file
- drex-max.per-language.json (Tacit fit on English dev)
3Mammoth-1API, size not disclosedBaseten Model APIsCertified coverage94.9%Automated712Errors10Realised error, 95% interval1.40% [0.68–2.57]Holds at 5%Accuracy, 95% CI96.7 [95.1–97.7]ECE raw / cal.022 / .011Brier raw / cal.050 / .047Latency p50 / p95247 / 448 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 247 / 448 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.per-language.json (Tacit fit on English dev)
5Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage94.0%Automated705Errors17Realised error, 95% interval2.41% [1.41–3.83]Holds at 5%Accuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.013 / .010Brier raw / cal.059 / .060Latency p50 / p95271 / 352 ms$ per 1k$0.008
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 271 / 352 ms on the provider endpoint; $0.008 per 1,000 decisions
- Calibration file
- d1-ultra.per-language.json (Tacit fit on English dev)
6pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage92.8%Automated696Errors19Realised error, 95% interval2.73% [1.65–4.23]Holds at 5%Accuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.010 / .009Brier raw / cal.074 / .073Latency p50 / p95294 / 385 ms$ per 1k$0.005
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 294 / 385 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.per-language.json (Tacit fit on English dev)
7Jev 1.14API, size not disclosedOpenRouterCertified coverage91.5%Automated686Errors11Realised error, 95% interval1.60% [0.80–2.85]Holds at 5%Accuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.032 / .012Brier raw / cal.060 / .058Latency p50 / p95251 / 308 ms$ per 1k$0.019
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 251 / 308 ms on the provider endpoint; $0.019 per 1,000 decisions
- Calibration file
- jev-1.14.per-language.json (Tacit fit on English dev)
8GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage87.6%Automated657Errors12Realised error, 95% interval1.83% [0.95–3.17]Holds at 5%Accuracy, 95% CI95.7 [94.0–97.0]ECE raw / cal.015 / .030Brier raw / cal.061 / .068Latency p50 / p95288 / 475 ms$ per 1k$0.028
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 288 / 475 ms on the provider endpoint; $0.028 per 1,000 decisions
9Elephant-1API, size not disclosedOpenRouterCertified coverage84.1%Automated631Errors19Realised error, 95% interval3.01% [1.82–4.66]Holds at 5%Accuracy, 95% CI94.0 [92.1–95.5]ECE raw / cal.032 / .020Brier raw / cal.074 / .073Latency p50 / p95269 / 363 ms$ per 1k$0.009
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 269 / 363 ms on the provider endpoint; $0.009 per 1,000 decisions
- Calibration file
- elephant-1.per-language.json (Tacit fit on English dev)
9basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage84.1%Automated631Errors4Realised error, 95% interval0.63% [0.17–1.62]Holds at 5%Accuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.012 / .017Brier raw / cal.075 / .072Latency p50 / p95201 / 292 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
11Mercury DecideAPI, size not disclosedOpenRouterCertified coverage82.7%Automated620Errors5Realised error, 95% interval0.81% [0.26–1.87]Holds at 5%Accuracy, 95% CI94.3 [92.4–95.7]ECE raw / cal.036 / .012Brier raw / cal.093 / .085Latency p50 / p95343 / 568 ms$ per 1k$0.004
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 343 / 568 ms on the provider endpoint; $0.004 per 1,000 decisions
- Calibration file
- mercury-decide.per-language.json (Tacit fit on English dev)
12basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage82.0%Automated615Errors5Realised error, 95% interval0.81% [0.26–1.89]Holds at 5%Accuracy, 95% CI94.5 [92.7–95.9]ECE raw / cal.016 / .028Brier raw / cal.078 / .076Latency p50 / p95182 / 268 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
12Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage82.0%Automated615Errors13Realised error, 95% interval2.11% [1.13–3.59]Holds at 5%Accuracy, 95% CI92.9 [90.9–94.6]ECE raw / cal.019 / .025Brier raw / cal.104 / .102Latency p50 / p95338 / 628 ms$ per 1k$0.017
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 338 / 628 ms on the provider endpoint; $0.017 per 1,000 decisions
- Calibration file
- clef.per-language.json (Tacit fit on English dev)
14Liquid d1.1API, size not disclosedOpenRouterCertified coverage78.9%Automated592Errors5Realised error, 95% interval0.84% [0.27–1.96]Holds at 5%Accuracy, 95% CI94.7 [92.8–96.1]ECE raw / cal.014 / .011Brier raw / cal.077 / .077Latency p50 / p95609 / 4770 ms$ per 1k$0.004
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 609 / 4770 ms on the provider endpoint; $0.004 per 1,000 decisions
- Calibration file
- d1.1.per-language.json (Tacit fit on English dev)
15basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage74.5%Automated559Errors6Realised error, 95% interval1.07% [0.39–2.32]Holds at 5%Accuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.036 / .017Brier raw / cal.112 / .102Latency p50 / p95255 / 376 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
16Respan Span-02API, size not disclosedOpenRouterCertified coverage72.7%Automated545Errors6Realised error, 95% interval1.10% [0.41–2.38]Holds at 5%Accuracy, 95% CI96.4 [94.8–97.5]ECE raw / cal.023 / .021Brier raw / cal.071 / .072Latency p50 / p95300 / 1264 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 300 / 1264 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.per-language.json (Tacit fit on English dev)
17basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage65.2%Automated489Errors7Realised error, 95% interval1.43% [0.58–2.93]Holds at 5%Accuracy, 95% CI91.5 [89.2–93.3]ECE raw / cal.045 / .022Brier raw / cal.128 / .116Latency p50 / p9557 / 74 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
18basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage59.7%Automated448Errors6Realised error, 95% interval1.34% [0.49–2.89]Holds at 5%Accuracy, 95% CI89.6 [87.2–91.6]ECE raw / cal.062 / .024Brier raw / cal.168 / .150Latency p50 / p95143 / 188 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
19basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage39.3%Automated295Errors10Realised error, 95% interval3.39% [1.64–6.15]Holds at 5%Accuracy, 95% CI83.3 [80.5–85.8]ECE raw / cal.093 / .033Brier raw / cal.252 / .231Latency p50 / p95165 / 223 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
20basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage25.9%Automated194Errors1Realised error, 95% interval0.52% [0.01–2.84]Holds at 5%Accuracy, 95% CI84.9 [82.2–87.3]ECE raw / cal.052 / .027Brier raw / cal.229 / .222Latency p50 / p9553 / 70 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
Vendor file: the calibration file the model author ships, applied unchanged. Only 8 of 20 models ship one. A model can reach a coverage and still miss the target on test; that is reported, not hidden.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 5%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage97.9%Automated734Errors13Realised error, 95% interval1.77% [0.95–3.01]Holds at 5%Accuracy, 95% CI97.9 [96.6–98.7]ECE raw / cal.008 / .014Brier raw / cal.032 / .032Latency p50 / p95504 / 750 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
2basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage92.3%Automated692Errors22Realised error, 95% interval3.18% [2.00–4.77]Holds at 5%Accuracy, 95% CI94.5 [92.7–95.9]ECE raw / cal.016 / .028Brier raw / cal.078 / .076Latency p50 / p95182 / 268 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
3basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage86.8%Automated651Errors26Realised error, 95% interval3.99% [2.63–5.80]Holds at 5%Accuracy, 95% CI91.5 [89.2–93.3]ECE raw / cal.045 / .022Brier raw / cal.128 / .116Latency p50 / p9557 / 74 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
4basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage84.5%Automated634Errors17Realised error, 95% interval2.68% [1.57–4.26]Holds at 5%Accuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.012 / .017Brier raw / cal.075 / .072Latency p50 / p95201 / 292 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
5basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage53.2%Automated399Errors3Realised error, 95% interval0.75% [0.16–2.18]Holds at 5%Accuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.036 / .017Brier raw / cal.112 / .102Latency p50 / p95255 / 376 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage52.3%Automated392Errors26Realised error, 95% interval6.63% [4.38–9.57]Misses the 5% targetAccuracy, 95% CI83.3 [80.5–85.8]ECE raw / cal.093 / .033Brier raw / cal.252 / .231Latency p50 / p95165 / 223 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
7basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage44.7%Automated335Errors5Realised error, 95% interval1.49% [0.49–3.45]Holds at 5%Accuracy, 95% CI89.6 [87.2–91.6]ECE raw / cal.062 / .024Brier raw / cal.168 / .150Latency p50 / p95143 / 188 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
8basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage19.2%Automated144Errors6Realised error, 95% interval4.17% [1.54–8.85]Holds at 5%Accuracy, 95% CI84.9 [82.2–87.3]ECE raw / cal.052 / .027Brier raw / cal.229 / .222Latency p50 / p9553 / 70 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
9DREX MaxAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.4 [94.8–97.5]ECE raw / cal.016 / .012Brier raw / cal.057 / .056Latency p50 / p951039 / 4649 ms$ per 1k$0.014
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 1039 / 4649 ms on the provider endpoint; $0.014 per 1,000 decisions
- Calibration file
- drex-max.pooled.json (Tacit fit on pooled dev)
9Mammoth-1API, size not disclosedBaseten Model APIsCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.7 [95.1–97.7]ECE raw / cal.022 / .011Brier raw / cal.050 / .047Latency p50 / p95247 / 448 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 247 / 448 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.pooled.json (Tacit fit on pooled dev)
9Microsoft-Decision-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.012 / .020Brier raw / cal.060 / .061Latency p50 / p95331 / 392 ms$ per 1k$0.008
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 331 / 392 ms on the provider endpoint; $0.008 per 1,000 decisions
- Calibration file
- microsoft-decision-1.pooled.json (Tacit fit on pooled dev)
9Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.013 / .010Brier raw / cal.059 / .060Latency p50 / p95271 / 352 ms$ per 1k$0.008
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 271 / 352 ms on the provider endpoint; $0.008 per 1,000 decisions
- Calibration file
- d1-ultra.pooled.json (Tacit fit on pooled dev)
9pplx-decider-v1.2API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.010 / .009Brier raw / cal.074 / .073Latency p50 / p95294 / 385 ms$ per 1k$0.005
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 294 / 385 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.pooled.json (Tacit fit on pooled dev)
9Jev 1.14API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.032 / .012Brier raw / cal.060 / .058Latency p50 / p95251 / 308 ms$ per 1k$0.019
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 251 / 308 ms on the provider endpoint; $0.019 per 1,000 decisions
- Calibration file
- jev-1.14.pooled.json (Tacit fit on pooled dev)
9GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.7 [94.0–97.0]ECE raw / cal.015 / .030Brier raw / cal.061 / .068Latency p50 / p95288 / 475 ms$ per 1k$0.028
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 288 / 475 ms on the provider endpoint; $0.028 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.pooled.json (Tacit fit on pooled dev)
9Elephant-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.0 [92.1–95.5]ECE raw / cal.032 / .020Brier raw / cal.074 / .073Latency p50 / p95269 / 363 ms$ per 1k$0.009
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 269 / 363 ms on the provider endpoint; $0.009 per 1,000 decisions
- Calibration file
- elephant-1.pooled.json (Tacit fit on pooled dev)
9Liquid d1.1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.7 [92.8–96.1]ECE raw / cal.014 / .011Brier raw / cal.077 / .077Latency p50 / p95609 / 4770 ms$ per 1k$0.004
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 609 / 4770 ms on the provider endpoint; $0.004 per 1,000 decisions
- Calibration file
- d1.1.pooled.json (Tacit fit on pooled dev)
9Mercury DecideAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.3 [92.4–95.7]ECE raw / cal.036 / .012Brier raw / cal.093 / .085Latency p50 / p95343 / 568 ms$ per 1k$0.004
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 343 / 568 ms on the provider endpoint; $0.004 per 1,000 decisions
- Calibration file
- mercury-decide.pooled.json (Tacit fit on pooled dev)
9Respan Span-02API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.4 [94.8–97.5]ECE raw / cal.023 / .021Brier raw / cal.071 / .072Latency p50 / p95300 / 1264 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 300 / 1264 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.pooled.json (Tacit fit on pooled dev)
9Cloudflare ClefAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI92.9 [90.9–94.6]ECE raw / cal.019 / .025Brier raw / cal.104 / .102Latency p50 / p95338 / 628 ms$ per 1k$0.017
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 338 / 628 ms on the provider endpoint; $0.017 per 1,000 decisions
- Calibration file
- clef.pooled.json (Tacit fit on pooled dev)
Tacit fit, pooled: temperature and threshold are fitted per question type on the public dev splits of all 4 languages together (2,852 items), then frozen and applied once to the English hidden test. This is the ranking column.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 1%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage87.7%Automated658Errors1Realised error, 95% interval0.15% [0.00–0.84]Holds at 1%Accuracy, 95% CI97.9 [96.6–98.7]ECE raw / cal.008 / .014Brier raw / cal.032 / .032Latency p50 / p95504 / 750 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
2Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage85.6%Automated642Errors4Realised error, 95% interval0.62% [0.17–1.59]Holds at 1%Accuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.013 / .010Brier raw / cal.059 / .060Latency p50 / p95271 / 352 ms$ per 1k$0.008
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 271 / 352 ms on the provider endpoint; $0.008 per 1,000 decisions
- Calibration file
- d1-ultra.pooled.json (Tacit fit on pooled dev)
3DREX MaxAPI, size not disclosedOpenRouterCertified coverage83.6%Automated627Errors5Realised error, 95% interval0.80% [0.26–1.85]Holds at 1%Accuracy, 95% CI96.4 [94.8–97.5]ECE raw / cal.016 / .012Brier raw / cal.057 / .056Latency p50 / p951039 / 4649 ms$ per 1k$0.014
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 1039 / 4649 ms on the provider endpoint; $0.014 per 1,000 decisions
- Calibration file
- drex-max.pooled.json (Tacit fit on pooled dev)
4Jev 1.14API, size not disclosedOpenRouterCertified coverage79.7%Automated598Errors0Realised error, 95% interval0.00% [0.00–0.61]Holds at 1%Accuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.032 / .012Brier raw / cal.060 / .058Latency p50 / p95251 / 308 ms$ per 1k$0.019
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 251 / 308 ms on the provider endpoint; $0.019 per 1,000 decisions
- Calibration file
- jev-1.14.pooled.json (Tacit fit on pooled dev)
5pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage77.6%Automated582Errors2Realised error, 95% interval0.34% [0.04–1.24]Holds at 1%Accuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.010 / .009Brier raw / cal.074 / .073Latency p50 / p95294 / 385 ms$ per 1k$0.005
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 294 / 385 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.pooled.json (Tacit fit on pooled dev)
6Mercury DecideAPI, size not disclosedOpenRouterCertified coverage75.9%Automated569Errors0Realised error, 95% interval0.00% [0.00–0.65]Holds at 1%Accuracy, 95% CI94.3 [92.4–95.7]ECE raw / cal.036 / .012Brier raw / cal.093 / .085Latency p50 / p95343 / 568 ms$ per 1k$0.004
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 343 / 568 ms on the provider endpoint; $0.004 per 1,000 decisions
- Calibration file
- mercury-decide.pooled.json (Tacit fit on pooled dev)
7basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage74.9%Automated562Errors0Realised error, 95% interval0.00% [0.00–0.65]Holds at 1%Accuracy, 95% CI94.5 [92.7–95.9]ECE raw / cal.016 / .028Brier raw / cal.078 / .076Latency p50 / p95182 / 268 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
8basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage74.7%Automated560Errors1Realised error, 95% interval0.18% [0.00–0.99]Holds at 1%Accuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.012 / .017Brier raw / cal.075 / .072Latency p50 / p95201 / 292 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
9Mammoth-1API, size not disclosedBaseten Model APIsCertified coverage63.2%Automated474Errors4Realised error, 95% interval0.84% [0.23–2.15]Holds at 1%Accuracy, 95% CI96.7 [95.1–97.7]ECE raw / cal.022 / .011Brier raw / cal.050 / .047Latency p50 / p95247 / 448 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 247 / 448 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.pooled.json (Tacit fit on pooled dev)
10Elephant-1API, size not disclosedOpenRouterCertified coverage58.7%Automated440Errors0Realised error, 95% interval0.00% [0.00–0.83]Holds at 1%Accuracy, 95% CI94.0 [92.1–95.5]ECE raw / cal.032 / .020Brier raw / cal.074 / .073Latency p50 / p95269 / 363 ms$ per 1k$0.009
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 269 / 363 ms on the provider endpoint; $0.009 per 1,000 decisions
- Calibration file
- elephant-1.pooled.json (Tacit fit on pooled dev)
11GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage49.5%Automated371Errors1Realised error, 95% interval0.27% [0.01–1.49]Holds at 1%Accuracy, 95% CI95.7 [94.0–97.0]ECE raw / cal.015 / .030Brier raw / cal.061 / .068Latency p50 / p95288 / 475 ms$ per 1k$0.028
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 288 / 475 ms on the provider endpoint; $0.028 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.pooled.json (Tacit fit on pooled dev)
12Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage46.0%Automated345Errors1Realised error, 95% interval0.29% [0.01–1.60]Holds at 1%Accuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.012 / .020Brier raw / cal.060 / .061Latency p50 / p95331 / 392 ms$ per 1k$0.008
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 331 / 392 ms on the provider endpoint; $0.008 per 1,000 decisions
- Calibration file
- microsoft-decision-1.pooled.json (Tacit fit on pooled dev)
13Respan Span-02API, size not disclosedOpenRouterCertified coverage42.0%Automated315Errors0Realised error, 95% interval0.00% [0.00–1.16]Holds at 1%Accuracy, 95% CI96.4 [94.8–97.5]ECE raw / cal.023 / .021Brier raw / cal.071 / .072Latency p50 / p95300 / 1264 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 300 / 1264 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.pooled.json (Tacit fit on pooled dev)
14Liquid d1.1API, size not disclosedOpenRouterCertified coverage33.3%Automated250Errors0Realised error, 95% interval0.00% [0.00–1.46]Holds at 1%Accuracy, 95% CI94.7 [92.8–96.1]ECE raw / cal.014 / .011Brier raw / cal.077 / .077Latency p50 / p95609 / 4770 ms$ per 1k$0.004
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 609 / 4770 ms on the provider endpoint; $0.004 per 1,000 decisions
- Calibration file
- d1.1.pooled.json (Tacit fit on pooled dev)
15Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage31.6%Automated237Errors0Realised error, 95% interval0.00% [0.00–1.54]Holds at 1%Accuracy, 95% CI92.9 [90.9–94.6]ECE raw / cal.019 / .025Brier raw / cal.104 / .102Latency p50 / p95338 / 628 ms$ per 1k$0.017
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 338 / 628 ms on the provider endpoint; $0.017 per 1,000 decisions
- Calibration file
- clef.pooled.json (Tacit fit on pooled dev)
16basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage26.9%Automated202Errors0Realised error, 95% interval0.00% [0.00–1.81]Holds at 1%Accuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.036 / .017Brier raw / cal.112 / .102Latency p50 / p95255 / 376 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-max.basal233.pooled.json (Tacit fit on pooled dev)
17basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage25.1%Automated188Errors1Realised error, 95% interval0.53% [0.01–2.93]Holds at 1%Accuracy, 95% CI91.5 [89.2–93.3]ECE raw / cal.045 / .022Brier raw / cal.128 / .116Latency p50 / p9557 / 74 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
18basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage15.6%Automated117Errors0Realised error, 95% interval0.00% [0.00–3.10]Holds at 1%Accuracy, 95% CI84.9 [82.2–87.3]ECE raw / cal.052 / .027Brier raw / cal.229 / .222Latency p50 / p9553 / 70 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-mini.basal233.pooled.json (Tacit fit on pooled dev)
19basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI89.6 [87.2–91.6]ECE raw / cal.062 / .024Brier raw / cal.168 / .150Latency p50 / p95143 / 188 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-2.0-4.5B.basal233.pooled.json (Tacit fit on pooled dev)
19basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI83.3 [80.5–85.8]ECE raw / cal.093 / .033Brier raw / cal.252 / .231Latency p50 / p95165 / 223 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-omni-pico.basal233.pooled.json (Tacit fit on pooled dev)
Per-language fit, a secondary view: the same protocol fitted on the English dev split alone (750 items). It shows how far a language-specific calibration differs from the pooled one.
No model is certified at 1% yet. Every decision goes to a person.
The per-language fit sees only the English dev split: 300 yes/no, 294 choice and 156 score items. A 1% bound needs at least 299 accepted dev decisions of one type, all of them correct, so this fit can usually certify 5% only. The table still shows each model's accuracy and calibration.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 1%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1DREX MaxAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI96.4 [94.8–97.5]ECE raw / cal.016 / .012Brier raw / cal.057 / .056Latency p50 / p951039 / 4649 ms$ per 1k$0.014
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 1039 / 4649 ms on the provider endpoint; $0.014 per 1,000 decisions
- Calibration file
- drex-max.per-language.json (Tacit fit on English dev)
1basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI97.9 [96.6–98.7]ECE raw / cal.008 / .014Brier raw / cal.032 / .032Latency p50 / p95504 / 750 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1Mammoth-1API, size not disclosedBaseten Model APIsCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI96.7 [95.1–97.7]ECE raw / cal.022 / .011Brier raw / cal.050 / .047Latency p50 / p95247 / 448 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 247 / 448 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.per-language.json (Tacit fit on English dev)
1Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.012 / .020Brier raw / cal.060 / .061Latency p50 / p95331 / 392 ms$ per 1k$0.008
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 331 / 392 ms on the provider endpoint; $0.008 per 1,000 decisions
1Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.013 / .010Brier raw / cal.059 / .060Latency p50 / p95271 / 352 ms$ per 1k$0.008
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 271 / 352 ms on the provider endpoint; $0.008 per 1,000 decisions
- Calibration file
- d1-ultra.per-language.json (Tacit fit on English dev)
1pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.010 / .009Brier raw / cal.074 / .073Latency p50 / p95294 / 385 ms$ per 1k$0.005
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 294 / 385 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.per-language.json (Tacit fit on English dev)
1Jev 1.14API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.032 / .012Brier raw / cal.060 / .058Latency p50 / p95251 / 308 ms$ per 1k$0.019
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 251 / 308 ms on the provider endpoint; $0.019 per 1,000 decisions
- Calibration file
- jev-1.14.per-language.json (Tacit fit on English dev)
1GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI95.7 [94.0–97.0]ECE raw / cal.015 / .030Brier raw / cal.061 / .068Latency p50 / p95288 / 475 ms$ per 1k$0.028
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 288 / 475 ms on the provider endpoint; $0.028 per 1,000 decisions
1Elephant-1API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.0 [92.1–95.5]ECE raw / cal.032 / .020Brier raw / cal.074 / .073Latency p50 / p95269 / 363 ms$ per 1k$0.009
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 269 / 363 ms on the provider endpoint; $0.009 per 1,000 decisions
- Calibration file
- elephant-1.per-language.json (Tacit fit on English dev)
1basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.012 / .017Brier raw / cal.075 / .072Latency p50 / p95201 / 292 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.5 [92.7–95.9]ECE raw / cal.016 / .028Brier raw / cal.078 / .076Latency p50 / p95182 / 268 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1Liquid d1.1API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.7 [92.8–96.1]ECE raw / cal.014 / .011Brier raw / cal.077 / .077Latency p50 / p95609 / 4770 ms$ per 1k$0.004
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 609 / 4770 ms on the provider endpoint; $0.004 per 1,000 decisions
- Calibration file
- d1.1.per-language.json (Tacit fit on English dev)
1Mercury DecideAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.3 [92.4–95.7]ECE raw / cal.036 / .012Brier raw / cal.093 / .085Latency p50 / p95343 / 568 ms$ per 1k$0.004
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 343 / 568 ms on the provider endpoint; $0.004 per 1,000 decisions
- Calibration file
- mercury-decide.per-language.json (Tacit fit on English dev)
1Respan Span-02API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI96.4 [94.8–97.5]ECE raw / cal.023 / .021Brier raw / cal.071 / .072Latency p50 / p95300 / 1264 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 300 / 1264 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.per-language.json (Tacit fit on English dev)
1basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI91.5 [89.2–93.3]ECE raw / cal.045 / .022Brier raw / cal.128 / .116Latency p50 / p9557 / 74 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.036 / .017Brier raw / cal.112 / .102Latency p50 / p95255 / 376 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI92.9 [90.9–94.6]ECE raw / cal.019 / .025Brier raw / cal.104 / .102Latency p50 / p95338 / 628 ms$ per 1k$0.017
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 338 / 628 ms on the provider endpoint; $0.017 per 1,000 decisions
- Calibration file
- clef.per-language.json (Tacit fit on English dev)
1basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI89.6 [87.2–91.6]ECE raw / cal.062 / .024Brier raw / cal.168 / .150Latency p50 / p95143 / 188 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI83.3 [80.5–85.8]ECE raw / cal.093 / .033Brier raw / cal.252 / .231Latency p50 / p95165 / 223 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
1basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI84.9 [82.2–87.3]ECE raw / cal.052 / .027Brier raw / cal.229 / .222Latency p50 / p9553 / 70 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
Vendor file: the calibration file the model author ships, applied unchanged. Only 8 of 20 models ship one. A model can reach a coverage and still miss the target on test; that is reported, not hidden.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 1%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4Apple M5 Max, localCertified coverage65.7%Automated493Errors0Realised error, 95% interval0.00% [0.00–0.75]Holds at 1%Accuracy, 95% CI97.9 [96.6–98.7]ECE raw / cal.008 / .014Brier raw / cal.032 / .032Latency p50 / p95504 / 750 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
6d5ba67- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
2basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage57.7%Automated433Errors1Realised error, 95% interval0.23% [0.01–1.28]Holds at 1%Accuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.012 / .017Brier raw / cal.075 / .072Latency p50 / p95201 / 292 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
3basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4Apple M5 Max, localCertified coverage55.6%Automated417Errors2Realised error, 95% interval0.48% [0.06–1.72]Holds at 1%Accuracy, 95% CI94.5 [92.7–95.9]ECE raw / cal.016 / .028Brier raw / cal.078 / .076Latency p50 / p95182 / 268 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
4basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4BApple M5 Max, localCertified coverage52.3%Automated392Errors3Realised error, 95% interval0.77% [0.16–2.22]Holds at 1%Accuracy, 95% CI91.5 [89.2–93.3]ECE raw / cal.045 / .022Brier raw / cal.128 / .116Latency p50 / p9557 / 74 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
5basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only)Apple M5 Max, localCertified coverage26.1%Automated196Errors2Realised error, 95% interval1.02% [0.12–3.64]Misses the 1% targetAccuracy, 95% CI83.3 [80.5–85.8]ECE raw / cal.093 / .033Brier raw / cal.252 / .231Latency p50 / p95165 / 223 ms$ per 1klocal
- Model id
Remek/basal-omni-pico- Revision
b31750a- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode eager --device mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-2.0-max (basal 2.3.3)11B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI92.8 [90.7–94.4]ECE raw / cal.036 / .017Brier raw / cal.112 / .102Latency p50 / p95255 / 376 ms$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI89.6 [87.2–91.6]ECE raw / cal.062 / .024Brier raw / cal.168 / .150Latency p50 / p95143 / 188 ms$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI84.9 [82.2–87.3]ECE raw / cal.052 / .027Brier raw / cal.229 / .222Latency p50 / p9553 / 70 ms$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.3.3 (v2.3.3 = 1eddd9a), basal-serve --mode mps --no-calibration, PyTorch MPS, bf16 full weights, batch size 1, one option order (harness adapter, 3 untimed warm-up requests per file)
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
9DREX MaxAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.4 [94.8–97.5]ECE raw / cal.016 / .012Brier raw / cal.057 / .056Latency p50 / p951039 / 4649 ms$ per 1k$0.014
- Model id
nace-ai/drex-max- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 1039 / 4649 ms on the provider endpoint; $0.014 per 1,000 decisions
- Calibration file
- drex-max.pooled.json (Tacit fit on pooled dev)
9Mammoth-1API, size not disclosedBaseten Model APIsCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.7 [95.1–97.7]ECE raw / cal.022 / .011Brier raw / cal.050 / .047Latency p50 / p95247 / 448 ms$ per 1knot billed
- Model id
baselabs/mammoth-1- Revision
not pinned (API)- Endpoint
- https://inference.baseten.co/v1/systemone (Baseten Model APIs)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 247 / 448 ms on the provider endpoint; not billed (free access until 2026-10-27)
- Calibration file
- mammoth-1.pooled.json (Tacit fit on pooled dev)
9Microsoft-Decision-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.012 / .020Brier raw / cal.060 / .061Latency p50 / p95331 / 392 ms$ per 1k$0.008
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 331 / 392 ms on the provider endpoint; $0.008 per 1,000 decisions
- Calibration file
- microsoft-decision-1.pooled.json (Tacit fit on pooled dev)
9Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.013 / .010Brier raw / cal.059 / .060Latency p50 / p95271 / 352 ms$ per 1k$0.008
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 271 / 352 ms on the provider endpoint; $0.008 per 1,000 decisions
- Calibration file
- d1-ultra.pooled.json (Tacit fit on pooled dev)
9pplx-decider-v1.2API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.8 [93.0–96.2]ECE raw / cal.010 / .009Brier raw / cal.074 / .073Latency p50 / p95294 / 385 ms$ per 1k$0.005
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 294 / 385 ms on the provider endpoint; $0.005 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.pooled.json (Tacit fit on pooled dev)
9Jev 1.14API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.1 [94.5–97.3]ECE raw / cal.032 / .012Brier raw / cal.060 / .058Latency p50 / p95251 / 308 ms$ per 1k$0.019
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 251 / 308 ms on the provider endpoint; $0.019 per 1,000 decisions
- Calibration file
- jev-1.14.pooled.json (Tacit fit on pooled dev)
9GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.7 [94.0–97.0]ECE raw / cal.015 / .030Brier raw / cal.061 / .068Latency p50 / p95288 / 475 ms$ per 1k$0.028
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 288 / 475 ms on the provider endpoint; $0.028 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.pooled.json (Tacit fit on pooled dev)
9Elephant-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.0 [92.1–95.5]ECE raw / cal.032 / .020Brier raw / cal.074 / .073Latency p50 / p95269 / 363 ms$ per 1k$0.009
- Model id
baseten/elephant-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 269 / 363 ms on the provider endpoint; $0.009 per 1,000 decisions
- Calibration file
- elephant-1.pooled.json (Tacit fit on pooled dev)
9Liquid d1.1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.7 [92.8–96.1]ECE raw / cal.014 / .011Brier raw / cal.077 / .077Latency p50 / p95609 / 4770 ms$ per 1k$0.004
- Model id
liquid/d1.1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 28 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 609 / 4770 ms on the provider endpoint; $0.004 per 1,000 decisions
- Calibration file
- d1.1.pooled.json (Tacit fit on pooled dev)
9Mercury DecideAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.3 [92.4–95.7]ECE raw / cal.036 / .012Brier raw / cal.093 / .085Latency p50 / p95343 / 568 ms$ per 1k$0.004
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 343 / 568 ms on the provider endpoint; $0.004 per 1,000 decisions
- Calibration file
- mercury-decide.pooled.json (Tacit fit on pooled dev)
9Respan Span-02API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.4 [94.8–97.5]ECE raw / cal.023 / .021Brier raw / cal.071 / .072Latency p50 / p95300 / 1264 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 300 / 1264 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.pooled.json (Tacit fit on pooled dev)
9Cloudflare ClefAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI92.9 [90.9–94.6]ECE raw / cal.019 / .025Brier raw / cal.104 / .102Latency p50 / p95338 / 628 ms$ per 1k$0.017
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls per model (different models ran concurrently); latency = client wall clock incl. network
- Test run date
- 22 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 338 / 628 ms on the provider endpoint; $0.017 per 1,000 decisions
- Calibration file
- clef.pooled.json (Tacit fit on pooled dev)
Automated shareDeferred to a personRealised error and 95% intervalError target
Accuracy by question type
Accuracy per question type on the English hidden test. Thresholds are fitted per type, so a model can be certified for one type and defer every decision of another.
| Model | Yes/no n=300 | Choice n=294 | Score n=156 |
|---|---|---|---|
| DREX Max | 93.7 | 97.3 | 100 |
| basal-omni-1.0-ultra (basal 2.3.3) | 96.3 | 98.3 | 100 |
| Mammoth-1 | 94.7 | 98.3 | 97.4 |
| Microsoft-Decision-1 | 94.3 | 97.3 | 97.4 |
| Liquid d1-ultra | 94.7 | 96.6 | 98.1 |
| pplx-decider-v1.2 | 91.7 | 95.9 | 98.7 |
| Jev 1.14 | 93.7 | 97.3 | 98.7 |
| GPT-6 Luna Decisions | 94.3 | 95.9 | 98.1 |
| Elephant-1 | 91.3 | 94.9 | 97.4 |
| basal-omni-1.0-max (basal 2.3.3) | 92 | 96.3 | 97.4 |
| basal-omni-1.1-max (basal 2.3.3) | 92.7 | 95.6 | 96.2 |
| Liquid d1.1 | 92 | 95.9 | 97.4 |
| Mercury Decide | 91.3 | 95.6 | 97.4 |
| Respan Span-02 | 95.3 | 98 | 95.5 |
| basal-omni-1.0-mini (basal 2.3.3) | 86 | 94.6 | 96.2 |
| basal-2.0-max (basal 2.3.3) | 87.3 | 96.3 | 96.8 |
| Cloudflare Clef | 90.3 | 94.2 | 95.5 |
| basal-2.0-4.5B (basal 2.3.3) | 84.3 | 94.6 | 90.4 |
| basal-omni-pico (basal 2.3.3) | 79.3 | 88.8 | 80.8 |
| basal-2.0-mini (basal 2.3.3) | 82 | 89.1 | 82.7 |
Accuracy in %, hidden test.
Accuracy by family
19 families, 30 to 80 test items each. Darker means more accurate. Small families move a lot with one item, so read single cells with care.
FamilyDREX Maxomni-1.0-ultra (basal 2.3.3)Mammoth-1Microsoft-Decision-1Liquid d1-ultrapplx-decider-v1.2Jev 1.14GPT-6 Luna DecisionsElephant-1omni-1.0-max (basal 2.3.3)omni-1.1-max (basal 2.3.3)Liquid d1.1Mercury DecideRespan Span-02omni-1.0-mini (basal 2.3.3)2.0-max (basal 2.3.3)Cloudflare Clef2.0-4.5B (basal 2.3.3)omni-pico (basal 2.3.3)2.0-mini (basal 2.3.3)
public administrationpublic_admin, n=4097.597.597.595.095.090.092.597.587.592.592.592.592.595.095.092.595.087.582.582.5
bank / AMLbank_aml, n=4097.597.595.095.092.597.590.097.597.597.597.590.095.0100.097.595.092.597.580.097.5
cybersecuritycybersecurity, n=8091.397.595.096.396.392.596.395.092.593.893.896.392.597.590.092.591.387.576.381.3
escalationescalation, n=30100.0100.0100.096.7100.0100.0100.0100.096.796.796.796.7100.096.796.790.096.793.396.790.0
HRhr, n=4092.597.595.092.592.585.092.597.587.595.092.592.587.597.585.092.585.082.570.075.0
customer irritationirritation, n=30100.0100.0100.0100.0100.096.7100.0100.0100.0100.0100.096.796.7100.096.796.7100.093.396.780.0
IT securityit_security, n=4095.0100.097.595.0100.095.097.597.587.592.592.587.592.595.090.090.092.587.585.087.5
law firmlaw_firm, n=4095.0100.095.095.092.5100.097.590.095.090.092.597.597.590.092.592.590.092.597.590.0
completenesscompleteness, n=3093.3100.096.796.7100.0100.0100.0100.0100.0100.0100.096.793.393.386.790.096.780.066.773.3
logisticslogistics, n=40100.097.597.595.097.597.595.092.595.0100.0100.095.097.595.097.595.095.090.085.075.0
moderationmoderation, n=4097.595.095.095.097.595.092.595.092.590.087.592.595.092.585.095.090.085.075.077.5
phishingphishing, n=30100.0100.0100.093.3100.096.796.7100.096.7100.0100.096.7100.0100.096.7100.093.396.790.0100.0
manufacturingmanufacturing, n=4097.595.097.597.592.592.597.592.595.095.092.597.592.595.085.090.092.582.582.585.0
clinicclinic_admin, n=4095.095.092.595.095.090.097.590.095.085.085.090.085.095.087.582.587.587.585.087.5
complaintscomplaint, n=30100.0100.0100.096.7100.096.7100.0100.0100.096.796.7100.096.7100.096.7100.0100.0100.086.796.7
routingrouting, n=40100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.095.097.5
insuranceinsurance, n=4092.597.595.0100.095.092.597.595.087.592.595.095.097.597.585.090.090.087.585.087.5
universityuniversity, n=4095.092.597.595.090.092.590.090.090.095.092.592.592.597.590.095.092.592.585.080.0
housing communityhousing, n=40100.0100.095.097.595.097.597.595.097.595.095.095.092.595.090.087.592.585.075.077.5