Leaderboard v0.1
This is the frozen v0.1 leaderboard: Polish only, fitted on the Polish dev split. It is kept as history and does not change. Current leaderboard, v0.2
Each row is the share of 652 hidden Polish test decisions a model takes on its own, with a Clopper-Pearson bound on its error. Ranked by coverage at the selected target. There is no composite score.
As of 20 October 2026, Liquid d1-ultra leads Tacit v0.1 with 94.2% certified coverage at a 5% error target (14 errors in 614 automated decisions, realised error 2.28%), ahead of Jev 1.14 at 92.0%. No model is certified at a 1% error target yet. 16 models are ranked.
- Benchmark
- v0.1, frozen
- Language
- Polish
- Last updated
- 20 October 2026 (what changed)
- Protocol
- tacit-devfit-1
- Hidden test
- n = 652
- Runs
- by the maintainer, no vendor approval
- Current
- v0.2
Error target
Calibration
Tacit fit: temperature and threshold are fitted per question type on the public dev split, then frozen and applied once to the hidden test.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 5%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage94.2%Automated614Errors14Realised error, 95% interval2.28% [1.25–3.80]Holds at 5%Accuracy, 95% CI96.9 [95.3–98.0]ECE raw / cal.012 / .007Brier raw / cal.051 / .049Latency p50 / p95297 / 580 ms$ per 1k$0.011
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 580 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- d1-ultra.json (Tacit fit on dev)
2Jev 1.14API, size not disclosedOpenRouterCertified coverage92.0%Automated600Errors20Realised error, 95% interval3.33% [2.05–5.10]Holds at 5%Accuracy, 95% CI94.9 [93.0–96.4]ECE raw / cal.022 / .013Brier raw / cal.076 / .075Latency p50 / p95287 / 402 ms$ per 1k$0.024
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 287 / 402 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.14.json (Tacit fit on dev)
3basal-omni-1.0-ultra31B, Gemma 4Apple M5 Max, localCertified coverage91.6%Automated597Errors9Realised error, 95% interval1.51% [0.69–2.84]Holds at 5%Accuracy, 95% CI95.6 [93.7–96.9]ECE raw / cal.024 / .016Brier raw / cal.060 / .056Latency p50 / p95549 / 963 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
e347c65- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-omni-1.0-ultra.json (Tacit fit on dev)
4pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage90.6%Automated591Errors14Realised error, 95% interval2.37% [1.30–3.94]Holds at 5%Accuracy, 95% CI95.2 [93.3–96.6]ECE raw / cal.017 / .011Brier raw / cal.069 / .068Latency p50 / p95350 / 448 ms$ per 1k$0.006
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 350 / 448 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.json (Tacit fit on dev)
4Mercury DecideAPI, size not disclosedOpenRouterCertified coverage90.6%Automated591Errors18Realised error, 95% interval3.05% [1.81–4.77]Holds at 5%Accuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.040 / .027Brier raw / cal.098 / .089Latency p50 / p95327 / 781 ms$ per 1k$0.006
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 327 / 781 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- mercury-decide.json (Tacit fit on dev)
6Jev 1.13API, size not disclosedOpenRouterCertified coverage89.7%Automated585Errors20Realised error, 95% interval3.42% [2.10–5.23]Holds at 5%Accuracy, 95% CI94.3 [92.3–95.9]ECE raw / cal.022 / .021Brier raw / cal.083 / .085Latency p50 / p95310 / 403 ms$ per 1k$0.024
- Model id
typesafe/jev-1.13- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 310 / 403 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.13.json (Tacit fit on dev)
7Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage89.1%Automated581Errors19Realised error, 95% interval3.27% [1.98–5.06]Holds at 5%Accuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.042 / .020Brier raw / cal.091 / .086Latency p50 / p95346 / 455 ms$ per 1k$0.010
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 346 / 455 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- microsoft-decision-1.json (Tacit fit on dev)
8basal-omni-1.1-max12B, Gemma 4Apple M5 Max, localCertified coverage87.1%Automated568Errors11Realised error, 95% interval1.94% [0.97–3.44]Holds at 5%Accuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.015 / .010Brier raw / cal.080 / .079Latency p50 / p95225 / 409 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-omni-1.1-max.json (Tacit fit on dev)
9basal-omni-1.0-max12B, Gemma 4Apple M5 Max, localCertified coverage86.0%Automated561Errors9Realised error, 95% interval1.60% [0.74–3.02]Holds at 5%Accuracy, 95% CI94.6 [92.6–96.1]ECE raw / cal.012 / .004Brier raw / cal.079 / .077Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-omni-1.0-max.json (Tacit fit on dev)
10basal-2.0-max11B, Bielik v3Apple M5 Max, localCertified coverage85.6%Automated558Errors27Realised error, 95% interval4.84% [3.21–6.96]Holds at 5%Accuracy, 95% CI91.3 [88.8–93.2]ECE raw / cal.050 / .025Brier raw / cal.133 / .122Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-2.0-max.json (Tacit fit on dev)
11Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage84.4%Automated550Errors11Realised error, 95% interval2.00% [1.00–3.55]Holds at 5%Accuracy, 95% CI94.2 [92.1–95.7]ECE raw / cal.016 / .019Brier raw / cal.087 / .086Latency p50 / p95322 / 474 ms$ per 1k$0.018
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 322 / 474 ms on the provider endpoint; $0.018 per 1,000 decisions
- Calibration file
- clef.json (Tacit fit on dev)
12Respan Span-02API, size not disclosedOpenRouterCertified coverage84.2%Automated549Errors17Realised error, 95% interval3.10% [1.81–4.91]Holds at 5%Accuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.035 / .016Brier raw / cal.108 / .102Latency p50 / p95296 / 2262 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 296 / 2262 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.json (Tacit fit on dev)
13basal-2.0-4.5B4.5B, Bielik v3Apple M5 Max, localCertified coverage69.2%Automated451Errors10Realised error, 95% interval2.22% [1.07–4.04]Holds at 5%Accuracy, 95% CI87.0 [84.2–89.3]ECE raw / cal.085 / .037Brier raw / cal.200 / .171Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-2.0-4.5B.json (Tacit fit on dev)
14basal-omni-1.0-mini4B, Gemma 4 E4BApple M5 Max, localCertified coverage64.9%Automated423Errors11Realised error, 95% interval2.60% [1.31–4.61]Holds at 5%Accuracy, 95% CI90.2 [87.7–92.2]ECE raw / cal.049 / .021Brier raw / cal.145 / .135Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-omni-1.0-mini.json (Tacit fit on dev)
15basal-2.0-mini1.5B, Bielik v3Apple M5 Max, localCertified coverage50.6%Automated330Errors14Realised error, 95% interval4.24% [2.34–7.02]Holds at 5%Accuracy, 95% CI83.3 [80.2–85.9]ECE raw / cal.080 / .024Brier raw / cal.258 / .238Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-2.0-mini.json (Tacit fit on dev)
16GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage47.5%Automated310Errors9Realised error, 95% interval2.90% [1.34–5.44]Holds at 5%Accuracy, 95% CI95.1 [93.2–96.5]ECE raw / cal.012 / .053Brier raw / cal.076 / .101Latency p50 / p95297 / 405 ms$ per 1k$0.036
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 405 ms on the provider endpoint; $0.036 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.json (Tacit fit on dev)
Vendor file: the calibration file the model author ships, applied unchanged. Only 7 of 16 models ship one. A model can reach a coverage and still miss the target on test; that is reported, not hidden.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 5%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1basal-omni-1.0-ultra31B, Gemma 4Apple M5 Max, localCertified coverage99.7%Automated650Errors27Realised error, 95% interval4.15% [2.75–5.99]Holds at 5%Accuracy, 95% CI95.6 [93.7–96.9]ECE raw / cal.024 / .016Brier raw / cal.060 / .056Latency p50 / p95549 / 963 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
e347c65- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
2basal-omni-1.1-max12B, Gemma 4Apple M5 Max, localCertified coverage92.9%Automated606Errors17Realised error, 95% interval2.81% [1.64–4.45]Holds at 5%Accuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.015 / .010Brier raw / cal.080 / .079Latency p50 / p95225 / 409 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
3basal-omni-1.0-max12B, Gemma 4Apple M5 Max, localCertified coverage91.9%Automated599Errors16Realised error, 95% interval2.67% [1.53–4.30]Holds at 5%Accuracy, 95% CI94.6 [92.6–96.1]ECE raw / cal.012 / .004Brier raw / cal.079 / .077Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
4basal-omni-1.0-mini4B, Gemma 4 E4BApple M5 Max, localCertified coverage87.0%Automated567Errors29Realised error, 95% interval5.11% [3.45–7.26]Misses the 5% targetAccuracy, 95% CI90.2 [87.7–92.2]ECE raw / cal.049 / .021Brier raw / cal.145 / .135Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
5basal-2.0-max11B, Bielik v3Apple M5 Max, localCertified coverage69.2%Automated451Errors5Realised error, 95% interval1.11% [0.36–2.57]Holds at 5%Accuracy, 95% CI91.3 [88.8–93.2]ECE raw / cal.050 / .025Brier raw / cal.133 / .122Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-2.0-4.5B4.5B, Bielik v3Apple M5 Max, localCertified coverage57.5%Automated375Errors5Realised error, 95% interval1.33% [0.43–3.08]Holds at 5%Accuracy, 95% CI87.0 [84.2–89.3]ECE raw / cal.085 / .037Brier raw / cal.200 / .171Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
7basal-2.0-mini1.5B, Bielik v3Apple M5 Max, localCertified coverage22.4%Automated146Errors2Realised error, 95% interval1.37% [0.17–4.86]Holds at 5%Accuracy, 95% CI83.3 [80.2–85.9]ECE raw / cal.080 / .024Brier raw / cal.258 / .238Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
8Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.9 [95.3–98.0]ECE raw / cal.012 / .007Brier raw / cal.051 / .049Latency p50 / p95297 / 580 ms$ per 1k$0.011
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 580 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- d1-ultra.json (Tacit fit on dev)
8Jev 1.14API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.9 [93.0–96.4]ECE raw / cal.022 / .013Brier raw / cal.076 / .075Latency p50 / p95287 / 402 ms$ per 1k$0.024
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 287 / 402 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.14.json (Tacit fit on dev)
8pplx-decider-v1.2API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.2 [93.3–96.6]ECE raw / cal.017 / .011Brier raw / cal.069 / .068Latency p50 / p95350 / 448 ms$ per 1k$0.006
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 350 / 448 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.json (Tacit fit on dev)
8Mercury DecideAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.040 / .027Brier raw / cal.098 / .089Latency p50 / p95327 / 781 ms$ per 1k$0.006
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 327 / 781 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- mercury-decide.json (Tacit fit on dev)
8Jev 1.13API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.3 [92.3–95.9]ECE raw / cal.022 / .021Brier raw / cal.083 / .085Latency p50 / p95310 / 403 ms$ per 1k$0.024
- Model id
typesafe/jev-1.13- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 310 / 403 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.13.json (Tacit fit on dev)
8Microsoft-Decision-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.042 / .020Brier raw / cal.091 / .086Latency p50 / p95346 / 455 ms$ per 1k$0.010
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 346 / 455 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- microsoft-decision-1.json (Tacit fit on dev)
8Cloudflare ClefAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.2 [92.1–95.7]ECE raw / cal.016 / .019Brier raw / cal.087 / .086Latency p50 / p95322 / 474 ms$ per 1k$0.018
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 322 / 474 ms on the provider endpoint; $0.018 per 1,000 decisions
- Calibration file
- clef.json (Tacit fit on dev)
8Respan Span-02API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.035 / .016Brier raw / cal.108 / .102Latency p50 / p95296 / 2262 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 296 / 2262 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.json (Tacit fit on dev)
8GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.1 [93.2–96.5]ECE raw / cal.012 / .053Brier raw / cal.076 / .101Latency p50 / p95297 / 405 ms$ per 1k$0.036
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 405 ms on the provider endpoint; $0.036 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.json (Tacit fit on dev)
Tacit fit: temperature and threshold are fitted per question type on the public dev split, then frozen and applied once to the hidden test.
No model is certified at 1% yet. Every decision goes to a person.
Thresholds are fitted per question type, and a 1% bound needs at least 299 accepted dev decisions, all of them correct. The v0.1 dev split has only 224 choice and 47 score questions, so those two types cannot be certified at 1%, whatever the model. Yes/no questions (331 on dev) can be, but no model keeps its bound at or under 1% on them yet. The table still shows each model's accuracy and calibration.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 1%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI96.9 [95.3–98.0]ECE raw / cal.012 / .007Brier raw / cal.051 / .049Latency p50 / p95297 / 580 ms$ per 1k$0.011
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 580 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- d1-ultra.json (Tacit fit on dev)
1Jev 1.14API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.9 [93.0–96.4]ECE raw / cal.022 / .013Brier raw / cal.076 / .075Latency p50 / p95287 / 402 ms$ per 1k$0.024
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 287 / 402 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.14.json (Tacit fit on dev)
1basal-omni-1.0-ultra31B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI95.6 [93.7–96.9]ECE raw / cal.024 / .016Brier raw / cal.060 / .056Latency p50 / p95549 / 963 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
e347c65- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-omni-1.0-ultra.json (Tacit fit on dev)
1pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI95.2 [93.3–96.6]ECE raw / cal.017 / .011Brier raw / cal.069 / .068Latency p50 / p95350 / 448 ms$ per 1k$0.006
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 350 / 448 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.json (Tacit fit on dev)
1Mercury DecideAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.040 / .027Brier raw / cal.098 / .089Latency p50 / p95327 / 781 ms$ per 1k$0.006
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 327 / 781 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- mercury-decide.json (Tacit fit on dev)
1Jev 1.13API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.3 [92.3–95.9]ECE raw / cal.022 / .021Brier raw / cal.083 / .085Latency p50 / p95310 / 403 ms$ per 1k$0.024
- Model id
typesafe/jev-1.13- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 310 / 403 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.13.json (Tacit fit on dev)
1Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.042 / .020Brier raw / cal.091 / .086Latency p50 / p95346 / 455 ms$ per 1k$0.010
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 346 / 455 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- microsoft-decision-1.json (Tacit fit on dev)
1basal-omni-1.1-max12B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.015 / .010Brier raw / cal.080 / .079Latency p50 / p95225 / 409 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-omni-1.1-max.json (Tacit fit on dev)
1basal-omni-1.0-max12B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.6 [92.6–96.1]ECE raw / cal.012 / .004Brier raw / cal.079 / .077Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-omni-1.0-max.json (Tacit fit on dev)
1basal-2.0-max11B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI91.3 [88.8–93.2]ECE raw / cal.050 / .025Brier raw / cal.133 / .122Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-2.0-max.json (Tacit fit on dev)
1Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.2 [92.1–95.7]ECE raw / cal.016 / .019Brier raw / cal.087 / .086Latency p50 / p95322 / 474 ms$ per 1k$0.018
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 322 / 474 ms on the provider endpoint; $0.018 per 1,000 decisions
- Calibration file
- clef.json (Tacit fit on dev)
1Respan Span-02API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.035 / .016Brier raw / cal.108 / .102Latency p50 / p95296 / 2262 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 296 / 2262 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.json (Tacit fit on dev)
1basal-2.0-4.5B4.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI87.0 [84.2–89.3]ECE raw / cal.085 / .037Brier raw / cal.200 / .171Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-2.0-4.5B.json (Tacit fit on dev)
1basal-omni-1.0-mini4B, Gemma 4 E4BApple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI90.2 [87.7–92.2]ECE raw / cal.049 / .021Brier raw / cal.145 / .135Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-omni-1.0-mini.json (Tacit fit on dev)
1basal-2.0-mini1.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI83.3 [80.2–85.9]ECE raw / cal.080 / .024Brier raw / cal.258 / .238Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-2.0-mini.json (Tacit fit on dev)
1GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI95.1 [93.2–96.5]ECE raw / cal.012 / .053Brier raw / cal.076 / .101Latency p50 / p95297 / 405 ms$ per 1k$0.036
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 405 ms on the provider endpoint; $0.036 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.json (Tacit fit on dev)
Vendor file: the calibration file the model author ships, applied unchanged. Only 7 of 16 models ship one. A model can reach a coverage and still miss the target on test; that is reported, not hidden.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 1%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1basal-omni-1.0-ultra31B, Gemma 4Apple M5 Max, localCertified coverage81.7%Automated533Errors1Realised error, 95% interval0.19% [0.00–1.04]Holds at 1%Accuracy, 95% CI95.6 [93.7–96.9]ECE raw / cal.024 / .016Brier raw / cal.060 / .056Latency p50 / p95549 / 963 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
e347c65- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
2basal-omni-1.0-max12B, Gemma 4Apple M5 Max, localCertified coverage71.9%Automated469Errors5Realised error, 95% interval1.07% [0.35–2.47]Misses the 1% targetAccuracy, 95% CI94.6 [92.6–96.1]ECE raw / cal.012 / .004Brier raw / cal.079 / .077Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
3basal-omni-1.1-max12B, Gemma 4Apple M5 Max, localCertified coverage68.7%Automated448Errors3Realised error, 95% interval0.67% [0.14–1.94]Holds at 1%Accuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.015 / .010Brier raw / cal.080 / .079Latency p50 / p95225 / 409 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
4basal-omni-1.0-mini4B, Gemma 4 E4BApple M5 Max, localCertified coverage63.3%Automated413Errors4Realised error, 95% interval0.97% [0.26–2.46]Holds at 1%Accuracy, 95% CI90.2 [87.7–92.2]ECE raw / cal.049 / .021Brier raw / cal.145 / .135Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
5basal-2.0-max11B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI91.3 [88.8–93.2]ECE raw / cal.050 / .025Brier raw / cal.133 / .122Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
5basal-2.0-4.5B4.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI87.0 [84.2–89.3]ECE raw / cal.085 / .037Brier raw / cal.200 / .171Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
5basal-2.0-mini1.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI83.3 [80.2–85.9]ECE raw / cal.080 / .024Brier raw / cal.258 / .238Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
8Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.9 [95.3–98.0]ECE raw / cal.012 / .007Brier raw / cal.051 / .049Latency p50 / p95297 / 580 ms$ per 1k$0.011
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 580 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- d1-ultra.json (Tacit fit on dev)
8Jev 1.14API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.9 [93.0–96.4]ECE raw / cal.022 / .013Brier raw / cal.076 / .075Latency p50 / p95287 / 402 ms$ per 1k$0.024
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 287 / 402 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.14.json (Tacit fit on dev)
8pplx-decider-v1.2API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.2 [93.3–96.6]ECE raw / cal.017 / .011Brier raw / cal.069 / .068Latency p50 / p95350 / 448 ms$ per 1k$0.006
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 350 / 448 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.json (Tacit fit on dev)
8Mercury DecideAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.040 / .027Brier raw / cal.098 / .089Latency p50 / p95327 / 781 ms$ per 1k$0.006
- Model id
inception/mercury-decide- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 327 / 781 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- mercury-decide.json (Tacit fit on dev)
8Jev 1.13API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.3 [92.3–95.9]ECE raw / cal.022 / .021Brier raw / cal.083 / .085Latency p50 / p95310 / 403 ms$ per 1k$0.024
- Model id
typesafe/jev-1.13- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 310 / 403 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.13.json (Tacit fit on dev)
8Microsoft-Decision-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.042 / .020Brier raw / cal.091 / .086Latency p50 / p95346 / 455 ms$ per 1k$0.010
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 346 / 455 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- microsoft-decision-1.json (Tacit fit on dev)
8Cloudflare ClefAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.2 [92.1–95.7]ECE raw / cal.016 / .019Brier raw / cal.087 / .086Latency p50 / p95322 / 474 ms$ per 1k$0.018
- Model id
cloudflare/clef- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 322 / 474 ms on the provider endpoint; $0.018 per 1,000 decisions
- Calibration file
- clef.json (Tacit fit on dev)
8Respan Span-02API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.035 / .016Brier raw / cal.108 / .102Latency p50 / p95296 / 2262 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 296 / 2262 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.json (Tacit fit on dev)
8GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.1 [93.2–96.5]ECE raw / cal.012 / .053Brier raw / cal.076 / .101Latency p50 / p95297 / 405 ms$ per 1k$0.036
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Endpoint
- https://openrouter.ai/api/alpha/decisions (raw, alpha)
- Client
- Poland, sequential calls; latency = client wall clock incl. network
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 405 ms on the provider endpoint; $0.036 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.json (Tacit fit on dev)
Automated shareDeferred to a personRealised error and 95% intervalError target
Accuracy by question type
Certified at 5%: yes/no and choice. Score items are too few on dev to fit a threshold, so they are reported but not certified.
| Model | Yes/no n=322 | Choice n=310 | Score n=20 |
|---|---|---|---|
| Liquid d1-ultra | 95 | 98.7 | 100 |
| Jev 1.14 | 92.5 | 97.1 | 100 |
| basal-omni-1.0-ultra | 93.2 | 97.7 | 100 |
| pplx-decider-v1.2 | 92.9 | 97.4 | 100 |
| Mercury Decide | 88.8 | 97.7 | 95 |
| Jev 1.13 | 91.6 | 96.8 | 100 |
| Microsoft-Decision-1 | 92.2 | 97.1 | 100 |
| basal-omni-1.1-max | 92.5 | 96.8 | 100 |
| basal-omni-1.0-max | 91.9 | 97.1 | 100 |
| basal-2.0-max | 87.6 | 94.8 | 95 |
| Cloudflare Clef | 91 | 97.1 | 100 |
| Respan Span-02 | 89.8 | 96.5 | 100 |
| basal-2.0-4.5B | 81.4 | 92.3 | 95 |
| basal-omni-1.0-mini | 85.7 | 95.5 | 80 |
| basal-2.0-mini | 79.5 | 87.4 | 80 |
| GPT-6 Luna Decisions | 93.5 | 97.1 | 90 |
Accuracy in %, hidden test.
Accuracy by family
19 families, 27 to 76 test items each. Darker means more accurate. Small families move a lot with one item, so read single cells with care.
FamilyLiquid d1-ultraJev 1.14omni-1.0-ultrapplx-decider-v1.2Mercury DecideJev 1.13Microsoft-Decision-1omni-1.1-maxomni-1.0-max2.0-maxCloudflare ClefRespan Span-022.0-4.5Bomni-1.0-mini2.0-miniGPT-6 Luna Decisions
public administrationadministracja, n=2796.388.992.688.988.988.988.992.692.692.681.596.392.685.277.892.6
bank / AMLbank_aml, n=2792.681.581.588.974.188.988.985.285.281.585.277.881.592.685.288.9
cybersecuritycyberbezpieczenstwo, n=7697.496.196.197.494.794.797.492.192.193.492.196.177.688.269.797.4
escalationeskalacja, n=38100.097.4100.097.4100.097.494.797.497.492.194.792.197.497.494.7100.0
HRhr, n=2792.688.992.692.688.988.988.992.685.281.592.692.677.877.881.588.9
customer irritationirytacja, n=38100.097.494.7100.097.494.7100.094.797.489.5100.092.192.194.784.297.4
IT securityit_security, n=27100.096.396.396.396.396.396.3100.0100.092.696.392.681.588.977.896.3
law firmkancelaria, n=2892.992.992.992.992.992.996.4100.0100.082.192.996.489.389.385.792.9
completenesskompletnosc, n=38100.092.1100.089.586.889.594.797.497.484.297.494.773.784.268.494.7
logisticslogistyka, n=2889.389.389.389.389.389.385.789.385.789.385.792.985.782.182.185.7
moderationmoderacja, n=2892.989.392.992.989.392.992.985.785.7100.092.992.992.992.975.089.3
phishingphishing, n=38100.097.4100.0100.0100.097.4100.0100.0100.097.492.1100.0100.0100.097.4100.0
manufacturingprodukcja, n=2896.492.992.996.489.389.389.389.389.389.392.989.385.782.182.192.9
clinicprzychodnia, n=2892.9100.092.996.489.3100.0100.096.496.489.396.489.378.685.775.0100.0
complaintsreklamacja, n=38100.0100.0100.0100.0100.0100.097.4100.0100.0100.0100.0100.097.4100.097.4100.0
routingrouting, n=54100.0100.0100.098.196.3100.098.1100.0100.098.1100.0100.098.1100.096.3100.0
insuranceubezpieczenia, n=2892.992.992.989.392.989.389.392.989.389.392.982.175.082.171.492.9
universityuczelnia, n=2896.4100.096.496.492.9100.096.492.9100.089.3100.089.389.378.692.992.9
housing communitywspolnota, n=28100.0100.0100.096.4100.092.992.996.496.489.396.489.382.196.489.389.3