# Tacit leaderboard v0.1

Source: https://tacitbench.ai/leaderboard/ (machine-readable: https://tacitbench.ai/results.json, https://tacitbench.ai/results.csv). Last updated 20 October 2026.

As of 20 October 2026, Liquid d1-ultra leads Tacit v0.1 with 94.2% certified coverage at a 5% error target (14 errors in 614 automated decisions, realised error 2.28%), ahead of Jev 1.14 at 92.0%. No model is certified at a 1% error target yet. 16 models are ranked.

Hidden test, n = 652. Ranked by certified coverage at a 5% error target under the Tacit fit (temperature and threshold fitted on the public dev split, frozen, applied once to test). There is no composite score. All runs by the maintainer; no vendor approved the numbers.

| # | Model | Size / base | Runs on | Coverage @5% | Automated | Errors | Realised error [95% CI] | Verdict | Accuracy [95% CI] | ECE raw / cal | Brier raw / cal | Latency p50 / p95 | $ per 1k |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | Liquid d1-ultra | API, size not disclosed | OpenRouter | 94.2% | 614 | 14 | 2.28% [1.25–3.80] | Holds at 5% | 96.9 [95.3–98.0] | .012 / .007 | .051 / .049 | 297 / 580 ms | $0.011 |
| 2 | Jev 1.14 | API, size not disclosed | OpenRouter | 92.0% | 600 | 20 | 3.33% [2.05–5.10] | Holds at 5% | 94.9 [93.0–96.4] | .022 / .013 | .076 / .075 | 287 / 402 ms | $0.024 |
| 3 | basal-omni-1.0-ultra | 31B, Gemma 4 | Apple M5 Max, local | 91.6% | 597 | 9 | 1.51% [0.69–2.84] | Holds at 5% | 95.6 [93.7–96.9] | .024 / .016 | .060 / .056 | 549 / 963 ms | local |
| 4 | pplx-decider-v1.2 | API, size not disclosed | OpenRouter | 90.6% | 591 | 14 | 2.37% [1.30–3.94] | Holds at 5% | 95.2 [93.3–96.6] | .017 / .011 | .069 / .068 | 350 / 448 ms | $0.006 |
| 5 | Mercury Decide | API, size not disclosed | OpenRouter | 90.6% | 591 | 18 | 3.05% [1.81–4.77] | Holds at 5% | 93.3 [91.1–94.9] | .040 / .027 | .098 / .089 | 327 / 781 ms | $0.006 |
| 6 | Jev 1.13 | API, size not disclosed | OpenRouter | 89.7% | 585 | 20 | 3.42% [2.10–5.23] | Holds at 5% | 94.3 [92.3–95.9] | .022 / .021 | .083 / .085 | 310 / 403 ms | $0.024 |
| 7 | Microsoft-Decision-1 | API, size not disclosed | OpenRouter | 89.1% | 581 | 19 | 3.27% [1.98–5.06] | Holds at 5% | 94.8 [92.8–96.2] | .042 / .020 | .091 / .086 | 346 / 455 ms | $0.010 |
| 8 | basal-omni-1.1-max | 12B, Gemma 4 | Apple M5 Max, local | 87.1% | 568 | 11 | 1.94% [0.97–3.44] | Holds at 5% | 94.8 [92.8–96.2] | .015 / .010 | .080 / .079 | 225 / 409 ms | local |
| 9 | basal-omni-1.0-max | 12B, Gemma 4 | Apple M5 Max, local | 86.0% | 561 | 9 | 1.60% [0.74–3.02] | Holds at 5% | 94.6 [92.6–96.1] | .012 / .004 | .079 / .077 | not timed | local |
| 10 | basal-2.0-max | 11B, Bielik v3 | Apple M5 Max, local | 85.6% | 558 | 27 | 4.84% [3.21–6.96] | Holds at 5% | 91.3 [88.8–93.2] | .050 / .025 | .133 / .122 | not timed | local |
| 11 | Cloudflare Clef | API, size not disclosed | OpenRouter | 84.4% | 550 | 11 | 2.00% [1.00–3.55] | Holds at 5% | 94.2 [92.1–95.7] | .016 / .019 | .087 / .086 | 322 / 474 ms | $0.018 |
| 12 | Respan Span-02 | API, size not disclosed | OpenRouter | 84.2% | 549 | 17 | 3.10% [1.81–4.91] | Holds at 5% | 93.3 [91.1–94.9] | .035 / .016 | .108 / .102 | 296 / 2262 ms | $0.003 |
| 13 | basal-2.0-4.5B | 4.5B, Bielik v3 | Apple M5 Max, local | 69.2% | 451 | 10 | 2.22% [1.07–4.04] | Holds at 5% | 87.0 [84.2–89.3] | .085 / .037 | .200 / .171 | not timed | local |
| 14 | basal-omni-1.0-mini | 4B, Gemma 4 E4B | Apple M5 Max, local | 64.9% | 423 | 11 | 2.60% [1.31–4.61] | Holds at 5% | 90.2 [87.7–92.2] | .049 / .021 | .145 / .135 | not timed | local |
| 15 | basal-2.0-mini | 1.5B, Bielik v3 | Apple M5 Max, local | 50.6% | 330 | 14 | 4.24% [2.34–7.02] | Holds at 5% | 83.3 [80.2–85.9] | .080 / .024 | .258 / .238 | not timed | local |
| 16 | GPT-6 Luna Decisions | API, size not disclosed | OpenRouter | 47.5% | 310 | 9 | 2.90% [1.34–5.44] | Holds at 5% | 95.1 [93.2–96.5] | .012 / .053 | .076 / .101 | 297 / 405 ms | $0.036 |

## 1% error target

No model is certified at a 1% error target under the Tacit fit. Every decision goes to a person.

## Vendor calibration files (applied unchanged)

- basal-omni-1.0-ultra: 5%: 99.7%, error 4.15%, holds at 5%; 1%: 81.7%, error 0.19%, holds at 1%
- basal-omni-1.1-max: 5%: 92.9%, error 2.81%, holds at 5%; 1%: 68.7%, error 0.67%, holds at 1%
- basal-omni-1.0-max: 5%: 91.9%, error 2.67%, holds at 5%; 1%: 71.9%, error 1.07%, misses the 1% target
- basal-2.0-max: 5%: 69.2%, error 1.11%, holds at 5%; 1%: not certified
- basal-2.0-4.5B: 5%: 57.5%, error 1.33%, holds at 5%; 1%: not certified
- basal-omni-1.0-mini: 5%: 87.0%, error 5.11%, misses the 5% target; 1%: 63.3%, error 0.97%, holds at 1%
- basal-2.0-mini: 5%: 22.4%, error 1.37%, holds at 5%; 1%: not certified

## Accuracy by question type (%, hidden test)

| Model | Yes/no | Choice | Score |
| --- | ---: | ---: | ---: |
| Liquid d1-ultra | 95.0 | 98.7 | 100.0 |
| Jev 1.14 | 92.5 | 97.1 | 100.0 |
| basal-omni-1.0-ultra | 93.2 | 97.7 | 100.0 |
| pplx-decider-v1.2 | 92.9 | 97.4 | 100.0 |
| Mercury Decide | 88.8 | 97.7 | 95.0 |
| Jev 1.13 | 91.6 | 96.8 | 100.0 |
| Microsoft-Decision-1 | 92.2 | 97.1 | 100.0 |
| basal-omni-1.1-max | 92.5 | 96.8 | 100.0 |
| basal-omni-1.0-max | 91.9 | 97.1 | 100.0 |
| basal-2.0-max | 87.6 | 94.8 | 95.0 |
| Cloudflare Clef | 91.0 | 97.1 | 100.0 |
| Respan Span-02 | 89.8 | 96.5 | 100.0 |
| basal-2.0-4.5B | 81.4 | 92.3 | 95.0 |
| basal-omni-1.0-mini | 85.7 | 95.5 | 80.0 |
| basal-2.0-mini | 79.5 | 87.4 | 80.0 |
| GPT-6 Luna Decisions | 93.5 | 97.1 | 90.0 |

## Accuracy by family (%, hidden test)

| Family | Liquid d1-ultra | Jev 1.14 | omni-1.0-ultra | pplx-decider-v1.2 | Mercury Decide | Jev 1.13 | Microsoft-Decision-1 | omni-1.1-max | omni-1.0-max | 2.0-max | Cloudflare Clef | Respan Span-02 | 2.0-4.5B | omni-1.0-mini | 2.0-mini | GPT-6 Luna Decisions |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| public administration (administracja) | 96.3 | 88.9 | 92.6 | 88.9 | 88.9 | 88.9 | 88.9 | 92.6 | 92.6 | 92.6 | 81.5 | 96.3 | 92.6 | 85.2 | 77.8 | 92.6 |
| bank / AML (bank_aml) | 92.6 | 81.5 | 81.5 | 88.9 | 74.1 | 88.9 | 88.9 | 85.2 | 85.2 | 81.5 | 85.2 | 77.8 | 81.5 | 92.6 | 85.2 | 88.9 |
| cybersecurity (cyberbezpieczenstwo) | 97.4 | 96.1 | 96.1 | 97.4 | 94.7 | 94.7 | 97.4 | 92.1 | 92.1 | 93.4 | 92.1 | 96.1 | 77.6 | 88.2 | 69.7 | 97.4 |
| escalation (eskalacja) | 100.0 | 97.4 | 100.0 | 97.4 | 100.0 | 97.4 | 94.7 | 97.4 | 97.4 | 92.1 | 94.7 | 92.1 | 97.4 | 97.4 | 94.7 | 100.0 |
| HR (hr) | 92.6 | 88.9 | 92.6 | 92.6 | 88.9 | 88.9 | 88.9 | 92.6 | 85.2 | 81.5 | 92.6 | 92.6 | 77.8 | 77.8 | 81.5 | 88.9 |
| customer irritation (irytacja) | 100.0 | 97.4 | 94.7 | 100.0 | 97.4 | 94.7 | 100.0 | 94.7 | 97.4 | 89.5 | 100.0 | 92.1 | 92.1 | 94.7 | 84.2 | 97.4 |
| IT security (it_security) | 100.0 | 96.3 | 96.3 | 96.3 | 96.3 | 96.3 | 96.3 | 100.0 | 100.0 | 92.6 | 96.3 | 92.6 | 81.5 | 88.9 | 77.8 | 96.3 |
| law firm (kancelaria) | 92.9 | 92.9 | 92.9 | 92.9 | 92.9 | 92.9 | 96.4 | 100.0 | 100.0 | 82.1 | 92.9 | 96.4 | 89.3 | 89.3 | 85.7 | 92.9 |
| completeness (kompletnosc) | 100.0 | 92.1 | 100.0 | 89.5 | 86.8 | 89.5 | 94.7 | 97.4 | 97.4 | 84.2 | 97.4 | 94.7 | 73.7 | 84.2 | 68.4 | 94.7 |
| logistics (logistyka) | 89.3 | 89.3 | 89.3 | 89.3 | 89.3 | 89.3 | 85.7 | 89.3 | 85.7 | 89.3 | 85.7 | 92.9 | 85.7 | 82.1 | 82.1 | 85.7 |
| moderation (moderacja) | 92.9 | 89.3 | 92.9 | 92.9 | 89.3 | 92.9 | 92.9 | 85.7 | 85.7 | 100.0 | 92.9 | 92.9 | 92.9 | 92.9 | 75.0 | 89.3 |
| phishing (phishing) | 100.0 | 97.4 | 100.0 | 100.0 | 100.0 | 97.4 | 100.0 | 100.0 | 100.0 | 97.4 | 92.1 | 100.0 | 100.0 | 100.0 | 97.4 | 100.0 |
| manufacturing (produkcja) | 96.4 | 92.9 | 92.9 | 96.4 | 89.3 | 89.3 | 89.3 | 89.3 | 89.3 | 89.3 | 92.9 | 89.3 | 85.7 | 82.1 | 82.1 | 92.9 |
| clinic (przychodnia) | 92.9 | 100.0 | 92.9 | 96.4 | 89.3 | 100.0 | 100.0 | 96.4 | 96.4 | 89.3 | 96.4 | 89.3 | 78.6 | 85.7 | 75.0 | 100.0 |
| complaints (reklamacja) | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 97.4 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 97.4 | 100.0 | 97.4 | 100.0 |
| routing (routing) | 100.0 | 100.0 | 100.0 | 98.1 | 96.3 | 100.0 | 98.1 | 100.0 | 100.0 | 98.1 | 100.0 | 100.0 | 98.1 | 100.0 | 96.3 | 100.0 |
| insurance (ubezpieczenia) | 92.9 | 92.9 | 92.9 | 89.3 | 92.9 | 89.3 | 89.3 | 92.9 | 89.3 | 89.3 | 92.9 | 82.1 | 75.0 | 82.1 | 71.4 | 92.9 |
| university (uczelnia) | 96.4 | 100.0 | 96.4 | 96.4 | 92.9 | 100.0 | 96.4 | 92.9 | 100.0 | 89.3 | 100.0 | 89.3 | 89.3 | 78.6 | 92.9 | 92.9 |
| housing community (wspolnota) | 100.0 | 100.0 | 100.0 | 96.4 | 100.0 | 92.9 | 92.9 | 96.4 | 96.4 | 89.3 | 96.4 | 89.3 | 82.1 | 96.4 | 89.3 | 89.3 |