tacit

An independent EU benchmark for System 1 AI

How much can you hand to a fast model?

Tacit measures how many business decisions in EU languages, under EU rules, a System 1 model can take on its own while staying under a fixed error rate. The answer is proven on a hidden test the model has never seen.

Model
Error target

Liquid d1-ultra takes 614 of 652 decisions on its own, with 14 wrong (2.3%, bound 3.8%). The other 38 go to a person.

Automated by the modelDeferred to a personWrong. The count is exact; where they sit in the grid is illustrative.

Independent. Not affiliated with or endorsed by EU institutions.

Languages

Items are written in each language, not translated from English. v0.1 is Polish only.

Leaderboard at 1 in 20

v0.1, hidden test, n = 652, run 20 October 2026 by the maintainer

Ranked by certified coverage at a 5% error target. There is no composite score. All 16 models hold the target on test.

#ModelCertified coverageErrorsRealised error, 95% interval (target 5%)
1Liquid d1-ultraAPI, size not disclosed, OpenRouterCoverage94.2%Errors14 of 614Realised error2.28% [1.25–3.80]
2Jev 1.14API, size not disclosed, OpenRouterCoverage92.0%Errors20 of 600Realised error3.33% [2.05–5.10]
3basal-omni-1.0-ultra31B, Gemma 4, Apple M5 Max, localCoverage91.6%Errors9 of 597Realised error1.51% [0.69–2.84]
4pplx-decider-v1.2API, size not disclosed, OpenRouterCoverage90.6%Errors14 of 591Realised error2.37% [1.30–3.94]
5Mercury DecideAPI, size not disclosed, OpenRouterCoverage90.6%Errors18 of 591Realised error3.05% [1.81–4.77]
6Jev 1.13API, size not disclosed, OpenRouterCoverage89.7%Errors20 of 585Realised error3.42% [2.10–5.23]
7Microsoft-Decision-1API, size not disclosed, OpenRouterCoverage89.1%Errors19 of 581Realised error3.27% [1.98–5.06]
8basal-omni-1.1-max12B, Gemma 4, Apple M5 Max, localCoverage87.1%Errors11 of 568Realised error1.94% [0.97–3.44]
9basal-omni-1.0-max12B, Gemma 4, Apple M5 Max, localCoverage86.0%Errors9 of 561Realised error1.60% [0.74–3.02]
10basal-2.0-max11B, Bielik v3, Apple M5 Max, localCoverage85.6%Errors27 of 558Realised error4.84% [3.21–6.96]
11Cloudflare ClefAPI, size not disclosed, OpenRouterCoverage84.4%Errors11 of 550Realised error2.00% [1.00–3.55]
12Respan Span-02API, size not disclosed, OpenRouterCoverage84.2%Errors17 of 549Realised error3.10% [1.81–4.91]
13basal-2.0-4.5B4.5B, Bielik v3, Apple M5 Max, localCoverage69.2%Errors10 of 451Realised error2.22% [1.07–4.04]
14basal-omni-1.0-mini4B, Gemma 4 E4B, Apple M5 Max, localCoverage64.9%Errors11 of 423Realised error2.60% [1.31–4.61]
15basal-2.0-mini1.5B, Bielik v3, Apple M5 Max, localCoverage50.6%Errors14 of 330Realised error4.24% [2.34–7.02]
16GPT-6 Luna DecisionsAPI, size not disclosed, OpenRouterCoverage47.5%Errors9 of 310Realised error2.90% [1.34–5.44]

At 1 in 100, no model is certified. Full leaderboard with calibration, accuracy, latency and cost

What an item looks like

A short business text and one structured question. This one is from the public dev split. Try it before you see what the models said.

R39Complaints (reklamacja)Yes or notricky

Customer message to a furniture shop

Chciałbym zamówić jeszcze dwa takie same fotele. Aha, i przy okazji: w tym, który już mam, po miesiącu pękł podłokietnik, więc proszę też o wymianę tej części.

I’d like to order two more of the same armchairs. Oh, and by the way: on the one I already have, the armrest cracked after a month, so please replace that part too.

Czy ta wiadomość jest reklamacją…?

Is this message a complaint, i.e. the customer reports a defect and asks for repair, replacement, a price cut or a refund?

Gold label: yes. Both blind labelers agreed.

Each model’s confidence in “yes”

After each model’s own temperature, fitted on dev. The tick is that model’s own threshold at 1 in 20. At or above it, the model answers on its own; below, the case goes to a person. A model can be very sure in raw terms and still be sent to a person once it is calibrated.

basal-omni-1.0-ultra0.995Answers on its own (threshold 0.800, raw 1.000)
GPT-6 Luna Decisions0.990Goes to a person (no yes/no threshold certified, raw 1.000)
Jev 1.140.989Answers on its own (threshold 0.771, raw 0.920)
Liquid d1-ultra0.988Answers on its own (threshold 0.662, raw 0.994)
Microsoft-Decision-10.985Answers on its own (threshold 0.837, raw 1.000)
basal-omni-1.1-max0.976Answers on its own (threshold 0.830, raw 0.995)
Jev 1.130.974Answers on its own (threshold 0.861, raw 0.900)
basal-omni-1.0-max0.974Answers on its own (threshold 0.842, raw 0.996)
Mercury Decide0.967Answers on its own (threshold 0.811, raw 0.996)
basal-2.0-max0.962Answers on its own (threshold 0.833, raw 0.998)
pplx-decider-v1.20.959Answers on its own (threshold 0.763, raw 0.977)
Respan Span-020.955Answers on its own (threshold 0.863, raw 0.993)
basal-omni-1.0-mini0.931Goes to a person (threshold 0.963, raw 0.997)
basal-2.0-4.5B0.922Answers on its own (threshold 0.906, raw 0.998)
basal-2.0-mini0.850Goes to a person (threshold 0.909, raw 0.975)
Cloudflare Clef0.825Goes to a person (threshold 0.901, raw 0.838)

Take part

Put your model on the board

Run the open harness on the dev split, then open a submission thread. The maintainer runs the hidden test once and publishes the row with full run details.

How to submit a model

Review items in your language

v0.2 adds English, German and Norwegian, about 1,000 items each, written in the language. Native speakers who know customer service, compliance, public administration or security are credited by name.

Offer to review

Supported byProsit ASSupporters get no access to the hidden test and no say in results. Independence