tacit

An independent EU benchmark for System 1 AI

How much can you hand to a fast model?

Tacit measures how many business decisions in EU languages, under EU rules, a System 1 model can take on its own while staying under a fixed error rate. The answer is proven on a hidden test the model has never seen.

Model
Error target

DREX Max takes 647 of 652 decisions on its own, with 23 wrong (3.6%, bound 5.3%). The other 5 go to a person.

Automated by the modelDeferred to a personWrong. The count is exact; where they sit in the grid is illustrative.

Polish hidden test of v0.2, one square per decision. Also in English, German and Norwegian Bokmål.

Independent. Not affiliated with or endorsed by EU institutions.

Languages

Items are written in each language, not translated from English. v0.2 covers four, each scored separately.

Leaderboard at 1 in 20

v0.2, Polish hidden test, n = 652, run 28 October 2026 by the maintainer

Ranked by certified coverage at a 5% error target, Polish table. There is no composite score. All 20 models hold the target on test. The same 20 models are ranked in English, German and Norwegian Bokmål.

#ModelCertified coverageErrorsRealised error, 95% interval (target 5%)
1DREX MaxAPI, size not disclosed, OpenRouterCoverage99.2%Errors23 of 647Realised error3.55% [2.27–5.29]
2Mammoth-1API, size not disclosed, Baseten Model APIsCoverage98.8%Errors22 of 644Realised error3.42% [2.15–5.13]
3basal-omni-1.0-ultra (basal 2.3.3)31B, Gemma 4, Apple M5 Max, localCoverage98.5%Errors16 of 642Realised error2.49% [1.43–4.02]
4Microsoft-Decision-1API, size not disclosed, OpenRouterCoverage98.2%Errors30 of 640Realised error4.69% [3.18–6.62]
5Liquid d1-ultraAPI, size not disclosed, OpenRouterCoverage97.7%Errors16 of 637Realised error2.51% [1.44–4.05]
6pplx-decider-v1.2API, size not disclosed, OpenRouterCoverage95.9%Errors20 of 625Realised error3.20% [1.97–4.90]
7Jev 1.14API, size not disclosed, OpenRouterCoverage94.5%Errors20 of 616Realised error3.25% [1.99–4.97]
7GPT-6 Luna DecisionsAPI, size not disclosed, OpenRouterCoverage94.5%Errors20 of 616Realised error3.25% [1.99–4.97]
7Elephant-1API, size not disclosed, OpenRouterCoverage94.5%Errors30 of 616Realised error4.87% [3.31–6.88]
10Liquid d1.1API, size not disclosed, OpenRouterCoverage93.4%Errors16 of 609Realised error2.63% [1.51–4.23]
11Mercury DecideAPI, size not disclosed, OpenRouterCoverage92.6%Errors17 of 604Realised error2.81% [1.65–4.47]
12basal-omni-1.0-max (basal 2.3.3)12B, Gemma 4, Apple M5 Max, localCoverage92.2%Errors15 of 601Realised error2.50% [1.40–4.08]
13Respan Span-02API, size not disclosed, OpenRouterCoverage91.0%Errors22 of 593Realised error3.71% [2.34–5.56]
14basal-omni-1.1-max (basal 2.3.3)12B, Gemma 4, Apple M5 Max, localCoverage90.6%Errors12 of 591Realised error2.03% [1.05–3.52]
15basal-2.0-max (basal 2.3.3)11B, Bielik v3, Apple M5 Max, localCoverage85.1%Errors22 of 555Realised error3.96% [2.50–5.94]
16basal-omni-1.0-mini (basal 2.3.3)4B, Gemma 4 E4B, Apple M5 Max, localCoverage80.2%Errors15 of 523Realised error2.87% [1.61–4.69]
17basal-2.0-mini (basal 2.3.3)1.5B, Bielik v3, Apple M5 Max, localCoverage50.9%Errors15 of 332Realised error4.52% [2.55–7.34]
18basal-2.0-4.5B (basal 2.3.3)4.5B, Bielik v3, Apple M5 Max, localCoverage50.3%Errors5 of 328Realised error1.52% [0.50–3.52]
19Cloudflare ClefAPI, size not disclosed, OpenRouterCoverage39.9%Errors5 of 260Realised error1.92% [0.63–4.43]
20basal-omni-pico (basal 2.3.3)0.8B, Qwen3.5-0.8B-Base (text only), Apple M5 Max, localCoverage33.9%Errors6 of 221Realised error2.71% [1.00–5.82]

Full leaderboard with calibration, accuracy, latency and cost

What an item looks like

A short business text and one structured question. This one is from the public dev split. Try it before you see what the models said.

R39Complaints (reklamacja)Yes or notricky

Customer message to a furniture shop

Chciałbym zamówić jeszcze dwa takie same fotele. Aha, i przy okazji: w tym, który już mam, po miesiącu pękł podłokietnik, więc proszę też o wymianę tej części.

I’d like to order two more of the same armchairs. Oh, and by the way: on the one I already have, the armrest cracked after a month, so please replace that part too.

Czy ta wiadomość jest reklamacją…?

Is this message a complaint, i.e. the customer reports a defect and asks for repair, replacement, a price cut or a refund?

Gold label: yes. Both blind labelers agreed.

Each model’s confidence in “yes”

After each model’s own temperature, fitted on dev. The tick is that model’s own threshold at 1 in 20. At or above it, the model answers on its own; below, the case goes to a person. A model can be very sure in raw terms and still be sent to a person once it is calibrated.

basal-omni-1.0-ultra (basal 2.3.3)0.998Answers on its own (threshold 0.690, raw 0.999)
Microsoft-Decision-10.996Answers on its own (threshold 0.674, raw 1.000)
GPT-6 Luna Decisions0.996Answers on its own (threshold 0.588, raw 1.000)
Liquid d1-ultra0.993Answers on its own (threshold 0.649, raw 0.994)
Liquid d1.10.992Answers on its own (threshold 0.699, raw 0.998)
Elephant-10.978Answers on its own (threshold 0.733, raw 0.973)
Jev 1.140.974Answers on its own (threshold 0.778, raw 0.920)
basal-omni-1.0-max (basal 2.3.3)0.970Answers on its own (threshold 0.772, raw 0.996)
basal-omni-1.1-max (basal 2.3.3)0.966Answers on its own (threshold 0.808, raw 0.995)
Respan Span-020.965Answers on its own (threshold 0.853, raw 0.993)
pplx-decider-v1.20.964Answers on its own (threshold 0.679, raw 0.977)
Mercury Decide0.946Answers on its own (threshold 0.808, raw 0.996)
basal-2.0-max (basal 2.3.3)0.942Answers on its own (threshold 0.840, raw 0.998)
Mammoth-10.940Answers on its own (threshold 0.607, raw 0.931)
basal-omni-1.0-mini (basal 2.3.3)0.933Answers on its own (threshold 0.862, raw 0.997)
basal-2.0-4.5B (basal 2.3.3)0.899Goes to a person (threshold 0.963, raw 0.998)
basal-2.0-mini (basal 2.3.3)0.844Goes to a person (threshold 0.903, raw 0.975)
DREX Max0.803Answers on its own (threshold 0.596, raw 0.858)
Cloudflare Clef0.793Goes to a person (threshold 0.895, raw 0.838)
basal-omni-pico (basal 2.3.3)0.786Goes to a person (threshold 0.940, raw 0.965)

Take part

Put your model on the board

Run the open harness on the dev split, then open a submission thread. The maintainer runs the hidden test once and publishes the row with full run details.

How to submit a model

Review items in your language

v0.2 added English, German and Norwegian Bokmål, 1,500 items each, written in the language. Volunteer native speakers are reviewing them now; corrections ship as point releases. Reviewers who know customer service, compliance, public administration or security are credited by name.

Offer to review

Supported byProsit ASSupporters get no access to the hidden test and no say in results. Independence