Take part

Submit a model

Any System 1 model that returns a probability per option can take part: a local checkpoint, an HTTP endpoint or a container.

1. 1

   Run the open harness on the dev split

   Install the harness (Python 3.10+, standard library only) and fetch the public dev split, then run, score and fit. This gives you the dev numbers and a calibration file. Adapters: basal, openrouter, or file for probabilities you produce yourself.

   python -m venv .venv
   .venv/bin/pip install "git+https://github.com/tacit-bench/tacit@v0.1.1#subdirectory=harness"
   curl -LO https://raw.githubusercontent.com/tacit-bench/tacit/v0.1.1/data/dev/items.jsonl
   
   .venv/bin/tacit run --adapter file --probs my_probs.json \
       --items items.jsonl --out preds/dev/my-model.jsonl
   .venv/bin/tacit score --items items.jsonl --preds preds/dev/my-model.jsonl
   .venv/bin/tacit fit   --items items.jsonl --preds preds/dev/my-model.jsonl \
       --out cal/my-model.json

   Or clone the repo and install from harness/. Full options in the README.

   Full options in the README.

2. 2

   Open a thread in GitHub Discussions

   Use the category “model submissions” and include:

   • the model id and exact revision
   • size and base model
   • access: a local checkpoint, an adapter-compatible HTTP endpoint, or a container
   • your dev score
   • the vendor calibration file, if you have one

   Open a submission thread

3. 3

   The maintainer runs the hidden test

   Nothing is fitted on test. The row is published with full run details: model id, revision, engine, hardware, date, who ran it and the calibration file.

Review items in your language

v0.2 adds English, German and Norwegian: about 1,000 items per language, written in the language, not translated. That needs people who know how a complaint, an incident report or a council letter really reads in their language.