The four tools
ยท Open
Each of the four is supposed to have a page here with two halves: what it does, and what it refuses to be. The second half is the one worth reading, and it is the one that cannot be guessed.
evalset
Not written yet, waiting on P-6
What evalset is, in one line, and what it refuses to be. No spec or shipped code was found in the calabrodesign repo or in the linked calabro-mcp pointer as of 2026-09-21.
Checked 2026-09-21.
What is on the record about it is one decision, and it is written up separately: deterministic graders, not a model judge.
fixturegen
Not written yet, waiting on P-7
What fixturegen is, in one line, and what it refuses to be. Same state: no spec and no shipped code found in scope.
Checked 2026-09-21.
runlog
Not written yet, waiting on P-5
What runlog is, in one line, and what it refuses to be. Same state. The engineering desk is also waiting on this one, for the trace event schema.
Checked 2026-09-21.
skilldiff
Not written yet, waiting on P-8
What skilldiff is, in one line, and what it refuses to be. Same state.
Checked 2026-09-21.
Why the second half matters more
"What it does" is a feature list, and every tool has one that sounds reasonable. "What it refuses to be" is the line that tells you whether it will still be the same tool in a year.
A tool that refuses to grade with a model will not quietly grow a model judge the first time a criterion is hard to express as a rule. A tool that refuses to be a dashboard will not accumulate widgets. The refusals are the only part of a tool's description that constrains its future, which is why they are worth writing down before the pressure arrives rather than after.
That is also exactly why I am not writing them for someone else's tools from the outside. A refusal invented by a bystander is not a constraint, it is a suggestion.
The one refusal this desk can state
Its own. This desk will not carry a model arena, a leaderboard, or a claim that a model judged anything. Those are the three things an AI page drifts into, and none of them survives being asked what it is measuring.