Check and Measure

Check written code

pnpm exec operator-ds check src

check reads every .tsx and .jsx file under the paths you give and reports where the code breaks the contract. It reads the source with the project's own TypeScript, so it needs no browser and adds no dependency. Tests and stories are skipped.

src/screen.tsx:14:17  error  INVALID_PROP_VALUE  Button variant="success" is not an option. Use one of: primary, secondary, tertiary, danger, ghost. "success" is a Badge variant.
1 error, 0 warnings in 1 file.

It exits 1 when there is an error, and 0 otherwise. --strict makes warnings fail the run too. --json returns the findings with their file, line, column and code.

Codes are only ever added to. A code's meaning does not change.

A rule is in the check only if it can be right from one file's source. What cannot be seen from there is left alone: a value that is computed, props that arrive by a spread, a part whose parent is rendered in another file. An anchor is never flagged, since it is how a Button becomes a link.

The dev-time errors remain: they catch a bad option value while the app runs. check covers more, and reaches an agent that never sees a browser console.

Measure an agent

Whether an agent builds correctly with OPERATOR is a thing to measure. The repository holds a set of screens to ask for (tests/agents/tasks.json: a status dashboard, an incident form, a table of 5,000 work orders and nine more) and a harness that scores what comes back.

pnpm build
node scripts/agent-eval.mjs prepare --out /tmp/eval/with
node scripts/agent-eval.mjs prepare --out /tmp/eval/bare --bare
# point an agent at each folder: it reads TASK.md and writes result.tsx
node scripts/agent-eval.mjs score --dir /tmp/eval/with
node scripts/agent-eval.mjs score --dir /tmp/eval/bare

prepare lays out a project folder per brief. The published packages are installed as copies in a store beside the folders, so an agent can read them the way it would in a real project and nothing links back into this repository. The folder holds the brief and, unless --bare is passed, the AGENTS.md block and the operator-ds command. That is the whole difference between the two conditions: the same briefs and the same packages, with the setup or without it. --sample <n> takes the first n briefs.

score judges each result twice and reports:

• contract errors, by code, from check;
• compiler errors, from TypeScript against the published types. This judge is not ours, so it does not share the contract's mistakes;
• escape hatches: native controls written where the system has a component;
• components used, and which of the purpose-built components the brief calls for were chosen or missed;
• pass: no contract error, no compiler error, no escape hatch, and at least one OPERATOR component on screen. Code that never touches the system has no contract errors either, which is why that last condition is there.

A brief says what the screen must do and never which component to use. Choosing is the test.

Scores say whether the code follows the contract and compiles. They do not say whether the screen is a good one: that still takes a person, or a browser.

The run of 13 October 2026

Two Claude models each wrote all twelve screens twice, once with the setup and once without: 48 screens.

• Agents build correct screens with the system. All 48 compile, follow the contract and use no escape hatch.
• The setup did not make them more correct. It made the work cheaper: about 16% fewer tokens, and fewer in 21 of the 24 paired runs.
• The first scoring was wrong about two screens. Both models, without the setup, put a ConfirmDialog directly inside a Toolbar, and the check called it an error. Tried for real, it works, so the rule was the mistake: the Toolbar's list of allowed children left out ConfirmDialog, DropdownMenu and Tooltip. The rule was corrected and every screen was scored again. A measurement tests the contract as much as it tests the agent.

One run per brief and model, two models from one maker, and no judgment of how the screens look. The screens, both scorings, the prompt and the limits are in docs/verification/2026-10-13-agent-measurement in the repository.