Skip to content
skill-be

The release gate

A skill is NOT READY when any enabled rule holds:

  1. Zero (gate.blockOnZero): a criterion scores 0% with the skill on every model under test.
  2. Critical regression (gate.regressionOnCritical): in a scenario marked "critical": true, a criterion scores lower with the skill than in the baseline, on any model.
  3. Errors (gate.blockOnErrors): any output could not be generated or graded, or its code tests could not run.

Otherwise it is READY. A criterion’s score for an arm is the share of reps whose verdict is YES, rounded to a whole percent. A generation that fails is not retried; it is an error, and its criteria count as 0.

A code test is a criterion for rules 1 and 2, named [test] <name> in the reasons, and an answer whose tests could not run is an error for rule 3 (Scores and the gate).

The Release Gate section of each report uses fixed strings, so scripts can grep them:

## Release Gate


**READY** — No universal 0% criteria. No critical regressions.

or, for a blocked skill:

## Release Gate


- **[ERROR]** 2 output(s) could not be generated or graded; their criteria were counted as 0 (see the scenario notes).
- **BLOCKED** (universal 0%): `order-total-with-vat` — criterion: *"Produces the display string with format, not with toFixed or string concatenation"*
- **BLOCKED** (regression): `split-the-bill` / claude-sonnet-5-5 — *"The shares it returns always add up exactly to the total"*: 100% → 33% (-67pp)


**NOT READY**

In a run with code tests, the error line names them too:

- **[ERROR]** 2 error(s): outputs that could not be generated or graded, or code tests that could not run; their lines were counted as 0 (see the scenario notes).

Per-output problems appear as [ERROR] and [NOTE] lines under the scenario tables.