Writing evals

An eval file holds the scenarios of one skill. The file name decides which skill it belongs to (evals/money.json for skills/money); the skill field is not read.

Field
scenarios	A non-empty list
id	A unique, non-empty string per scenario
critical	Optional, default false. A regression on a critical scenario blocks the skill
prompt	The task given to the model
criteria	A non-empty list of strings, graded together but each judged separately. Optional when the scenario has an output
output	Optional. The names the model’s code must export, to run code tests against it. A scenario needs criteria, an output, or both

Good scenarios and criteria are most of the value:

• A realistic task. Write the prompt the way a user would ask for the code, with enough detail to be unambiguous. Mark the scenarios the skill must not make worse as critical.
• Do not name the API. If the prompt says “split it with allocate()”, the baseline does it too, and the scenario measures reading comprehension instead of whether the skill taught anything.
• Gradeable from the code alone. The grader sees the output text, not a running program. “The shares add up exactly to the total” can be read from code; “is robust” or “performs well” cannot.
• Accept equivalent idiomatic forms. If the skill allows two ways to do it, say so in the criterion: “Adds the lines with add or sum”, not “Uses sum”. A criterion that demands one form fails correct code, and the failure shows up as a skill gap that is really an eval problem.
• One check per criterion. Each criterion is judged on its own, as if it were the only one. Split “uses A and B” into two.

When a criterion fails with the skill, read the grader’s answers (*.grading-<n>.md next to the output) before changing the skill, or run skill-be triage. One model with one rep is noise: keep several reps, and look at the delta between the arms, not only the score with the skill.