Commands

Usage: skill-be <command> [options]

Benchmarks Agent Skills: baseline versus with-skill, LLM-graded and gated.

Commands:
  run [skill...]     Benchmark every skill that has an eval file, or the named ones. Writes
                     the outputs, the grader answers, a report per skill and summary.json
                     under the results directory, updates the result file, then runs the checks.
  verify             Without a model call: checks that the result file matches the current
                     skills, evals and config and records every skill READY, then runs the checks.
  check              Runs the typecheck and coverage checks, and parses the code tests.
  triage [skill...]  Proposes a classification for each failing criterion or code test of the
                     latest run. Never blocks.

Options:
  --config <path>    The config file (default: ./skill-be.yaml)
  --reps <n>         run: generations per arm per scenario, instead of the config's
  --concurrency <n>  run: model calls in flight at once, instead of the config's
  --html             run: also writes results.html beside summary.json, with every answer
  --keep-deps        run, triage: keeps the installed test packages in .skill-be/deps for the next run
  --below <pct>      triage: also triage criteria that scored under pct% with the skill
  --samples <n>      triage: fresh with-skill samples per failing criterion (default: 2)
  -h, --help         Shows this help

Exit codes: 0 when every skill is READY and every blocking check passes, 1 when anything
blocks, 2 for a usage or config error.

A skill folder with no eval file is skipped with a warning (warning: skill "x" has no eval file) by every command. Naming a skill that has no SKILL.md is a config error. Options take --name value or --name=value.

run and triage install the package lists of code tests under .skill-be/deps and delete them when they end, unless --keep-deps is given (Libraries the code may import). verify installs nothing and takes no --keep-deps. check parses the test files, reads the package lists, warns about hint words in the output contracts, and installs and runs nothing.

--reps and --concurrency apply to that run only. The result file does not record reps or gradings, so run the benchmark you commit with the config’s values.

A run makes, per skill, scenarios x models x reps x 2 generations (two arms), and each output is graded gradings times (a failed grading is retried once). Size reps and gradings with that in mind.

Exit codes

Code	Meaning
0	Success: every skill READY and every blocking check passing (run, verify, check), triage finished, or --help
1	A skill is NOT READY, verify found a problem with the result file, or a blocking check failed. An unexpected crash also exits 1, with its stack on stderr
2	A usage or config error: unknown command or option, a missing or invalid config or eval file, an unknown skill name, no skill with both a SKILL.md and an eval file, a broken result file, a misconfigured check or adapter (an apiKeyEnv variable unset in CI, missing provider, bad thinking, unknown adapter), a model or grader CLI that cannot start (not installed, a wrong bin or run), code tests that cannot be set up (a test file that matches no scenario, a missing package list, an install that fails, a test file that throws when it loads), or triage with no earlier run
130	Interrupted by SIGINT (Ctrl+C)
143	Terminated by SIGTERM

On SIGINT or SIGTERM the model calls in flight are killed before skill-be exits. run writes the summary and the result file after every skill, so an interrupted run keeps the skills it finished.

What run writes

Everything lands under results, in a directory named for the start time of the run:

.skill-be/results/2026-10-31T16-20-05-112Z/
  summary.json                          the JSON summary of the run
  results.html                          with --html: the run as a page to open in a browser
  money.md                              the markdown report of a skill
  money/<scenario>/<model>/
    rep-1-baseline.md                   a generated output
    rep-1-baseline.grading-1.md         a raw grader answer for that output
    rep-1-baseline.tests.json           with code tests: the result of each test on that output
    rep-1-with-skill.md
    rep-1-with-skill.grading-1.md

Add .skill-be/ to your .gitignore: the results directory is for inspection, and only skill-be.result.json is committed.

The HTML page

skill-be run --html also writes results.html, the run as a page to open in a browser. It holds what summary.json holds, plus every answer:

• a banner with the verdict of the run, how many skills block, the date, the models and the grader (or that nothing was graded, when no scenario has criteria);
• an overview table: each skill’s verdict, errors, duration and why it blocks, linked to its section;
• a section per skill, open when the skill blocks: the gate reasons, then each scenario with its prompt, its scores per model and the grader’s notes, and, folded, every rep’s answer from both arms with the grader replies and, for code tests, the result and failure message of each test.

The page follows the system’s light or dark setting. Cyan marks the baseline and magenta the with-skill arm, in the score tables and around the answers; red is kept for what blocks: NOT READY, failed tests, [ERROR] notes and scores that drop with the skill.

The page is one file that loads nothing and runs no script, so it opens offline and still works when you move it or download it from CI (Verify in CI). Its fonts, Press Start 2P and IBM Plex Mono (SIL Open Font License), are embedded in it, about 57 KB. Like summary.json, it is rewritten after every skill.

run ends with a summary of the verdicts, the checks and the files it wrote (sample). The checks run after the benchmark, over every skill that has an eval file. A bad check setting (for example a library that does not exist) stops run with exit code 2 before the first model call.