Configuration

skill-be.yaml at the project root; use --config <path> for another file. Relative paths inside it resolve against the config file’s directory. An unknown key is an error (exit code 2), at the top level and inside gate, checks, checks.typecheck, checks.coverage, triage and codeTests, so a typo such as bloking: true cannot pass unnoticed. The adapter entries (model, grader, triage.model) are the exception: they accept any key, since their settings belong to the adapter.

This is the minimum. A skill is a folder under skills with a SKILL.md; its eval file is <evals>/<skill>.json.

skills: ./skills
evals: ./evals
model:
  adapter: claude-cli
  id: claude-sonnet-5-5
grader:
  adapter: claude-cli
  id: claude-haiku-5-5

Key	Default	Meaning
skills	required	Directory with one folder per skill, each holding a SKILL.md
evals	required	Directory with one <skill>.json eval file per skill
model	required	The model under test: one entry or a list (see below)
grader	required	The judge: a single entry
reps	3	Generations per arm per scenario (a positive integer)
gradings	3	Grader calls per output; the majority wins (a positive integer)
concurrency	4	Model calls in flight at once, across generation and grading
timeoutSeconds	120	Per call; the child process (and what it started) is killed on timeout. At most 2147483 (about 24.8 days), the longest a Node timer holds
results	./.skill-be/results	Where outputs, grader answers and reports go
resultFile	./skill-be.result.json	The committed result that verify reads
withSkillTemplate	see below	How the skill text and the task are put together; must contain {{skill}} and {{task}}
graderPrompt	built in	Replaces the grader’s role sentence and rules (see Grading)
gate.regressionOnCritical	true	Block when a critical scenario’s criterion scores lower with the skill than without
gate.blockOnZero	true	Block when a criterion scores 0% with the skill on every model
gate.blockOnErrors	true	Block when any output could not be generated or graded
checks.typecheck	{ preset: typescript, blocking: true } when typescript is installed, else off	Typecheck the code examples (see Checks); false turns it off
checks.coverage	off	Report library exports that no skill mentions; blocking defaults to false
triage.model	the failing output’s own model under test	An adapter entry that classifies failures (see Triage)
tests	none	The folder of the code tests, one subfolder per skill; required for a skill whose scenarios have an output (see Writing the tests)
codeTests.packages	test-packages.json next to the config, when it exists	The default package list the code tests install (see Libraries the code may import)
codeTests.packagesBySkill	{}	A map from a skill name to its own package list, instead of the default
codeTests.testTimeoutSeconds	10	Per test; a test that runs longer fails. At most 2147483
codeTests.answerTimeoutSeconds	60	Per answer, for all its tests; the process is killed after it and the unfinished tests fail. At most 2147483
codeTests.allow	[]	Words the hint-word check does not report

The default withSkillTemplate ({{skill}} is the full text of SKILL.md, {{task}} is the scenario prompt):

You have access to the following library. Use its APIs when writing the code.

--- LIBRARY DOCS ---
{{skill}}
--- END LIBRARY DOCS ---

Task: {{task}}

Write your own as a YAML block scalar. For example, to describe the skill as a TypeScript library:

withSkillTemplate: |-
  You have access to the following TypeScript library. Use its APIs when writing the code.

  --- LIBRARY DOCS ---
  {{skill}}
  --- END LIBRARY DOCS ---

  Task: {{task}}

Model entries

model and grader entries have the same shape: adapter, id, and that adapter’s own settings. adapter is a built-in name (claude-cli, pi-cli, opencode-cli, command) or a path to your own adapter module. id is the model id passed to the harness (for command it is only a label). model takes a list to test several models in one run; the gate rules are defined across them.

model:
  - adapter: claude-cli
    id: claude-sonnet-5-5
  - adapter: pi-cli
    provider: anthropic
    id: claude-haiku-5-5
grader:
  adapter: claude-cli
  id: claude-haiku-5-5

In the tables of a report and in the result file a model is labeled by its id, or adapter:id when two entries share an id.

Credentials

Credentials never go in the YAML. An adapter setting such as apiKeyEnv: ANTHROPIC_API_KEY names the environment variable skill-be reads an API key from. Keys are replaced by *** in error messages. Which credential a CLI adapter uses:

apiKeyEnv	Its variable	CI	Credential
set	set and non-empty	any	The key from the variable
set	unset or empty	not set	The CLI’s stored login; no key is passed
set	unset or empty	set	None: the command stops with exit code 2, naming the variable
not set	n/a	any	The CLI’s stored login

For claude-cli, “stored login” assumes no ANTHROPIC_API_KEY in your shell: the environment is inherited, and claude prefers that variable to its login (see claude-cli).

CI counts as set when it holds any non-empty value other than false or 0; GitHub Actions, GitLab CI and most other CI services set it to true. So one config serves both places: on your machine the CLIs use your logins, in CI the secret is the key, and a forgotten secret stops the command with exit code 2 instead of quietly turning every output into an error.

model:
  adapter: pi-cli
  provider: anthropic
  id: claude-sonnet-5-5
  apiKeyEnv: ANTHROPIC_API_KEY
grader:
  adapter: claude-cli
  id: claude-haiku-5-5
  apiKeyEnv: ANTHROPIC_API_KEY

A key exported in your shell is used on your machine too. To use the stored login instead, unset the variable for the run: env -u ANTHROPIC_API_KEY npx skill-be run, or in PowerShell Remove-Item Env:ANTHROPIC_API_KEY; npx skill-be run (which unsets it for the rest of that session).

How the key reaches each CLI differs:

Adapter	With a key	Without one
claude-cli	The key is set as ANTHROPIC_API_KEY in claude’s environment; claude prefers it over its login	claude’s own login
pi-cli	The key is passed as --api-key, so it is visible in the process list while pi runs	pi’s stored login
opencode-cli	The key goes in OPENCODE_AUTH_CONTENT, which replaces the stored logins for that call	opencode’s stored login (opencode auth list)
command	Not used; your command finds its own credentials	n/a