Skip to content
skill-be

Example: test a skill end to end

This example builds a complete project from scratch: tiny-money, a small TypeScript library for amounts of money; money, a skill that teaches a model to use it; and the evals that test the skill. Every file is shown in full, so you can recreate the project by copy and paste. The same files are in skill-be’s own tests (test/fixtures/demo), which check that they match this section and pass skill-be check.

The folder layout:

tiny-money/
  package.json
  skill-be.yaml
  lib/
    index.ts
  skills/
    money/
      SKILL.md
  evals/
    money.json

The library

Section titled “The library”

tiny-money keeps amounts as whole cents, so adding and splitting them never drifts. A model that has not seen it will reach for floating-point numbers, which is what the skill is meant to change.

package.json

Section titled “package.json”
{
  "name": "tiny-money",
  "private": true,
  "type": "module"
}

The name matters: the typecheck imports the library by the name of the nearest package.json, so an example that writes import { Money } from 'tiny-money' compiles against lib/index.ts. npm install adds the dev dependencies when you set it up.

lib/index.ts

Section titled “lib/index.ts”
// tiny-money: amounts are whole cents, so adding and splitting them never drifts.


export type Currency = 'EUR' | 'USD' | 'GBP';


/** Thrown when two amounts in different currencies are combined. */
export class CurrencyMismatchError extends Error {
  constructor(expected: Currency, actual: Currency) {
    super(`cannot combine ${expected} with ${actual}`);
    this.name = 'CurrencyMismatchError';
  }
}


export class Money {
  private constructor(
    /** The amount in minor units: 1250 is 12.50. */
    readonly cents: number,
    readonly currency: Currency,
  ) {}


  /** An amount in major units, such as `12.5` for 12.50, rounded to the nearest cent. */
  static of(amount: number, currency: Currency): Money {
    return new Money(Math.round(amount * 100), currency);
  }


  static zero(currency: Currency): Money {
    return new Money(0, currency);
  }


  /** Throws `CurrencyMismatchError` when `other` is in another currency. */
  add(other: Money): Money {
    if (other.currency !== this.currency) {
      throw new CurrencyMismatchError(this.currency, other.currency);
    }
    return new Money(this.cents + other.cents, this.currency);
  }


  /** Multiplies by a factor (a quantity, or `1.2` for 20% tax), rounded to the nearest cent. */
  multiply(factor: number): Money {
    return new Money(Math.round(this.cents * factor), this.currency);
  }


  /**
   * Splits a non-negative amount by the given ratios without losing a cent: the shares always
   * add up to the original amount. `allocate([1, 1, 1])` splits it three ways.
   */
  allocate(ratios: readonly number[]): Money[] {
    const total = ratios.reduce((a, b) => a + b, 0);
    if (this.cents < 0 || ratios.some((ratio) => ratio < 0) || total <= 0) {
      throw new RangeError('allocate needs a non-negative amount and positive ratios');
    }
    const floors = ratios.map((ratio) => Math.floor((this.cents * ratio) / total));
    const remainder = this.cents - floors.reduce((a, b) => a + b, 0);
    return floors.map((cents, i) => new Money(cents + (i < remainder ? 1 : 0), this.currency));
  }


  /** The amount for display, such as `€12.50`; `locale` defaults to `en-US`. */
  format(locale = 'en-US'): string {
    return new Intl.NumberFormat(locale, { style: 'currency', currency: this.currency }).format(
      this.cents / 100,
    );
  }
}


/** Adds amounts of one currency; an empty list is zero in that currency. */
export function sum(amounts: readonly Money[], currency: Currency): Money {
  return amounts.reduce((total, amount) => total.add(amount), Money.zero(currency));
}

The skill

Section titled “The skill”

A skill is a folder with a SKILL.md. The front matter (name, description) is what a harness uses to pick the skill; skill-be gives the model the whole file. The ts blocks are typechecked against the library, so an example that calls an API wrongly fails skill-be check.

skills/money/SKILL.md

Section titled “skills/money/SKILL.md”
---
name: money
description: Use when TypeScript code adds, multiplies, splits or displays amounts of money. Covers the tiny-money library, which keeps amounts as whole cents so totals never drift.
---


# tiny-money


Never do arithmetic on money as floating-point numbers: `0.1 + 0.2` is not `0.3`, and dividing
100.00 by 3 loses a cent. Use `Money` from `tiny-money` instead. A `Money` holds whole `cents` and
a `currency` (`'EUR'`, `'USD'` or `'GBP'`, the `Currency` type), and every operation returns a new
`Money`.


| API | What it does |
|---|---|
| `Money.of(12.5, 'EUR')` | An amount in major units, rounded to the nearest cent |
| `Money.zero('EUR')` | Zero in a currency |
| `a.add(b)` | The sum; throws `CurrencyMismatchError` if the currencies differ |
| `a.multiply(3)` | Times a quantity or a rate (`1.2` adds 20% tax), rounded to the nearest cent |
| `a.allocate([1, 1, 1])` | Splits by ratios; the shares always add up to `a`, no cent lost |
| `a.format('de-DE')` | A display string such as `12,50 €`; the locale defaults to `en-US` |
| `sum(amounts, 'EUR')` | Adds a list of amounts; an empty list is zero |
| `a.cents`, `a.currency` | The raw amount in cents, and the currency |


## Totals


Turn every price into `Money` at the edge, compute with `multiply`, `add` and `sum`, and only call
`format` for display.


```ts
import { Money, sum } from 'tiny-money';


type Line = { unitPrice: number; quantity: number };


function orderTotal(lines: Line[]): string {
  const net = sum(
    lines.map((line) => Money.of(line.unitPrice, 'EUR').multiply(line.quantity)),
    'EUR',
  );
  const gross = net.multiply(1.2); // 20% VAT
  return gross.format('en-GB');
}
```


## Splitting


Split with `allocate`, never with `/`. Equal ratios split evenly; the leftover cents go to the
first shares.


```ts
import { Money } from 'tiny-money';


const [alice, bob, carol] = Money.of(100, 'EUR').allocate([1, 1, 1]);
// 33.34, 33.33 and 33.33: together exactly 100.00


const [platform, seller] = Money.of(59.99, 'USD').allocate([15, 85]); // a 15% commission
```


## Mixed currencies


`add` and `sum` throw `CurrencyMismatchError` rather than adding euros to dollars. Catch it, or
check `currency` first, when the input may mix currencies.


```ts
import { CurrencyMismatchError, Money, sum, type Currency } from 'tiny-money';


function walletTotal(payments: Money[], currency: Currency): Money | undefined {
  try {
    return sum(payments, currency);
  } catch (error) {
    if (error instanceof CurrencyMismatchError) return undefined;
    throw error;
  }
}
```

The evals

Section titled “The evals”

The eval file of a skill is <evals dir>/<skill>.json. Each scenario is a task, written the way a user would ask, and the criteria a good answer meets. The prompts do not name the library’s API: if they did, the baseline would use it too, and the scenario would not show what the skill teaches. critical marks the scenarios the skill must not make worse. See Writing evals.

evals/money.json

Section titled “evals/money.json”
{
  "skill": "money",
  "scenarios": [
    {
      "id": "split-the-bill",
      "critical": true,
      "prompt": "Write a TypeScript function that splits a restaurant bill between a number of people. It takes the total in euros (for example 100.00) and the number of people, and returns each person's share. The shares must add up exactly to the total, to the cent.",
      "criteria": [
        "Represents the bill and the shares as Money values (or whole cents), not as floating-point euro amounts",
        "Splits the total with allocate (or another whole-cent method that hands out the leftover cents), not by dividing a floating-point total",
        "The shares it returns always add up exactly to the total"
      ]
    },
    {
      "id": "order-total-with-vat",
      "critical": true,
      "prompt": "Write a TypeScript function for a checkout page. It receives the cart lines, each with a unit price in euros and a quantity, adds 20% VAT to the total, and returns the total as a string ready to show to the customer.",
      "criteria": [
        "Converts each unit price into a Money value (Money.of) instead of keeping euro amounts as plain numbers",
        "Computes the line totals and the VAT with multiply, and adds the lines with add or sum",
        "Produces the display string with format, not with toFixed or string concatenation"
      ]
    },
    {
      "id": "wallet-in-several-currencies",
      "critical": false,
      "prompt": "A user's wallet holds payments that may be in EUR, USD or GBP. Write a TypeScript function that returns the total of the payments in a given currency, and reports a clear error instead of silently adding amounts in different currencies together.",
      "criteria": [
        "Adds the amounts with Money's add or sum rather than adding plain numbers",
        "Detects mixed currencies by catching CurrencyMismatchError or by comparing each payment's currency, and reports it instead of adding the amounts"
      ]
    }
  ]
}

The config

Section titled “The config”

skill-be.yaml is the only config file. Paths in it are relative to it.

skill-be.yaml

Section titled “skill-be.yaml”
skills: ./skills
evals: ./evals


# The model under test and the judge. For apiKeyEnv, see "Credentials" in skill-be's README.
model:
  adapter: claude-cli
  id: claude-sonnet-5-5
  apiKeyEnv: ANTHROPIC_API_KEY
grader:
  adapter: claude-cli
  id: claude-haiku-5-5
  apiKeyEnv: ANTHROPIC_API_KEY


# Small, so a first run is quick and cheap: 3 scenarios x 2 reps x 2 arms = 12 generations,
# each graded once. The defaults are 3 reps and 3 gradings.
reps: 2
gradings: 1


withSkillTemplate: |-
  You have access to the following TypeScript library. Use its APIs when writing the code.


  --- LIBRARY DOCS ---
  {{skill}}
  --- END LIBRARY DOCS ---


  Task: {{task}}


checks:
  typecheck:
    library: ./lib/index.ts
  coverage:
    entry: ./lib/index.ts
    blocking: true
  • model is the model under test and grader the judge; here both are Claude Code models.
  • apiKeyEnv names the environment variable that may hold an API key; while it is unset, claude uses your own login (Credentials).
  • withSkillTemplate says how the skill and the task are put together in the with-skill prompt.
  • checks.typecheck.library is the library the examples compile against, and checks.coverage.entry the file whose exports must each be mentioned by a skill or an eval. Coverage only warns by default; here it blocks.

Set it up

Section titled “Set it up”

Install skill-be and TypeScript, and keep the results directory out of git:

Terminal window
npm install --save-dev @ngwes/skill-be typescript
echo ".skill-be/" >> .gitignore

In PowerShell, the second line is Add-Content .gitignore ".skill-be/".

Check the examples

Section titled “Check the examples”

skill-be check typechecks the examples and checks coverage. It calls no model, so it is the quick loop while you write the skill.

Terminal window
npx skill-be check
typecheck: passed (1 skill(s))
coverage: every API name is mentioned
Result: pass

A mistake in an example fails it with exit code 1. With allocate(1, 1, 1) in place of allocate([1, 1, 1]) in the first example under “Splitting” in SKILL.md:

typecheck: 1 problem(s)
  - skill "money", SKILL.md line 52: error TS2554: Expected 1 arguments, but got 3.
coverage: every API name is mentioned
Result: blocked by the typecheck

Run the benchmark

Section titled “Run the benchmark”

run calls claude, so it must be installed (npm install --global @anthropic-ai/claude-code) and logged in (run claude once and follow the login prompt). claude -p "say hi" should print an answer before you go on.

Terminal window
npx skill-be run

This makes 12 generations (3 scenarios, 2 reps, 2 arms) and 12 gradings, a few at a time. Progress goes to stderr:

money: 3 scenarios × 1 model(s) × 2 reps, concurrency 4
  [split-the-bill] claude-sonnet-5-5 rep 1/2 baseline: 1/3 YES (16.4s)
  [split-the-bill] claude-sonnet-5-5 rep 1/2 with skill: 3/3 YES (13.9s)
  [split-the-bill] claude-sonnet-5-5 rep 1/2 done (1/6 reps, 17.1s elapsed)
  …
money: finished in 61.8s

The outputs in this section are samples: every run calls the model afresh, so your timings, scores, timestamps and fingerprints will differ.

The report goes to stdout, and to money.md in the run’s directory. Each table compares the baseline with the with-skill arm, criterion by criterion:

## 2026-10-31 — money benchmark


Method: automated eval. Models: claude-cli:claude-sonnet-5-5. Reps: 2. Grading: majority of 1 claude-haiku-5-5 calls per output (via claude-cli).
Note: "with skill" puts the SKILL.md content in the user prompt.


### Scenario: split-the-bill *(critical)*


*Write a TypeScript function that splits a restaurant bill between a number of people. It takes the total in euros (for example 100.00) and the number of people, and returns each person's share. The shares must add up exactly to the total, to the cent.*


**claude-sonnet-5-5**


| Criterion | Baseline | With skill | Delta |
|-----------|----------|------------|-------|
| Represents the bill and the shares as Money values (or whole cents), not as floating-point euro amounts | 50% | 100% | +50pp |
| Splits the total with allocate (or another whole-cent method that hands out the leftover cents), not by dividing a floating-point total | 0% | 100% | +100pp |
| The shares it returns always add up exactly to the total | 50% | 100% | +50pp |


### Scenario: order-total-with-vat *(critical)*


*Write a TypeScript function for a checkout page. It receives the cart lines, each with a unit price in euros and a quantity, adds 20% VAT to the total, and returns the total as a string ready to show to the customer.*


**claude-sonnet-5-5**


| Criterion | Baseline | With skill | Delta |
|-----------|----------|------------|-------|
| Converts each unit price into a Money value (Money.of) instead of keeping euro amounts as plain numbers | 0% | 100% | +100pp |
| Computes the line totals and the VAT with multiply, and adds the lines with add or sum | 0% | 100% | +100pp |
| Produces the display string with format, not with toFixed or string concatenation | 0% | 50% | +50pp |


### Scenario: wallet-in-several-currencies


*A user's wallet holds payments that may be in EUR, USD or GBP. Write a TypeScript function that returns the total of the payments in a given currency, and reports a clear error instead of silently adding amounts in different currencies together.*


**claude-sonnet-5-5**


| Criterion | Baseline | With skill | Delta |
|-----------|----------|------------|-------|
| Adds the amounts with Money's add or sum rather than adding plain numbers | 0% | 100% | +100pp |
| Detects mixed currencies by catching CurrencyMismatchError or by comparing each payment's currency, and reports it instead of adding the amounts | 100% | 100% | +0pp |


---


## Release Gate


**READY** — No universal 0% criteria. No critical regressions.

Then a summary, whose last line matches the exit code:

Skills:
  money: READY
typecheck: passed (1 skill(s))
coverage: every API name is mentioned
Written:
  .skill-be/results/2026-10-31T16-20-05-112Z/money.md
  .skill-be/results/2026-10-31T16-20-05-112Z/summary.json
  skill-be.result.json
Result: pass

The skill is READY: no criterion is at 0% with the skill, and no critical scenario got worse. The outputs and the grader’s answers are under .skill-be/results/<run>/money/, one file each, if you want to see what the model wrote.

Look at weak criteria

Section titled “Look at weak criteria”

Formatting with format passed only half the time. skill-be triage takes the criteria that blocked the latest run and, with --below, also those that scored under a threshold with the skill, generates fresh with-skill samples, and asks a model why they fail:

Terminal window
npx skill-be triage --below 80
Triage: 1 failure(s)
  money [order-total-with-vat] claude-sonnet-5-5 "Produces the display string with format, not with toFixed or string concatenation": skill gap — the samples format cents / 100 with Intl.NumberFormat; the skill's table lists format but never says not to format by hand
Report: .skill-be/results/2026-10-31T16-20-05-112Z/triage-2026-10-31T16-23-40-518Z.md

The proposal is for you to confirm. Here it is right: say plainly in SKILL.md that amounts are displayed only with format, never with toFixed or Intl.NumberFormat, then run again. Triage never blocks.

Commit the result

Section titled “Commit the result”

run wrote skill-be.result.json: per skill, the verdict and a fingerprint of its SKILL.md, its eval file and skill-be.yaml.

{
  "skillBeVersion": "1.0.0",
  "skills": {
    "money": {
      "date": "2026-10-31",
      "fingerprint": "342ccc50…",
      "ready": true,
      "reasons": [],
      "models": ["claude-sonnet-5-5"],
      "grader": "claude-haiku-5-5"
    }
  }
}

Commit it with the skill, the evals and the config it covers; .skill-be/ stays out of git. In a git repository (git init if needed):

Terminal window
git add package.json package-lock.json .gitignore skill-be.yaml skill-be.result.json lib skills evals
git commit -m "feat: add the money skill"

skill-be verify checks that commit without calling a model:

Terminal window
npx skill-be verify
result file skill-be.result.json: 1 skill(s) match their last run, every one READY
typecheck: passed (1 skill(s))
coverage: every API name is mentioned
Result: pass

Edit SKILL.md, the eval file or skill-be.yaml without running again, and verify fails with exit code 1:

result file skill-be.result.json: 1 problem(s)
  - skill "money": changed since its last run (its SKILL.md, its eval file or the config); run "skill-be run money"
typecheck: passed (1 skill(s))
coverage: every API name is mentioned
Result: blocked by the result file

Verify in CI

Section titled “Verify in CI”

verify needs no API key, so it can run on every pull request. To let CI run the benchmark itself as well, store an Anthropic API key as the repository secret ANTHROPIC_API_KEY. The config does not change: apiKeyEnv: ANTHROPIC_API_KEY makes claude use your login on your machine (when ANTHROPIC_API_KEY is not exported there) and the secret in CI, where a missing secret stops skill-be run with exit code 2 (Credentials).

.github/workflows/skills.yml
name: skills


on:
  pull_request:
  workflow_dispatch:


permissions:
  contents: read


jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 22
      - run: npm ci
      - run: npx skill-be verify


  benchmark:
    # Started by hand from the Actions tab; it calls the model and costs tokens.
    if: github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 22
      - run: npm ci
      - run: npm install --global @anthropic-ai/claude-code
      - run: npx skill-be run --html
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      # Keeps the page of the run, also when a skill is NOT READY.
      - uses: actions/upload-artifact@v7
        if: ${{ !cancelled() }}
        with:
          path: .skill-be/results/*/results.html
          archive: false

The benchmark job’s exit code is the gate for that run. Its skill-be.result.json stays on the runner; to commit results, run the benchmark on your machine. The last step saves results.html (The HTML page) on the run’s page in the Actions tab; archive: false keeps it a plain file rather than a zip.

The verify step can also be the skill-be GitHub Action:

      - run: npm ci
      - uses: ngwes/skill-be@v1

The action runs the skill-be the project installs, here with npm ci. Without one, it runs the npm release that matches its tag: @v1 runs the latest 1.x. Install the project’s dependencies first when the checks are on, since they typecheck against the library. Its inputs:

  • command: the skill-be command, verify by default.
  • args: more arguments, separated by spaces, such as --config path.
  • working-directory: the folder that holds skill-be.yaml, . by default.
  • version: the skill-be version to run when the project does not install it.

Use another model

Section titled “Use another model”

The model under test and the grader are separate entries, so either one can come from another harness. With pi, using your pi login or the key (Credentials):

model:
  adapter: pi-cli
  provider: anthropic
  id: claude-sonnet-5-5
  apiKeyEnv: ANTHROPIC_API_KEY

With opencode, using your opencode login (opencode auth list) or the key:

model:
  adapter: opencode-cli
  provider: google
  id: gemini-2.5-flash
  apiKeyEnv: GEMINI_API_KEY

model also takes a list, to test several models in one run. The fingerprint covers skill-be.yaml, so after changing it, run the benchmark again before verify. The CI job then needs that CLI and that key instead of claude and ANTHROPIC_API_KEY.