Skip to content
skill-be

Related tools

skillgrade is an eval harness for coding agents, with or without skills. The two tools answer different questions:

  • skill-be asks whether a skill makes a model better than it is without the skill. Every scenario runs twice, without and with the skill, and the gate blocks regressions. The answer is one turn of text, graded by an LLM judge, by code tests that run the code in it, or by both. The release gate runs in CI without a model call: verify checks a committed result file.
  • skillgrade asks how often an agent completes a task. The agent works on real files with its tools, in Docker or locally. The skill is installed where the agent discovers it, and the pass rate is compared with a threshold.

skillgrade is the better fit when the agent has to act: tasks that edit a workspace, run commands or take several turns, a skill the agent has to find on its own, or grading by scripts, rubrics and human review. It supports more agents (Gemini CLI, Codex CLI, ACP-compatible agents) and writes JSON, JUnit, CSV and Markdown reports.

skill-be is the better fit when the question is whether a skill is worth shipping: its effect against a baseline, regressions on critical scenarios, and a gate that costs nothing to run in CI.

Some of skillgrade’s strengths are open ideas for skill-be: deterministic graders, JUnit reports, drafting evals from a SKILL.md, agentic tasks, a discovery mode and more adapters.

promptfoo is a general tool for testing prompts and models. It runs every prompt against every provider and checks each answer with assertions: string and JSON checks, a rubric graded by a model (llm-rubric), and custom JavaScript or Python functions. It shows the results in a local web viewer, runs in CI, and also does red teaming.

The two overlap. Two prompts in promptfoo, one with the skill text and one without, are a baseline and a with-skill arm; llm-rubric plays the part of skill-be’s criteria; a JavaScript assertion can take the code out of an answer and test it. What skill-be adds is what is specific to skills:

  • baseline versus with-skill as the shape of every scenario, with the skill pasted in by withSkillTemplate;
  • a gate that blocks a criterion at 0% with the skill and a critical scenario that gets worse with it, not only a pass rate;
  • verify, which checks a committed result file in CI without a model call;
  • the typecheck of the examples in each SKILL.md, the coverage check, and triage of failing criteria;
  • adapters that run Claude Code, pi and opencode without the user’s settings, tools or project files;
  • code tests that run each answer in its own process, with a package list and an output contract that both arms get.

promptfoo is the better fit for comparing many prompts or providers at once, for red teaming, or when skills are a small part of what you test. skill-be is the better fit when the question is whether a skill is worth shipping.