claude plugin eval: scoring a Claude Code plugin before you ship it
In brief
Claude Code v2.1.269, released September 11, 2026, added `claude plugin eval` — a command that runs a plugin's eval suite against Claude Code and returns scored, reproducible results as JSON and an HTML report. If you ship plugins to a team or a client, this is the test suite you have been writing by hand.
Contents
A Claude Code plugin bundles skills, slash commands, hooks, subagents, and MCP server config into something you can hand to someone else. The hard part has never been building one. The hard part is knowing whether it still works after you change a skill description, or whether it works at all on someone else's machine.
claude plugin eval, shipped in Claude Code v2.1.269 on September 11, 2026, runs a plugin's eval suite against Claude Code and gives you a score. Output comes back as JSON and as an HTML report.
claude plugin eval --help
Why a plugin needs evals at all
Plugins fail in ways ordinary software does not. The failures are rarely exceptions:
- A skill does not trigger. The description is too narrow, or it overlaps with another skill and loses. The plugin is installed, the user asks the thing it was built for, and nothing fires. No error — just the default behavior.
- A skill triggers when it should not. The description is too broad and it hijacks unrelated requests.
- A slash command works for you and not for them. It depends on a binary, an env var, or a directory that happens to exist on your machine.
- A change to one skill breaks another. Two descriptions that used to be distinguishable stop being distinguishable.
None of these are caught by reading the code. They are caught by running the plugin against realistic prompts and checking what actually happened — which is what an eval suite is. If you have read Writing evals that catch regressions, the logic is identical; the unit under test is a plugin instead of an agent.
What "reproducible scoring" buys you
The word doing the work in the release note is reproducible. A model's output varies run to run, so a single pass tells you little — you can flip a skill description for the worse and still get a green run by luck.
Scored, repeatable runs give you two things you cannot get by trying it once:
- A baseline. Score the plugin today, change a skill description, score it again. The delta is the signal. This is the only honest way to answer "did that edit help?"
- A gate. A score threshold is something CI can enforce. Without a number, "the plugin works" is a claim somebody made on a Tuesday.
The HTML report is the artifact you show someone else. The JSON is what you assert against in a pipeline.
Where this matters most
If you ship plugins to clients. This is the piece that has been missing from client handoff. A plugin you hand over with no eval suite is a plugin that degrades quietly after you leave — someone edits a skill, nobody notices the trigger stopped firing, and six weeks later the team concludes "the AI thing didn't work." An eval suite in the repo turns that into a failing check.
If you maintain an internal plugin for a team. The same problem at a smaller radius. The person who edits the plugin in month four is not the person who wrote it.
If your plugin wraps an MCP server. MCP tool descriptions are prompts. Changing one changes model behavior, and nothing in your type system notices.
Practical setup
A reasonable starting point, whatever your suite format ends up looking like:
- Write the cases from real transcripts, not imagination. Pull the prompts people actually typed. Your guesses about how someone will phrase a request are worse than the log.
- Include negative cases. For each skill, at least one prompt that should not trigger it. Over-triggering is the more common failure and the one you will never notice by hand.
- Run it before every release, not after a bug. A suite you only run when something is broken is a debugging tool, not a gate.
- Wire the JSON output into CI.
claude plugin evalprinting a good number on your laptop is not a guarantee about anyone else's.
Run claude plugin eval --help for the current flags — this landed in September 2026 and the interface is young.
Also in v2.1.269
Two other changes in the same release worth knowing:
/output-stylelists and switches output styles, and works in Remote Control, cloud sessions, and other headless sessions — so the style is no longer locked to what an interactive session picked.- Bash diff output shows which files a Bash command changed, when
bashEditDiffEnabledis set. Useful when an agent runs a script and you want to see the blast radius without diffing by hand.