How a pack earns its proof
1.Same tasks, same model, same repo
A release report runs each task twice: once with the agent alone (baseline), once with the pack installed. The only difference between the two halves is the pack file. Task set, model version, seeds and the repo snapshot are pinned in the report so anyone can rerun it.
2.Repetitions, not anecdotes
Every task runs three times per condition. Agents are nondeterministic; a single lucky run proves nothing. The report shows every repetition - the ticks, not just the totals.
3.Regressions stay in the report
A pack that improves five tasks and breaks one is a pack that breaks one. Regressions and no-change results are printed next to the wins, with a named fix version. A release that hides a failure does not ship.
4.Not measured means not measured
Packs still in testing show an empty, dashed meter - never a partial score. A status of "partial" means exactly which half of the runs is done. Nothing rounds up.
5.What a pack is
A pack is a small workflow document your agent reads - rules, order of operations, hard boundaries. It is not a plugin, not a binary, and it never runs code on your machine. Installing is downloading one file into your agent's skills folder; removing is deleting it.
- Reads your repo and the diff you ask about
- Writes only inside the pack's own folder
- No account, no telemetry, no network at run time
Concept stage note: the registry currently ships staged sample evidence so the report format can be reviewed. Live benchmark runs are the next milestone.