Packs / Backend / api-change-safety-net / release report v0.3.1
Release report
api-change-safety-net v0.3.1 · 12 Nov 2026 · agent: claude-code · report pp-2026-1112-034 (sample)
6 tasks × 3 repetitions × 2 conditions = 36 runs. Same tasks, same model, same repo. The pack is the only difference. This is a staged demo report showing the format - the numbers are not from live runs.
Sample data
Tasks passed, baseline vs pack
BASE8/18 · 44%
PACK10/15 attempted · 67% · 3 correctly idle
3 improved1 regression1 no change1 correctly idle
Sample data
Every run, no cherry-picking
| Task | Baseline reps | Pack reps | Verdict |
|---|---|---|---|
| endpoint-edit | ✓✓✗ 2/3 | ✓✓✓ 3/3 | Improved |
| add-query-param | ✓✗✗ 1/3 | ✓✓✗ 2/3 | Improved |
| rename-field | ✗✓✗ 1/3 | ✓✗✓ 2/3 | Improved |
| deprecate-route | ✓✓✗ 2/3 | ✓✗✗ 1/3 | Regression |
| add-middleware | ✓✓✗ 2/3 | ✓✓✗ 2/3 | No change |
| change-response-shape | ✗✗✗ 0/3 | idle idle idle 3 idle | Correctly idle |
Kept failures - deprecate-route regressed: with the pack, 2 of 3 runs missed the docs update. Kept in the report, not hidden - fix planned for v0.3.2. Legacy DELETE 500 stays documented, not fixed. Run 14 timeout was a flake; the rerun passed.
How to reproduce
Pinned in this report (sample values): seed pp034-20261112, model claude-sonnet-4.6, task-repo snapshot pp-tasks-2026-11-12. Environment notes and raw logs are attached to the release. This report is staged sample data for the concept stage - the structure is final, the numbers are not from live runs.