Concept demoThe pack files are real and downloadable. All versions, counts and evidence numbers are staged sample data - no live benchmark runs yet.
Packs / Backend / api-change-safety-net / release report v0.3.1

Release report

api-change-safety-net v0.3.1 · 12 Nov 2026 · agent: claude-code · report pp-2026-1112-034 (sample)

6 tasks × 3 repetitions × 2 conditions = 36 runs. Same tasks, same model, same repo. The pack is the only difference. This is a staged demo report showing the format - the numbers are not from live runs.

Sample data

Tasks passed, baseline vs pack

BASE
8/18 · 44%
PACK
10/15 attempted · 67% · 3 correctly idle
3 improved1 regression1 no change1 correctly idle
Sample data

Every run, no cherry-picking

TaskBaseline repsPack repsVerdict
endpoint-edit✓✓✗ 2/3✓✓✓ 3/3Improved
add-query-param✓✗✗ 1/3✓✓✗ 2/3Improved
rename-field✗✓✗ 1/3✓✗✓ 2/3Improved
deprecate-route✓✓✗ 2/3✓✗✗ 1/3Regression
add-middleware✓✓✗ 2/3✓✓✗ 2/3No change
change-response-shape✗✗✗ 0/3idle idle idle 3 idleCorrectly idle
Kept failures - deprecate-route regressed: with the pack, 2 of 3 runs missed the docs update. Kept in the report, not hidden - fix planned for v0.3.2. Legacy DELETE 500 stays documented, not fixed. Run 14 timeout was a flake; the rerun passed.

How to reproduce

Pinned in this report (sample values): seed pp034-20261112, model claude-sonnet-4.6, task-repo snapshot pp-tasks-2026-11-12. Environment notes and raw logs are attached to the release. This report is staged sample data for the concept stage - the structure is final, the numbers are not from live runs.
Concept stage - counts, versions and statuses marked as sample data are staged examples, not live benchmark results.Release report · Ajay Mathuriya · @codesitbetter