---
name: flaky-test-hunter
description: Prove whether a suspect test is actually flaky before anyone edits it or the code under it. Use when a test passes and fails without code changes - re-runs it under shuffled order, random seeds, and repeated execution, then separates order-dependence from time-dependence from genuine product bugs.
---

# Flaky Test Hunter

## When to use

- A test fails intermittently with no related code change
- Someone is about to "fix" it by deleting or weakening it

Do NOT use on tests that fail consistently. A consistent failure is a
bug report, not a flake - hand it back as one.

## Phase 1: Reproduce the flake

1. Run the suspect test alone, N times. Record pass/fail per run -
   never just the summary. Pick N from the failure rate you need to
   rule out: N=20 misses a 5% flake about 36% of the time and a 1%
   flake about 82% of the time. State your N and the miss probability
   you accepted.
2. Run the full file it lives in, N times, same recording.
3. Run the whole suite once with a random order seed, twice with two
   different fixed seeds.
4. Run the suite (or the file it lives in) once serial and once with
   parallel workers. A test that passes alone but fails only under
   parallelism points at shared ports, databases, or files - add that
   to the classification.

If the test never fails in any of these, report "not reproduced" and
name the conditions tried. Do not call it fixed.

## Phase 2: Classify the flake

Match the failure pattern to a cause before touching anything:

- **Order-dependent**: fails only after certain other tests run.
  Isolate the polluter: bisect by running the suspect test after each
  candidate predecessor (or the test runner's bisect mode) until one
  predecessor flips the result. Hunt for the shared state - globals,
  databases, files, env vars - that predecessor leaves dirty, and name
  the polluting test in the report.
- **Time-dependent**: fails under load, at certain wall-clock times,
  or near timeouts. Hunt for sleeps, fixed dates, and races.
- **Environment-dependent**: fails on one OS, CI runner, or machine
  only. Diff the environments before blaming the test.

## Phase 3: Fix the cause - never the symptom

The fix goes where the cause lives: reset the dirty state, inject the
clock, await the async work. Quarantining or retrying the test is a
last resort and must be labeled as such in the report.

## Output format

Report: the reproduction rate (x of N, with N and its miss
probability), the classification, the smallest condition that triggers
it, the polluting test if order-dependent, and the shared state or
race responsible.

## Hard boundaries

- Never delete, skip, or loosen a flaky test to make CI green.
- Never report a flake without the reproduction rate and the seeds
  tried.
- If the flake turns out to be a real product bug, say so plainly -
  that is a win, not a scope violation.
