Almost every team we talk to has at least one test that fails now and then for no obvious reason. The usual response is to hit rerun and move on. That works until the reruns pile up and nobody trusts the suite anymore. Here is a simple process for working through one flaky test this week, start to finish.
1. Pick one test and write down what you know
Do not try to fix every flaky test at once. Choose the one that fails most often or blocks the most builds. Before touching code, note when it fails, on which machines or agents, and what the error message says. Pull the last several failures from your CI history if you can. Patterns often show up right away, like failures only on one runner or only in the first job of the morning.
2. Reproduce it by running it many times
A test that fails one time in twenty will not fail on demand, so run it in a loop. Most frameworks have a way to do this. Playwright has a --repeat-each option, JUnit 5 has @RepeatedTest, and pytest has the open source pytest-repeat plugin with a --count option. Run it 50 or 100 times locally, then try the same in CI. If it never fails locally but fails in CI, that difference is your first real clue.
3. Change one condition at a time
Once you can make it fail, start narrowing things down. Try these one at a time:
- Run it alone, then with the full suite. If it only fails alongside other tests, look for shared state such as leftover database rows, cached files, or a global setting another test changes.
- Change the order. Randomizing test order can expose hidden dependencies between tests.
- Slow the machine down or speed it up. Running under load, or with fewer CPUs, often exposes timing problems.
- Check the clock and the data. Tests that depend on the current date, time zones, or random values tend to fail on specific days or inputs.
4. Look for the usual causes
Most flaky tests come down to a handful of things: fixed sleeps instead of waiting for a real condition, tests that depend on each other, shared test data, external services that are slow or unavailable, and assumptions about ordering in lists or async results. When you find the cause, fix that cause. Raising a timeout from five seconds to thirty usually just hides the problem until the day it does not.
5. Prove the fix and record what you learned
Run the same loop again. If it was failing one time in twenty before, a clean run of 100 or more gives you some confidence. Then write two or three sentences in the commit message or ticket about what caused it. That note helps the next person who sees a similar failure, and over time it becomes a useful list of the problems your codebase tends to have.
If you cannot fix it today
Quarantining a test (moving it out of the blocking suite while it gets fixed) is reasonable, as long as it has an owner and a ticket. A quarantine list with no owners just becomes a list of tests nobody runs.
We would like to hear what has worked for your team. Tell us your best flaky test story at info@nosqc.org, and if you work in software quality in Northern Ohio, join us at nosqc.org/join.
Dave