Flaky Automated Tests (Dev Quality Guide)
A flaky automated test passes and fails without a meaningful change to the code or test environment. To fix one, record the run conditions, reproduce the failure, identify its cause, and verify a targeted change with repeated runs and normal CI. Retries can help diagnose a problem, but they do not prove the test is reliable or the software is correct.
A test that fails once, passes on rerun, then fails again can stall a release and leave the team unsure what to trust. The first task is to separate a real product defect from an unstable test or test environment. The next is to preserve enough evidence to find the cause.
I approach flaky-test reports as a quality investigation, not a request to make CI green at any cost. The steps below work for coursework, team projects, and production test suites. There is no universal acceptable flake rate; teams should set a local target and state how they measure it.
Why flaky tests matter
A flaky test gives different results when the code and relevant environment conditions are materially the same. It weakens confidence in test results: a failure may be real, while a passing rerun may simply hide it. Treat flakiness as a quality issue to investigate, not just a CI nuisance.
Unstable tests can waste developer time, slow delivery, and conceal regressions when teams rely on retries or quarantine tests indefinitely. A test suite is useful only when people can act on its results. That does not mean every intermittent failure comes from the test itself: infrastructure, resource limits, or external services may also be involved.
No universal flake-rate limit or accreditation requirement applies to every team. Use any target as a local service objective, not as an industry-wide rule. Keep the test’s purpose visible, too: map tests to project requirements or course outcomes, and apply ISO/IEC/IEEE 29119 guidance where your organization has adopted it.
Measure flakiness before changing tests
A flake rate describes the share of test executions that produce an unexplained intermittent outcome within a stated window. Report both the count and denominator, along with the time period and conditions. Without those details, a percentage can hide whether the suite ran ten times or ten thousand.
Start with a baseline:
- Inventory the test suites and note their runtimes, runners, dependencies, and retry settings.
- Record failures and reruns over a defined period, such as one week.
- Identify tests that fail and pass under materially equivalent conditions.
- Separate product failures from confirmed infrastructure incidents where evidence allows.
For example, five unexplained intermittent outcomes in 1,000 executions equal 0.5%. A team might set ≤0.5% across at least 1,000 repeated executions as a local objective. This is an example threshold, not a standard. Define what counts as an execution and decide whether the metric covers one test, a suite, or all CI tests.
| Metric | What to report | Why it helps |
|---|---|---|
| Flaky outcomes per execution | Count, denominator, and time window | Shows the rate and the amount of evidence |
| Rerun recovery rate | Failures that pass on rerun, with total initial failures | Finds intermittent behavior but does not prove its cause |
| Test duration | Median or typical runtime, plus unusually long runs | Helps reveal timeouts and resource pressure |
| Quarantined tests | Test name, owner, start date, and removal date | Makes temporary risk visible |
Do not combine every red CI result into a flake metric. A repeatable failure may point to a product defect or a broken test expectation, not flakiness. Keep categories distinct where possible.
Diagnose intermittent failures with evidence
Diagnosis means collecting the conditions around a failure before changing code. Preserve the failing output, then compare repeated runs under controlled conditions. This helps distinguish timing, shared state, nondeterminism, order dependence, external-service behavior, resource contention, and infrastructure problems.
For each event, record:
- Test name and commit identifier.
- Runner, operating system, runtime, and dependency versions.
- Test seed, timestamps, logs, and relevant resource or network conditions.
- Whether the run was local, on CI, or on an independent worker.
- Retry policy and the result of every attempt.
A fixed seed helps reproduce a known run. Randomized seeds can help discover failures tied to generated data or execution order. Record each seed so you can repeat a failure rather than losing the evidence when randomness changes.
Reproduce the failure in a controlled environment
Run the failing test at least 20 times in the same recorded environment. This is a practical diagnostic protocol, not a guarantee that a rare failure will appear. Then compare results in a clean environment or on an independent worker to check whether the instability follows the test or the infrastructure.
For a Python test using pytest, a simple loop can collect repeated results:
i=1
while [ "$i" -le 20 ]; do
printf 'Run %s at ' "$i"
date -u
pytest tests/test_checkout.py -q
printf 'Exit code: %s\n' "$?"
i=$((i + 1))
done
Save the full output to a log in a real investigation. Keep the commit, environment, and test data fixed for this first pass. If the test depends on a random seed, pass the same seed each time using the option supported by your test framework or plugin.
Classify the likely cause
A useful classification narrows the next experiment. Timing failures may depend on a sleep or timeout; shared-state failures may depend on leftover files or database rows. Order dependence may appear only after another test runs. External-service failures may track network or service conditions, while resource contention may appear under a busy worker.
| Suspected cause | Clue to investigate | First experiment |
|---|---|---|
| Timing | Failure changes with machine load or run speed | Replace fixed waits with a condition-based wait |
| Shared state | Test passes alone but fails in a suite | Isolate files, records, and global state |
| Nondeterminism | Results vary by seed or iteration | Fix and record the seed, then compare |
| Order dependence | Failure follows a particular test | Run the test alone and in different suite positions |
| External service | Failures align with network or service events | Use a fake for unit tests; keep focused contract tests |
| Resource contention | Failures rise on busy workers | Compare worker load and control concurrency |
| Infrastructure | Multiple unrelated tests fail together | Compare independent workers and runner health |
These clues are not proof by themselves. Change one factor at a time and keep the run evidence, so you can tell whether the experiment supports or weakens a theory.
Remediate and verify the fix
A remediation should remove the source of nondeterminism where possible, not simply make failures less visible. Common actions include isolating test state, controlling concurrency, using deterministic clocks and data, and replacing uncontrolled external dependencies with suitable fakes or contract tests.
Avoid fixed sleeps when the test can wait for a clear condition, such as a completed response or visible state change. Clean up files and records after each run. For time-based logic, inject a controllable clock if the design allows it. For concurrency, make shared resources explicit and verify whether the test needs parallel execution.
Once changed, repeat the test under the recorded conditions, then run it through normal CI. Include adjacent tests if order or shared state was suspected. A passing set of reruns is evidence that the change helped; it is not proof that every rare failure is gone. Keep watching the test after the fix.
Use retries and quarantine only as temporary controls
A retry repeats a failed test, often to keep a pipeline moving. It can help gather evidence, but a retry-passing result does not establish reliability or rule out a real defect. Indefinite retries can increase CI time and compute use while making failures harder to interpret.
Quarantine may be reasonable when a test blocks other work and the risk is understood. Give the test an owner, a reason, and an expiry or review date. Keep its failures visible in a separate report, and do not treat a passing retry as a fix. Remove the quarantine after the cause is addressed and the test is verified.
Put repeatable practices into CI
CI, or continuous integration, runs tests as changes enter a shared codebase. Reliable CI makes its environment and test behavior visible. Record runner and dependency versions, preserve useful logs, and make retry behavior clear so that developers can tell an initial failure from a later pass.
A practical team workflow is:
- Establish a baseline. Inventory suites, current failure and rerun rates, environments, and retry settings.
- Diagnose and classify. Reproduce the issue, preserve evidence, and test likely causes.
- Remediate and verify. Make a focused change, repeat the test, and check normal CI behavior.
- Govern and improve. Review the agreed local flake objective, quarantine dates, and unresolved tests during release readiness and follow-up.
Teams can adapt this sequence to their test policy. The key is to make ownership and evidence part of the process. A release decision should account for unresolved flaky tests rather than quietly treating them as green.
Practical failure walkthroughs
The examples below are illustrative scenarios, not reports from named organizations. They show how the same investigation method can lead to different fixes. A test that fails intermittently does not reveal its cause on its own; the run conditions and comparisons matter.
Scenario 1: A checkout test fails under load. The test waits a fixed number of seconds for a response. Repeated runs fail more often on a busy CI worker. The team replaces the fixed delay with a wait for the expected response, then repeats the test and checks it in normal CI. This supports a timing cause; it does not prove all timing issues are gone.
Scenario 2: A data test fails after a suite run. The test passes alone but fails after another test writes to a shared database. The team gives each run isolated test data and adds cleanup, then checks both tests together and in different orders. The important evidence is the contrast between isolated and suite runs.
Scenario 3: A service test fails when the network is unstable. The test calls a live service, so results vary with conditions outside the code under test. The team uses a fake for the unit-level behavior and retains a focused contract test for the service boundary. That separates local logic checks from integration risk without claiming the external service can never fail.
Build reliability skills and evidence
Flaky-test work develops practical skills in test design, debugging, CI operations, and technical communication. A useful project record shows the failure, the environment, the experiment, and the outcome. That gives classmates, teammates, reviewers, and future maintainers a way to judge the reasoning rather than just the final green check.
For a course or portfolio project, include a small reliability report with the test name, run count, observed failures, environment details, likely cause, fix, and post-fix results. Be clear about limits: a short run may not expose a rare event. This is stronger evidence than reporting only that a test passed once.
Frequently asked questions
These answers cover common decisions when a test passes and fails across runs. Use them as practical guidance, then follow the project’s own test and release policies. If evidence is incomplete, keep the uncertainty visible instead of declaring the test fixed.
What is a flaky automated test?
It is a test that produces different outcomes under materially equivalent code and environment conditions.
How many times should I rerun a failing test?
Run it at least 20 times in the same recorded environment as a practical starting point, then compare with a clean environment or independent worker.
Does passing on retry mean the test is reliable?
No. A pass on retry shows that results vary; it does not prove the test or product is correct.
What is an acceptable flake rate?
There is no universal rate. Set a local target, state the measurement window and denominator, and review whether the threshold fits your project’s risk.
Should I use fixed or random seeds?
Use a fixed, recorded seed to reproduce a known failure. Use randomized seeds to search for issues, and record any seed that reveals one.
When should I quarantine a test?
Only when needed as a temporary control. Assign an owner, document the risk, keep failures visible, and set a review or removal date.
Are flaky tests always caused by bad test code?
No. Timing, shared state, external services, resource contention, and infrastructure can also produce intermittent outcomes.
How do I know whether a fix worked?
Repeat the test under recorded conditions, run it through normal CI, and monitor subsequent results. Repeated success is evidence, not a guarantee.
Should every external-service test use a fake?
No. Fakes can isolate unit behavior, while focused contract or integration tests can check service boundaries. Choose the test type that matches the behavior you need to verify.
The durable goal is not a perfectly quiet dashboard. It is a test suite whose failures are understood, whose limits are visible, and whose results help the team make sound decisions.
(This article was written by one of our staff writers, James Holloway. Visit our Meet the Team page.)
