Secrets of flaky test detection and fixing in CI pipelines

A flaky test passes and fails without a relevant code change. That behaviour makes a CI pipeline difficult to trust: a green build can hide a defect, while a red build can send developers chasing an environmental accident. The damage grows over time because teams learn to rerun jobs, ignore warnings, or quarantine tests indefinitely.

Flakiness is rarely a single problem. Timing assumptions, shared state, unstable test data, network dependencies, parallel execution, resource limits, and defects in the test framework can all produce intermittent results. A reliable response requires evidence across many pipeline runs rather than a quick guess based on the latest failure.

For Australian engineering teams, the operational context can make the pattern harder to see. A build triggered in Sydney may exercise services hosted in a different region, while colleagues in Perth or Melbourne investigate it hours apart. Daylight saving changes, distributed teams, and variable CI workloads can expose timing problems that never appear on a developer laptop.

What makes a test genuinely flaky

A test is flaky when the outcome changes under materially equivalent conditions. The key phrase is “materially equivalent”: a test that fails because an API correctly rejects invalid input is not flaky, even if the test author expected success. A test that alternates between success and failure for the same commit, inputs, and environment is a stronger candidate.

The first step is to distinguish flakiness from a legitimate intermittent production defect. Compare the failure message, stack trace, request payload, database state, browser or service logs, and commit history. If every failure points to the same reproducible defect, the test is exposing a real problem. If the failure location moves between an assertion, timeout, connection error, and missing record, unstable execution is more likely.

A useful measure is the test’s historical pass rate. A test that fails twice in 200 runs is different from one that fails once every five runs, but both create delivery risk. Track retries separately from first-attempt results. A retry that turns red to green is evidence of instability, not a successful test run. Store the original outcome so the pipeline does not erase the signal.

Build evidence into the CI pipeline

Flaky test detection works best when the pipeline records enough context to compare runs. Persist the commit SHA, branch, runner image, operating system, Java or Groovy version, dependency lockfile, test shard, duration, retry count, and relevant environment variables. For REST and asynchronous tests, capture request metadata, response status, correlation IDs, queue names, and consumer lag without exposing secrets.

Run suspicious tests repeatedly in a controlled job. A simple repetition mode can reveal intermittent failures, but it should vary execution order and parallel grouping as well. A test that passes 100 times in isolation may fail when another suite holds a database lock or consumes a shared port. Reproducing the production-like schedule is often more useful than merely increasing the loop count.

Classify failures by signature rather than by test name alone. Timeout exceptions, connection resets, assertion mismatches caused by ordering, missing files, and port binding errors each suggest different causes. A dashboard should show first-run failures, retry recoveries, failure frequency, duration changes, and the environments where failures cluster. This turns a vague complaint into a test reliability profile.

For teams using GitHub Actions, GitLab CI, Jenkins, or Buildkite, keep raw artefacts available after the job finishes. Screenshots, thread dumps, container logs, database logs, and service traces often explain an event that the short CI summary cannot. A retention period of several weeks helps reveal whether failures started after a runner image, dependency, or infrastructure change.

Find the source instead of adding retries

Timing races are among the most common causes of flaky backend and asynchronous tests. A test may sleep for two seconds and assume a message has been consumed, even though the queue, worker, database, and network are operating at different speeds. Replace fixed sleeps with bounded polling that checks a meaningful condition, such as an expected state transition, while retaining a clear timeout and diagnostic output.

A polling loop must be designed carefully. It should use a sensible interval, stop when the condition is satisfied, and report the last observed state when it times out. Blindly extending the timeout can make the suite slower without removing the race. For Java and Groovy systems, awaitility-style patterns or a small domain-specific wait helper are usually clearer than scattered Thread.sleep calls.

Shared mutable state is another frequent source of intermittent failures. Tests may reuse database rows, files, ports, queues, environment variables, static caches, or singleton objects. Parallel workers expose these assumptions quickly. Give each test isolated data, unique identifiers, independent resources, and reliable cleanup. When cleanup fails, make that failure visible instead of allowing the next test to inherit corrupted state.

External services deserve particular attention. A test that depends on a public API, SaaS provider, DNS, or a remote cloud endpoint can fail for reasons unrelated to the application under test. Use contract tests, controlled stubs, service virtualisation, or a local test container where appropriate. Keep a small set of explicit integration tests for real dependencies, and label them so their slower and less predictable nature does not contaminate every pull request.

Make test data and concurrency deterministic

Random data can find valuable edge cases, but it must be reproducible. Record the random seed, generator version, locale, timezone, and generated payload whenever a test fails. Without that information, a failure can disappear before a developer has a chance to inspect it. Property-based tests should print the smallest failing example after shrinking the input.

Time is another hidden input. Tests that use the current clock can fail around midnight, month-end, leap years, daylight saving transitions, or delayed event processing. Inject a clock and set it explicitly in the test. This matters in Australia because Sydney and Melbourne switch between AEST and AEDT, while Brisbane stays on standard time and Perth uses AWST. A test that assumes one local offset can behave differently across runners.

Ordering assumptions should be removed from both data and assertions. Database queries without an explicit ORDER BY, JSON comparisons that treat object order as meaningful, and assertions over unordered collections are common causes of false failures. Sort data deliberately when order matters, or compare sets and maps using semantics that match the API contract.

Concurrency needs deliberate control. Run the test suite with multiple workers, then repeat it with one worker and with altered shard boundaries. If only parallel runs fail, inspect locks, mutable fixtures, resource pools, and test cleanup. If only serial runs fail, look for state leaking from one test to another. Containerised databases and per-test schemas can cost more setup time, but they usually cost less than investigating invisible coupling.

Use retries as a measurement tool

Retries have a legitimate role, but they should not decide whether a change is safe. A single automatic retry can reduce disruption from a transient runner or network event, while the pipeline still marks the job as unstable and records the initial failure. A policy that silently retries until green converts a quality signal into noise.

Quarantine can be useful when a test blocks urgent delivery, but it needs ownership and an expiry date. Keep quarantined tests running in a separate visibility lane, report their failure rate, and link them to a tracking issue. A quarantine without a deadline becomes a permanent bypass, especially when the original author has moved to another team or project.

Prioritise repairs using impact rather than irritation. A test that fails once a month but protects payment processing may deserve attention before a frequently flaky cosmetic browser test. Consider pipeline interruption, developer time, release risk, customer impact, and the test’s diagnostic value. In a regulated or high-assurance system, preserve the complete history even when a test is temporarily excluded from merge blocking.

Retries can also help identify infrastructure faults. Compare the original and retry environments, runner host, network route, resource usage, and service health. If failures correlate with a particular container image or executor pool, the fix may belong in platform engineering rather than in the test code. A flaky test report should therefore include both software and infrastructure ownership.

Create a sustainable reliability practice

Set a reliability budget for the suite. For example, define an acceptable first-attempt failure rate for merge checks and a maximum age for unresolved flaky tests. Measure the rate per test, suite, repository, and pipeline stage. This gives engineering leaders a way to see whether reliability is improving rather than relying on anecdotes from the morning stand-up.

Treat test fixes like production changes. Reproduce the failure, write down the suspected mechanism, make the smallest correction, and run the test under the conditions that previously exposed it. Then run the broader suite to check for side effects. A useful pull request explains why the test was unstable, what evidence supports the fix, and how future failures will be diagnosed.

Keep CI environments close to the systems they represent, but remove accidental variation. Pin tool versions, control locale and timezone, define resource limits, and make service readiness explicit. For Australian teams using cloud regions in Sydney or Melbourne, check latency and failover paths rather than assuming the local region makes network behaviour deterministic. A local endpoint can still experience throttling, cold starts, or overloaded shared infrastructure.

Make reliability visible during delivery. A weekly review of recovered retries, new flaky signatures, quarantined tests, and mean time to repair can identify trends before trust collapses. Use plain language in team discussions: “the test failed on the first attempt and passed on retry” is more useful than calling the build “a bit dodgy”. Clear terminology encourages people to fix causes rather than normalise noise.

A dependable CI pipeline does not require every test to be perfectly isolated from reality. It requires the suite to reveal genuine failures, expose environmental limits, and provide enough evidence to separate the two. Start with the ten tests that have the highest retry-recovery rate, run each under varied order and parallelism, and open a repair issue containing its first failure artefacts and reproduction details.