How to Fix Flaky Tests in GitHub Actions

Key Takeaways

• A test that passes on a laptop and fails on ubuntu-latest is usually a machine problem before it is a test problem. Private-repo Linux runners ship 2 vCPU and 8 GB, half the public allocation.
• Google measured about 1.5% of all test runs returning a flaky result, with roughly 16% of its tests carrying some flakiness, and 84% of pass-to-fail transitions involving a flaky test.
• Auto-retry is the most-reached-for fix and the one that guarantees the underlying defect survives the sprint.
• Quarantine is a gate decision, not a deletion. Move the test off the blocking path, keep it executing, and set an expiry date.
• Autosana removes the selector layer by re-planning against the current UI every run, and hosted real devices remove emulator boot flake. Fixture and concurrency flake stay your pipeline's problem.

Which Layer Does Your Flaky Test Live In?

Why Do Tests Pass Locally and Fail in GitHub Actions?

Your laptop has performance cores, 32 GB of RAM, a warm GPU context, and a browser process resident for six hours. A private-repo ubuntu-latest runner, per GitHub's hosted runner reference, gives you 2 vCPU and 8 GB. Public repos on the same label get 4 vCPU and 16 GB. No GPU on either. The filesystem was empty ninety seconds ago.

Same workflow file. Same image. Two very different machines.

The Actions log does not tell you which difference caused the red. It tells you a step exited non-zero and prints 50 lines of stdout. Everything after that is inference, and the first inference is usually "the test is flaky" rather than "the environment is different."

What Are the Five Layers of Flaky Tests in GitHub Actions?

Luo and colleagues at Illinois analysed flaky-test fixes across 51 open-source projects and found the three largest categories were async wait, concurrency, and test-order dependency, in that order. Two of three are environment problems in a test costume.

How Do You Know If the GitHub Actions Runner Is Causing the Flake?

Signature: failure correlates with wall-clock duration, not code.

On Android, the usual culprit is hardware acceleration. The default emulator-options in reactivecircus/android-emulator-runner include -gpu swiftshader_indirect, which is software rasterisation. Every frame your test waits on is composited by the CPU, on a runner that has two of them. The action's emulator-boot-timeout defaults to 600 seconds, which tells you how long boot can legitimately take.

GitHub shipped hardware-accelerated Android virtualisation on larger hosted runners in 2023, but KVM is off by default. Grant it:

Three checks confirm this layer. Compare median step duration on passing against failing runs; slower failures mean the machine. Re-run the same SHA three times and watch whether the failing step moves; if it does, the code is not at fault. Then run the identical job on a 4-vCPU runner. If it stays green, you have your answer and your invoice.

Why Do Parallel GitHub Actions Jobs Cause Flaky Tests?

Signature: stable in isolation, unstable in a matrix.

A six-way matrix against one staging environment means six jobs logging into qa-user@example.com, six mutating the same cart, six racing the same "mark all read" endpoint. The test is correct. The fixture is a shared mutable global with six writers.

This layer punishes scaling. It is invisible at matrix width one and appears the week someone adds iOS 17 and 18 to the device list.

Two fixes. Namespace every fixture by ${{ github.run_id }}-${{ strategy.job-index }} so no two jobs touch the same row, and use workflow concurrency groups to stop a new push running against an environment a previous run is still mutating.

How Do Test Fixtures Cause Flaky Tests in CI?

Signature: failures correlate with the calendar, or with what ran before.

Suspects: a test hitting live staging that inherits whatever shipped at 4pm, an OTP or JWT expiring mid-suite, a seeded record another test consumed, a date assertion that holds in UTC and breaks on a locale change. Test-order dependency belongs here too, the third-largest category in the Illinois data.

If your suite calls a third-party sandbox, that vendor's uptime is now part of your merge gate. Service containers stand the dependency up inside the job instead of reaching across the internet.

When Is a Flaky Test Really a Bad Assertion?

Signature: the same step fails intermittently, on a different element each time.

Every article covers this layer, so the short version. A fixed sleep is a bet about a machine you do not control. On a 2-vCPU runner you lose that bet between 2% and 10% of the time, exactly the frequency that makes a test annoying rather than obviously broken. Wait on the condition. Autosana's breakdown of why E2E suites fail at waiting goes deeper.

Why Does a Test Break Every Time the UI Changes?

Signature: not intermittent at all. It starts at one commit and never stops.

Strictly this is not flake. It is breakage on a schedule nobody controls, filed as flake because it surfaces as a red E2E job after a merge that touched no test code. A selector pinned to .btn-primary-v2 is a contract with a stylesheet, and nobody in the frontend review thread knows it exists.

It belongs in this taxonomy because of where triage hours go: selector breakage eats the "is this real or is this flake" budget while looking identical to the other four layers in the Actions summary.

Why Does Auto-Retry Make Flaky Tests in GitHub Actions Worse?

Retry is one line: retries: 2 in Playwright, --rerun-tasks in Gradle, a nick-fields/retry wrapper. The job turns green by lunchtime and the ticket closes.

The cost, in Google's numbers. Google reported about 1.5% of test runs returning a flaky result, with almost 16% of tests showing some flakiness. Then this, on their own three-strikes retry policy: it "reduces false positives, but encourages developers to ignore flakiness in their own tests unless their tests start failing 3 times in a row."

That is the mechanism. Retry does not make tests reliable. It raises the threshold at which unreliability becomes visible, and engineers calibrate to whatever threshold they are given.

The compounding problem is on the other side. Google also found 84% of observed pass-to-fail transitions involved a flaky test. When most reds are noise, the genuine regression arrives in the same costume, and somebody closes it with a re-run.

Retry is defensible as measurement: run it, record that it happened. Playwright's retry documentation describes the flaky outcome for exactly this, reporting a failed-then-passed test separately rather than folding it into green. Treat that count as a bug backlog. If a retry is indistinguishable from a first-attempt pass in your reporting, you bought silence.

How Do You Quarantine a Flaky Test Without Losing Coverage?

Quarantine sits between "block every merge" and "delete the test."

• Move it out of the required check into a non-blocking job, so the gate works again.
• Keep executing it on the same trigger. A quarantined test that stops running stops producing the data you need to fix it.
• Tag it with the layer from the diagnostic table above. "Flaky" as a label tells the next engineer nothing.
• Set an expiry in the issue, usually two sprints.

The expiry is the part teams skip, and skipping it turns quarantine into a graveyard. A test parked six months is dead code with a CI bill.

One rule on what never gets quarantined. If the flow is in your smoke suite, it does not go to quarantine. A flaky login test is not a quarantine candidate, it is an incident, because the thing being measured is the thing that gates everything else.

How Do You Measure Flake Rate as a CI Metric?

Per test, over a rolling 30 days:

flake rate = (runs that failed then passed on the same SHA) / (total runs)

Keyed on the SHA, which makes it a flake rather than a regression. Below 0.5%, leave it. Between 0.5% and 2%, ticket the layer. Above 2%, quarantine today: at 2%, a 200-test suite reports red on roughly 98% of runs from noise alone.

Three numbers belong beside it: suite-level first-attempt pass rate, which predicts whether engineers trust the gate; mean retries per green run, the number that quietly climbs for a year; and flake rate split by runner label, since a gap between ubuntu-latest and a 4-vCPU runner is a free runner-layer diagnosis.

Flake rate needs an owner the same way latency does. Unowned, it drifts to whatever level is individually tolerable and collectively fatal.

Which GitHub Actions Settings Actually Reduce Flake?

Six changes, each mapped to a layer.

• timeout-minutes on the job and long steps. The default is 360, so a hung emulator burns six hours of billed time before anyone notices. Ten minutes converts a hang into a fast, readable failure. (Runner)
• fail-fast: false on the matrix. The default cancels sibling jobs the moment one fails, destroying the evidence that separates a one-device problem from a six-device problem. (Concurrency)
• Larger runner with KVM for Android emulator jobs, per the snippet above. Highest-yield single change for mobile suites. (Runner)
• A concurrency group keyed on the ref, so a rapid second push cannot run against an environment the first is still mutating. (Concurrency)
• Verify caches actually hit rather than merely exist. A cache-hit of false on every run is a silent multi-minute cost. Run caching covers the same arithmetic on the execution side. (Runner)
• Always upload artifacts on failure: if: always() with screenshots, video, and device log. Triage without artifacts is guesswork. (All layers)

A QA lead on r/QualityAssurance made the point that automation reliability is mostly an infrastructure discipline rather than a test-writing one, which is where this list lands from the other direction.

Which Flake Fix Should You Apply First?

Route by signature, not preference.

• If a re-run of the identical SHA passes and the failing step moves, start at the runner layer. Try a larger runner before touching test code.
• Stable at matrix width one, unstable at six: concurrency layer. Namespace the fixtures.
• Failures cluster at a time of day or follow a specific job: fixture layer. Seed per run, freeze the clock.
• One step failing on a different element each time: assertion layer. Replace the duration wait with a condition wait.
• Failure began at one commit and never stopped: selector layer, and not flake. Rewrite the selector or move that flow to an intent-based runner.

How Autosana Fits

Autosana is an AI-native E2E testing agent for iOS, Android, and web. Flows are intent, not selectors, and the agent re-plans against the current UI every run.

That removes the selector layer outright: a renamed test ID or restructured navigation stack cannot produce a red job, because no pinned selector exists to break. It removes much of the assertion layer too, since the agent decides a screen is ready by looking at it rather than counting down a sleep tuned on somebody's laptop.

For the runner layer, Autosana runs on hosted real devices rather than emulators inside your Actions container. No KVM config, no swiftshader compositing, no boot timeout. Results return through the GitHub integration as a PR comment with session replay attached, so triage happens where the review already is. Coding agents can invoke the same flows through the MCP server.

Two layers it does not fix. Fixture flake is yours: if six parallel runs share one staging account, an agent races itself as cleanly as a scripted test does. Concurrency flake is yours for the same reason. Autosana gives you environments to separate configuration, but cannot make your staging database multi-tenant.

Autosana deletes the two layers that generate the most maintenance work, and neither of the two behind the most mysterious 3am reds.

Conclusion

Pull your last 200 Actions runs and count how many greens needed a second attempt. That ratio is your real pass rate, and it is what your merge gate actually enforces.

Then assign your three noisiest tests a layer from the table above. Where all three land in one layer, there is one fix, not three. Start with a larger runner on the mobile jobs: one line, and it settles the runner-layer question in an afternoon.

FAQ

Why do my tests pass locally but fail in GitHub Actions?

The runner is a different machine. Private-repo ubuntu-latest provides 2 vCPU and 8 GB with no GPU, against a laptop with many times that. Timing-sensitive steps that complete in 200ms locally can exceed a timeout under CPU contention, which reads as flake but is resource starvation.

How do I stop flaky tests in GitHub Actions without using retries?

Diagnose the layer first. Re-run the same SHA three times and watch whether the failing step moves. A moving failure points to the runner or concurrency layer, and the fix is a larger runner or namespaced fixtures. A fixed failure point is an assertion or selector problem.

Does adding retries fix flaky tests?

No. Retries raise the threshold at which unreliability becomes visible. Google reported that its own three-strikes retry policy encouraged engineers to ignore flakiness below that bar. Use retries to label and count flaky outcomes, never to convert them into silent passes.

What is an acceptable flake rate for a CI test suite?

Below 0.5% per test over 30 days needs no action. Between 0.5% and 2%, open a ticket. Above 2%, quarantine immediately, because at that rate a 200-test suite reports red on nearly every run from noise alone.

Why does the Android emulator time out in GitHub Actions?

Hardware acceleration is usually off. The common emulator action defaults to -gpu swiftshader_indirect, which composites in software, and KVM requires an explicit udev rule in your workflow. Its boot timeout defaults to 600 seconds, which indicates how slow an unaccelerated boot can be.

How do you quarantine a flaky test properly?

Move it out of the required check into a non-blocking job, keep it running on the same trigger, tag it with the failing layer rather than with "flaky," and set a written expiry of about two sprints. Smoke-suite flows are never quarantined.

Can Autosana fix flaky tests in GitHub Actions?

It removes the selector layer and much of the assertion layer, because the agent re-plans against the current UI each run instead of binding to selectors or sleeps. Hosted real devices remove emulator boot flake. It does not fix shared-fixture or shared-environment flake.

Where does an AI testing agent not help with CI flakiness?

Anywhere the flake comes from shared mutable state. Parallel jobs writing to one test account, an unseeded database, an expiring OTP, or a third-party sandbox outage are pipeline design problems. No agent can resolve a race between two jobs competing for the same row.