End-to-End Testing: Why Most Suites Fail at Waiting

TL;DR

End-to-end testing verifies a complete user journey through a real build against real services. The received wisdom is to have very few of them because they are slow and flaky. Slow is true. Flaky is not inherent: it comes almost entirely from synchronization, meaning the framework guesses wrong about when your app has finished working. Espresso solves part of this automatically and documents exactly where it stops. XCUITest does not solve it at all. Understanding that boundary is most of what separates an E2E suite people trust from one they rerun.

Every team that has been burned by end-to-end tests arrives at the same conclusion, which is to have fewer of them. It is reasonable advice, and it is treating a symptom.

The reason it feels right is that the pain is real and correlated with count. Ten E2E tests are annoying; two hundred are a second job. But count is not the variable that determines whether the suite is trusted. Plenty of teams run two hundred and believe the results, and plenty run twelve and re-run all twelve when one goes red. The variable that actually separates those two teams is whether the tests know when to look.

What is end-to-end testing?

End-to-end testing exercises a complete user journey through the real application, against real or realistic backing services, asserting what the user would see. Sign in, search, add to cart, check out, confirm the order appears.

It is the only layer that tests the wiring. Unit tests verify components in isolation, and API tests verify contracts, and a product can pass both completely while being broken, because the failure lives in how the pieces were connected. That is not a gap you can close by adding more of the cheaper layers, which is why the E2E layer survives despite being the most expensive one to own.

End-to-end testing vs integration testing vs unit testing

The distinction people get wrong is between integration and end-to-end. Integration tests can and usually do stub the network. An end-to-end test that stubs the network has stopped being an end-to-end test, because the thing it was uniquely positioned to catch was whether the real call works from the real client.

The real reason E2E tests flake

Flakiness gets attributed to breadth, as though touching more of the system makes a test statistically likelier to break. That is not the mechanism. The mechanism is that the test acts before the app is ready, and every framework has a different answer to the question of how it knows.

Espresso's answer is the most developed, and its documentation is unusually honest about the limits. By default Espresso waits for UI events in the current message queue to be processed and for the default AsyncTask thread pool to complete before moving to the next operation. That covers a lot. But Android's own guidance states plainly that because Espresso is not aware of any other asynchronous operations, including those running on a background thread, it cannot provide its synchronisation guarantees in those situations.

Read that against how apps are actually written now. Almost nobody uses AsyncTask any more. Work happens in coroutines, in RxJava, and in a custom thread pool executor. All of that is invisible to Espresso's idle detection, so the test proceeds while the app is still fetching, and the failure surfaces as a view not matching because what is on screen is not yet what the test was told to expect. The fix is to register idling resources, which means your production code now carries test awareness so that the test can ask it whether it is busy.

The registration mechanism and its performance envelope are laid out in Android's IdlingResource guide, including the exact interface a production component implements to tell Espresso it is still working. The guide is also honest that anything running outside the default AsyncTask thread pool is invisible until you register it, which is the load-bearing detail behind most coroutine-era flake reports.

XCUITest does not attempt this at all. It runs out of process and offers existence-with-timeout waiting, which is polling with a nicer name. Every wait is a number someone guessed, and every guessed number is a race that resolves differently on a loaded CI machine than on a laptop.

So the honest statement about E2E flakiness is that it is a synchronization problem wearing a coverage costume. And it is worth knowing how much of your red is this rather than real: analysis at Google found that 84% of transitions from pass to fail involved a flaky test rather than a genuine breakage, against a background rate of roughly 1.5% of test runs reporting flaky results. Most red is noise, and the noise is mostly timing.

The 84% and 1.5% numbers are both from Google's post Flaky Tests at Google and How We Mitigate Them, which analyzes test-outcome telemetry across thousands of projects and remains the reference point for what "flake" actually is at scale. Reading it once is a defense against the common instinct to add tests to fix red, when the honest response is almost always to fix a synchronization bug.

What to test end to end

Because the layer is expensive, selection matters more here than anywhere else. Three filters:

Journeys that cross a boundary the lower layers cannot see. Sign-in, checkout, anything where the client, the network, and the backend all have to agree. This is the whole justification for the layer.

Journeys where failure is unacceptable rather than merely annoying. Revenue paths, data-writing paths, auth. Weight by consequence.

Journeys a real user actually completes. Not a screen, not a component state. If you cannot describe it as something a person came to the app to do, it probably belongs one layer down.

What to leave out: field validation rules, error message wording, anything with many variants and one code path. Those are cheaper and more thorough as unit tests, and putting them at the E2E layer is how a twelve-test suite becomes a two-hundred-test suite without becoming more useful.

How to write end-to-end tests that survive

Two failure sources, and they compound: the test waits wrong, and the test is bound to how the interface is built rather than what it does.

The second one is a phrasing discipline. An assertion on an element identifier breaks when someone renames it, and that break is not a regression; it is a maintenance tax that arrives looking exactly like a regression. Describing the outcome a user would observe survives redesigns that identifier-bound assertions do not. Our guidance on writing effective flow instructions makes the case for journey-level phrasing over step-by-step UI phrasing, and the argument is the same one: the durable thing is the intent, not the implementation.

The first one is why Autosana writes tests as natural-language flows interpreted by an agent against the running app rather than as scripts with waits in them. An agent evaluating whether the expected state has arrived does not need an idling resource registered in production code, and it does not need a number someone guessed. That does not make timing free, but it moves the problem out of your app's source.

Two more practices that matter regardless of tooling:

Isolate every test's data. Shared accounts are the single largest source of phantom failures in parallel runs, and two tests mutating one cart fail intermittently in a way indistinguishable from a real bug. Provision per session.

Set state through the back door. Getting a user into a specific condition by clicking through six screens is slow and adds six failure points to a test about the seventh. Hooks run scripts and API calls before or during a flow for exactly this.

Running end-to-end tests in CI/CD

E2E is the slowest layer, so it does not run on everything. Tier it.

Suites are the grouping of stage points, and automations fire them on new builds or on a schedule. Parallelism is what makes the nightly tier viable, and it is also what surfaces every isolation defect you have, so expect the first parallel run to fail for reasons unrelated to your app.

Conclusion

The advice to write few end-to-end tests is a reasonable response to a bad experience, but it identifies the wrong cause. Breadth is what makes the layer valuable, since crossing boundaries is the only thing it does that cheaper layers cannot. Timing is what makes it painful, and timing is a solvable engineering problem rather than a property of the layer. Work out where your framework's idle detection stops, because that boundary is where your flakiness lives. Then pick journeys by consequence, isolate their data, and set up state through the back door rather than through the interface.

FAQ

How many end-to-end tests should we have?

Enough to cover every journey whose failure you could not ship, and no more. For most mobile apps that is somewhere between fifteen and sixty. Count matters less than whether each one earns its runtime.

What is the difference between end-to-end testing and system testing?

They overlap heavily. System testing traditionally means verifying the assembled system against requirements, often including non-functional ones. End-to-end testing is scoped to user journeys through it. In practice most teams use the terms interchangeably.

Should end-to-end tests use mocks?

Not for the services under test. Mocking the backend removes the only thing this layer uniquely verifies. Third-party dependencies you do not control, such as a payment sandbox, are the reasonable exception.

Are end-to-end tests worth it if they are flaky?

A flaky E2E suite is worse than none, because it trains the team to dismiss red. But flakiness is usually fixable through synchronization and data isolation rather than a reason to abandon the layer.

Can end-to-end tests replace manual QA?

They replace the repetitive part, which is re-verifying that known journeys still work. They do not replace exploratory testing, where a human notices something nobody thought to assert on.

Where does end-to-end testing sit in the testing pyramid?

At the top, meaning the fewest tests. Worth remembering that the pyramid was formulated when E2E meant a brittle script bound to selectors, and the shape is a heuristic about cost rather than a law.