Regression Testing: How to Decide What to Re-Run

TL;DR

Regression testing re-verifies working behavior after a change. The hard part is selection, and the common strategy (run the tests your diff touches) is structurally blind to transitive dependency resolution, targetSdk-gated OS behavior, and server-driven config, which is where a large share of mobile regressions come from. Select the trigger type, not the diff size. And check whether automatic retries are deleting your flake rate before trusting any of it.

Every team that has run a mobile suite for more than a year has a version of the same story. A release goes out on a green pipeline. Something breaks in production that has worked fine for eight months. Someone pulls up the diff, reads it three times, and finds nothing that should have touched the broken screen.

That story is usually told as a coverage gap, and it usually is not one. The test existed. It simply was not selected to run, because the thing that changed the behavior was never in the diff at all. This post is about that gap and how to close it.

What is regression testing?

Regression testing is re-verifying behavior that already worked to confirm a change did not break it. The change can be a feature, a bug fix, a dependency bump, a config edit, or an OS target increase.

The definition is uncontroversial. Selection is where teams actually struggle: once a suite passes a few hundred cases, running all of it on every commit stops being viable, and every strategy for running less than all of it has blind spots.

Why regressions happen: the four causes

A regression is a side effect: one intended change producing a second unintended one. On mobile there are four distinct mechanisms, and they differ in one way that determines everything downstream, which is whether your diff can see them.

Only the first is a source-code problem. The other three change behavior while your source stands still, which is why selection strategies built on the diff report green for them.

Regression testing vs. retesting

Retesting, also called confirmation testing, verifies one specific defect is fixed, using the reproduction steps from the ticket. It is scoped to the known bug.

Regression testing verifies that fixing it did not break something else. It is scoped to everything around the change.

They run in that order, and only one of them is plannable. Retesting depends on which bugs were found, so it cannot be scheduled in advance. Regression can be, which is why it is the part worth automating first.

Types of regression testing

The taxonomy matters less than the selection logic behind it, but the terms come up in planning:

Complete runs everything. Correct before major releases and after any platform, framework, or dependency movement.

Partial (regional) runs tests for the changed module plus modules that interact with it. The common default, and only as good as your knowledge of the dependency graph.

Selective narrows further using change-impact analysis: only tests whose execution path includes the changed code. Efficient, and the subject of most of this post's criticism.

Progressive applies when requirements change, so existing tests are no longer valid as written and must be revised before they can serve as a baseline.

Corrective applies when requirements did not change, as in a refactor. Tests are reused unmodified, making this the cheapest kind and the one every refactor should be paired with.

Most teams run partial on merge and complete nightly. That is a reasonable default; the failure is treating selective as a complete strategy.

How to select regression test cases

Three inputs are worth combining: change impact, risk weighted by consequence rather than likelihood, and historical failure data. A test that has caught real regressions is worth more than one that has passed 4,000 consecutive times on a screen nobody touches. The latter belongs in the nightly tier, not the pull-request tier.

Change impact is the most automatable of the three and the most trusted. It is also the one with the three blind spots below.

Transitive dependency resolution

You bump one library. Gradle re-resolves the graph and a dependency you never named moves with it. The Gradle dependencies task prints the resolved tree with conflict resolution annotated, so a version Gradle upgraded appears with an arrow from the requested version to the one it settled on. Your build file diff shows one line; the resolved tree shows forty. Diff the resolved output, not the manifest.

The algorithm that produces that resolved tree is walked through step by step in Gradle's own dependency-resolution reference, including how conflict resolution picks between the version you asked for and the version a transitive pin dragged in. Reading the reference alongside the output of gradlew dependencies is the fastest way to see which of the forty moved pins are load-bearing for a regression run.

This matters for tests specifically because network client defaults change test timing in ways that look like application bugs. OkHttp retries on connection failure by default, and Square documents what it silently recovers from: unreachable IPs among a multi-homed host's addresses, stale pooled connections that timed out in the connection pool, and unreachable proxy servers. Note what that list excludes: HTTP status codes. It is connection-level recovery, not 5xx retry, and conflating the two is a common misreading.

Square publishes the exact behavior in OkHttp's connections reference, which enumerates the classes of failure the client transparently retries and confirms that response-status codes are not among them. A test-side wait that fires before the connection-level retry completes is the mechanism behind most flaky-looking latency failures after a dependency bump.

The consequence is latency, not logic. A stale pooled connection is silently re-established, a 200ms request returns in 900ms, and an assertion with a fixed wait fails on a screen that was never broken. If your flows describe intent rather than timing (Autosana's natural-language flows are interpreted against the running app rather than executing fixed waits), this class of false failure largely disappears, but the underlying resolution change still needs a trigger.

On iOS the equivalent surface is the resolved package file. It pins transitive versions; it is not always touched by the PR that changes behavior, and a teammate's merge can move a pin your branch inherits. Treat any change to it as a full-suite trigger.

targetSdk and OS behavior changes

Android 13 (API 33) introduced the notification posting permission as a runtime permission, and the behavior branches on targetSdk. At 32 or lower, the OS shows the prompt itself when the app posts its first notification. At 33 or higher, you must declare the permission and request it yourself, and if it is not granted, the OS silently drops the app's notifications.

As a selection problem: a target SDK 32 to 33 bump is a one-line diff that changes whether notifications arrive at all. Every push and notification-deep-link journey is in the blast radius, and a diff-derived selector sees one Gradle line with no mapped tests.

This generalizes. OS behavior changes are gated on targetSdk by design, so the platform guarantees that behavioral change and source change are decoupled. Any targetSdk, compileSdk, or minimum-iOS movement is a full-suite trigger, and the run must happen at the new API level. Executing it against an older system image tells you nothing about the gate you just crossed.

Server-driven config and feature flags

Remote config, flag state, API versions, and A/B assignment all change behavior with no app release. Your selection logic is reading the wrong repository.

Two consequences. The flag state belongs pinned in the test definition, or you are testing whatever the config service happened to serve and a green run is not reproducible; environment variables are the usual place to hold that. And config deploys need their own trigger. A flag flip is a production change, and if it can break checkout, it deserves the same gate as code that can break checkout.

When to run regression tests in CI/CD

Tier on the trigger type, not on how large the change looks.

TriggerScopeWhySource-only PRChange-impact subset, minutes The one-case impact analysis is valid for the lockfile change. Full suite resolution moved dependencies; you did not name the targetSdk or min-iOS change. Full suite at the new API level The platform gates behavior on exactly this config or flag deploy. Affected journeys, flag pinned, behavior changed with no app diff, nightly full suite, wide device matrix Net, and under the cases no trigger anticipated

Suites are the unit a trigger points at, and automations fire them on new builds or on a schedule. The source-only row can be handled automatically: the GitHub integration analyses the diff, runs the flows it maps to, and posts results onto the PR. The lockfile, targetSdk, and config rows have to be wired deliberately, because the whole point is that the diff does not contain the signal.

When nightly catches something with no trigger fired for, that is a bug in your trigger list, not in the test.

Regression testing metrics that matter

Xcode 13 added test repetition flags to the xcodebuild command line: a fixed iteration count, retry on failure, and run until failure, with the iteration count acting as the ceiling when combined with either of the others. As a diagnostic, running ten iterations against one suspected flaky test, they are the right tool. As a blanket CI setting, they destroy your ability to measure.

The reporting layer is where it breaks. In the Xcode result bundle, a test that fails on attempt one and passes on attempt two is recorded as passed and counted once. Downstream JUnit and HTML converters have counted retried tests once per attempt instead, producing contradictory failure counts from the same run. Separately, retry on failure has re-run every Swift Testing case in a suite rather than only the failures, so passing tests get executed repeatedly.

So track two numbers, both resistant to that.

Escaped defect rate. Regressions found in production rather than by the suite. The only measure of whether the suite works.

Attempt-adjusted flake rate. Failures that were not real defects, computed from attempt counts rather than final status.

Test count and coverage percentage can both rise while the suite gets less useful. Runtime and triage time are operationally useful but say nothing about whether anything gets caught.

Regression testing best practices

Tier from day one. Retrofitting triggers onto a 500-test monolith is far harder than starting with five tiers of ten.

Provision test data per session. Shared accounts are the largest source of phantom failures in parallel runs: two tests mutating one cart fail intermittently in a way indistinguishable from a real bug.

Write assertions against user-visible outcomes, not UI structure. A test asserting an element identifier breaks on a rename, and that break arrives as a red run consuming the triage budget real regressions need. The instruction-writing guidance covers the phrasing difference.

Delete tests. A suite that only grows gets abandoned wholesale. Removing coverage of a deprecated screen is maintenance.

Pair every escaped regression with a new test and a trigger review. Ask which trigger should have fired, not just which test was missing.

Conclusion

The green pipeline that shipped a broken build was not lying. It ran the tests it was told to run, and nothing told it that a lockfile, an API level, or a flag had moved. That is a fixable problem, and fixing it is mostly clerical: write down the triggers, attach a scope to each one, keep a nightly full run underneath as the net. Then measure escaped defects and attempt-adjusted flake rate, because those are the two numbers that go bad when the suite quietly stops working, and almost every other test metric will keep looking fine while it does.

FAQ

Is change-impact selection wrong?

No. It is valid for source-only changes, which are most PRs. The error is using it as the only strategy, because it silently returns an empty set exactly when the diff is not where the change is.

How often should the full regression suite run?

Nightly, plus on every non-source trigger above. Running the full suite per commit is only realistic if it finishes in minutes, and on mobile it usually does not.

Should we retry failed tests automatically?

As a diagnostic, yes. As a permanent CI default, it costs you the ability to measure flakiness, and it has known over-retry behavior with Swift Testing. If you keep it on, record attempt counts.

Can regression testing be done manually?

Yes, and it is common early on. It stops scaling once release frequency rises, because the same cases are re-executed every cycle. A reasonable start and a poor steady state.

Does this apply to the web?

The mechanism does; the specifics do not. Web has no targetSdk gate, but it has browser auto-update, which is the same structural problem of platform behavior changing with no diff on your side.

Does regression testing replace unit tests?

No. Unit tests catch logic errors in seconds. This is the end-to-end layer, where integration and platform-behavior breakage live. Different failure classes.