Regression Testing Without Maintaining 500 Scripts

TL;DR

• Suite cost rises faster than suite size, because maintenance, runtime, flake probability and triage load all compound as tests accumulate.
• Teams keep adding tests and skip deleting them, so suites grow in one direction only and the oldest tests are frequently the least examined.
• A test bound to a selector breaks whenever the interface changes, whereas a test bound to an outcome survives redesigns, so binding predicts maintenance cost better than count does.
• An audit sorts the suite into keep, consolidate and retire by asking which tests have caught a real defect and which have only ever reported their own breakage.
• Autosana runs flows by intent, so UI changes leave the flow intact, and the 500-script problem stops reproducing itself.

Why does a regression suite grow to 500 scripts?

Every mechanism in a normal engineering process pushes the count upward, and few push it down.

A production incident arrives, the postmortem asks for a regression test, and one gets written. That is correct practice, repeated two hundred times over three years.

Coverage targets reward volume. A team held to a percentage adds tests to satisfy the number, and the easiest tests to add are the ones that exercise paths already covered by heavier tests above them.

Feature work adds its own tests, while the tests belonging to the feature it replaced stay in the repository, since deleting them feels like discarding safety. The suite accumulates strata, with the oldest layers written by people who left.

Nobody owns removal. Adding a test earns approval in review, whereas removing one invites the question of what happens when that path breaks, and that question is uncomfortable enough that most engineers withdraw it.

An engineer on r/QualityAssurance described the familiar end state, where the suite is large enough that its results are consumed as a ratio rather than read as individual outcomes, which is the point at which the signal stops functioning.

The reframe this article runs on: suite size is a proxy, and binding is the cause. Five hundred tests bound to outcomes cost far less to own than fifty bound to selectors, because the first set survives interface change and the second set breaks on contact with it.

Which costs scale as a suite grows?

Four, and they compound rather than add.

Flake deserves attention because the arithmetic surprises people. Where a single test holds a 0.5 percent chance of failing spuriously, a suite of 500 such tests reports at least one false red on roughly 92 percent of runs. The individual tests look excellent, while the suite as a whole reports red almost always, so the team learns to read a red build as noise.

That learned response is the real loss. A suite that reports constantly stops being consulted, and a gate that goes unconsulted protects nothing while still consuming CI minutes and salary.

Triage compounds on top. Each red requires someone to establish whether a genuine defect sits underneath it, and the cost per incident is set by how quickly that question resolves. A stack trace and fifty lines of stdout resolve it slowly, whereas a recording of the run resolves it in a minute.

Why do teams keep tests that cost more than they return?

Three reasons, and all three are rational from the individual's position.

Deleting a test looks like reducing coverage. The coverage number falls, the dashboard turns the wrong way, and explaining why that is healthy takes longer than leaving the test in place.

The downside is asymmetric and personal. An engineer who removes a test and later sees that path break in production carries the blame, whereas the cost of keeping a useless test is spread across the whole team and shows up as a vague sense that CI is slow.

Nobody knows which tests are useless. Suites rarely record which test caught which defect, so the evidence required to delete with confidence sits outside the system.

The fix is to make removal evidential rather than brave. Record which tests have caught genuine regressions, and deletion becomes a decision about data instead of a decision about courage.

How do you audit a regression suite?

Run this over a rolling ninety days, per test, and the suite sorts itself.

Three categories emerge.

Keep. Tests holding true catches and few repairs. These earn their place, and they are usually fewer than anyone predicts.

Consolidate. Clusters exercising one path through slightly different entry points, typically accumulated through successive feature work. One outcome-bound test replaces six mechanics-bound ones.

Retire. Tests with zero true catches and several repairs over ninety days. Each one has reported only its own fragility, so retiring it removes cost and preserves signal.

The number that persuades a sceptical room is repairs per true catch. A test repaired eleven times to catch nothing is straightforward to discuss, whereas "I think this test is useless" invites debate.

Archive rather than delete on the first pass. Move retired tests to a quarantined directory excluded from the gate, leave them for a quarter, and confirm that no incident arrives in the paths they covered. That converts an argument about risk into an observation.

What should a regression test bind to?

Here is the mechanism underneath everything above.

A mechanics-bound test names interface elements. Click #submit-v2, assert that .badge-success rendered. It breaks when the button is renamed, when the component is extracted, when a design system lands, and when a Compose or framework migration changes the addressing model. Each of those events is routine engineering work, so the test accrues repairs at the rate the product improves.

An outcome-bound test names a result a customer would recognise. Place an order, confirm the charged amount matches the expected total. That assertion holds through every one of the changes above, because the product kept its promise even while the interface moved.

The distinction has a practical consequence for suite size. Mechanics-bound tests multiply, since each interface variant needs its own, while outcome-bound tests consolidate, since one assertion covers every interface that delivers the outcome.

A developer on r/Playwright made the cross-framework version of this point, observing that locator strategy debates are really debates about which string a team prefers to maintain, and that teams with the least churn write assertions about what the user achieved.

Self-healing locators sit between the two. They repair a path that a human already chose, which reduces repair events and leaves the binding intact, so the cost curve flattens while keeping its shape.

Consolidation patterns that reduce count honestly

Collapse variants into one parameterised flow. Six tests covering six promo codes become one flow executed with six data rows, so the maintenance surface shrinks by five sixths while coverage holds.

Move assertions down the pyramid. Validation rules and calculation logic belong in unit tests, which run in milliseconds and stay stable, so an end-to-end test asserting a rounding rule is paying end-to-end prices for unit-level information.

Delete tests that duplicate a path already covered above them. Where a checkout flow traverses the cart, a separate cart-rendering test adds runtime and repair surface while contributing little.

Retire tests for removed features. This sounds obvious and survives in most suites anyway, since the test passes against a code path kept alive largely to satisfy it.

A tester on r/softwaretesting noted the version of this that matters commercially: the tools that genuinely helped removed ceremony, while the ones that promised to remove thinking created a second problem.

How do you stop the suite growing back?

An audit buys a quarter. A policy keeps the gain.

Attach an expiry to incident-driven tests. A regression test written after an incident carries value proportional to the risk of recurrence, and that risk declines once the underlying defect is fixed and the code refactored. Review those tests at six months rather than keeping them permanently by default.

Require a deletion candidate alongside each addition. Teams adopting this report that the conversation changes immediately, since proposing an addition now prompts a look at what already exists.

Measure repairs per true catch as a standing metric. Publishing it per quarter makes suite health visible, and visible numbers attract ownership.

Separate the gate from the archive. A merge gate holds a small set of outcome-bound flows, while the broad suite runs nightly and reports. That split removes the pressure to keep the gate comprehensive, which is what drives the growth in the first place.

A developer on r/webdev put the architectural principle plainly: the layers that change most often deserve the fewest things pinned to them.

What does the migration look like when the suite is already past 500?

Replacing a large suite wholesale invites the one outcome everyone fears, which is a gap in coverage during the transition. A staged path avoids it.

Stage one, freeze the growth. Stop adding to the existing suite for a sprint, and route every new regression test through the outcome-bound approach. The suite stops expanding while the audit runs, and the team gets practice on low-stakes work before touching anything load-bearing.

Stage two, cover the top twenty flows in parallel. Identify the twenty paths carrying the most revenue or access risk, and author outcome-bound flows for them alongside the existing tests. Both run. Where results agree for two weeks, the outcome-bound version has earned trust on evidence rather than on argument.

Stage three, retire the duplicated originals. Those twenty flows usually correspond to somewhere between eighty and two hundred existing tests, since a single checkout journey tends to be covered by a dozen narrower scripts accumulated over years. Archive them outside the gate, hold for a quarter, and confirm that the paths stayed healthy.

Stage four, tier what remains. Whatever survives the audit splits between a small blocking gate and a broad nightly run. That split relieves the pressure that produced the original growth, because a gate expected to be comprehensive will always expand toward comprehensiveness.

The sequencing matters more than the speed. Teams that begin at stage three and work backwards are deleting before they have evidence, and the first production escape in a retired path ends the programme regardless of whether the two events were related.

How long does the transition take?

For a suite of five hundred, budget one quarter and expect the shape rather than the schedule to hold. The audit consumes a fortnight of part-time work, parallel authoring of twenty flows runs two to three sprints, and the archive hold lasts a quarter by definition.

The number worth watching throughout is engineering hours spent on suite repair per sprint, captured before the programme begins. Where that figure declines while escaped defects hold steady, the transition is working. Where escapes rise, the retired tests were carrying more than the audit credited, and restoring them from the archive takes minutes precisely because they were archived rather than deleted.

One caution on sequencing. Run the transition during a period of ordinary feature work rather than alongside a redesign, since a redesign breaks the mechanics-bound originals en masse and removes your ability to compare the two suites against each other.

How does Autosana change the maintenance curve?

Autosana is an AI-native E2E testing agent for iOS, Android and web. Flows are stated as intent, and the agent decides the interaction sequence at run time against whatever the interface currently shows.

Applied to the four costs, three of them change.

Maintenance falls because the flow holds no stored selector, so a renamed element, an extracted component or a design-system migration leaves the flow intact. The repair events that dominate a mature suite arise from exactly those changes, and removing the binding removes the events.

Flake falls for the same reason, since the largest contributor to red builds in a mature suite is interface drift reported as failure. Timing and environment flake persist, and those belong to the pipeline.

Triage falls because session replay arrives with the result through the GitHub integration, so the reviewer watches what happened on the device rather than reconstructing it from a log. Runs execute on hosted real devices, and flows can be invoked over MCP where a coding agent opened the pull request.

Runtime stays yours, since a flow that takes ninety seconds of real interaction takes ninety seconds regardless of how it was authored, so parallelism and tiering remain your levers.

Two further limits worth stating. Unit and component tests keep their place, as they run in milliseconds and catch classes of defect that an end-to-end layer reaches slowly. And an audit still requires a human to judge which outcomes matter, which is product knowledge that stays with your team.

Where to start

Pick the twenty tests that failed most often last quarter and label each failure as a true catch or a repair. Most teams discover that a small group of tests generates the majority of their maintenance, and that the same group has caught remarkably little.

Then take the six noisiest and ask what outcome each one exists to protect. Where several share an outcome, one outcome-bound flow replaces the group. That single exercise usually removes more maintenance hours than a quarter of tooling work, and it costs an afternoon.

FAQ

How many regression tests should a suite have?

Count matters less than binding, and regression testing automation tools vary more in what they bind to than in what they execute. A suite of 500 outcome-bound tests costs less to own than 50 selector-bound ones. Measure repairs per true catch instead, and let that figure guide whether the suite is sized correctly.

When should you delete an automated test?

Where ninety days of history shows zero true catches alongside several repair events, the test has reported only its own fragility. Archive it outside the gate for a quarter first, then confirm that the paths it covered stayed healthy.

Why do large regression suites produce so many false failures?

Probability accumulates. Where each test carries a 0.5 percent chance of spurious failure, a 500-test suite reports at least one false red on roughly 92 percent of runs, so the team learns to treat red as noise.

What is the difference between a mechanics-bound and an outcome-bound test?

A mechanics-bound test names interface elements, so it breaks when the interface changes. An outcome-bound test names a result a customer would recognise, so it survives renames, component extraction and design-system migrations.

Does self-healing solve regression maintenance?

Partially. Self-healing repairs a path a human already chose, which lowers repair events while keeping the binding. Intent-based execution removes the binding entirely, which changes the shape of the curve rather than its slope.

How do you justify deleting tests to leadership?

Present repairs per true catch per test. A test repaired eleven times that caught zero defects is a straightforward conversation, whereas an opinion about usefulness invites debate that evidence settles quickly.

Should every production incident get a regression test?

Write one, then attach a review date. The value of an incident-driven test declines once the underlying defect is fixed and the surrounding code refactored, so a six-month review keeps the suite proportional to current risk.

Can Autosana reduce an existing 500-script suite?

It changes what replaces them. Flows stated as intent consolidate groups of selector-bound tests that shared an outcome, and they resist the interface changes that generate repairs. Unit and component tests keep their place beneath.