Closed-Loop Verification: Stopping AI Hallucinations in QA

TL;DR

• A QA hallucination is a false pass, meaning the agent reports success while the underlying outcome stayed absent, which is more dangerous than a false failure because it produces silence.
• Screen evidence alone is insufficient, since a confirmation message can render from cached state, from an optimistic update, or from a path that skipped the operation entirely.
• Open-loop testing asks the agent what it saw, while closed-loop testing confirms the effect through an independent channel such as network payloads, application state or backend records.
• Verification depth should track blast radius, so a payment deserves a backend assertion while a navigation check reasonably stops at the screen.
• Autosana pairs intent-based execution with evidence beyond the rendered screen, because a system that decides needs its decisions auditable.

What does a hallucination look like in a testing context?

The term arrives from language models, where it describes confident output unsupported by fact. Applied to a testing agent, it means something narrower and more consequential: the agent reports an outcome that failed to happen.

Four shapes recur.

The wrong path to a plausible screen. The agent intends to apply a promo code, misidentifies the field, enters the code somewhere harmless, and lands on a checkout screen that renders normally. Nothing visibly failed, and the discount never applied.

The optimistic render. Modern interfaces update before the server confirms. The agent observes "Order placed", which the client displayed on submission rather than on acknowledgement, and the request subsequently failed.

The cached echo. The screen shows a state from a previous session, so a profile update appears saved because the old value matched the new one, or a list renders from cache rather than from the write that was meant to occur.

The satisfied-by-accident goal. The agent intends to verify that a restricted user is denied access, encounters an unrelated error page, and treats that as the denial. The assertion passed, and the permission check stayed untested.

Each one produces a green result, which is the defining property. A tester on r/QualityAssurance made the adjacent point about gating decisions generally, observing that a check only earns trust once someone has established what its passing actually proves, and these four cases describe passes proving considerably less than they appear to.

Why does a false pass cost more than a false failure?

Because of who learns about it.

A false failure is loud. Someone investigates, discovers the test was wrong, and repairs it. The cost is an hour of annoyance, and the system corrects itself through ordinary attention.

A false pass is silent. Nobody investigates a green build. The defect proceeds to production, surfaces through support tickets or monitoring, and the postmortem traces back to a suite that reported healthy throughout. The cost includes the incident, plus the subsequent erosion of confidence in every other green result the suite produces.

The asymmetry resembles the permission-testing asymmetry, where testing that an administrator can act is comfortable while testing that a viewer cannot is what actually protects you. Here the comfortable test confirms the happy path rendered, and the protective test confirms the effect occurred.

An engineer on r/devsecops described the general principle that controls which fail open are more dangerous than controls which fail closed, and a testing system reporting a pass it cannot substantiate fails open by construction.

What separates open-loop from closed-loop testing?

The distinction concerns where the evidence comes from.

Open loop means the agent acts and then reports its own observation, so the system trusts the actor's account of its own work. For many assertions that is entirely reasonable, since confirming a menu opened needs nothing more.

Closed loop means the system confirms the effect through a channel the agent cannot influence by misreading a screen. The loop closes because the action and its verification arrive from different places.

That independence is the whole mechanism. Verification performed by asking the agent whether it succeeded adds confidence without adding evidence.

What should a verification layer actually inspect?

Four channels, each catching a different hallucination shape.

Network activity. Did the request fire, and what did the server return? This catches optimistic renders directly, since a client-side "Order placed" accompanied by a 500 response is unambiguous. It also catches the wrong-path case, where the promo endpoint was never called.

Application state. What does the client believe after the interaction? Inspecting store contents or persisted state catches cases where the visual layer and the state layer disagree, which is common during partial failures.

Backend records. Does the order exist, with the expected total, against the expected account? This is the strongest evidence and the most expensive to arrange, since it requires access the test harness frequently lacks. Test oracle design is the formal name for this choice, meaning the decision about what source establishes truth for a given assertion.

Negative conditions. What should be absent? An error toast, a second charge, a duplicate row. Asserting absence catches the satisfied-by-accident case, where something unintended produced a superficially acceptable outcome.

A developer on r/programming made the broader observation that reliability tends to come from observing a system rather than from asking it to describe itself, which is precisely the distinction between these channels and the rendered screen.

Matching verification depth to blast radius

Verifying everything through backend records would be prohibitively slow and would require access most suites lack, so depth should follow consequence.

The useful rule: where a false pass would cost money, reputation or data, the evidence must come from beneath the interface. Everything else reasonably stops at the screen.

How do you design verification for your own flows?

Verifying AI test results begins with three questions per flow, and most teams can answer them in a sentence each.

What is the observable effect? Something changed somewhere beyond the pixels. A row exists, a balance moved, a token was issued, an email queued. Name it, because that is the thing worth asserting.

Which channel exposes it most cheaply? Backend records are strongest and costliest, network responses are usually sufficient, and state inspection sits between. Choose the cheapest channel genuinely proving the effect rather than the strongest available.

What would a wrong path look like? This question is the one teams skip, and it produces the negative assertions. Imagine the agent succeeding incorrectly, describe the residue that attempt would leave, then assert its absence.

A worked example: one promo code, four verdicts

Trace a single flow through four verification designs and the differences become concrete.

The intent: apply promo code SAVE10 to a cart totalling 80 and confirm the discount applied.

What actually happens on this run: the agent misidentifies the promo field, enters SAVE10 into the gift-message box, proceeds to checkout, and the page renders normally showing a total of 80.

Design one, screen observation only. The assertion checks that a checkout screen rendered. It renders. Verdict: pass. The discount never applied and nobody learns anything.

Design two, screen observation plus text assertion. The assertion additionally checks for the string "Discount applied". That string is absent, so the test fails. Verdict: correct failure, arrived at for a fragile reason, since a copy change to "You saved 8.00" would break this assertion while the product worked perfectly.

Design three, network assertion. The assertion checks that a request to the promo endpoint returned a success response. No such request fired. Verdict: correct failure, with a diagnostic message pointing at the actual problem rather than at a missing string.

Design four, network assertion plus independent total check. The assertion checks the promo response and independently confirms the authorised amount equals a separately computed expected value of 72. Verdict: correct failure, with evidence that would also have caught a subtler case where the promo endpoint succeeded and the discount was computed against the wrong subtotal.

Design one is where most suites sit. Design two is where careful suites sit, and it is fragile against copy change. Designs three and four are what closed loop means, and the difference in cost between them and design two is smaller than teams expect, since both reuse evidence the browser already holds.

The pattern generalises. Assertions about appearance catch presentation defects, and assertions about effect catch behaviour defects. A suite composed entirely of the first kind will pass through an entire class of failure without noticing.

How does verification differ from retry and self-healing?

Three mechanisms get conflated because all three respond to a test behaving unexpectedly, and separating them clarifies what each actually buys.

Retry re-executes after a failure, so it addresses transience. It assumes the test was correct and the environment misbehaved, which is sometimes true. Retry provides no new evidence, and it raises the threshold at which unreliability becomes visible rather than removing the unreliability.

Self-healing repairs a broken locator at runtime, so it addresses drift. It assumes the intended target still exists under a changed identifier, which is usually true and occasionally wrong. A heal onto the wrong element produces a passing test asserting against something nobody intended, which is a false pass arriving through a different door than the one this article describes.

Verification confirms an effect, so it addresses correctness. It assumes nothing about why something happened and inspects whether it did.

The three compose rather than compete. A suite can retry transient failures, heal drifted locators and verify effects, and each handles a category the others leave open. Agentic testing reliability depends on holding all three rather than on any one of them. The risk sits in adopting the first two while skipping the third, since retry and healing both convert red into green while verification is the only one of the three that can convert a wrongly-green into red.

That asymmetry is worth stating outright. Two of these mechanisms make your suite quieter, and one makes it more honest. A team adopting the quiet ones without the honest one has improved its dashboard rather than its coverage.

Where is verification unnecessary?

Three cases, because over-verification carries its own cost in runtime and in brittleness.

Purely presentational assertions. Confirming that a modal opened, a menu expanded or a tooltip appeared has no effect beneath the interface to verify, so the screen is the complete evidence.

Read-only navigation. Moving between pages that mutate nothing leaves nothing to confirm, and adding network assertions here produces maintenance without protection.

Flows already covered beneath. Where a contract test or an integration test already proves the backend behaves correctly, the end-to-end layer reasonably stops at confirming the interface reached that behaviour rather than re-proving it.

Over-verifying also creates a subtler problem. Assertions against backend records bind the test to a schema, so a legitimate database refactor breaks tests that were meant to assert about product behaviour. The binding lesson from selectors applies to verification channels too.

An accessibility practitioner on r/accessibility made a related point about automated checks generally, observing that a tool reporting compliance proves only that it tested what it knows how to test, which is the honest limit on any verification layer including this one.

What a verification failure should tell you

A verification layer earns its cost through the quality of its failure messages rather than through the assertions alone.

A screen assertion failing reports that an expected string was absent, which leaves the reader to establish whether the feature broke, the copy changed or the agent went somewhere unexpected.

An effect assertion failing reports which channel disagreed and how. The promo endpoint returned 422 rather than 200, or the authorised amount was 80 where 72 was expected. That message names the defect rather than describing a symptom, which collapses triage from an investigation into a reading.

Design the message alongside the assertion. Where a failure would send someone to reproduce the scenario manually before they understand it, the assertion is incomplete regardless of how correct it is.

How does Autosana handle verification?

Autosana is an AI-native E2E testing agent for iOS, Android and web, and the verification question is structural for this category rather than optional.

Because flows state intent and the agent decides the interaction sequence at run time, the agent genuinely can take a path nobody anticipated. That property is what makes the approach resilient to interface change, and it is simultaneously what creates the false-pass risk described above. Treating those as the same property, viewed from two sides, is the honest framing.

Evidence accordingly extends past the rendered screen. Session replay arrives with every result through the GitHub integration, which matters more for a deciding system than for a resolving one, since a reviewer needs to see which path the agent actually took rather than only its verdict. Runs execute on hosted real devices, so the evidence reflects real platform behaviour, and flows are invocable over MCP where a coding agent opened the pull request. The documentation covers flow authoring and execution.

The limits stated plainly. A verification layer catches what it inspects, so an effect occurring in a channel nobody configured stays invisible. Deciding which effects matter is product judgement that stays with your team. And no verification design removes the need to review what the agent did on flows where the cost of being wrong is high, which is why replay exists rather than as a convenience.

Where to start

Take your three highest-consequence flows and name the observable effect each one produces beneath the interface. A row, a balance, a token, a queued message. Most teams find they have been asserting against the confirmation screen rather than against the effect, which is the gap this article describes.

Then write one negative assertion. Pick a flow, imagine the agent succeeding through a wrong path, and assert the absence of whatever residue that attempt would leave. That single assertion usually catches more than the three positive ones surrounding it.

FAQ

What is an AI hallucination in software testing?

A false pass, where the agent reports success while the underlying outcome stayed absent. It arises when the agent reaches a screen that superficially satisfies the goal through a path that skipped the operation being tested.

Why is a false pass worse than a false failure?

A false failure is loud, so someone investigates and repairs it within an hour. A false pass is silent, so the defect proceeds to production and erodes confidence in every other green result the suite produced.

What is closed-loop verification in testing?

Confirming an effect through a channel independent of the actor, such as network responses, application state or backend records. The loop closes because the action and its verification arrive from different sources rather than from the agent's own account.

Can you trust what a testing agent reports about itself?

For presentational assertions, reasonably. For transactions, permissions and data writes, the report needs corroboration, since a system that decides its own path can also misjudge whether that path achieved the intended effect.

Which channels should a verification layer inspect?

Network activity catches optimistic renders, application state catches visual and state disagreement, backend records provide the strongest evidence, and negative assertions catch outcomes satisfied accidentally by something unintended.

Does every test need closed-loop verification?

No, and over-verification costs runtime and introduces brittleness. Match depth to blast radius, so navigation stops at the screen while anything touching money, permissions or data gets evidence from beneath the interface.

How do negative assertions prevent false passes?

They assert what should be absent, such as an error toast, a duplicate charge or a second row. A wrong path frequently leaves residue, so asserting its absence catches success that arrived through an unintended route.

How does Autosana make agent decisions auditable?

Session replay accompanies every result through the GitHub integration, so a reviewer observes which path the agent took rather than only its verdict, and runs execute on hosted real devices so the evidence reflects real platform behaviour.