Agentic QA Testing: 5 Levels of AI Testing Autonomy

Key Takeaways

• The dividing line is runtime authority, not AI involvement. Ask what the AI is permitted to decide while the test is running, not whether AI touched the test at any point.
• Self-healing is not agentic. Self-healing repairs a path a human already chose. An agent chooses the path. Most tools marketed as agentic in 2026 sit at Level 2.
• The cleanest single test is the artifact test: if something is stored between runs that enumerates the steps, the tool is Level 2 or below, whatever the homepage says.
• Level 3 trades determinism for resilience. That is a real trade, not a free upgrade, and it disqualifies agentic testing from compliance assertions, pixel-exact visual regression, and load testing.
• Autosana runs at Level 3 by default, with no stored step list and a fresh plan against the current UI every run, and reaches Level 4 through its shipped MCP server, which lets a coding agent start a run and read back a structured verdict.

What is agentic QA testing?

Agentic QA testing is end-to-end testing in which an AI agent receives an intent rather than a step list, and decides the interaction sequence at runtime by observing the application's current state. There is no stored script. The agent re-derives the path on every run against whatever the interface actually shows, and reports pass or fail against the intent rather than against a recorded sequence.

AI test automation is the superset: every product where a model touched the process at any point, including tools that use a model once at authoring time and never again. Agentic testing is the narrow subset where the model holds decision authority during execution.

That distinction only becomes operational when you grade it. Here is the ladder.

What are the five levels of the agentic testing capability ladder?

Columns three and four settle every argument. If a human picked the order of operations, or an artifact enumerating them survives between runs, the tool is Level 0 to Level 2. Levels 3 and 4 are defined by the absence of both.

Level 0 remains correct for a large amount of testing. The W3C WebDriver specification and native runners like XCTest exist because a pinned, deterministic assertion is the right tool when the target is stable and the assertion is exact. This ladder classifies. It is not a hierarchy of virtue.

Level 1 is where most of the recent "AI testing" wave landed. A model drafts a Playwright file from a Jira ticket. A human reads it, fixes it, commits it. From that moment the artifact is an ordinary script with ordinary maintenance costs, and the AI has left the building. A QA engineer on r/QualityAssurance made the point that no-code and AI-generated automation moves the work rather than removing it: authoring gets faster, and the maintenance queue that follows is the same queue it always was.

Level 2 is where the marketing budget currently sits. The next section explains why.

Level 3 removes the artifact. The input is "sign up a new user with an invalid promo code and confirm the error shows at the field, not as a toast." Nothing stored can go stale, because nothing is stored. On run 400 the agent does what it did on run 1: look, decide, act, check.

Level 4 removes the human trigger. A coding agent that just opened a PR starts the run through a machine interface such as an MCP server and gets back a structured verdict it can branch on, not a dashboard a person has to open. A human-triggered suite becomes the bottleneck the moment the code author stops being human.

Why is self-healing not the same as agentic testing?

The reframe that does the most work, stated bluntly:

Self-healing repairs a path the human already chose. An agent chooses the path.

A self-healing tool holds a stored step list: tap the element with this accessibility identifier, type into this XPath, assert this text. When a locator stops resolving, the healing layer finds the most probable replacement using sibling attributes, text, position, or a visual embedding, rebinds it, and continues. Valuable, and strictly a repair operation on a fixed plan.

The limit appears the moment a step changes rather than a selector. Ship a consent modal in front of checkout, split one screen into two, or add a required field, and there is no broken locator to heal. There is a missing step that was never in the artifact. Level 2 fails here by construction, because nothing in its design may invent an action the author did not write. Level 3 handles it without noticing.

Two corollaries:

• A tool can be excellent at Level 2 and still not be agentic. Independent judgments.
• A tool claiming Level 3 while storing a step list is not partially agentic. It is Level 2 with better copywriting.

What changes operationally when a team moves to Level 3 agentic testing?

The architectural change is one line. The operational consequences are not.

The maintenance backlog disappears and reappears somewhere else. There is no artifact to update after a UI change, so the "fix the tests after the redesign" ticket stops existing. In its place comes intent quality: writing unambiguous instructions and oracles precise enough that the agent knows what counts as success. "Test the checkout flow" is a wish. "Add two items, apply code SAVE10, confirm the total drops by exactly 10 percent and the discount line is itemized" is an intent.

The review surface changes from a diff to a run. Below Level 3, code review of test changes enforces quality. At Level 3 there is no test diff, so the artifact of record becomes the session recording plus the agent's action log. Autosana posts session replay to the PR through its GitHub integration for exactly this reason: evidence has to arrive where the decision is made.

Failure modes shift category. The old mechanical taxonomy does not vanish: stale selector, race condition, timing, including Espresso's inability to synchronize on work outside the message queue (Android developer docs), and Luo et al.'s breakdown of async wait at 45 percent and concurrency at 20 percent across 201 flaky-test fix commits. Level 3 adds a semantic class: the agent misread the intent, satisfied the literal instruction on the wrong screen, or reached the end state by a valid but unintended path. Harder to triage, because the run looks reasonable throughout.

False passes get more expensive than false fails. An agent told to "complete signup" can complete it through a route your users never take and return green. Google's reliability data is the warning: when roughly 84 percent of pass-to-fail transitions involve a flaky test, teams learn to discount failures, and a discounted suite is already dead. The agentic version is worse, because an unintended route produces no failure to discount at all.

The cost model inverts. Below Level 3 the cost is human: seats plus engineer-hours against the maintenance queue. At Level 3 it is compute, per step, per run. Agent runs are slower per step than a pinned selector. Caching recovers much of that by replaying a known-good action sequence instead of reasoning through it again, which is what run caching does, but the economics differ enough to model before migration rather than after.

Determinism is the real trade. A stored script run twice does the same thing twice. An agent run twice may take two valid paths to one verdict, and LLM inference is not bit-reproducible even at temperature zero (Thinking Machines Lab). The response is to measure verdict stability the way Meta measures ordinary flakiness, as a probability rather than a boolean, and gate merges only on flows that clear a threshold.

An engineer on r/opensource raised the related concern of depending on vendor-hosted AI tooling for infrastructure-critical workflows. That tension is real at Level 3, because the decision-making component is a hosted model. The mitigation is practical: demand exportable run logs, replayable action sequences, and a verdict format you can parse yourself.

When is agentic testing the wrong choice for QA automation?

A category definition that never says "not this" is marketing. Level 3 is the wrong tool in at least five places.

• Deterministic compliance assertions. Regulated disclosure text, legal copy, and financial arithmetic need an exact assertion that fails on one changed character. Brittleness is the feature here. Pin it at Level 0.
• Pixel-exact visual regression. An agent reads meaning, which is precisely why it will accept a layout that shifted eight pixels. If the question is "did anything move," use a baseline image differ, not a reasoner.
• Load and performance testing. Throughput work needs thousands of cheap, identical clients. Per-step inference cost scales in exactly the wrong direction.
• Accessibility conformance auditing against a rule set. Conformance is rule-evaluation against WCAG and the W3C ACT rules, and the automation ceiling did not move because agents arrived. A contributor on r/accessibility noted that automated checks only ever cover the machine-checkable fraction, and that correct use of platform accessibility APIs matters more than any scanner. An agent reads the same tree.
• Unit, contract, and integration layers. Nothing on this ladder applies below the end-to-end layer.

There is a timing answer too. If the UI has not changed in a year and the suite is green, Level 3 solves nothing. It pays off in proportion to how fast the interface moves.

How do you evaluate a vendor's agentic testing claim in three questions?

Skip the demo narration. Ask these three and classify from the answers.

1. "What is stored between runs, and can you show it to me?" The artifact test decides Level 2 versus Level 3 on its own. A step list, selector map, recorded DOM path, or YAML flow file means Level 2 or below. Watch for the hedge "we store it, but the AI maintains it." That is self-healing, which is Level 2 by definition. A Level 3 tool stores an intent and a run history, not a sequence to execute.

2. "Insert a screen the test has never seen, then run it untouched." Add a consent modal or a new required field between two existing steps. Update nothing. Run it. Level 2 cannot pass, because there is no broken locator to heal and no mechanism to invent the missing action. Level 3 passes without being told. Ten minutes to set up on staging, and it separates the ladder cleanly.

3. "Can another agent start a run and read the result without a human?" Level 4 requires a machine-callable interface and a parseable verdict. If the only entry points are a UI button and a CI cron, and the only output is a dashboard, the tool is not closing the loop with your coding agents, whatever the integrations page claims. Ask to see the tool contract.

Two further filters. A tester on r/softwaretesting gave the durable rule for evaluating tool claims: run the trial against your own ugliest flow, never the vendor's reference app, which was chosen to make the tool look good. Second, ask where the tool is the wrong choice. A vendor who cannot name three disqualifying cases has not thought about the boundary, and the boundary is where you will meet them.

Where does Autosana sit on the agentic capability ladder?

Autosana is Level 3 by default and Level 4 through its MCP server.

• Level 3 by default. Flows are intent-based. No selectors, no scripts, no stored step list, and the agent re-plans against the current UI on every run across iOS, Android, and web, including on hosted real devices.
• Level 4 through MCP. The shipped MCP server lets coding agents in Cursor, Claude Code, or Devin invoke a run directly and receive a structured verdict, closing the loop when the PR author is itself an agent.
• Evidence by default. Session replay posts to every PR, standing in for the test diff that no longer exists.
• Priced per agent run, with a self-serve trial, matching the compute-shaped cost model Level 3 actually has.

The honest version: Level 3 buys resilience to UI change and pays in determinism and per-run cost. Teams with exact compliance assertions should keep those at Level 0 and reserve agentic runs for flows that break every sprint. The migration that works is hybrid, not wholesale, roughly the shape of Gobi's move off Maestro.

Should your team adopt agentic QA testing in 2026?

• Stay at Level 0 or 1 if the UI is stable, the assertions are exact, and the suite is green. No pain to solve.
• Level 2 is enough if the only recurring breakage is cosmetic: renamed identifiers, restyled components, attribute churn.
• Move to Level 3 if structural change is the problem: screens split, steps inserted, flows reordered, maintenance queue growing faster than the feature queue. Start with the three flows that break most often, not the whole regression suite.
• Require Level 4 if a meaningful share of PRs are opened by coding agents. A human-triggered suite is the bottleneck there, and the verdict has to be machine-readable or the loop never closes.

Frequently asked questions

What is agentic QA testing?

End-to-end testing where an AI agent receives an intent instead of a step list and decides the interaction sequence at runtime by reading the current UI. No script is stored between runs. The agent re-plans every execution and reports pass or fail against the intent.

Is self-healing the same as agentic testing?

No. Self-healing repairs a broken locator inside a path a human already chose, so the stored step list still exists. An agent chooses the path itself, with no step list. Self-healing is Level 2 on the capability ladder; agentic execution is Level 3.

What is the difference between AI test automation and agentic testing?

AI test automation is the superset: any tool where a model touched the process, including tools that use AI once at authoring time. Agentic testing is the subset where the model holds decision authority during execution. What matters is what the AI decides at runtime, not whether it was used.

Does agentic testing replace Selenium, Appium, or Playwright?

Not entirely. Scripted frameworks remain correct for exact assertions, compliance text, performance work, and anything below the end-to-end layer. Agentic testing replaces the scripted E2E flows that break on every UI change. Most teams run both, with agentic coverage on the fastest-moving flows.

How can you tell if a testing tool is actually agentic?

Ask what is stored between runs. A step list, selector map, or flow file means Level 2 or below. Then insert a screen the test has never seen and run it untouched. Self-healing cannot pass that: there is no broken locator, only a missing action.

Is agentic testing reliable enough to block a merge?

Conditionally. Gate only on flows that clear a verdict-stability threshold measured over repeated runs against a frozen build. Treat stability as a probability, not a yes or no. Flows below the threshold report without blocking until the environment or the intent wording is tightened.

When should a team not use agentic testing?

Five cases: deterministic compliance assertions, pixel-exact visual regression, load and performance testing, accessibility conformance auditing against a rule set, and anything below the end-to-end layer. In each, brittleness is the desired property rather than a defect.

Is Autosana agentic testing, and at what level?

Autosana operates at Level 3 by default: intent-based flows, no stored step list, re-planned against the current UI every run on iOS, Android, and web. It reaches Level 4 through a shipped MCP server that lets coding agents start a run and read back a structured verdict.