E2E Test Results as PR Comments, With Video

TL;DR

• Test failure triage time dominates the cost of a failing suite, and it scales with the distance between the result and the evidence rather than with the difficulty of the defect.
• A red check in a separate dashboard is a notification competing for attention, whereas test results in pull request form arrive inside work a reviewer is already doing.
• Video changes triage specifically because a recording answers what happened, while a stack trace answers only where execution stopped.
• A useful PR comment test report leads with a verdict, names the failing step in product language, and links evidence within one click.
• Autosana posts session replay with every result through its GitHub integration, so the evidence travels with the verdict rather than waiting in an artifact store.

Why does delivery location change behavior?

Because attention is allocated by context rather than by importance.

A reviewer opening a pull request is already engaged with that change. They have the diff in front of them, the intent in their head, and a decision to make. A result appearing in that context becomes an input to a decision already underway.

The same result sitting in a CI dashboard requires a context switch, a navigation, and a deliberate act of looking. Each step loses people, and the loss compounds. A result nobody opens protects nothing while still consuming compute.

There is a sharper version of this. A reviewer who sees a green check inside the pull request forms a judgement about the change. A reviewer who sees nothing forms a judgement about the change anyway, using only the diff. The second reviewer is making the same decision with less information, and neither of them experienced a failure.

Where do the triage minutes actually go?

Breaking it down reveals that the defect itself is rarely the expensive part.

Reconstructing dominates, and it is the phase a stack trace serves worst. A trace reports where execution stopped, which is a different question from what the application was doing at that moment. An assertion failing on a missing element says nothing about whether the page errored, redirected, rendered empty or displayed a modal nobody anticipated.

That gap is why the same failure costs one engineer four minutes and another forty. The difference is evidence rather than skill.

A practitioner on r/Everything_QA made the related point that a check's value is bounded by how quickly someone can act on its output, and reconstruction time is precisely where that bound binds.

What does video change?

It collapses the reconstructing phase, which is the expensive one.

A recording answers the question a trace cannot. The reviewer watches the run reach the checkout screen, observes the promo field rendering empty, and sees the submit button remaining disabled. Diagnosis arrives in the time it takes to watch, and no hypothesis-forming is required.

Three secondary effects follow, and each matters more than it first appears.

Flake becomes identifiable rather than assumed. A recording showing the page still loading when the assertion fired identifies a timing problem immediately. Without it, the same failure gets labelled flaky and retried, which is how genuine defects enter the flake bucket.

Review participation widens. A product manager cannot read a stack trace and can absolutely watch a recording, so a reviewer who understands intended behaviour can say the flow is wrong even where the code is right.

Disputes resolve quickly. The conversation about whether a failure is real ends when someone watches it, which removes an entire category of back-and-forth from the review thread.

CI video recording tests carry a storage cost, which is real and modest beside the engineering hours they remove. Retention of a fortnight covers the window in which anyone will look.

What belongs in a useful PR comment?

Comment design determines whether the mechanism helps or becomes noise.

That last row deserves attention. A reviewer facing a red check needs to know whether their change caused it, and a comment answering that question directly removes the most common first step in triage.

Three things to leave out. Full logs pasted inline, which bury the verdict. Passing-test detail, since a green result needs one line. And a fresh comment per run, which produces a thread nobody can read.

Update one comment in place rather than appending. A pull request receiving six pushes should carry one result comment reflecting the latest state, since a chronological pile of outdated verdicts actively misleads anyone scrolling.

How is this implemented in practice?

Three shapes, with different costs.

Artifacts only. The pipeline uploads screenshots, video and logs, and the reviewer navigates to the run to retrieve them. GitHub Actions test artifacts work this way by default, and the approach is better than nothing while leaving the orienting phase intact.

Comment with links to artifacts. A bot posts a verdict with links pointing at the stored evidence. This removes the noticing and orienting phases, which is most of the avoidable cost, and it is the pragmatic default for teams building their own.

Comment with inline evidence. Verdict, failing step, and a playable recording inside the comment. Removes reconstruction too, and requires evidence hosted somewhere renderable within the comment.

A practical note on artifacts regardless of shape: upload on failure unconditionally, because the run holding the evidence is already gone by the time anyone wishes they had it.

Two engineering details worth getting right early. Keep the comment identity stable, so updates replace rather than accumulate, which usually means a marker in the comment body the bot searches for. And fail the check rather than only commenting, since a comment carries information while a check status carries enforcement, and conflating them leaves the gate open.

A tester on r/softwaretesting described the familiar end state of noisy reporting, where the signal is technically present and nobody reads it any more, which is the failure mode this pattern invites when comment hygiene is skipped.

One failure, three reporting setups

Tracing an identical defect through three setups makes the cost difference concrete.

The defect: a promo code field fails to render because a feature flag evaluated incorrectly for the test account, so the checkout flow reaches a page where the expected input is absent.

Setup one, dashboard only. The suite runs nightly and a check appears red in the CI interface. Nobody is engaged with the change at the moment it fails, so the result waits until morning. Someone opens the dashboard, locates the run among the night's jobs, opens the failing job, scrolls the log, finds an assertion reporting a missing element, forms a hypothesis about a selector change, checks the frontend commits from yesterday, finds nothing relevant, then reproduces locally and discovers the field is genuinely absent. Elapsed: most of a morning, across two sittings.

Setup two, comment with artifact links. The same failure posts a comment on the pull request within minutes, naming the failing step as "could not find the promo code field" and linking a screenshot. The author, still holding context, opens the screenshot, observes an empty area where the field belongs, and recognises the flag immediately. Elapsed: under ten minutes, one sitting.

Setup three, comment with inline video. The comment carries a playable recording. The author watches the run reach checkout, sees the field never render, and notices in passing that a banner above it also failed to appear, which points at the flag evaluation rather than at the field specifically. Elapsed: around three minutes, and the diagnosis is more precise than in setup two.

Three observations from this.

The defect was identical in all three cases. Only evidence proximity varied, and it varied the cost by roughly two orders of magnitude.

Setup one risks a wrong conclusion. Hypothesising a selector change was reasonable given the available evidence, and following it consumed most of the elapsed time. Thin evidence generates plausible wrong paths.

Setup three surfaced something extra. The missing banner appeared in the recording without anyone looking for it, which is the characteristic benefit of evidence a reviewer can observe rather than query.

What should you measure?

Four figures establish whether the change worked, and the first is the one that justifies the effort.

Median time from failure to first human action. Capture it before changing anything, since the before-and-after comparison is the entire business case. Most teams have never measured it and guess low by a wide margin.

Retry rate on failing checks. A retry is a vote that investigating costs more than re-running. Where retries fall while failures hold steady, triage genuinely became cheaper rather than failures merely becoming rarer.

Distinct people who commented on a test failure. Review participation widening is the signal that evidence became readable by people beyond the suite's owner, and a count of one has a bus-factor problem independent of tooling.

Proportion of failures labelled flaky without investigation. This usually falls sharply once recordings exist, because the label was frequently shorthand for an unaffordable diagnosis rather than a judgement about transience.

What to avoid measuring: comment volume, which rewards noise, and time-to-green, which rewards re-running until the problem hides.

A QA lead on r/QualityAssurance made the broader point that teams instrument what is convenient rather than what would catch the problem, and triage time is the clearest instance, since it sits entirely in human minutes that no pipeline reports.

One organisational note. Present the before-and-after on time-to-first-action rather than on failure counts, because failure counts rise initially for the reason described below and a leadership conversation anchored on the wrong number starts badly.

What does this change about team behaviour?

Three shifts appear within weeks, and they compound.

Failures get investigated rather than re-run. When diagnosis costs two minutes, people diagnose. When it costs forty, they retry and hope, which is a rational response to the cost rather than a discipline failure.

The suite earns trust. A team that can see what happened develops a calibrated view of which failures matter, replacing the binary of either trusting everything or trusting nothing.

Test quality improves through visibility. A recording showing a test clicking through a flow in an obviously wrong way gets noticed by whoever watches it, so bad tests surface rather than persisting silently behind green results.

The counter-effect worth anticipating: visibility initially increases the apparent failure rate, because failures that were previously re-run into invisibility now get seen and counted. That rise is measurement improving rather than quality declining, and saying so in advance prevents an uncomfortable conversation later.

Where this pattern reaches its limit

Three cases where better reporting changes little, worth naming so the effort lands where it pays.

Suites nobody gates on. Where a failing check blocks nothing, improving its presentation improves the quality of a signal people are already entitled to ignore. Fix the gate first, then the reporting.

Failures concentrated in one flaky test. A single test generating most of your red builds needs removing or repairing rather than documenting more clearly, since better evidence about a known-bad test is a faster route to the same conclusion.

Teams reviewing asynchronously across time zones. Evidence proximity helps less when the author is asleep, so the noticing phase stays long regardless of placement. Those teams benefit more from improving the comment content, since whoever reads it next will be working from the comment alone.

How does Autosana deliver results?

Autosana is an AI-native E2E testing agent for iOS, Android and web, and the reporting path is built into the execution model rather than assembled alongside it.

Results arrive through the GitHub integration as a pull request comment carrying session replay, so the verdict and the evidence travel together. For an agent-based system that property is structural rather than convenient, since a system deciding its own path needs those decisions reviewable, and a recording is what makes them so.

Runs execute on hosted real devices, which means the recording shows genuine platform behaviour rather than an emulator approximation, and that distinction matters when the failure involves a system dialog or a device-specific layout. Flows are invocable over MCP, so a coding agent opening a pull request can trigger the run whose result lands on it.

The honest limits. Reporting improves triage rather than coverage, so a flow nobody wrote stays untested regardless of how well results are presented. Retention policy and storage remain your decision. And a comment carries information rather than enforcement, so the check status still has to gate the merge.

Where to start

Time your next three failure investigations with a stopwatch, splitting noticing, orienting, reconstructing and deciding. Most teams discover that reconstructing consumes more than the other three combined, and that number is the business case for everything above.

Then make one change. Upload video on failure unconditionally and link it from a comment carrying the verdict. That single step removes the noticing and orienting phases immediately, and it takes an afternoon to wire up.

FAQ

Why post test results as pull request comments?

Because attention follows context. A reviewer is already engaged with the change, so a result arriving there becomes an input to a decision underway, whereas a dashboard result requires a context switch that loses people at every step.

What should an E2E test result comment contain?

A verdict first, the failing step described in product language, evidence within one click, the scope of the run, and whether the failure is new or already present on the base branch.

Does video recording in CI cost much?

Storage is modest beside the engineering hours it removes, and a fortnight of retention covers the window in which anyone will look. Upload on failure unconditionally, since the run holding the evidence disappears.

Why does a stack trace fall short for E2E failures?

It reports where execution stopped rather than what the application did. An assertion failing on a missing element says nothing about whether the page errored, redirected, rendered empty or displayed an unexpected modal.

Should the bot post a new comment per run?

No. Update one comment in place using a stable marker the bot searches for, because a chronological pile of outdated verdicts misleads anyone scrolling and buries the current state.

Does a PR comment replace a failing check?

No. A comment carries information while a check status carries enforcement. Conflating them leaves the gate open, so post the comment and fail the check independently.

Why does the failure rate rise after improving reporting?

Because failures previously re-run into invisibility now get seen and counted. That is measurement improving rather than quality declining, and anticipating it prevents an uncomfortable conversation afterwards.

How does Autosana attach evidence to results?

Session replay arrives with every result through the GitHub integration, so the verdict and its evidence travel together. Runs execute on hosted real devices, so the recording shows genuine platform behaviour.