A Playwright reporter · self-hosted

An LLM reads your
failed CI runs
so you don't have to.

Every Playwright failure classified — real bug, flake, selector drift, or env — with a short, human-readable summary posted straight to your PR.

self-hosted·BYO API key·your logs never leave your infra

🐞 REAL_BUG 🎯 SELECTOR_DRIFT 🎲 FLAKY 🌩 ENV_ISSUE

playwright.config.ts

// npm i -D playwright-ai-triage

export default defineConfig({

reporter: [['list'], ['playwright-ai-triage']],

});

playwright-ai-triage monthly downloads on npm

✓ No platform between you and your logs ✓ Bring your own Anthropic key ✓ ~$0.01 per run ✓ Never fails your build

🤖

github-actions bot commented

now

🔍 AI triage — 3 failure(s) triaged

✅ 2 failure(s) resolved since the last run.

🐞 REAL_BUG (1)

🆕 checkout total updates — asserted 3 items, cart shows 2 (94%)

tests/checkout.spec.ts:42

🎯 SELECTOR_DRIFT (1)

🆕 nav opens account menu — button text changed (88%)

fix: getByRole('button', { name: 'Account' })

tests/nav.spec.ts:17

🎲 FLAKY (1)

⏳ dashboard loads chart (91%)

tests/dashboard.spec.ts:63

failure classes, every run

~$0.01

per run, printed honestly

<60s

install → first triage

builds ever failed by us

How it works

Three steps, zero dashboards

It runs inside your CI. Nothing leaves except redacted failure text, sent to the Anthropic API on your key.

🧪

CI run fails

The reporter attaches to Playwright. When tests fail, it collects error text, stacks, retry history, and step titles — locally.

🔍

The LLM classifies

Obvious cases (retry-passed, network errors) are decided locally for free. The rest get a verdict, a confidence score, and a reason.

💬

Posted to your PR

One auto-updating comment on your PR — or stdout, or Slack. Fixed findings flip to ✅, repeats stay muted, only new failures shout.

The output

One summary. Three destinations.

Same triage, rendered for wherever your team reads it. Auto-detected from your CI environment.

🤖 github-actions bot · auto-updating comment

🔍 AI triage — 2 failure(s) triaged

🐞 REAL_BUG (1)

checkout total updates — asserted 3 items, cart shows 2 (94%)

🎲 FLAKY (1)

dashboard loads chart — passed on retry (91%)

$ npx playwright test

🔍 AI triage — 2 failure(s)

🐞 REAL_BUG checkout total updates (94%)

tests/checkout.spec.ts:42

🎲 FLAKY dashboard loads chart (91%)

tests/dashboard.spec.ts:63

Triaged · claude-haiku-4-5 · $0.0123

🔍

AI triage APP

2 failures triaged on #build-1284

🐞 REAL_BUG — checkout total updates

🎲 FLAKY — dashboard loads chart (passed on retry)

The verdicts

It tells you why it failed, not just that it did

🐞

REAL_BUG

A genuine regression in your app. Assertion diffs and logic errors that a human should look at now.

🎯

SELECTOR_DRIFT

The UI moved, not the logic. Comes with a suggested locator fix so you can patch the test in seconds.

🎲

FLAKY

Passed on retry or matched a known flake signature. Classified locally, for free — never hits the API.

🌩

ENV_ISSUE

Network blips, expired credentials, infra hiccups. Not your code — and it says so, so you stop debugging it.

Data boundary

You can see exactly what's sent

Self-hosted by design. Text-only, secret-patterns redacted, on your own Anthropic key. No hosted platform in the middle.

↗

Sent to the Anthropic API

• Test id, title, file path & line

• Error message & truncated stack

• Failing step title & retry history

• Optional: redacted DOM snapshot, git diff summary

⊘

Never sent anywhere

× Screenshots, videos, traces

× Your source code beyond stack frames

× Media files — referenced by local path only

× Anything, when there's no API key (degrades gracefully)

Why this, not a platform

A reporter, not a service

playwright-ai-triage

Hosted CI platform

Where data lives

Your CI only

Their servers

Cost model

~$0.01/run, BYO key

Per-seat / per-run tier

Can it fail your build?

Never

Sometimes

Setup

One config line

Account, tokens, webhooks

Flaketrace · coming soon

Run history, flakiness scores
& suite trends

The reporter stays free and self-hosted forever. Flaketrace, the optional hosted add-on for playwright-ai-triage, adds cross-run memory: which tests flake most, whether the suite is getting healthier, and trends over time.

📊 Flakiness leaderboard 📈 Suite health trends 🔁 Cross-run memory

Ship it in one line

Add the reporter, set your key, open a PR. That's the whole thing.

$ npm i -D playwright-ai-triage

An LLM reads yourfailed CI runsso you don't have to.