Open-source AI regression testing

The open-source Playwright library for AI regression testing.

Write end-to-end tests in plain English. Passmark drives Playwright with an AI agent, caches each action after the first pass, self-heals when the UI changes, and verifies assertions with multiple vision models.

Your tests stay stable without needing to update AI prompts or retrain models.

Get Started How it works

~/your-project

❯ npm install passmark

added 1 package in 0.8s

cart.spec.ts

import { test, expect } from "@playwright/test"
import { runSteps } from "passmark"

test("Shopping cart tests", async ({ page }) => {
  await runSteps({
    page,
    test,
    expect,
    userFlow: "Add product to cart",
    steps: [
      { description: "Navigate to https://demo.vercel.store" },
      { description: "Click Acme Circles T-Shirt" },
      { description: "Select color", data: { value: "White" } },
      { description: "Select size", data: { value: "M" } },
      { description: "Add to cart", waitUntil: "My Cart is visible" },
    ],
    assertions: [{ assertion: "Cart shows the Acme Circles T-Shirt" }],
  });
});

❯ npx playwright test

12 passed · 0 failed · 5 minutes total

❯

The problem

Most browser suites end up in one of three states.

Teams ship faster than ever, but end-to-end testing stayed manual and brittle. Whether you skipped it, watched it rot, or tried pure-AI runners, the failure mode is the same: nothing reliably catches regressions before users do.

How Passmark runs them instead. ↓

No tests.

End-to-end coverage never happened. Writing and maintaining a Playwright suite was more work than the team had time for. Regressions ship and surface in production.

Brittle tests.

The suite you wrote six months ago is half-skipped because the flows changed, and the rest flakes on selector drift. Failures get muted. The suite stops gating anything.

Pure-AI tests, too slow.

AI agents fix authoring, but routing every action through an LLM on every run is fine for one PR and untenable for hundreds of regression tests per commit: slow, costly, and nondeterministic in CI.

How it works

Discover with AI. Replay from cache.

Describe

Write test steps in plain English. No selectors. No page objects. No boilerplate.

steps: [
  { description: "Add the first product to cart" },
  { description: "Open cart and proceed to checkout" },
  { description: 'Fill shipping address with zip "90210"' },
  { description: "Complete payment with test card" },
]

Execute

On the first run, an AI agent drives your app through Playwright using the accessibility snapshots and screenshots. Each single-action step is cached to Redis once it lands cleanly.

First run:  AI agent → ~30s per action step

Replay

Later runs replay cached actions straight through Playwright, with no AI for those steps. Caching covers single-action steps only; multi-action steps and every assertion still run. When a cached action breaks because the UI moved, the agent re-discovers it and updates the cache.

First run:    AI agent → ~30s per action step
Cached run:   Replay action → ~ms, no AI
Assertions:   Vision models run on every pass
Auto-heal:    Replay fails → AI re-discovers → re-caches

Code

This is a Passmark test.

Plain-English steps, no selectors. Runs on Playwright, self-heals when the UI changes, and caches its action steps after the first run.

Compare that to the 40-line Playwright test with data-testid selectors, waitForSelector calls, and hardcoded timeouts you'd write by hand. Then imagine maintaining 200 of those.

cart.test.ts

import { test, expect } from "@playwright/test";
import { runSteps } from "passmark";

test("Shopping cart tests", async ({ page }) => {
  await runSteps({
    page,
    test,
    expect,
    userFlow: "Add product to cart",
    steps: [
      { description: "Navigate to https://demo.vercel.store" },
      { description: "Click Acme Circles T-Shirt" },
      { description: "Select color", data: { value: "White" } },
      { description: "Select size", data: { value: "M" } },
      { description: "Add to cart", waitUntil: "My Cart is visible" },
    ],
    assertions: [{ assertion: "Cart shows the Acme Circles T-Shirt" }],
  });
});

Comparison

Where Passmark sits.

Hand-written Playwright

Plain English test authoring
Caches actions for repeat runs
Self-heals on UI changes
Built for CI (500+ test suites)
Cross-test state sharingManual wiring
Email and OTP testingExternal tools
AssertionsBrittle
Cheap at scale
Open-source

Passmark

Recommended

Plain English test authoring
Caches actions for repeat runsYes
Self-heals on UI changes
Built for CI (500+ test suites)
Cross-test state sharingBuilt-in (Redis)
Email and OTP testingBuilt-in
AssertionsVision models with consensus
Cheap at scale
Open-source

Real-time AI tools

Plain English test authoring
Caches actions for repeat runs
Self-heals on UI changesPartial
Built for CI (500+ test suites)
Cross-test state sharing
Email and OTP testing
AssertionsVision models
Cheap at scale
Open-sourceSome

Built for regression

Built for the whole suite, not one test.

Cross-test state

// Test 1: Sign up
data: { email: "{{global.dynamicEmail}}" }

// Test 2: Log in (same execution, same email)
data: { email: "{{global.dynamicEmail}}" }

Built-in email testing

Comes bundled with disposable inboxes via emailsink integration. Extract OTPs, magic links, or any content from incoming emails with natural language prompts.

{ description: "Sign up", data: { email: "{{run.dynamicEmail}}" } },
{ description: "Enter OTP", data: { otp: "{{email.otp:Extract the 6-digit code}}" } }

Consensus assertions

Every assertion is evaluated by Claude and Gemini independently. A third model breaks ties. Passes only on consensus. Returns a confidence score (0-100).

assertions: [
  { assertion: "Dashboard shows a welcome message for the new user" }
]
// Evaluated by multiple models. Passes only on agreement.

Dynamic test data

Every run gets unique IDs, emails, names, phone numbers. Parallel tests never collide. Three scopes: {{run.*}} (per test), {{global.*}} (per execution), {{data.*}} (per project).

// Per test
data: { email: "{{run.dynamicEmail}}" }

// Per execution (shared across tests)
data: { email: "{{global.dynamicEmail}}" }

// Per project
data: { apiKey: "{{data.stagingApiKey}}" }

Computer-use mode (CUA)

Switch any step, or the whole run, to OpenAI's computer-use agent for pixel-coordinate automation: canvas, drag, and visual-only UI that accessibility snapshots can't reach. Snapshot and CUA steps mix in one run. Locked to gpt-5.5 and direct OpenAI (gateway: "none").

import { configure } from "passmark";

// Whole run, via configure()
configure({ ai: { mode: "cua", gateway: "none" } });

// Or one step: snapshot and CUA mix freely
{ description: "Drag the price slider to $40",
  ai: { mode: "cua", gateway: "none" } }

Video assertions

Pass video: true and Passmark records the entire runSteps run, uploads the WebM to Gemini, and checks your assertion against the full timeline, catching transient toasts, snackbars, and flash errors a single screenshot misses.

assertions: [
  { assertion: "A success toast appears, then fades", video: true },
]
// Evaluated over the whole recording, not one frame.

Good to know

What to expect when you run it.

The honest operating model, so nothing surprises you in CI.

Assertions always run AI

Every assertion is checked by Claude and Gemini on each run, and is never cached. Budget for model calls on repeat runs, not just the first.

Only single-action steps cache

A step that maps to one action caches and replays. Multi-action steps re-run through the agent every time.

Redis is optional

Without REDIS_URL, caching and cross-test state turn off and everything else still works. Add Redis when you want fast, deterministic replays.

Give steps room to run

A single step's AI budget can exceed Playwright's 30s default. Set test.setTimeout(60_000) at minimum so first runs don't time out.

CI integration

npx playwright test. That's it.

Your existing Playwright config, fixtures, and CI setup work as-is. Passmark is a library you import inside Playwright tests - not a separate runner.

GitHub Actions, GitLab CI, Jenkins, any CI runner
Parallel execution with zero shared-state collisions
Tracing for every step, cache hit, and assertion
Redis (optional) for action caching and cross-test state

test.yml

# .github/workflows/test.yml
- name: Run regression tests
  run: npx playwright test

About

What is Passmark?

Passmark is an open-source AI regression testing library built on Playwright. Developers write end-to-end tests in plain English. On the first run, an AI agent drives the browser and caches each single-action step to Redis. Later runs replay those cached actions directly through Playwright with no AI for those steps, while assertions are always verified by multiple models. When a UI change breaks a cached action, the agent re-discovers the correct interaction and updates the cache. Passmark includes built-in email testing, cross-test state, dynamic test data generation, and multi-model consensus assertions. It requires Node.js 18+, Playwright 1.59+, and API keys for Anthropic and Google AI; Redis is optional and enables caching and cross-test state.

Team

Built by the team behind Bug0 and Hashnode.

Passmark is the open-source core of Bug0, an AI-native QA platform trusted by 200+ engineering teams. Built by Fazle Rahman and Sandeep Panda, who previously created Hashnode, a developer blogging platform with millions of monthly readers.

Backed by Accel, Guillermo Rauch (Vercel), and Naval Ravikant.

Star on GitHub

Go managed

Run it yourself, or have Bug0 run it.

Passmark is the open-source core. Bug0 Managed is the service built on top of it. A dedicated QA engineer writes your tests, triages every failure, and gates your releases. Same engine. Zero maintenance on your end.

Open-source

Passmark

Test enginePassmark
Who writes testsYou
Who triages failuresYou
InfrastructureYours
PriceFree

Get Started

Service

Bug0 Managed

Test enginePassmark
Who writes testsDedicated QA engineer
Who triages failuresHuman-verified reports
InfrastructureBug0 AI agents for testing
PriceFrom $2,500/mo

Learn more at bug0.com