Regression testing optimized

Building an automated regression harness and bringing it to some form of regularity is what I’d argue is the most important testing task for any non-trivial project. The amount of work, time, money and tension saved by automated regression is so huge that I could compare it only with the recent developments in agentic test documentation support (test documentation has been the biggest pain since the dawn of software testing).

Obviously, there are myriads of caveats when implementing something as big as a regression suite, and as complex as an automated one. The history of testing knows dozens of best practices, dos and don’ts. For example:

Minmaxing the regression suite

I’d say this practice is a bit outdated nowadays but it certainly helped to keep the growing monstrosity at bay in the past. The idea is straightforward: add as few tests to the regression suite as you can.

×

Three ways to keep the regression suite thin: test techniques, siblings and causality, the testing pyramid

Exploit the testing techniques. To put it roughly, test techniques are statements (or rather assumptions) for minimizing the number of tests required to cover something in a proper manner. For instance, the well-known Equivalence Partitioning (EP) is an assumption that the application behavior will be the same for all the inputs within the range defined by you based on your understanding of internal application logic and external business logic. You make educated assumptions, thus reducing the amount of work for presumably the same coverage.

Assume siblings, cause and effect. It looks like an upscaled version of testing techniques but it’s rather a mindset that you develop the more you work with a particular application. At some point you will learn that some logic might have siblings across the application, so a change in one implies the others are affected. You will learn that some pieces of logic are tightly connected so that if you prove the later one (the effect) works properly, you can assume the earlier one (the cause) works too. Of course, there is no rigid science behind this knowledge. It’s made of intuition, experience and risk-based decisions. This is why a human might stay in the loop for a while.

Follow the testing pyramid. If you monitor lower levels of testing closely enough or you’re on good terms with developers, you might have a picture of the whole testing that happens in the app besides what testers are doing (yeah, not only testers are responsible for testing the app). And by the way, I see this point neglected a lot because of testers completely ignoring unit tests and other tests supported by developers. Perhaps they think that developers’ tasks are none of their concern. But if you don’t know what’s being covered on lower levels of pyramid, you can’t deduplicate the higher levels. Testers should dedupe against what developers already cover at the unit, integration and API levels, and aim for testing only the emergent or user-facing behavior.

So following these three and a thousand more principles we can minimize our regression suite. Cool. If your regression suite is about 100 tests running up to an hour, then perhaps you can stop reading here. But my reality was always about heavy 500+ regression suites, so simply thinning them out was never a complete answer to them being big frustrating gates.

Test Impact Analysis

Test Impact Analysis (TIA) is basically about having application monitors that thoroughly trace any call to a particular code element while your automated tests are being executed. The idea is simple: if frontend element Header and backend method authenticateUser were reached while executing test #5, it means they are tied logically. After mapping these facts together, from now on any change to the Header or authenticateUser would imply that our test #5 is required to execute since that test is exactly about them (the V8 setup below covers only the frontend side).

Playwright + V8

Playwright has a straightforward solution for JS code in its ecosystem - a bridge with V8. Playwright can directly ask Chromium’s JS engine called V8 for data which lets us know exactly what is being executed.

The idea is not new. I personally stumbled upon it back in 2024 in this rather concise Medium post by Cenfun. Recently my curiosity was reignited after watching this practical presentation by Nataliia Pyslar (in Ukrainian). She also points out several fine points about deduplication and overly wide traces.

Typically you would want some kind of fixture that wraps each test in a coverage monitor:

// tests/fixtures.ts
import { test as base } from '@playwright/test'
import { writeFile } from 'node:fs/promises'
import path from 'node:path'
import v8toIstanbul from 'v8-to-istanbul'

const distDir = path.resolve('dist')

export const test = base.extend<{ collectImpact: void }>({
  collectImpact: [
    async ({ page, browserName }, use, testInfo) => {
      // V8 coverage exists only in Chromium
      if (browserName !== 'chromium') return use()
      // The default (true) wipes coverage on every goto()
      await page.coverage.startJSCoverage({ resetOnNavigation: false })
      await use()
      const coverage = await page.coverage.stopJSCoverage()
      // A failed test stops early, so its partial coverage would drop files
      if (testInfo.status !== testInfo.expectedStatus) return

      const touchedFiles = new Set<string>()
      for (const entry of coverage) {
        const { pathname } = new URL(entry.url)
        if (!pathname.startsWith('/assets/')) continue
        // Point at the built file: v8-to-istanbul reads the .map next to it
        const converter = v8toIstanbul(path.join(distDir, pathname), 0, {
          source: entry.source!,
        })
        await converter.load()
        converter.applyCoverage(entry.functions)
        for (const [file, data] of Object.entries(converter.toIstanbul())) {
          // Count called functions: statements mark every bundled module as hit
          if (Object.values(data.f).some((calls) => calls > 0)) {
            touchedFiles.add(path.relative(process.cwd(), file))
          }
        }
      }
      await writeFile(
        testInfo.outputPath('impact.json'),
        JSON.stringify({
          test: testInfo.titlePath.join(' › '),
          files: [...touchedFiles].sort(),
        }),
      )
    },
    { auto: true },
  ],
})
export { expect } from '@playwright/test'

Raw V8 output is just intermediate data. What you want is a literal (usually JSON) trace from test to files and/or from files to tests:

// build-tia-map.ts
import { glob, readFile, writeFile } from 'node:fs/promises'

const impactMap: Record<string, string[]> = {}
for await (const file of glob('test-results/**/impact.json')) {
  const { test, files } = JSON.parse(await readFile(file, 'utf8'))
  impactMap[test] = files
}
await writeFile('tia-map.json', JSON.stringify(impactMap, null, 2))

×

Playwright's V8 coverage for one test becomes a test impact map: a changed file selects only the tests that executed it

What about non-JS backends?

The V8 bridge is surely convenient but life isn’t always about having a transparent full-JS app. For non-JS backends there are still plenty of options - from external solutions (SeaLights, Parasoft etc.) to native language features (like Ruby coverage) to the ones you can vibecode yourself.

The one thing to keep in mind is that backend coverage is counted per server process, not per test. Tools attribute it to a test either by resetting and dumping coverage around it, or by matching hits to test start and end times. Either way, parallel tests against one server get mixed, so for TIA runs reduce your workers to exactly 1 (or run one backend instance per worker).

Flexible regression in PR pipeline

After building the TIA map your next step is to just build a harness for PR pipeline that will select regression tests based on the code diff. Just make sure the harness falls back to the full suite whenever a changed file isn’t in the map, otherwise a change that no test traced selects nothing. That’s basically it! You just turned the regression suite from a final sign-off before release into a routine part of every PR!

TIA is not a complete solution

Traceability can be elusive. There are a lot of possible complications that can make direct traceability difficult. To name a few:

• the change isn’t code (configuration, feature flags, environment variables, database migrations, seed data, lockfile upgrades, CSS, i18n strings);
• the coverage tool can’t see the code (stored procedures, database triggers, server-side templates, another team’s service, a third-party SaaS, generated clients);
• code is found at runtime, not called directly (reflection, dependency-injection registration, plugin registries, routes loaded by a file glob);
• a test doesn’t run the same code every time (random data, feature flags, A/B splits, code that depends on the date);
• code runs outside the test’s window (queue consumers, cron jobs, webhooks, email sent after the test ends);
• the map is out of date (code is moved between files, renamed or split after the map was built);
• the code has nothing to call (constants, type tables, static initializers);
• some code runs for every test (app bootstrap, auth middleware), so changing it selects the whole suite;
• the file is new, so no test maps to it yet.

So at the end of the day you still need to run the full suite, at least once after the deliverable code freeze. But in my opinion there is an even better addition or alternative for TIA.

Regression as a cron job

I strongly advocate running the full regression suite as a recurring cron job as regularly as you can while your team sleeps. Ideally, every night.

Running full regression nightly completely relieves you of the burden of minimization. You don’t have to be shy anymore about the size and execution time of your suite. I don’t want to say that minmaxing principles should be ignored in this case though. Just that you can afford a few extra tests for peace of mind.

If you’re working in a small-to-medium project, regression as cron can be your whole setup. You don’t need the whole TIA harness since not that many PRs land in your app each day. If a nightly or weekly run has 5 failed tests, it’s obviously one of the few merged PRs. It’s really easy to pinpoint a culprit at such scale of development so TIA, while nice, might be overkill.

I would say a proper TIA harness is a pretty complex thing. You need to make a lot of decisions on how you deal with uncertainties and pair TIA with full regression execution which still has to happen. If you’re not sure yet, start with a regular cron job scheduled outside of working hours.