Background
Over the last week, I’ve been implementing a risk-based framework to assist with testing software products and documenting evidence for audits.
The products I work on typically deal with PII, worker and employer data, payrolling, staffing and placement, benefits, prescreening, compliance, Vendor Management Systems (VMSs), Managed Service Providers (MSPs), and all kinds of other fun stuff.
Additionally, the products serve clients domestically in the US and globally in jurisdictions across the EU, the UK, and Canada.
The Problem
The friction I was initially facing was how I could realistically keep testing and QA on pace with the speed of the developers on my team in the age of agentic coding.
Until recently, when Claude Code was rolled out to my company’s QA team, testing our products was done mostly manually.
Regression testing was painful and slow, taking days on end to complete with a team of 4 QA engineers.
Automation was essentially non-existent, and what did exist were a handful of unit and UI tests that weren’t running in CI/CD, so they weren’t providing any real value in catching regressions or ensuring quality.
The Solution
Manual testing was immediately deprioritized, but not completely eliminated, and risk-based automation testing was immediately prioritized.
Leveraging testing frameworks and best practices such as The Test Pyramid, functional testing, and non-functional testing, I developed a single workflow with five phases for testing software products:
- Classify: High-Risk or Standard. HR products are High-Risk by default.
- Measure: 20 test categories from the tool output.
- Score: Invest / Keep / Hold / Drop
- Gate: Test suites earn merge-blocking status after an advisory period. The smoke test suite blocks from day 1.
- Govern: Dual sign-off, escalation paths, and quarterly reviews.
The Defaults
Every project includes the following default test suites:
- Unit
- Security - Role-based access control (RBAC)
- UI - Smoke & E2E
- Regression
The Definitions
| Category | Definition | Coverage signal to report |
|---|---|---|
| Unit | Smallest testable units in isolation | Line/branch % (v8 or equivalent) |
| Integration | Components working together.Name the layer(s): database (RLS/pgTAP), service (internal API), cross-system (payroll/identity connectors); they are different tests Multi-jurisdiction / EOR connectors are a distinct sub-layer; decompose and report per jurisdiction in the audit fill | % of entities or integration points covered, per layer |
| Integration — Third-Party Data Handling | Compliance posture of data exchanged with external vendors (payroll processors, background-check vendors, E-Verify, tax-filing APIs): encryption in transit, audit trail of what was sent/received, retention alignment, SLA verification; distinct from whether the connector works. | Drop-by-default for early products, but the row must exist so the question has a home when a vendor integration ships |
| System / E2E | Full user journeys through the real stack (Playwright) | % of high-risk modules with ≥1 spec |
| Smoke | Critical-path checks pre-release. | Absorbs Sanity by default. Split only if the deploy pipeline diverges from the release pipeline |
| API / Interface | Merged category: route-level functional behavior and contract/schema validation. | % of routes tested; schema validation present Y/N |
| Regression | Existing suites re-run on a cadence; usually satisfied via unit/E2E gates | Cadence + gate reference |
| Mutation | Do the existing tests actually assert anything (Stryker)? | Mutation score (% mutants killed) |
| Security — Access Control & Data Boundaries | AuthZ/RLS correctness plus data-boundary assertions: right data visible to right roles across PII / pay / identity tiers; no cross-tenant or cross-client bleed. | For SOC 2-scoped products, note which controls (e.g., CC6.1; Logical Access & Information Protection) these tests evidence |
| Security — Vulnerability Scanning | SAST, dependency scanning, secrets detection. Three distinct failure modes; report each | Per sub-scope: SAST Y/N, deps Y/N, secrets Y/N |
| Performance | Response-time measurement against a stated budget | Stated budget + measured p95 |
| Load | Behavior at expected concurrent volume | Pass/fail at stated volume |
| Stress | Behavior beyond expected volume; failure modes and recovery | Pass/fail at stated ceiling |
| Reliability | Sustained-operation correctness: concurrency, retries, idempotency, crash recovery | Count of verified failure-mode scenarios |
| Compatibility | Browser/device/version matrix actually exercised | # of targets in CI vs. supported list |
| Accessibility | WCAG conformance at component level (vitest-axe) and flow level (E2E + axe) | % of components scanned; # of flows scanned |
| Usability | Formal user studies or heuristic reviews | Sessions run |
| Globalization | i18n readiness: framework, encoding, RTL, date/number formats | N/A until an i18n framework exists |
| Localization | Translated-content correctness per locale | Locales verified |
| Exploratory | Structured session-based exploration beyond scripted tests | Sessions logged |
The Framework
| Category | Tooling & pipeline ref | Coverage signal | CI gate | Priority | Owner | Revisit Trigger |
|---|---|---|---|---|---|---|
| Unit | ||||||
| Integration | ||||||
| Integration — Third-Party Data Handling | ||||||
| System / E2E | ||||||
| Smoke | ||||||
| API / Interface | ||||||
| Regression | ||||||
| Mutation | ||||||
| Security — Access Control & Data Boundaries | ||||||
| Security — Vulnerability Scanning | ||||||
| Performance | ||||||
| Load | ||||||
| Stress | ||||||
| Reliability | ||||||
| Compatibility | ||||||
| Accessibility | ||||||
| Usability | ||||||
| Globalization | ||||||
| Localization | ||||||
| Exploratory |
The Example
| Category | Tooling & pipeline ref | Coverage signal | CI gate | Priority | Owner | Revisit trigger (Hold & Drop) |
|---|---|---|---|---|---|---|
| Unit | Vitest 4 + v8 | >= 80% line/branch coverage | Required-PR (ci.yml) — coverage threshold gate enabled | Keep | QA Lead | — |
| Integration | RLS (test/rls) + pgTAP — database layer | 88% of entities (15/17) | Advisory-PR (ci.yml) — intended Required, branch protection pending | Hold | QA Lead | Pending flip from Advisory-PR to Required-PR |
| System / E2E | Playwright, 23 specs | 83% of HIGH-risk modules (5/6) | Nightly critical subset (nightly-e2e.yml); full run Weekly (weekly-e2e.yml) | Invest | QA Lead | — |
| Smoke | Playwright smoke config (playwright.smoke.config.ts) | 100% of critical paths | Advisory-PR — intended Required from day one | Keep | QA Lead | Sanity merged: no separate deploy pipeline, so post-deploy sanity = smoke. Split only if deploy pipeline diverges |
| API / Interface | Partial route tests (Vitest) + 2nd Playwright project overlapping | Partial route coverage; schema/contract validation 0% | None | Hold — consolidate the two overlapping suites before growing | QA Lead | Contract tier: first external API consumer |
| Regression | Unit + E2E suites on cadence | Currently satisfied via unit/E2E gates (no separate suite needed while those run on cadence) | Advisory-PR + Nightly (via unit/E2E gates; branch protection pending) | Keep | QA Lead | — |
| Mutation | Stryker (stryker.config.json, stryker.integration.config.json) | Break threshold enforced at 60 (high:80 / low:60 / break:60, stryker.config.json), unit scope; latest run’s score not recorded, pull from mutation.yml artifact | Weekly (mutation.yml — cron Sun 06:00 UTC + manual dispatch); break:60 enforced on unit scope. Integration scope (stryker.integration.config.json) wired but intentionally not in CI until PR3/PR4 integration tests exist | Hold — record the latest score, then re-score (Keep or Invest) | QA Lead | Promote once the latest mutation score is recorded in the audit |
| Security — Access Control & Data Boundaries | RLS (test/rls) + pgTAP; evidences SOC 2 CC6.1 (Logical Access & Information Protection) | 88% of entities with RLS tests (same RLS suite as the Integration row — two lenses, not double-counted); cross-tenant / pay-tier boundary assertions not yet named | Advisory-PR (ci.yml) — intended Required, branch protection pending | Hold | QA Lead | Promote to Invest when cross-tenant and pay-tier boundary assertions are named, written, and CI-gated. SOC 2: these tests evidence CC6.1 only — map the rest of the CC6/CC7/C1 cluster (RBAC, audit logging, encryption, access review, classification) to test rows or named non-test controls before any SOC 2 audit (owner: QA Lead) |
| Security — Vulnerability Scanning | Dependabot (deps). Secrets: GitHub native secret scanning — enable this sprint (free on Team tier, one config change). SAST: none | Deps: Y · Secrets: N (enabling — Invest) · SAST: N | None (Dependabot runs async; secret scanning enforces platform-side once enabled) | Invest (secrets) / Hold (SAST) | QA Lead | SAST: adoption decision (Semgrep or CodeQL), owner Infra team — wire as Advisory-PR, escalate per the Appendix A gate lifecycle. Secrets: no trigger enable now |
| Performance | None | 0% — no tooling | None | Drop | QA Lead | Real traffic scale number |
| Load | None | 0% — no tooling | None | Drop | QA Lead | Real traffic scale number |
| Stress | None | 0% — no tooling | None | Drop | QA Lead | Real traffic scale number |
| Reliability | 1 targeted concurrency test | 1 verified failure-mode scenario | Advisory-PR (ci.yml unit job — branch protection pending) | Hold | QA Lead | Expand scenarios at real scale or first concurrency incident |
| Compatibility | Playwright — chromium only | 1 of 1 CI-exercised browser (chromium); no other targets declared | Advisory-PR (smoke; branch protection pending) + Nightly (E2E), chromium only | Hold | QA Lead | Multi-browser demand signal (stakeholder request or non-chromium usage in analytics) |
| Accessibility | vitest-axe (component tier); 0 E2E flows | ~71 components scanned; 0 flows scanned | Advisory-PR (component tier — branch protection pending) / None (E2E tier) | Invest — E2E tier (org-wide internal tool: near-external weight, §2) | QA Lead | — |
| Usability | None | 0 sessions | None | Drop | QA Lead | Dedicated QA/UX hire |
| Globalization | None — no i18n framework | N/A | None | Drop | QA Lead | Multi-country UI rollout |
| Localization | None | N/A | None | Drop | QA Lead | Multi-country UI rollout |
| Exploratory | Informal QA triage (session notes, no formal charters) | 0 formal sessions logged | None | Drop | QA Lead | Dedicated QA hire; keep lightweight session notes meanwhile |