Pa11y CI Setup: Automated Accessibility Testing in Your Pipeline

Pa11y CI is a command-line accessibility test runner built for continuous integration. Where the core pa11y command scans one URL and prints a report, pa11y-ci reads a list of URLs (or a sitemap), scans them all in a single run, and exits with a non-zero status code when any page exceeds its violation threshold. That exit code is the whole point: it is what makes a CI system mark the build as failed.

Under the hood Pa11y CI drives a headless Chromium instance through Puppeteer, loads each page, and evaluates the rendered DOM against either the HTML CodeSniffer ruleset or axe-core. It is the natural next step after the automated accessibility testing overview: that guide compares the tools; this one sets one of them up end to end.

Install Pa11y CI as a dev dependency so the exact version is pinned in your lockfile and every contributor and CI run uses the same one. It pulls in its own copy of Chromium, so no separate browser install is needed.

# Install as a dev dependency (recommended)
npm install --save-dev pa11y-ci

# Run it through npx
npx pa11y-ci --help

# Or scan a single URL with no config to confirm it works
npx pa11y-ci https://example.com

Add a script to package.json so the command is short and consistent across local runs and CI:

{
  "scripts": {
    "test:a11y": "pa11y-ci --config .pa11yci.json"
  }
}

Pa11y CI reads its settings from a JSON file — by convention .pa11yci or .pa11yci.json. It has two parts: defaults, applied to every URL, and urls, the list of pages to scan. A URL entry can be a plain string or an object that overrides the defaults for that one page.

{
  "defaults": {
    "standard": "WCAG2AA",
    "runners": ["axe"],
    "timeout": 30000,
    "threshold": 0,
    "hideElements": "#cookie-banner, .third-party-widget"
  },
  "urls": [
    "http://localhost:3000/",
    "http://localhost:3000/patterns/",
    {
      "url": "http://localhost:3000/legacy-page",
      "threshold": 8
    }
  ]
}

Maintaining a hand-written URL list goes stale the moment someone adds a page. Point Pa11y CI at your sitemap.xml instead and every page in the sitemap is covered automatically:

# Scan every URL listed in the sitemap
npx pa11y-ci --sitemap https://example.com/sitemap.xml

# Test a local build by rewriting the production host to localhost
npx pa11y-ci \
  --sitemap https://example.com/sitemap.xml \
  --sitemap-find https://example.com \
  --sitemap-replace http://localhost:3000

The --sitemap-find and --sitemap-replace flags rewrite each URL's hostname on the fly, so you can reuse your production sitemap to test a freshly built local copy or a deploy preview. Defaults from a .pa11yci file still apply when you scan from a sitemap, so you keep your standard, runners, and timeout settings.

The highest-value place to run Pa11y CI is on every pull request, against a freshly built copy of the site. The workflow below builds the site, serves it locally, waits for the server, then scans it. A non-zero exit from pa11y-ci fails the job and blocks the merge.

name: Accessibility (Pa11y CI)

on:
  pull_request:
    branches: [main]

jobs:
  pa11y:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm

      - name: Install dependencies
        run: npm ci

      - name: Build site
        run: npm run build

      - name: Serve site
        run: npx http-server ./dist -p 3000 &

      - name: Wait for server
        run: npx wait-on http://localhost:3000

      - name: Run Pa11y CI
        run: npm run test:a11y

Because the config points at http://localhost:3000, the same npm run test:a11y script runs identically on a developer's machine and in CI — no environment-specific branching.

Pa11y CI prints a per-URL summary and, for each failing page, the list of violations with the rule that fired, a human-readable message, and a CSS selector pointing at the offending element:

Running Pa11y on 3 URLs:
 > http://localhost:3000/ - 0 errors
 > http://localhost:3000/patterns/ - 0 errors
 > http://localhost:3000/contact - 2 errors

Errors in http://localhost:3000/contact:
 • Elements must have sufficient color contrast (color-contrast)
   (#newsletter-cta > .btn)
 • Form elements must have labels (label)
   (#email)

✘ 2 errors found across 3 URLs

To capture results for a build artifact or a dashboard, add --reporter json and redirect the output to a file. The JSON reporter emits a machine-readable record of every URL and violation, which you can attach to the CI run or feed into your own reporting:

npx pa11y-ci --config .pa11yci.json --reporter json > pa11y-results.json

Turning on a zero-tolerance gate against an existing site floods the first build with failures and tempts the team to disable the whole thing. The fix is a baseline: set each page's threshold to its current violation count so the build passes today, then ratchet the numbers down as you fix issues. New pages start at zero.

{
  "defaults": {
    "standard": "WCAG2AA",
    "runners": ["axe"],
    "threshold": 0
  },
  "urls": [
    "http://localhost:3000/",
    { "url": "http://localhost:3000/dashboard", "threshold": 4 },
    { "url": "http://localhost:3000/legacy-report", "threshold": 11 }
  ]
}

The discipline is one-directional: a threshold may only ever go down. Every time you fix a violation, lower the number so the gap cannot silently refill. Record the totals over time — a steadily shrinking sum is proof your accessibility debt is being paid down rather than just contained.

A per-build pass/fail answers "did this change break anything?" but not "are we getting better or worse over time?" Pa11y Dashboard answers the second question. It is a separate self-hosted web app that runs Pa11y against your URLs on a schedule, stores each run in a database, and charts the violation count per URL over time.

It is a deliberately different tool from pa11y-ci:

Most teams do not need the dashboard on day one. Start with pa11y-ci in the pipeline to stop regressions, and add Pa11y Dashboard later when someone needs to see the trend line — for a compliance report, a quarterly review, or to prove the backlog is shrinking. Both tools share the same underlying engine, so a violation looks identical in either.