Screen Reader Regression Testing

What Is an Accessibility Regression?

An accessibility regression is any code change that degrades the assistive technology experience. Unlike a bug introduced in new code, a regression breaks something that previously worked. These are particularly insidious because they often pass code review (the visual appearance is unchanged) and automated rule checks (axe still passes because no WCAG rule is technically violated).

Examples of Screen Reader Regressions

Here are concrete examples of regressions that predictive testing can catch:

Each of these regressions is invisible in a visual diff. The page looks identical. But the screen reader experience is degraded: users lose context, cannot identify elements, or encounter broken navigation patterns.

How Speakable Detects Regressions

Speakable's diff mode compares the predicted screen reader output of two HTML files (typically a "before" version and an "after" version) and reports any differences in the announcements.

The Diff Algorithm

Speakable builds the accessibility tree for both HTML inputs, runs both through the same heuristic renderers, and then compares the output line by line. It identifies:

• Removed announcements: Lines present in "before" but missing in "after" (elements that lost accessibility).
• Added announcements: Lines present in "after" but not in "before" (new elements, which may or may not be intentional).
• Changed announcements: Lines that exist in both but differ in content (name changes, role changes, state changes).

Here is the diff command in action:

npx @reticular/speakable before.html --diff after.html -f text

# Output:
# NVDA Differences:
# - Submit form, button
# + button
#
# VoiceOver Differences:
# - Submit form, button
# + button
#
# Summary: 2 regressions detected across 2 readers

The minus sign indicates what the element used to announce. The plus sign indicates what it now announces. In this case, the button lost its accessible name across all readers.

Exit Codes for CI

Speakable uses exit codes to signal results to CI systems:

• Exit code 0: No differences detected. The screen reader output is identical.
• Exit code 1: Audit issues found (when using -f audit mode).
• Exit code 2: Diff detected changes in screen reader output.

In a CI pipeline, exit code 2 causes the job to fail, blocking the pull request until the regression is reviewed. The developer can then either fix the regression or update the baseline if the change is intentional.

JSON Output for Programmatic Use

For integration with custom tooling or dashboards, use JSON output:

npx @reticular/speakable before.html --diff after.html -f json -s all

# Returns structured JSON:
# {
#   "hasDifferences": true,
#   "readers": {
#     "nvda": {
#       "removed": ["Submit form, button"],
#       "added": ["button"],
#       "changed": []
#     },
#     "voiceover": {
#       "removed": ["Submit form, button"],
#       "added": ["button"],
#       "changed": []
#     }
#   },
#   "summary": { "totalRegressions": 2, "readersAffected": 2 }
# }

Setting Up Regression Testing

Regression testing requires two things: a baseline (the known-good output) and a way to compare against that baseline on every change.

Creating Baselines

Generate baselines for each component after you have verified the output is correct (either through manual testing or Speakable's text output):

# Generate baseline files for your components
npx @reticular/speakable src/Button.html -f json -s all > baselines/Button.json
npx @reticular/speakable src/NavBar.html -f json -s all > baselines/NavBar.json
npx @reticular/speakable src/LoginForm.html -f json -s all > baselines/LoginForm.json

# Commit these baselines to version control
git add baselines/
git commit -m "Add screen reader output baselines"

These baseline files serve the same purpose as snapshot files in visual regression testing. They represent the expected screen reader output for each component.

Diffing Against Baselines on Every PR

On each pull request, regenerate the output and compare against the stored baselines. If anything changed, the CI job fails and the PR author sees exactly what regressed:

# Compare current output against baseline
npx @reticular/speakable src/Button.html --diff baselines/Button.json -f json -s all

# If output matches: exit code 0 (pass)
# If output differs: exit code 2 (fail with diff details)

GitHub Actions Example

Here is a complete GitHub Actions workflow for screen reader regression testing:

name: Screen Reader Regression
on: [pull_request]

jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - run: npm ci

      - name: Check for screen reader regressions
        run: |
          EXIT_CODE=0
          for baseline in baselines/*.json; do
            component="src/$(basename "$baseline" .json).html"
            if [ -f "$component" ]; then
              npx @reticular/speakable "$component" \
                --diff "$baseline" -f json -s all || EXIT_CODE=$?
            fi
          done
          exit $EXIT_CODE

      - name: Comment on PR if regressions found
        if: failure()
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '⚠️ Screen reader regression detected. Run `npx @reticular/speakable <file> --diff baselines/<file>.json -f text -s all` locally to see details.'
            })

Interpreting Diff Output

When Speakable detects a regression, the output shows what changed for each screen reader. Here is how to read the diff:

npx @reticular/speakable form.html --diff baselines/form.json -f text -s all

# NVDA Differences:
# - Email address, edit, required
# + edit, required
#   Password, edit, required        (unchanged)
# - Remember me, check box, not checked
# + Remember me, check box
#   Sign in, button                 (unchanged)
#
# 2 changes detected for NVDA

Reading this diff: the email field lost its accessible name ("Email address" became just "edit"). The checkbox lost its checked state announcement. Both are regressions that would confuse screen reader users. The password field and submit button are unchanged.

Common causes of regressions include: removing or renaming label elements, changing HTML element types (nav to div, button to span), removing ARIA attributes during refactoring, and CSS changes that affect display property (which can hide elements from the accessibility tree).

False Positives and How to Handle Them

Not every diff indicates a real regression. Some changes are intentional. Here are common false positive scenarios and how to handle them:

• Intentional label changes: You renamed a button from "Save" to "Save Changes". The diff shows the old name removed and new name added. This is expected. Update the baseline.
• New elements added: You added a new field to a form. The diff shows additions. Verify they announce correctly, then update the baseline.
• Structural improvements: You changed a div to a nav element. The diff shows a new landmark announcement. This is an improvement, not a regression. Update the baseline.
• Component reordering: You moved a section higher on the page. The announcement content is identical but the order changed. Verify the new order is logical, then update the baseline.

When a diff appears, the workflow is:

1. Read the diff. Is this change intentional?
2. If yes: regenerate the baseline with npx @reticular/speakable src/Component.html -f json -s all > baselines/Component.json and commit the updated baseline.
3. If no: fix the regression in your code.
4. If unclear: run the component in a real screen reader to verify the new behavior is acceptable.

The baseline update step is comparable to updating snapshot tests in Jest or visual regression baselines in Chromatic. The diff is a signal that something changed; your judgment determines whether the change is acceptable.