Accessibility Testing Beyond Automation: The Manual Testing Playbook

The Automation Ceiling

Every design system team we work with has automated accessibility testing in their CI pipeline. They run axe-core on every pull request. Lighthouse scores are tracked in dashboards. Pa11y crawls staging environments nightly. And they still ship components that are unusable for people with disabilities.

This is not a tooling failure. It is a category limitation. Automated accessibility testing tools are excellent at detecting structural violations — missing alt text, insufficient color contrast, absent form labels, invalid ARIA attributes. These are pattern-matching problems, and computers are very good at pattern matching. The issue is that a large portion of real-world accessibility barriers are not structural violations. They are interaction failures, comprehension failures, and context failures.

Consider a modal dialog that traps focus correctly (automated tools will confirm the focus trap exists) but returns focus to the page body instead of the triggering element when dismissed. Or a live region that announces updates (automated tools will confirm the aria-live attribute is present) but fires so frequently that it drowns out everything else a screen reader user is trying to do. Or a form that provides error messages (automated tools will confirm they exist) but places them in a location that makes no spatial or logical sense.

What Automated Tools Cannot Evaluate

Automated testing is necessary but not sufficient. It is the foundation, not the structure. What follows is the manual testing protocol we layer on top of automation for every component we build or audit.

Screen Reader Testing Protocol

Screen reader testing is where most teams struggle the most because it requires learning an entirely different interaction paradigm. Sighted users navigate visually — scanning layouts, recognizing patterns, clicking targets. Screen reader users navigate structurally — moving through headings, landmarks, form controls, and links. If your component does not make sense when consumed as a linear stream of announcements, it does not work.

We test with three screen readers because they behave differently and have different user bases. Skipping any one of them means skipping a significant portion of your disabled user population.

VoiceOver (macOS / iOS)

VoiceOver is the default screen reader on Apple platforms and the most common screen reader for mobile testing. It pairs with Safari on macOS and is the only option on iOS. Key behaviors to understand: VoiceOver uses a virtual cursor (the "VoiceOver cursor") that is independent of the keyboard focus. This means elements can be reachable by VoiceOver but not by keyboard, or vice versa — both are bugs.

Cmd + F5          → Toggle VoiceOver on/off
VO + Right Arrow  → Move to next element
VO + Left Arrow   → Move to previous element
VO + U            → Open rotor (headings, links, landmarks)
VO + Space        → Activate current element
VO + Shift + Down → Enter a group/web area
VO + Shift + Up   → Exit a group/web area
Ctrl              → Stop speaking

NVDA (Windows)

NVDA is free, open-source, and the second most popular screen reader on Windows after JAWS. It uses a "browse mode" for reading web content and a "focus mode" for interacting with form controls. Understanding when NVDA switches between these modes — and when it fails to — is critical for testing interactive components.

TalkBack (Android)

TalkBack is the screen reader for Android devices. It uses a gesture-based navigation model — swipe right to move forward, swipe left to move backward, double-tap to activate. Mobile screen reader testing is essential because touch interaction introduces entirely different accessibility challenges than desktop keyboard navigation.

Key Flows to Test

Do not test screen reader compatibility on isolated components. Test complete user flows. A button that announces correctly in isolation may produce a confusing experience when embedded in a form wizard with inline validation, error summaries, and progress indicators. We test these flows on every project:

Keyboard Navigation Audit

Keyboard accessibility is the most fundamental layer of interactive accessibility. If a component cannot be operated with a keyboard alone, it is broken for screen reader users, switch device users, voice control users, and power users who prefer keyboard workflows. Every interactive element must be reachable, operable, and have a visible focus indicator.

Tab Order Validation

Put your mouse away. Starting from the browser address bar, press Tab repeatedly through the entire page. The focus should move through interactive elements in a logical, predictable order that matches the visual layout. Common failures we find:

• Focus jumps to a footer link before reaching the main content
• Visually hidden elements receive focus (off-screen menus, collapsed accordions)
• Custom components built with div or span are not focusable at all
• Positive tabindex values create a chaotic navigation order
• Focus order does not match visual order after content reflows at different viewport sizes

Focus Trap Testing

Modal dialogs, dropdown menus, and other overlay components must trap focus — keeping Tab and Shift+Tab cycling within the component until it is dismissed. But focus traps are tricky to implement correctly. Here is what to verify:

Skip Links

Skip navigation links allow keyboard users to bypass repeated content blocks (headers, navigation menus) and jump directly to main content. They should be the first focusable element on the page, become visible on focus, and target the main content area. Test that they actually work — many implementations have broken anchor targets or fail to move focus after activation.

Roving Tabindex Patterns

Composite widgets like tab lists, toolbars, and menu bars should use the roving tabindex pattern. The group has a single tab stop. Arrow keys move focus between items within the group. This matches the interaction pattern users expect from native OS controls and prevents a toolbar with 20 buttons from requiring 20 Tab presses to pass through.

Tab list:     Tab into → Arrow Left/Right → Tab out
Menu bar:     Tab into → Arrow Left/Right between menus
              Arrow Down to open → Arrow Up/Down within
Toolbar:      Tab into → Arrow Left/Right → Tab out
Radio group:  Tab into → Arrow Up/Down → Tab out
Tree view:    Tab into → Arrow Up/Down to move
              Arrow Right to expand → Arrow Left to collapse

Focus Management in SPAs

Single-page applications break the browser's built-in accessibility model. In a traditional multi-page website, a link click triggers a full page load. The browser announces the new page title, resets focus to the top of the document, and the screen reader user knows they have arrived somewhere new. SPAs do none of this by default. The URL changes, the content swaps, but the browser treats it as the same page — because technically it is.

This is the single most common accessibility failure we encounter in modern web applications, and it affects every SPA framework: React, Vue, Angular, Svelte, and their meta-frameworks.

Route Change Announcements

When the route changes in a SPA, you must explicitly inform assistive technology that navigation has occurred. There are two primary approaches:

function handleRouteChange(pageTitle) {
  // Update document title
  document.title = `${pageTitle} | My App`;

  // Find the main heading or content area
  const target = document.querySelector('h1')
    || document.querySelector('main');

  if (target) {
    // Make it focusable if it isn't already
    if (!target.hasAttribute('tabindex')) {
      target.setAttribute('tabindex', '-1');
    }
    // Move focus
    target.focus();
  }
}

Focus Restoration After Modal Close

When a modal, dialog, or popover closes, focus must return to the element that triggered it. This sounds simple, but edge cases abound. What if the trigger was inside a list item that has since been removed? What if the trigger was a delete button and the item it belonged to no longer exists? You must have a fallback strategy:

• Store a reference to the trigger element before opening the modal
• On close, check if the trigger still exists in the DOM
• If it does, focus it
• If it does not, focus the nearest logical parent or the next element in the list
• As a last resort, focus the main content area

Managing Focus in Dynamic Content

Toast notifications, inline alerts, loading indicators, and live data updates all present focus management challenges. The key principle is: notify without disrupting. Moving focus to a toast notification is almost always wrong — it rips the user away from what they were doing. Instead, use aria-live regions with appropriate politeness levels:

Cognitive Accessibility

Cognitive accessibility is the least tested and least understood dimension of accessibility work. WCAG addresses it through success criteria around reading level, consistent navigation, error identification, and input purpose — but cognitive accessibility extends far beyond what any specification can capture. It is about whether real people with varying cognitive abilities can actually understand and use your interface.

Reading Level Assessment

WCAG 2.1 Success Criterion 3.1.5 recommends that content be understandable at a lower secondary education reading level (roughly age 12-14). For UI text — labels, instructions, error messages, help text — we aim even simpler. Test your component text by asking: could a stressed, distracted, non-native English speaker understand this on the first reading?

Information Density

Every additional element on screen competes for cognitive resources. Users with ADHD, anxiety disorders, or traumatic brain injuries are particularly affected by information-dense interfaces. Evaluate each component for cognitive load:

• How many distinct pieces of information does the user need to process simultaneously?
• Can any information be deferred to a secondary view or progressive disclosure pattern?
• Are related items visually grouped to reduce parsing effort?
• Is there sufficient whitespace to prevent elements from blending together?
• Can the user predict what will happen before they take action?

Motion Sensitivity

Animations and transitions that seem subtle to most users can cause nausea, dizziness, or seizures in users with vestibular disorders or photosensitive epilepsy. This is not an edge case — approximately 35% of adults over 40 have experienced vestibular dysfunction.

/* Default: animations enabled */
.component {
  transition: transform 300ms ease;
}

/* Reduced motion: remove or minimize animations */
@media (prefers-reduced-motion: reduce) {
  .component {
    transition: none;
  }
}

/* Test checklist for motion: */
/* 1. Does prefers-reduced-motion disable all non-essential animation? */
/* 2. Are parallax effects removed? */
/* 3. Are auto-playing videos/carousels paused? */
/* 4. Do page transitions use crossfade instead of slide? */
/* 5. Are loading spinners replaced with static indicators? */

Error Message Clarity

Error messages must answer three questions: What happened? Why did it happen? What should the user do now? We review every error message in the design system against these criteria. Vague messages like "An error occurred" or "Invalid input" fail all three.

Progressive Disclosure

Show users only what they need at each step. Collapse advanced options. Break long forms into multiple steps with clear progress indicators. Let users expand sections on demand rather than presenting everything at once. This reduces cognitive load for everyone and is essential for users with cognitive disabilities.

The Component Testing Matrix

Not every component needs the same testing depth. A static heading does not need keyboard testing. A complex date picker needs everything. This matrix maps component types to testing requirements, distinguishing what automation handles from what requires human evaluation.

Testing with Real Users

WCAG compliance is necessary but not sufficient for actual accessibility. You can build a component that passes every WCAG success criterion and still creates a frustrating, unusable experience for disabled users. Specifications define minimum technical requirements; usability testing reveals whether those requirements translate into a workable experience.

We have seen date pickers that are technically keyboard-accessible but so cumbersome to operate that screen reader users abandon the task. We have seen forms that meet every ARIA specification but present information in an order that makes no sense when consumed linearly. Compliance testing tells you whether you followed the rules. User testing tells you whether the rules were enough.

Recruiting Participants

Recruiting disabled participants requires intentionality and respect. Some practical approaches:

• Partner with disability organizations (NFB, ACB, RNIB) who maintain tester networks
• Use accessibility-focused research platforms like Fable or Access Works
• Include disability in your general user research recruitment criteria rather than treating it as a separate study
• Compensate participants fairly — rates should reflect the specialized expertise they bring
• Allow participants to use their own devices and assistive technology, not lab equipment they are unfamiliar with

Session Structure

Accessibility user testing sessions require some adjustments from standard usability testing:

Common Findings That Specs Miss

Across dozens of accessibility user testing sessions, these are the issues that consistently surface that no automated tool or manual WCAG audit would catch:

Building a Testing Culture

Manual accessibility testing is only sustainable when it is embedded in your development process, not bolted on as a final gate. Teams that treat accessibility as a pre-release checklist consistently fail to maintain it. The findings come too late, the fixes are too expensive, and the testing itself becomes associated with delay and friction.

Integrating into Sprint Workflows

Accessibility testing should happen at three points in the development cycle:

Accessibility Champions Program

Centralized accessibility experts create bottlenecks. Instead, train accessibility champions within each team. These are developers, designers, or QA engineers who receive deeper accessibility training and serve as the first line of defense. They are not solely responsible for accessibility — that responsibility belongs to the entire team — but they have the expertise to catch issues early and mentor others.

• One champion per squad or feature team
• Monthly knowledge-sharing sessions across champions
• Champions participate in design reviews and provide accessibility guidance
• Champions have allocated time (10-15% of sprint) for accessibility work
• Champions maintain the team's accessibility testing documentation and checklists

Definition of Done

If accessibility is not in your definition of done, it will be treated as optional. Add explicit criteria:

[ ] Automated tests pass (axe-core, no new violations)
[ ] Keyboard walkthrough completed (all interactions work)
[ ] Screen reader tested with VoiceOver or NVDA
[ ] Focus management verified (modals, route changes)
[ ] Color contrast verified for all states (hover, focus, active, disabled)
[ ] Content reviewed for reading level and clarity
[ ] Zoom tested at 200% (no horizontal scroll, no overlap)
[ ] Touch targets meet 44x44px minimum

Quarterly Audit Cadence

Even with strong sprint-level testing, conduct a quarterly comprehensive audit. This catches issues that emerge from component interactions, tests complete user flows rather than individual components, and provides a baseline for tracking accessibility progress over time. Each quarterly audit should include both automated scanning and manual testing of the top 10 user flows, with findings tracked in the same backlog as feature work — not in a separate accessibility backlog that nobody prioritizes.

Our Per-Component Checklist

This is the exact checklist we run against every component in a design system. It is not exhaustive for every possible context, but it covers the critical dimensions that we have found catch the vast majority of accessibility issues. We evaluate each item as Pass, Fail, or Not Applicable.

Accessibility testing is a skill that improves with practice. The first time you test with a screen reader, it will feel foreign and slow. Within a few weeks of regular practice, you will develop fluency — and you will start noticing accessibility issues instinctively, before you even begin formal testing. That instinct, built through repeated manual testing, is something no automated tool can replicate.