Summary: A discussion of some problems with VPATs and ACRs and their role in the procurement process for government and large organizations. From that discussion, I’ll also put forward the kind of accessibility documentation that I think would be more valuable when purchasing a product.
In the Ad Hoc Insights blog post about state and local government web accessibility that I co-wrote with Mark Headd, I included this bullet point:
Ask vendors tough questions. In recent years, unscrupulous companies have begun selling automated “one line of code” products that promise to make a website ADA-compliant — products that don’t actually work. Likewise, some vendor-provided Voluntary Product Accessibility Templates (VPATs) offer an incomplete picture of their product. When reviewing proposals, state and local governments should expect detailed answers about WCAG conformance, findings from user research, and a vendor’s own assistive technology testing capabilities.
The swipe at overlays is pretty self-explanatory for anyone who works in digital accessibility. Overlays don’t work, will get you sued, and the companies that make them seem to routinely have significant ethical lapses.
But I also threw some shade at VPATs, which may seem unjustified. VPATs are a mainstay of digital accessibility in government, so why the negativity? Surely they don’t belong in the same category as overlays!
Yet here I am in an internal chat at my company earlier this year, where I was even more blunt:
My super-cynical [opinion] is still that VPATs should be treated as marketing materials.
Oof. Harsh words. But this is a topic that’s been slowly burning in the back of my mind for a long time, and I’d like to explore my anti-VPAT stance a little more. What’s wrong with VPATs? Why do I have such a negative reaction to them? What might be better?
What’s a VPAT? What’s an ACR?
Let’s start with some definitions. I’ll let Wikipedia do the heavy lifting here:
A Voluntary Product Accessibility Template (VPAT) is a template containing information regarding how an Information and communications technology product or service conforms with Section 508 of the Rehabilitation Act of 1973, as amended (29 U.S.C. § 794 (d)). Section 508 provides guidelines for rendering ICT accessible to, and therefore usable by, people with disabilities. The VPAT was originally designed as a tool for vendors to document product compliance to Section 508 and facilitate government market research on ICT with accessible features. Many people started to call the completed document a “VPAT” but the wider procurement community would prefer to call it a product Accessibility Conformance Report, or ACR. The distinction is that the VPAT is the incomplete form, and the ACR is the completed report using the VPAT template.
Feeling a little called out by Wikipedia here! Like many others in the field, I feel like I’ve always referred to both the template and the completed report as “VPAT.” Even if the two terms are largely interchangeable in my head, I’ll try to be a little more deliberate in how I use VPAT and ACR in this post:
- ACR = compliance report documenting accessibility features and deficiencies of a product
- VPAT = blank template for generating an ACR
The Information Technology Industry Council (ITI) maintains and periodically updates the template, with Word documents for different flavors available depending on your audience. Most ACRs that I’ve encountered have been written using these templates or a close variation, then exported as PDF files. The General Services Administration (GSA) in the United States maintains the OpenACR editor, which is essentially the VPAT presented as a web form that publishes ACRs in HTML and YAML.
Correction, June 12, 2024: The preceding paragraph previously stated that the OpenACR YAML file could be imported into Drupal, which is not correct. I’m not sure exactly how I ended up making that claim, except that some combination of misreading and misremembering led me there. Thank you to Mike Gifford for pointing out the error!
(And also apologies to Mike, who pointed out my error five days ago and I’m only just now getting around to correcting it.)
Sample ACR entry
Regardless of publishing format, an ACR contains:
- Basic information about the product that’s been evaluated and who’s evaluating it
- How it was evaluated (methodology, assistive technology used, testing tools used)
- What standard it’s being evaluated against (eg. WCAG 2.1)
- For each success criterion or requirement in the standard:
- A conformance level (supports, partially supports, does not support, not applicable, or not evaluated)
- Remarks explaining the assessment—specific pages of concern, identified barriers, limitations in what could be tested, caveats about proper usage, and so on
Per Wikipedia, the VPAT/ACR standard has been in use since 2001. Reading an ACR is a pretty straightforward exercise, if you can stomach giant tables in PDFs. Here’s an entry from Google’s current ACR for Gmail:
| Criteria | Conformance level | Remarks and explanations |
|---|---|---|
| 1.4.1 Use of Color (Level A) Also applies to: EN 301 549 Criteria – 9.1.4.1 (Web) – 10.1.4.1 (Non-web document) – 11.1.4.1 (Open Functionality Software) – 11.1.4.1 (Closed Software) – 11.8.2 (Authoring Tool) – 12.1.2 (Product Docs) – 12.2.4 (Support Docs) Revised Section 508 – 501 (Web) (Software) – 504.2 (Authoring Tool) – 602.3 (Support Docs) | Partially Supports | Use of alternates to color is widely supported across Gmail. However, there are exceptions: – Color alone is used to convey info on the “Don’t miss any important messages”, “I want to find information from a message received earlier”, “Keep my Gmail account secure”, “Remember to get back to a message later”, “Send a quick short message” and “Share content with someone” pages. – Link contrast is not at least 3:1 with surrounding text on the “Keep my Gmail account secure”, “Send a quick short message” and “Understand a conversation” pages. |
Seems pretty straightforward. If I’m responsible for procuring a product and I want to determine if it’s accessible, reading an ACR gives me a good first look at what the people who made the product are self-reporting.
Where things go wrong
Over the course of my career, I’ve been involved with both the production of digital products and the procurement of them. I’ve read ACRs as part of evaluating bids, and I’ve provided recommendations on whether or not to buy products based in part on those ACRs. But that’s the sort of thing I’ve been brought into occasionally and as needed. Reading and writing ACRs has never been a core day-to-day job responsibility.
That’s the round about way of saying, I don’t actually know anything about anything. I have thoughts about the ACRs that I’ve read, the teams that I’ve worked on when building things, and the procurement processes that I’ve been a part of. And apparently I feel strongly enough about those experiences that this blog post has eventually bubbled up out of them. But if I’m off-base on any of this, I am absolutely interested in hearing that feedback.
With that said, here are some strongly-held and possibly misinformed opinions about where the VPAT/ACR model falls apart.
Conflicts of interest
Some vendors bring in a third-party to complete an accessibility audit, and completing audits in the VPAT framework is the bread-and-butter of the accessibility industry. For example, here’s TPGi’s page about hiring them for your VPAT. While accessibility consulting firms often offer more than just an audit and ACR, this is something well within the wheelhouse of the industry.
But some vendors complete their ACRs internally, completed either by an in-house accessibility specialist or by whichever developer or designer gets told to do it as part of their other responsibilities. There’s nothing wrong with assessing your own products for accessibility, or with non-accessibility specialists being involved in accessibility testing. In fact, that’s kind of what you hope every company does! But when you’re creating a compliance report in hopes of landing a lucrative contract, it’s easy to imagine management pressure to present the most positive picture possible. Even if that doesn’t happen often, the risk (or just perception of risk) is still there—and gives me that uneasy feeling about what I’m reading.
Mental model mismatch
Software often does a lot of stuff. When you’re buying it, you’re not always buying it for all of the features. There are many user flows through a product, but not all of them are equally relevant to your organization.
Looking at that “partially supports” Gmail ACR entry on use of color again, here’s part of the explanation provided:
Color alone is used to convey info on the “Don’t miss any important messages”, “I want to find information from a message received earlier”, “Keep my Gmail account secure”, “Remember to get back to a message later”, “Send a quick short message” and “Share content with someone” pages.
When you think of an email client, these are likely not the core user paths that are motivating your procurement decision. You’re probably much more interested in the feature set you described in your request for proposals—whether the product can do the things you said you’re looking for, and whether the experience when doing those things accessible. ACRs aren’t built around that mental model.
That’s not to say secondary features don’t need to be accessible! Every user should be able to take advantage of every part of the product. I don’t at all mean to excuse poor practices or say disabled users should be satisfied with something that’s “good enough.”
But accessibility is about risk assessment, and that’s especially true for procurement. In this context, risk assessment isn’t just about legal liability. It’s also about:
- Choosing a product that will empower the people in your organization to accomplish tasks, and not slow them down or prevent them from doing their jobs.
- Choosing a vendor that will respond to feature requests and bug reports in a timely manner, and won’t leave your organization operating at a diminished capacity as you wait for those improvements.
If you’re already familiar with the product, the details provided in the ACR may be enough to piece together that information. But if you don’t already have a strong understanding of how the product works, it takes some close reading and some reading between the lines to gain the insights you need to assess risk.
Stretched definitions of conformance levels
I don’t know that ITI or GSA have ever defined each conformance level, but Harvard University’s accessibility team describes the conformance levels as:
- Supports: The functionality of the product has at least one method that meets the criterion without known defects or meets with equivalent facilitation.
- Partially Supports: Some functionality of the product does not meet the criterion.
- Does Not Support: The majority of product functionality does not meet the criterion.
- Not Applicable: The criterion is not relevant to the product.
These seem like fairly clear definitions, but there’s a lot of space for well-intentioned (or malicious!) interpretations that paint a deceptive picture.
- Having “at least one method that meets the criterion” or “meets with equivalent function” are squishy enough that you can talk yourself into some pretty arduous workarounds meeting the requirement.
- Determining the line between “some functionality” and “the majority” not meeting the criterion is likely to be subjective. What’s the line for “the majority” of times color is used in a web app?
- A “not applicable” requirement may not be relevant to the intended use case for a product, but the reality of how your users actually use it is less predictable.
Having your ACR completed by a third party mitigates these somewhat. But it’s easy to imagine a motivated interpretation of these conformance levels in an ACR completed internally by an engineer under pressure from management to help win a contract. It certainly explains ACRs that I’ve read with lengthy exceptions documented for success criteria labeled with “supports.”
Incomplete testing
I once used a product whose error states had multiple accessibility issues. Like, every single occurrence of an error message failed multiple WCAG success criteria. When I went back and reviewed the ACR? Error states weren’t mentioned once, except to note that they existed under WCAG criteria like 3.3.1.
One of my primary responsibilities in my current job is completing accessibility audits for products that are about to launch. I’m absolutely up front with everyone I talk to about this: I miss stuff! It’s hard to catch it all! Testing big complicated products takes forever, and after a while you get bored or you start tuning subtle issues out. So I’m sympathetic to ACRs that don’t document everything. It’s absolutely understandable that sometimes things slip through the cracks.
But there’s an important difference between human error and testing that excludes key parts of the product, like:
- Unhappy user paths like error states, warning messages, and temporary dead ends
- Secondary user flows besides the primary purpose of the product
- Opt-in beta features, including beta features that may be adopted by admins for all of an organization’s users
A keyboard-only user is going to trigger an error state at some point. A screen reader user is going to try to use a secondary feature. But once again, it’s easy to imagine excuses being made to exclude things from the scope of testing—exclusions that are not obvious from reading an ACR but become clear after some period of use.
Out-of-date ACRs
Google is big enough that I’ll call them out for this. That Gmail ACR I shared an excerpt from? It’s the current ACR listed on Google’s accessibility page, and at the time of this writing it’s about ten months out of date. Google’s public development blog would suggest that Gmail has had quite a few new user-facing features launch since August 2023, but these are not accounted for in the ACR.
And there’s a lot of variation in how frequently ACRs are updated. GitHub has committed to regular updates to their ACRs and all but one has been updated within the last month. But as I was writing this post, someone in the web-a11y Slack space posted about a vendor providing an eight year old ACR. You never know what you’re going to get.
I get it! Completing a comprehensive ACR is a ton of effort, and Agile software development practices are built around constant, small iterations—not big version releases building up to a major compliance report. It’s not practical (or even desirable!) for most organizations to complete a new ACR after each two-week sprint. But as a customer that limits the value of the ACR in assessing whether I should purchase the product now, and it limits the value of the ACR in predicting whether the product will continue to meet the needs of my organization.
VPATs and ACRs are artifacts of how software was written 20 years ago, and not how much of the industry approaches it today. At a time when software was released in discrete versions—and the next version might involve another procurement process—having a report that’s a few months old might not be an issue. But we now live in a software landscape of continuous deployment, public betas, A/B testing in production, and feature roll-outs that reach different users at different times. It’s hard to imagine an ACR that can keep up.
The wrong people reviewing the ACR
I’m going to keep this anecdote extra vague, because it involves people I’ve had professional relationships with who really were trying to do their due diligence.
A new product was adopted by a team at a place where I worked. After it was announced, I asked if any accessibility review had occurred prior to purchasing the product. Someone replied explaining that they had reviewed the product’s ACR, as provided by the vendor. They provided a link to the ACR for me to review as well.
When I reviewed the ACR, I saw a lot of red flags: the ACR wasn’t particularly recent, there were a lot of “supports” and “partially supports” that had numerous documented exceptions, and there were things I wasn’t seeing in the ACR that made me think some user paths weren’t included in the scope of the testing.
At worst, an ACR is a self-evaluation that tells the story a vendor wants to tell. At best, an ACR is a starting point for determining whether the product is accessible. It’s a road map to guide your own testing of the product. Each “partially supports” is a yellow flag that you need to test yourself to validate the explanation provided by the vendor. The ACR itself isn’t the end of your accessibility assessment, it’s the beginning.
But the team that purchased the product in my vague anecdote didn’t include anyone with an accessibility background—or, possibly, anyone who had ever read an ACR before. They didn’t do any of their own testing. Instead, they saw a lot of “support” and “partially support” labels in that second column of the tables and interpreted it as the product being accessible.
As I’m sure you can guess, this was not the case.
Imagining something better
What kind of documentation would actually be useful when purchasing a product? What should you actually look for to decide if the thing you want to buy is actually going to be accessible to your users?
An ACR approaches accessibility as though it’s an end goal, where you a product is building towards a state of completion. I feel like it implies that the product is finished, and that the “partially supports” and “does not support” items are bugs in the backlog for future dot-releases. It implies that there may be some future state where every entry in the ACR is labeled with “supports” and then the product will be good to go forever.
But just as that doesn’t really reflect how software is built, it doesn’t really reflect how accessibility works either. Accessibility is really better understood an ongoing process, a set of practices that ensures the best possible outcomes for users. No sufficiently-complex product is ever going to be 100% accessible to 100% of users. But the organization building the product should have safeguards in place to assess risk, minimize the issues that make it into production, and fix barriers as they’re identified.
Vendors should provide accessibility documentation that speaks to those practices. You want enough information to be confident that the product was built with accessibility in mind, and that the organization is built to fix issues quickly. You want documentation that gives you confidence in the product both when it’s purchased and after you’ve been using it for years.
Real-time ACRs
In my current role, I create a lot of GitHub issues describing accessibility barriers. As we’ve refined our internal processes, we’ve developed a complicated labeling system that tags each barrier with:
- The product and feature of the product where the barrier is located
- Any relevant WCAG success criteria
- Impacted interaction modalities (eg. keyboard-only, voice, screen magnification, screen reader, etc)
These labels are useful internally for tracking issues and finding examples of past solutions to help teams fix bugs faster. But a labeling scheme like this could generate something equivalent to an ACR that updates in real time, as issues are identified and fixed.
If a product is already open source in a public source control repository—or even in a closed repo, but with an API that could be used for reporting—it would be relatively straightforward to create an ACR-like dashboard displaying all of the known open issues. Filtering by feature could enhance the real-time ACR well beyond what text in a published document can do, to help explore how widespread the issues are and assess risk.
While I’ve seen plenty of projects with GitHub issues labeled for accessibility, I’m not currently aware of anyone that’s doing this kind of robust live reporting. But I would be very interested in seeing someone attempt it!
Public reporting and tracking of accessibility barriers
Many vendors provide a mechanism for reporting bugs. Some vendors provide a mechanism for reporting accessibility barriers in particular, sometimes with a promise that those issues will be given priority.
But if someone wants to take me up on that live ACR idea, let’s go the extra mile and tie the public accessibility bug reporting form directly into it. When I identify an accessibility issue and I fill out your reporting form, the very next step in my user path should be where my issue is in your real-time ACR. I should be able to track it as it’s triaged, fixed, and released.
Organizational maturity and culture reporting
Besides a report on the current status of the product, I would also be interested in a report on the vendor’s accessibility culture.
When I say accessibility culture, I mean: A detailed description of the accessibility practices of the teams building the product.
ACRs describe how testing was conducted and what tools were used. But if I’m buying something, I’m much more interested in the day-to-day product development practices. Do any of the QA engineers have their DHS Trusted Tester certification? Are accessibility specialists embedded on product teams? Are prototypes annotated for their accessibility features by designers? Are you adopting the practices described in Melanie Sumner’s Continuous Accessibility? Vendors should provide a clear picture of their approach to accessibility and their maturity as an organization.
The report should also include: A rubric for how accessibility defects are prioritized.
Not all accessibility barriers are as consequential as others, and any organization has to make choices about how to prioritize their work. A vendor with strong accessibility support for its products should be able describe how they make those choices, what kinds of issues get addressed immediately, and what kinds of issues land on the back-burner. And I should be able to decide if the vendor’s prioritization process is consistent with my own organizational needs and values.
Knowing something about whether the product is accessible right now is useful. But knowing something about how the vendor approaches accessibility can tell you whether it’s going to be accessible for the whole duration of your contract with the vendor. Let’s call this the Accessibility Culture Report—wait, no, that’s still just “ACR.” How about the Accessibility Practices Report? Someone out there is hopefully better at naming things than me.
The power of procurement
I think we all collectively assume that right now, anyone who cares about accessibility is mainly looking for an ACR. And often what we’re really talking about here is governments, universities, and selected major companies. ACRs satisfy compliance requirements, and they provide a snapshot of how accessible the product is—even if that snapshot is flawed. It’s something.
But what would happen if the people buying digital products started asking for more? How could that change the industry?
In this line of work, we spend a lot of time talking about building accessibility into the culture of our teams and our workplaces. We spend time pitching leadership on the moral case for accessibility, and when that fails we pitch the business case. We host lunch-and-learns and hold office hours post tips of the day in Slack and put a lot of emotional labor into making other people care enough to prioritize accessibility. Sometimes it sticks, sometimes it doesn’t.
(Important disclaimer: My current employer is pretty good for this! Come work for Ad Hoc!)
All of that effort is worthwhile. But the incentive structures of capitalism are pretty well understood at this point, and the most consistent driver of change is a clear financial benefit or cost.
Suppose, hypothetically, the United States federal government began to require real-time ACRs and some kind of reporting on the accessibility practices of its vendors. Suppose procurement decisions were made at least in part based on that information. Suppose vendors that don’t have mature and transparent accessibility practices started losing multi-million dollar contracts because of that.
I hypothesize that there would be significant ripple effects through the industry. The vendors that get those multi-million dollar contracts wield influence by example, and as they adapt I would bet others would follow their lead.
Or maybe not! I don’t think this would lead to an accessibility utopia, but I might be putting too much faith in tech company leadership to respond to this kind of procurement pressure.
Anyway, it’s fun to imagine—and it’s nice to get my negative feelings about VPATs off my chest. I’d love to see someone start asking for more than just another out-of-date, not-totally-accurate ACR, just to see what happens.