Methodology: Accessibility data generated with Bobby

The methodology used in this survey is similar to that used in my previous studies (1999, 2000 and 2001). The following remarks are not meant to be exhaustive; they are meant to highlight and clarify some important aspects:

1. Scope of investigation:

Three sets of web pages were included in this survey: general campus pages,  library pages and academic department/unit pages. The general campus and library set each consists of the respective home pages and pertinent pages directly linked to them. If a campus web site contained a page with links that, for the most part, connected to the various academic department homepages, that page was used as the starting point for evaluating academic department/unit pages; all pages directly linked to it were included in the academic department/unit set.  If such a page could not be found, a page with links to academic departments was created and used for the purpose of this evaluation.

Text-only or non-frame versions were used in this survey only when they were accessible from the top of the home page.

As in the previous year, the focus was on the thirteen UW campuses that offer four-year programs are included in this survey. However, some data on the UW two-year campuses and its central institution ("UW-Colleges") were also collected. 

2. Evaluation tools

I used downloaded, PC-run versions of Bobby, which is an accessibility validator originally created by the Center for Applied Special Technology (CAST) and further developed by Watchfire. Bobby was developed to assist people in checking the accessibility of their web pages. For each page checked, Bobby provides information pertaining to the type, number, and location of accessibility errors--both minor and major ones. Bobby also issues a summary report for each set of web pages.

Only some accessibility problems can be detected by Bobby's automated checking function, and it is only information pertaining to these problems that are reflected in the collected data. No systematic manual checks were performed.

Each set of pages was evaluated with two different Bobby versions: Bobby 3.1.1, a once free and now no longer available version developed by CAST, and Bobby 4.01 purchased from Watchfire.

Bobby 3.1.1

Bobby 3.1.1 was used for checking compliance with the Priority 1 component within the W3C/WAI-issued Web Content Accessibility Guidelines (WCAG) . For Bobby 3.1.1 findings, the term "Bobby-approved" refers to those Web pages that are free of major ("Priority 1") errors. Even though Bobby 3.1.1 is an older version, its continued use as an evaluation tool is warranted for three reasons:

First, Bobby 3.1.1 permits a direct comparison of the current data with those collected in previous years (which were also collected with this Bobby version).

Second, only the summary report provided by Bobby 3.1.1 includes a total count of the instances of detected "Priority 1" errors within a given set of pages. This figure, which can be used to compute the "average number of errors per page" (which, next to the "percentage of Bobby-approved pages," is an important indicator of web accessibility) is not provided by later versions of Bobby. Watchfire's Bobby 4.01 version, for example, only reports the number of error types (not the total number of each and every error instance) found in a given set of web pages.

Third,  a preliminary test run conducted in December 2001 in connection with a similar research project involving campuses nation-wide, revealed only minor differences between the various Bobby versions then available to me: Bobby 3.1.1, Bobby 3.2, and Bobby WW, the version most similar to Bobby 4.01. Among the 12 web sites (a total of 288 pages), which were evaluated with each of these three Bobby versions, the closest similarity was found to be between the Bobby 3.1.1 and the Bobby WW evaluation results. While the correlations for the numbers of errors detected per site between Bobby 3.1.1 and Bobby WW, Bobby 3.1.1 and Bobby 3.2, and Bobby 3.2 and Bobby WW, were all close to one (the Pearson product-moment coefficient was .9999 for all three pairs), the total numbers of errors detected (1105 by Bobby 3.1.1, 1098 by Bobby 3.2 and 1106 by Bobby WW) showed more similarity between the first pair.

Bobby 4.01 (Watchfire)

My previous studies focused exclusively on compliance with the Priority-1 components within Web Content Accessibility Guidelines (WCAG) developed by the W3C/WAI. With more campuses within the UW systems incorporating Section 508 Standards  into their Web accessibility policies, data reflecting compliance with these standards are badly needed.  For this reason, all sets of web pages were also evaluated with Bobby 4.01, which can be used to check for compliance with both WCAG and Section 508 Standards. The data collected and reported on this site provide information about compliance with Section 508, WCAG (level A), WCAG (level AA) and WCAG (Level AAA), each broken down by data set (library, campus, departments) and by individual campus.


In this study, the term "Bobby-approved" is used in a rather lax manner.  It only refers to those pages which passed the automated Bobby check.  (No systematic manual checks were performed.) Please be also aware that "Bobby-approved" is always relative to the standard or guidelines against which Bobby checks. In connection with Bobby 3.1.1, as it is used in this study, Bobby-approved refers to error-free "Priority-1" components. In connection with Bobby 4.01, talk about Bobby-approval" makes only sense if the standard or level of compliance is specified (e.g., "Section 508 Bobby-approved," WCAG-Level A Bobby-approved," etc.)

3. Limitations associated with Bobby as evaluation tool

As Bobby's creators freely admit, their product is not a perfect tool. While Bobby checks for compliance with the W3C-WAI’s W3C/WAI Web Content Accessibility Guidelines and Techniques for html documents, it automatically checks for compliance with only a subset of these. For the features not included in its automatic test, Bobby prompts the user to perform a “manual” check. Bobby is also unable to check for the accessibility of script (such as Javascript) or script-generated content. Some features can only be partially checked with Bobby. When encountering images, for example, Bobby will not report an error as long as some alternative text is provided—no matter how meaningless or non-descriptive this text may be. Thus, for various reasons, reliance on Bobby’s automatic checking facility alone is prone to produce some falsely positive (error-free) findings.

In addition to falsely positive results, Bobby, on occasion, also produces falsely negative results (reported errors where none exist), as I found out during my earlier studies. For example, pages that, at the very beginning, provide a "text-only version" link may not get Bobby's approval. Bobby simply checks the graphics versions for violation of accessible design principles. If it discovers a violation, Bobby considers this page to be inaccessible—regardless of how perfectly accessible the text-only version may be.

Another problematic feature of Bobby is its inability to distinguish between degrees of impact between different manifestations of the same error. For example, a bullet icon without an ALT tag (containing alternative text) registers as equal in status (i.e., as being a “Priority 1” error) to that of an image (also without an ALT tag) that is packed with crucial information. Similarly, Bobby may classify different types of accessibility errors as equal in severity even if the barriers they constitute differ to a significant degree. For example, the lack of alternative text associated with a purely decorative image registers as an error equal in need of correction to the lack of frame labels in multi-frame pages.

Despite its shortcomings, Bobby is a good evaluation tool in studies like this, where the accessibility of thousands of individual web pages are evaluated and a rough measure of accessibility suffices. In fact, the vast majority of web page accessibility studies known to this author rely exclusively on Bobby’s automatically generated data. 

Back to the top of this page

Back to the page entitled: Accessibility data generated with Bobby

Back to the Contents/Overview page

Researched and created by Axel Schmetzke, Library, University of Wisconsin-Stevens Point.
Last updated 03/30/03 .
Comments are welcome!

Valid HTML 4.0! Bobby 3.0 Approved Valid CSS!