Methodology: Accessibility data generated with Bobby

The methodology used in this survey is essentially identical to that used in my previous web accessibility studies. The following remarks are not meant to be exhaustive; they are intended to highlight and clarify some important aspects. 

1. Scope of investigation:

For each of the 56 campuses, I collected accessibility data on the top-layer web pages for the general campus, the library and the library school (SLIS). The top-layer pages included the homepage and all the pages within the same domain that were directly linked to it.

Text-only or non-frame versions were used in this survey only when they were accessible from the top of the home page. 

The universities that were in included in this study do not constitute a random sample. Since I was particularly curious about the accessibility of SLIS pages, I selected the campuses that offer ALA-accredited programs in library and information science. The data are presented in three sets: one set involving the 24 campuses already included in my 2000 study (those that have the highest ranked LIS programs, according to U.S. News & World Report); a second set involving all other 25 U.S. campuses (incl. the Universidad de Puerto Rico); and a third set with 7 Canadian campuses.

While the validity of U.S. News & World Report rankings is highly questionable, one may reasonably assume that these rankings shape the perceptions of prospective library science students and influence their decisions. Many students will enroll in these highly-ranked LIS schools with the expectation that these are the leaders in the field.

2. Evaluation tool

I used the downloadable version of Bobby (3.1.1), which is an accessibility validator created by the Center for Applied Special Technology. Bobby was developed to help people to check the accessibility of their web pages. For each page checked, Bobby provides information pertaining to the type, number, and location of accessibility errors--both minor and major ones. Bobby also issues a summary report for each set of web pages. Web pages that contain any major ("priority 1") error do not receive Bobby's approval.

Bobby checks for compliance with the W3C/WAI Web Content Accessibility Guidelines (or, if selected, with the Access Board standards under Section 508). The developers of Bobby  emphasize that passing the automated Bobby check is not sufficient proof for a web page's accessibility. To get full "Bobby approval," a manual check involving human judgment is also required.

When preparing for this study, much thought was given as to which version of Bobby to use. I had used Bobby 3.1.1 for my previous investigation of library and library school web sites. Since then, three other versions—Bobby 3.2, Bobby 3.3 and Bobby WW—had been developed. Since the difference between Bobby 3.3 and Bobby WW consisted mainly of the addition of US Government Section 508 compliance checking, which is not utilized in this study, the author faced a choice between Bobby 3.1.1, Bobby 3.2 and Bobby 3.3/Bobby WW. The final choice fell on Bobby 3.1.1— for the following reasons: First, a preliminary test run, in which 12 web sites (a total of 288 pages) were evaluated with each of the three Bobby versions, revealed the closest similarity between the Bobby 3.1.1 and the Bobby WW evaluation results. While the correlations for the numbers of errors detected per site between Bobby 3.1.1 and Bobby WW, Bobby 3.1.1 and Bobby 3.2, and Bobby 3.2 and Bobby WW, were all close to one (the Pearson product-moment coefficient was .9999 for all three pairs), the total numbers of errors detected (1105 by Bobby 3.1.1, 1098 by Bobby 3.2 and 1106 by Bobby WW) showed more similarity between the first pair. Using either Bobby 3.1.1 or Bobby WW thus seemed to promise slightly more consistency. Second, for unknown reasons, Bobby WW resulted in frequent crashes of my computer (running Windows ME). Third, unlike Bobby 3.2 and Bobby WW, Bobby 3.1.1, provides a total count of the instances of Priority-1 detected errors, broken down by error types, in the summary report. The summary report in the two later Bobby versions does not provide this information. Collecting it with these versions is excruciatingly time-consuming; it would involve looking at the individual page reports and then adding the figures provided therein.

3. Limitations of this study

In this study, the term "Bobby-approved" is used in a rather lax manner.  It refers to those pages which passed the automated Bobby check.  No systematic manual checks were performed. Similarly, the error data presented on this site refer exclusively to those errors that were detected during the automated Bobby check.

The exclusive use of Bobby’s automated function for evaluative purposes is also problematic for other reasons. Bobby is unable to check for the accessibility of script (such as Javascript) or script-generated content. Some features can only be partially checked with Bobby. When encountering images, for example, Bobby will not report an error as long as some alternative text is provided—no matter how meaningless or non-descriptive this text may be. Thus, for various reasons, reliance on Bobby’s automatic checking facility alone is prone to produce some falsely positive (error-free) findings.

In addition to falsely positive results, Bobby, on occasion, also produces falsely negative results (reported errors where none exist), as this author found out during earlier studies. For example, pages that, at the very beginning, provide a "text-only version" link may not pass Bobby's muster. Bobby simply checks the graphics versions for violation of accessible design principles. If it discovers a violation, Bobby considers this page to be inaccessible—regardless of how perfectly accessible the text-only version may be. In order to eliminate this particular type of false results, each homepage was checked for a “text-only” version. Where a link to a text-only version was provided close to the top of the graphical homepage version, the text-only homepage was used as the starting point for the Bobby test. In cases where the text-only homepage contained a link back to the graphical homepage, the test results reported by Bobby were adjusted so that they would not reflect the latter’s accessibility errors.

Another problematic feature of Bobby is its inability to distinguish between degrees of impact between different manifestations of the same error. For example, a bullet icon without alternative text registers as equal in status (i.e., as being a “priority 1” error) to that of an image (also without the ALT tag) that is packed with crucial information. Similarly, Bobby may classify different types of accessibility errors as equal in severity even if the barriers they constitute differ to a significant degree. For example, the lack of alternative text associated with a purely decorative image registers as an error equal in need of correction to the lack of frame labels in multi-frame pages.

Despite its shortcomings, Bobby is a good evaluation tool in studies like this, where the accessibility of  thousands of individual web pages needs to be evaluated and a rough measure of accessibility suffices. In fact, with very few exceptions, all web site accessibility studies known to this author relied exclusively on Bobby’s automatically generated data.

Back to the top of this page

Back to the Contents/Overview page

Researched and created by Axel Schmetzke, Library, University of Wisconsin-Stevens Point.
Last updated 03/13/02 .
Comments are welcome!

Valid HTML 4.0! Bobby 3.0 Approved Valid CSS!