Methodology: Accessibility data generated with Bobby

With some minor variations, the methodology used in this survey is essentially identical to that used in my previous web accessibility studies. The following remarks are not meant to be exhaustive; they are meant to highlight and clarify some important aspects. 

1. Scope of investigation:

Two sets of sites are included: (a) sites of distance-education providers as listed in Marcie Kisner Thorson's book "Campus-free College Degrees," a "guide to accredited college degrees through distance learning." (Tulsa, Oklahoma: Thorson Guides, 2000), and (b) the sites of twelve North American organizations/associations concerned with distance learning. For the sites included in both sets, I collected accessibility data on the homepages as well as the next layer of hyperlinked pages.

With online learning programs mushrooming all over the nation, Thorson's guide is not exhaustive. Also, the URLs I used for this study are not necessarily identical to those provided in Thorson's guide, which were often either invalid or linking to the larger institution. Every effort was made to select the URLs of the relevant distance-learning units. 

2. Evaluation tool

For the sites listed by Thorson, I used the downloadable version of Bobby 3.1.1., which, unlike the most recent release, Bobby 3.2,  includes summaries of error-type data.  For the twelve distance-education organizations, I used the downloadable version of Bobby release (3.2).  Several trial runs revealed that both program versions provide almost identical result as far as "priority 1" errors are concerned.

Bobby is an accessibility validator created by the Center for Applied Special Technology. Bobby was developed to help people check the accessibility of their web pages. For each page checked, Bobby provides information pertaining to the type, number, and location of accessibility errors--both minor and major ones. Bobby also issues a summary report for each set of web pages. Web pages that contain any major ("priority 1") error do not receive Bobby's approval. 

Bobby checks for compliance with the W3C/WAI Web Content Accessibility Guidelines. The developers of Bobby emphasize that passing the automated Bobby check is not sufficient proof for a web page's accessibility. To get full "Bobby approval," a manual check involving human judgment is also required.

3. Limitations of this study

In this study, the term "Bobby-approved" is used in a rather lax manner.  It refers to those pages which passed the automated Bobby check.  No systematic manual checks were performed. Similarly, the error data presented on this site refer exclusively to those priority-1 type errors that were detected during the automated Bobby check.

Bobby's automated checks provide only a rough measure of web page accessibility. As indicated above, the automated Bobby checker will miss certain access problems that, at this point, only  humans can detect. Since Bobby checks for compliance with the W3C accessibility standards, it may determine a page to be inaccessible even though its authors have employed creative, nonstandard ways to ensure universal access.

One particular shortcoming of Bobby is its inability to distinguish between degrees of impact between different manifestations of the same error.  For example, a bullet icon without alternative text registers as equal in status (i.e., as a "level 1" error) to that of an image (also without the ALT tag) that is packed with crucial information. Furthermore, Bobby classifies certain types of accessibility errors as equally severe even though the barriers they constitute may differ to a significant degree. For example, the lack of alternative text associated with a purely decorative image and the lack of meaningful frame labels in multi-frame pages both register as "level 1" errors. 

Back to the top of this page

Back to the Contents/Overview page

Researched and created by Axel Schmetzke, Library, University of Wisconsin-Stevens Point.
Last updated 12/29/00 .
Comments are welcome!

Valid HTML 4.0! Bobby 3.0 Approved Valid CSS!