The methodology used in this survey is similar to that used in my previous five studies (1999-2003). The following remarks are not meant to be exhaustive; they are meant to highlight and clarify some important aspects:
Three sets of web pages were included in this survey: general campus pages, library pages and academic department/unit pages. The general campus and library set each consists of the respective home pages and pertinent pages directly linked to them. If a campus web site contained a page with links that, for the most part, connected to the various academic department homepages, that page was used as the starting point for evaluating academic department/unit pages; all pages directly linked to it were included in the academic department/unit set. If such a page could not be found, a page with links to academic departments was created and used for the purpose of this evaluation.
Text-only or non-frame versions were used in this survey only when they were accessible from the top of the home page.
As in the previous year, the focus was on the thirteen UW campuses that offer four-year programs are included in this survey. However, some data on the UW two-year campuses and its central institution ("UW-Colleges") were also collected.
I used downloaded, PC-run versions of Bobby, which is an accessibility validator originally created by the Center for Applied Special Technology (CAST) and further developed by Watchfire. Bobby was developed to assist people in checking the accessibility of their web pages. For each page checked, Bobby provides information pertaining to the type, number, and location of accessibility errors--both minor and major ones. Bobby also issues a summary report for each set of web pages.
Only some accessibility problems can be detected by Bobby's automated checking function, and it is only information pertaining to these problems that are reflected in the collected data. No systematic manual checks were performed.
Each set of pages was evaluated with two different Bobby versions: Bobby 3.1.1, a once free and now no longer available version developed by CAST, and Bobby 4.01 purchased from Watchfire. Unfortunately, the latest Bobby version, Bobby 5, has two major drawbacks that excluded its use for this study (see my comments at the Bobby discussion forum)
Bobby 3.1.1 was used for checking compliance with the Priority 1 component within the W3C/WAI-issued Web Content Accessibility Guidelines (WCAG) . For Bobby 3.1.1 findings, the term "Bobby-approved" refers to those Web pages that are free of major ("Priority 1") errors. Even though Bobby 3.1.1 is an older version, its continued use as an evaluation tool is warranted for three reasons:
First, Bobby 3.1.1 permits a direct comparison of the current data with those collected in previous years (which were also collected with this Bobby version).
Second, only the summary report provided by Bobby 3.1.1 includes a total count of the instances of detected "Priority 1" errors within a given set of pages. This figure, which can be used to compute the "average number of errors per page" (which, next to the "percentage of Bobby-approved pages," is an important indicator of web accessibility) is not provided by later versions of Bobby. Watchfire's Bobby 4.01 version, for example, only reports the number of error types (not the total number of each and every error instance) found in a given set of web pages.
Third, a preliminary test run conducted in December 2001 in connection with a similar research project involving campuses nation-wide, revealed only minor differences between the various Bobby versions then available to me: Bobby 3.1.1, Bobby 3.2, and Bobby WW, the version most similar to Bobby 4.01. Among the 12 web sites (a total of 288 pages), which were evaluated with each of these three Bobby versions, the closest similarity was found to be between the Bobby 3.1.1 and the Bobby WW evaluation results. While the correlations for the numbers of errors detected per site between Bobby 3.1.1 and Bobby WW, Bobby 3.1.1 and Bobby 3.2, and Bobby 3.2 and Bobby WW, were all close to one (the Pearson product-moment coefficient was .9999 for all three pairs), the total numbers of errors detected (1105 by Bobby 3.1.1, 1098 by Bobby 3.2 and 1106 by Bobby WW) showed more similarity between the first pair.
My previous studies focused exclusively on compliance with the Priority-1 components within Web Content Accessibility Guidelines (WCAG) developed by the W3C/WAI. With more campuses within the UW systems incorporating Section 508 Standards into their Web accessibility policies, data reflecting compliance with these standards are badly needed. For this reason, all sets of web pages were also evaluated with Bobby 4.01, which can be used to check for compliance with both WCAG and Section 508 Standards. The data collected and reported on this site provide information about compliance with Section 508, WCAG (level A), WCAG (level AA) and WCAG (Level AAA), each broken down by data set (library, campus, departments) and by individual campus.
In this study, the term "Bobby-approved" is used in a rather lax manner. It only refers to those pages which passed the automated Bobby check. (No systematic manual checks were performed.) Please be also aware that "Bobby-approved" is always relative to the standard or guidelines against which Bobby checks. In connection with Bobby 3.1.1, as it is used in this study, Bobby-approved refers to error-free "Priority-1" components. In connection with Bobby 4.01, talk about Bobby-approval" makes only sense if the standard or level of compliance is specified (e.g., "Section 508 Bobby-approved," WCAG-Level A Bobby-approved," etc.)
In addition to falsely positive results, Bobby, on occasion, also produces falsely negative results (reported errors where none exist), as I found out during my earlier studies. For example, pages that, at the very beginning, provide a "text-only version" link may not get Bobby's approval. Bobby simply checks the graphics versions for violation of accessible design principles. If it discovers a violation, Bobby considers this page to be inaccessible—regardless of how perfectly accessible the text-only version may be.
Another problematic feature of Bobby is its inability to distinguish between degrees of impact between different manifestations of the same error. For example, a bullet icon without an ALT tag (containing alternative text) registers as equal in status (i.e., as being a “Priority 1” error) to that of an image (also without an ALT tag) that is packed with crucial information. Similarly, Bobby may classify different types of accessibility errors as equal in severity even if the barriers they constitute differ to a significant degree. For example, the lack of alternative text associated with a purely decorative image registers as an error equal in need of correction to the lack of frame labels in multi-frame pages.
For reasons unknown to me, Bobby 3.1.1 and Bobby 4.01 often do not visit the same number of pages when the settings (URL; domain, folder, all; and link level) are identical. Bobby 4.01 tends to visit less sites. In a few cases, the differences is quite extreme.
Bobby 4.01 seems to report errors when images are associated with null ALT tags (ALT=""). Interestingly, the free online version (with the v. 4.0.1 core) does not report errors in these cases. When spot-checking revealed Bobby 4.01 to record errors (missing ALT tags) in cases where null tags were used appropriately, I adjusted my data so that they would not reflect these falsely reported errors.
The latest version of Bobby, Bobby 5 (mine happens to be version 5.10), was not used in this study. Bobby 5 does no longer provide summary data for whole sets of web pages; nor does its settings include the option to select a specific "link level" within the folder or domain of the starting page. (Other options are provided, but none of these instructs Bobby 5 to visit the same sets of pages that I have been evaluation for the previous five years. Without relative stability in the set composition, the data would make a year-to-year comparison less meaningful.)
Despite its shortcomings, Bobby 3.1.1 and 4.0.1 are good evaluation tools in studies like this, where the accessibility of thousands of individual web pages are evaluated and a rough measure of accessibility suffices. In fact, the vast majority of web page accessibility studies known to this author rely exclusively on Bobby’s automatically generated data.
Back to the top of this page
Back to the page entitled: Accessibility data generated with Bobby
Back to the Contents/Overview page
Researched and created by Axel
Schmetzke, Library, University of Wisconsin-Stevens Point.
Last updated 04/12/04 .
Comments are welcome! firstname.lastname@example.org