Federal Website Assessment

OK, so this is not the first time I have assessed a website. This is actually a topic I have taught to undergrads. And I completed a usability test of a non-federal but still government website last year in Chuck’s Evaluation class (with Aaron). So, some of this week’s reading are familiar territory. But, I have never really thought about web assessment from an information policy perspective.

I know what works and what doesn’t when it comes to web design, based on usability readings and my own background in printing and newspaper layout (bet y’all didn’t know that). But knowing what works and what doesn’t has very little to do with actual implementation. I am forever arguing with my undergrads about the difference between making a website for yourself, where you can use any design you like, and making a website at the behest of a client. With a client, you HAVE to do what THEY want. All you can do is offer suggestions, provide options, and try to make the strongest possible case for a website that follows usability and accessibility guidelines (i.e. the U.S. Department of Health and Human Services usability guide on Usability.gov and Jakob Nielsen’s recommendations on UseIt.com).

[Side note on functionality: No matter what you have to give in to regarding usability, accessibility and aesthetics, all features of any system should always FUNCTION. For goodness sake, what’s the point of non-functioning features?]

In class, my students usually choose a for-profit business or a FSU department for their web redesign project. In the fall, though, one group chose the FAFSA website, which I agree is pretty awful. [FAFSA is the Free Application for Federal Student Aid in case any of you are blessed not to need student aid and don’t know about the FAFSA.] Anyway, the group made changes to the website that would put it in violation of federal laws. For example, they didn’t know they have to include a privacy statement and a link for FOIA requests.

When you are designing a web site for a Federal agency or department, the first step should be identification of all information policies and regulations that relate to web site assessment (McClure, Sprehe, & Eschenfelder, 2000). At the very least, start with the Office of Management and Budget (OMB) Policies for Federal Websites (2004). Obviously, not all Federal web sites are designed this way since so many violate the Section 508 accessibility regulations (West, 2008), let alone usability issues.

But failure to follow existing guidelines is not the only information policy issue related to web assessment. The scope, implementation, and enforcement of existing policies are also critical (Eschenfelder, Beachboard, McClure, & Wyman, 1997).

Yes, there is Section 508, but is it current? Is it relevant with current assistive technologies? I am guessing no, but someone needs to actually verify my guess.

Also, we know the potential value of E-Government to open government, cost savings, etc. (Freed, 2009), but do we know exactly how much return on investment (ROI) the Federal government is getting by utilizing electronic technologies? And do we know how much ROI they COULD be getting, e.g. are they getting less ROI than they possibly could get? Nielsen explains how much ROI governments could get by increasing usability, something that suggests that E-Government services are not getting the overall ROI they could get. Nielsen says that on average, websites increase desired metrics by 135% following a usability redesign (2003), although this figure has declined some (to 83% in 2008). He also explains the ROI that government agencies get from usability, especially when they engage in e-commerce (2007), something Freed tells us they do fairly well (2009).

Meanwhile, we know from the American Customer Satisfaction Index that Americans seem satisfied with E-Government services (Freed, 2009), but the satisfaction level of 74.1 out of 100. That’s a C. Hardly what I’d call satisfaction, more like complacency. There is a BIG difference between satisfaction and success. Aaron and I found that out last year when we did a usability test on MyFlorida.com. The five test participants said they were satisfied with certain aspects of MyFlorida.com that they did not use successfully. For example, there were tasks that the testers “completed” because they found an answer, but it was the WRONG answer. Yet, they were SATISFIED. This is satisficing, and that is a far cry from success.

And there’s another issue with Americans’ satisfaction with E-Government – the comparison between that and traditional government service. Apparently, satisfaction with E-Government is higher than satisfaction with traditional government. But who is reporting their satisfaction? You know that voluntary surveys are full of methodological holes so how reliable are the ACSI data (Grob’s argument for good enough evaluations notwithstanding)? Consider the fact that last year they reported a three-year low for E-Government satisfaction but not it’s an all-time high (Freed, 2008, 2009). How weird does that seem? And there’s the possibility that Americans have higher expectations of traditional government than E-Government so even equal service quality could get a higher satisfaction rating for E-Government than traditional government. Going back to the usability test Aaron and I did last year, most of our testers indicated they started with REALLY LOW expectations of MyFlorida.com. So even though they were “satisfied” with the services, that’s based on a low starting point!

I obviously have a lot to say on this topic. Good thing I am presenting this week!

References

Eschenfelder, K. R., Beachboard, J. C., McClure, C. R., & Wyman, S. K. (1997). Assessing U.S. federal government websites. Government Information Quarterly, 14(2), 173-189. Retrieved March 29, 2008, from Science Direct database.

Freed, L. (2009). American customer satisfaction index: E-Government satisfaction index. Ann Arbor, MI: ForSee Results. Retrieved March 12, 2009, from http://www.theacsi.org/images/stories/images/govsatscores/0109q4.pdf

Freed, L. (2008). American customer satisfaction index e-government satisfaction index [Electronic resource]. Available from http://www.theacsi.org/index.php?option=com_content&task=view&id=178&Itemid=181

Grob, G. F. (2003). A truly useful bat is one found in the hands of a slugger. American Journal of Evaluation, 24(4), 499-505. Retrieved March 12, 2009, from Science Direct database.

McClure, C. R., Sprehe, J. T., & Eschenfelder, K. (2000). Performance measures for Federal agency websites: Final report. Tallahassee, FL: Florida State University, School of Information Studies, Information Use Management & Policy Institute.

Nielsen, J. (2008, January 22). Usability ROI declining, but still strong. Alertbox [Electronic resource]. Retrieved March 29, 2008, from http://www.useit.com/alertbox/roi.html

Nielsen, J. (2007, February 12). Do government agencies and non-profits get ROI from usability? Alertbox [Electronic resource]. Retrieved March 29, 2008, from http://www.useit.com/alertbox/government-nonprofit.html

Nielsen, J. (2003, August 25). Usability 101: introduction to usability. Alertbox [Electronic resource]. Retrieved March 29, 2008, from http://www.useit.com/alertbox/20030825.html

Nielsen, J. (2003, January 7). Return on investment for usability. Alertbox [Electronic resource]. Retrieved March 29, 2008, from http://www.useit.com/alertbox/roi-first-study.html

Office of Management and Budget. (2004). Memorandum for the heads of executive departments and agencies: Policies for Federal agency public websites. Washington, DC: Office of Management and Budget. Retrieved March 12, 2009, from http://www.whitehouse.gov/omb/memoranda/fy2005/m05-04.pdf

U.S. Department of Health and Human Services. (2006). Research-based web design and usability guidelines [Electronic version]. Washington, DC: U.S. Department of Health and Human Services. Retrieved March 12, 2009, from http://www.usability.gov/pdfs/guidelines.html#1

U.S. Department of Health and Human Services. (n.d.). Usability.gov [Electronic resource]. Washington, DC: U.S. Department of Health and Human Services. Retrieved March 12, 2009, from http://www.usability.gov

West, D. M. (2008). State and federal electronic government in the United States, 2008. Washington, DC: Brookings Institution.

Accessibility

The topic of accessibility is extremely important – not just when talking about website evaluation. It is important for ensuring equal access to information for all persons, regardless of their physical and cognitive abilities, and especially when you see the numbers – 54 million Americans have disabilities (Jaeger, 2006). Also, note Jaeger’s reminder that this number will only grow as the baby boomers age.

John presented on accessibility last year in Evaluation class (I believe that website accessibility is his preferred research stream), and I remember that he said this about accessibility – the area has relatively clear standards, but little evaluation is being done. Thank goodness for Paul Jaeger and (not brown-nosing here) the Information Institute, or probably even less would be done!

Most of you probably know that 1. Chuck loves multi-method iterative evaluation and 2. the Institute conducted an evaluation of the Florida Electronic Library, so you can probably guess that #2 involved #1 (Bertot, Snead, Jaeger, & McClure, 2006). That study combined functionality, usability and accessibility testing, getting back to a point Chuck highlighted in class when I discovered the problem with the button location on (I think) USASpending.gov. I had enlarged the text on the webpage but then couldn’t click the “Go” button because the link hadn’t shifted with the text. Chuck asked if that was a usability, functionality, or accessibility issue. Now that I have been properly indoctrinated, I of course knew the answer – D. All of the above. So not only is accessibility important in its own right, it’s important to overall usability and functionality as well.

I do find it interesting that Section 508 has exemptions since I don’t really see why any federal electronic and information technology system should NOT provide equal access for all persons, regardless of ability (IT Accessibility & Workforce Division, n.d.a, n.d.b). And I was especially interested to see what the exemptions are – systems used for military command, weaponry, intelligence, and cryptology, also back office equipment used by service personnel for maintenance, repair, etc. Um, the military and intelligence departments can discriminate against differently-abled persons, and that’s OK? And repair men (and women, I suppose), cannot be people with any disabilities? What? Congress didn’t see the problems with this? Meanwhile, there’s the “undue burden” exemption, yet another example of vague language. Section 508 actually includes a definition of undue burden [I was really excited], but it means “significant difficulty or expense” with NO definition of the ever-popular significant [suddenly I was less excited] (IT Accessibility & Workforce Division, n.d.a, n.d.b).

But, it’s great that we even have Section 508. That’s a huge step forward. Now, if only it were implemented, which Jaeger reminds us is not happening in any standardized, government-wide way (2006). Jaeger’s policy analysis revealed that Section 508 requirements and guidelines, if correctly implemented, should produce websites accessible to most or all disabled persons. So why aren’t they? One possibility that agencies don’t all give accessibility the same level of importance. Another is lack of funding. And education. So, what can we do about this? There needs to be stronger enforcement of Section 508, and that probably requires some punitive impact on non-compliant agencies since we know the carrot approach to motivation does not work well with civil servants.

References

Bertot, J. C., Snead, J. T., Jaeger, P. T., & McClure, C. R. (2006). Functionality, usability, and accessibility: iterative user-centered evaluation strategies for digital libraries. Performance Measurement and Metrics, 7(1), 17-28. Retrieved March 14, 2008, from Emerald Insight database.

IT Accessibility & Workforce Division, Office of Governmentwide Policy, U.S. General Services Administration. (n.d.a). Section 508 standards [Electronic resource]. Washington, D.C.: U.S. General Services Administration, Office of Governmentwide Policy, IT Accessibility & Workforce Division. Retrieved February 27, 2009, http://www.section508.gov/index.cfm?FuseAction=Content&ID=12

IT Accessibility & Workforce Division, Office of Governmentwide Policy, U.S. General Services Administration. (n.d.b). Summary of Section 508 standards [Electronic resource]. Washington, D.C.: U.S. General Services Administration, Office of Governmentwide Policy, IT Accessibility & Workforce Division. Retrieved February 27, 2009, http://www.section508.gov/index.cfm?FuseAction=Content&ID=11

Jaeger, P. T. (2006). Assessing Section 508 compliance on federal e-government Web sites: a multi-method, user-centered evaluation of accessibility for persons with disabilities. Government Information Quarterly, 23(2), 169-190. Retrieved February 27, 2009, from WilsonWeb database.