|
4.0 Measuring Success
4.1 Visitor Counts
4.2 Online Feedback
4.3 Usability Testing
4.4 Self Evaluation
Section 4.0 Measuring Success
In the discussion in Section 3.2 - Topics, some review and update schedules for various types of content were discussed. Along with the content review, it is often helpful to review the Web site as a whole and evaluate the project’s goals. Reviewing user patterns can help shift emphasis away from unused areas and focus content development in sections where viewers are helped the most. It can also point out sections that are difficult to find. User feedback is an invaluable evaluation tool, and most importantly, self-evaluation is a necessary part of any Web project. A self-evaluation form is presented in Appendix D.
4.1 Visitor Counts
One method for measuring the success of a Web site is to track the number of visitors that the site serves. Accurately calculating this number can be a difficult task; however, it can give some overall indications as to whether a Web site is having the desired impact. A number of different methods can be employed. The two most common are hit counters and Web server log files. In order to track visitors, Web site administrators need a high level of access to their Web server, and often a working knowledge of server side applications (see Appendix A.2 - Other Types of Internet Files: Perl, Java, CGI - "Server Side Scripting").
Hit Counters
Many Web sites use an online "Hit Counter" or similar graphical method for posting the number of visitors to a Web site. A hit counter uses a server side application to track each visit to a certain page. Many count the number of times a certain graphic is loaded - often an invisible graphic embedded in the page for just this purpose - while others track the number of times a file is opened. Most hit counters are set up to only log traffic to the first page (or home page) of a Web site, and don’t have a way to track visitors who enter through a bookmark to a page deeper into the Web site.
Hit counters by themselves do not provide a complete, or accurate picture of the number of visitors to a Web site. Although they are fairly easy to implement and provide almost instant feedback, best practice suggests avoiding them altogether.
Log Files
Most Web servers keep track of each page that is requested by a user. Commonly, the server logs the IP address of the request, which files were requested, the date of the request, and the status (i.e., a page returned or an error was generated, etc.). There are many applications available to analyze these log files and provide a statistical picture of the visitors to a site. Appendix E lists a typical log file from an Apache Web server. Often, a basic Web site statistics package is a part of the basic set of services provided by professional Web hosting companies.
Analyzing log files is an effective method for determining the success of the various sections of a Web site. Which sections users frequent can help determine how often to update content in a section and can help focus development. Sections that are not visited often may be hard to find, or are possibly sections users simply do not find helpful. Unused sections could be eliminated in order to focus maintenance on sections that are being used. Sections that are underused but deemed important could be highlighted on the front page, or on pages that see more frequent use.
On-line search applications should keep track of the keywords that are being used. It is important to review these keywords periodically. They can provide insight into the user experience, and can help determine why users are visiting in the first place and which areas of the Web site are harder to find.
Using visitor counts to evaluate a Web site, however, is not a perfect solution. Proxy servers, page caches, search engines that index the Web site can all interfere with accurate counting and analysis. Visitor counts should not be the sole evaluation tool, but they can illuminate visitor trends.
4.2 Online Feedback
Most Web sites provide contact information for user feedback, or attempt to survey visitors in an effort to solicit feedback. The practice is encouraged, but user feedback is as difficult to quantify as visitor counts. It can be difficult to entice a viewer to interrupt their online experience to fill out information about the site that they are visiting. Many times, users who are discouraged with their experience will use an online survey to vent their frustration. While such feedback is important, the other viewers who were happy with their experience are less likely to say so. Therefore, online surveys can help pinpoint problem spots, but may not be a good measure of overall experience.
Like visitor counts, data from user feedback is generally gathered by server side applications. While programming a user feedback form is a straightforward task, there are a number of subtle errors that can trip up inexperienced administrators. A programmer or consultant with a good working knowledge of server side applications should develop the feedback form. There are also a number of free, low-cost or off-the-shelf commercial applications that require less knowledge to install and operate than a customized solution.
What to Survey
Web site surveys are ideal for getting subjective answers from users. The overall satisfaction level of Web site could be the subject of a survey question. What motivated users to the site in the first place, and which sections of the site users find helpful or which features they like or dislike can also be good questions to ask. Keep in mind that a Web survey measures perceptions, not reality. A survey will only return an opinion about a user’s likes and dislikes, and the actual data (see Visitor Counts above) may be different than the user reports.
Questions should be answered with a yes/no button, a list of boxes to check, or a drop-down menu of options. Overall satisfaction could be represented with a scale from 1 to 5. Questions where users fill in the answers themselves often produce responses that are vague, cryptic or off-topic. Users are also more likely to ignore questions that require them to create their own answers.
What Not to Survey
Dr. Jakob Nielsen, Author of the web site www.useit.com advises:
Surveys don't work for getting feedback on new design ideas. Users can't comment on things they haven't seen, and their hypothetical answers to questions like "would you like a persistent navigation bar on every page with buttons for every section of the site?" are worthless. The way to get user feedback on new designs is to build a prototype and get a few test users in to try out the prototype. Only by observing how people actually use a new design can you learn whether it works.
Users can't predict their own future behavior, and it's common for people to say that they would love a certain feature only to discover that they never use it after it is launched.[1]
Designing an On-line Survey
Most users are not motivated to help evaluate a Web site. They tend to visit a Web site, scan it quickly for the information that they need and move on. Most users will ignore an online survey, especially if they feel as though it will interfere with their primary objective. The users who do respond are likely to give up if the survey is too long, too personal or too vague.
- Keep the survey to one page. Ideally, users should not have to scroll to see questions. If possible make the button to submit the data visible without scrolling. Users who can see how short the survey is are more likely to participate. Keep in mind that Web server logs (see Appendix E - Server Log Files) often capture data about users, and avoid asking for information that you can get from other sources. Surveys that are spread out over multiple pages are also likely to be abandoned.
- Inform the users how their responses are to be used. Privacy is a primary concern to most on-line users. Any page on a Web site that solicits information from a viewer should contain a link to the privacy policy (see Section 3.2.3 - Policies). It might also be appropriate to state the goals of the survey either on the form, or to a clearly marked link on the survey form. If users are satisfied that their answers are safe, they are more likely to participate.
- Keep the questions appropriate to the subject. When questions are too personal, users tend to give up or make up data. Subjects to avoid are income, gender, occupation and education level.
- Have a plan for the results. Know in advance what you will do with the data, and only ask users to participate in a survey if their responses will change your site design, philosophy or goals.
4.3 Usability Testing
No single designer or design team can anticipate every problem with a Web site interface, layout and navigation scheme. Users have the uncanny ability to defy logic, reason and careful planning with one click of the mouse. It is important to prototype a Web site design and test it with a representative group that is detached from the design process.
A usability test is usually conducted with a focus group of four to eight people. Five is an ideal number, and often more than five people will produce diminishing results. Users should be given a list of tasks to complete and should be observed during the testing. It is important to determine if the users are able to complete the tasks and if they are able to complete the tasks fast enough to meet their expectations. The choices that they make should be observed for deficiencies in design. It is also important to determine if the site performs fast enough to meet their satisfaction. Testing in ideal conditions and less-than-ideal conditions (i.e., a slow modem) might be necessary.
Iterative testing - testing a prototype, fixing the prototype and testing again - is the best practice to achieve good results. Usability testing with focus groups is not a trivial task and should only be part of an initial design or major redesign project.
4.4 Self Evaluation
In the end, the most effective - and perhaps cost efficient - method of Web site design evaluation may be a self-evaluation. One of the goals of this document was to provide such a tool. To conduct the research for these guidelines, an evaluation form was developed to rate a number of representative city and county public works and highway Web sites across Minnesota and the U.S. The form was refined to be used with these guidelines as a self-evaluation tool. This form is presented in Appendix D - Evaluation Form.
[1]Surveying Users(Internet), April 2000 (cited February 2002), http://www.zdnet.com/devhead/stories/articles/0,4413,2211547,00.html
(Content has changed since sited).
[2]See Test with 5 Users (Internet), http://www.useit.com/alertbox/20000319.html, April 19, 2000 (cited June 2002) for a mathematical test model.
|