Desktop Network Services
Hamilton Library Pilot Study, May 2001

Prepared by Beth Tillinghast

The Effects of Label Terminology and Scope Notes on a Library's Gateway Page for Accuracy and Efficiency of Accessing Information

More and more in the world of academic libraries, professionals are feeling the need to provide services to an invisible clientele. In this environment patrons rely greatly on the virtual gateway that is the library home page or gateway page. Unfortunately the interaction that must exist as remote users access the gateway page is not thoroughly understood, and the majority of design guidelines and standards seem to be based more on general ideas of what users might need than on empirical research: opinion and personal observation seem to prevail in this area. The literature suggests that as university libraries expand their services in the virtual environment, it is critical to clearly focus on the ability to communicate to diverse, remote users. This study looks at two ways in which an academic library can more clearly label links and therefore enhance that critical communication: through use of simplified terminology and use of descriptive scope notes. Participants completed a series of tasks designed to test navigation accuracy, efficiency, and ease of use on four versions of the gateway page for Hamilton Library at the University of Hawaii. Though no definitive data emerged, indications suggest a certain amount of revision would enhance communication for remote users.

Introduction

More and more in the world of academic libraries, professionals are feeling the need to provide services to an invisible clientele. As more users attempt to remotely use a library's resources via the Internet, Web developers need to be sensitive to the dependency of these patrons while serving their needs. In this environment patrons rely greatly on the virtual gateway that is the library home page or gateway page. This page is actually the shell to a depth of information sources found on subsequent library Web pages.

Unfortunately the interaction that must exist as remote users access the gateway page is not thoroughly understood, and the majority of design guidelines and standards seem to be based more on general ideas of what users might need than on empirical research: opinion and personal observation seem to prevail in this area.

D'Angelo presented a researched-based set of guidelines for design of Web pages (D Angelo, 1998). Though a number of guidelines were covered, three major categories used by Cocheonour, Lee, and Wilkins were highlighted (Cocheonour et al, 1995). One of the essential categories focused on navigational characteristics. The emphasis is on making sure that menus and sequences are understandable to users no matter how they enter the site.

A few researchers have focused their study of essentials in Web design for purposes of communication on particular elements. Spivey suggests that the trend in library home page development is to emphasize efficiency and simplicity and that some libraries have developed a minimalist aesthetic design. The home page, or gateway page, becomes a shell that links to a vast array of information: specific collections, services, personnel, policies, and links to other networks. He indicates that the brevity of some home pages places a semantic burden on the chosen vocabularies used on these gateway pages (Spivey, 2000).

To complicate matters, librarians have long been aware that the jargon we use as professionals can be confusing to our patrons. Naismith and Stein completed a study that measured students' comprehension of a selection of library terms. The results indicated that in a test situation almost half of the time subjects were not able to identify correct definitions for commonly-used library terms (Naismith, 1989). Barbara Dewey reports that the second major barrier to locating information services is terminology on a number of university library Web pages (Dewey, 1999). Labels may be obscure to users: "library outreach services" for "distance education" or "lending services" for "circulation". Another study focusing on the navigation of online menus found that for undergraduates to make efficient choices in a complex information environment, more descriptive text is necessary (Eliasen, 1997).

The literature suggests that as university libraries expand their services in the virtual environment, it is critical to clearly focus on the ability to communicate to diverse, remote users. Hamilton Library at the University of Hawaii at Manoa serves as a key information provider in its unique geographical environment. The library's mission statement reads that "Regardless of location, students and faculty expect convenient access to electronic resources such as online library catalogs, digital full-text literature, and image databanks," (University of Hawaii, 2001). It further states that the access to electronic resources will be designed to meet the needs of users and enhance their productivity. Digital information will be presented through an interface that is flexible, powerful, and easy to use. In order to fulfill this aspect of the mission statement and to provide access for researchers, staff, students and the university community, the gateway to this information must provide clear and informative access. Therefore the Desktop Network Services at the University of Hawaii at Manoa in Hamilton Library decided to look at the library's gateway page to see how usable it was and to see if it helped to address the mission of the library.

Methodology and Procedures

Needs Assessment

Initially a needs assessment was conducted with the intention of learning if the functional needs of the users were being met through the gateway page. The instrument used was a survey that focused on the gateway page used by Hamilton Library. (See Appendix A.) It evaluated the following criteria: content, structure and navigation, visual design, functionality, and interactivity. The survey was administered to second semester freshman enrolled in a Library & Information Science class. Though the students had been briefly introduced to the library's gateway page, they had not worked with it extensively. The results of the needs assessment indicated that there was some confusion in accessing information. The most commonly reported source of confusion revolved around the terminology used for the links and the lack of scope note information for those links. The DNS team therefore decided to focus on terminology and scope note information in a quantitative pilot study.

Participants

Twenty-eight undergraduate students participated in the experiment. No incentive was offered as they were enrolled in an LIS 100 or LIS 102 course. These courses are offered at the University of Hawaii as part of a writing-intensive program. They were all second semester freshman. Eight students enrolled in LIS 102 were Special Studies students, a pre-honor's track. Through the course all of the students had been exposed to the library gateway pages and some of the resources they could access via that page. The amount of experience with computers ranged from rarely to daily, though the majority reported using computers on a weekly basis.

Materials

For the pilot study paper prototypes were created that simulated the computer screens for the library's gateway page. Paper screen shots were also made for those key pages that were one link deep within the hierarchy of Web pages. It was decided to use paper prototypes in order to eliminate the interference and inconsistency that might occur due to differing computer systems. These inconsistencies could affect the rate of transfer of information and alter the efficiency score.

Students were given an introductory sheet, which requested demographic information. They were also given ten separate packets. Each of the packets included a cover sheet. This sheet listed the task, a rating system for rating the ease or difficulty for the task performance, and a space for an accounted rating. Attached to the task cover page were the prototype pages representing a library gateway page and those pages that were linked to it at the next level.

Tasks

Students were required to complete a total of ten tasks. These tasks represented real search situations as can be seen in Table #1.

Table 1. List of Pilot Study Tasks

The same researcher gave the subjects the instructions and examples for each test situation. This helped to assure consistency during the introduction. Students were told that they needed to use the main part of the gateway page (which included the link to Hawaii Voyager Library Catalog, Databases and Indexes, Electronic Journals, Newspapers and E-books, Reference Tools, Digital Archives Collection, and Learning the Library) and the subsequent page only in order to navigate to the place on the page where they thought they could find the information required by the task. It was stressed that the goal was not to actually find the information but rather to find the place on the page where they could most likely access the information in order to complete the task if using the library Web pages. The students were asked to indicate their thought processes while completing each task by numbering the steps they took as they sought an access point to complete the task. Starting with the gateway page, they were asked to number the first link they thought could be used to navigate to the next page where they might find access to the information. On the subsequent page they were then to indicate where they might find the information with a number two. If they had to retrace their steps and go back to the gateway page, they were then to mark a number three on the link they would try next and continue in this manner, marking each step with a consecutive number. Our logic was that we would be able to determine the students' mental progress as they solved each task and therefore give us an indication of the efficiency of the links. We recognize that in the problem-solving process, reformulation often takes place, especially when the initial problem representation may not be adequate for the task of planning a solution (Glass, 1986). The scoring system we employed would help us determine how often reformulation needed to take place. The third step was to rate the difficulty or ease in accomplishing the task using an attitude assessment. A difficult task was listed as a number one and at the other end of the scale, a task considered easy would be scored with a number five. Finally an accounted rating technique was used, requiring the students to briefly explain why they selected a particular rating.

Design

The Effects of Label Terminology and Scope Notes on a Library's Gateway Page for Accuracy and Efficiency of Accessing Information

Table #2. Research Design

As shown in Table #2 four different designs of paper prototypes were used for the study. (See Appendix B.) The first represented the control group, with no change in the screen from the current gateway page. All subsequent pages for that group were current as well. In the second paper prototype labels were changed in order to use more communicative terminology, with less of an emphasis on library jargon. The third prototype used no labels but instead provided scope notes briefly describing the subsequent page as links to the those pages. Finally the fourth prototype included both clearer terminology and scope notes for labeling. The ten tasks and the scoring system remained the same for all groups. Two tasks were created for each of the five main access points, and these were later grouped together when analyzing the data.

We hypothesized that if ease of navigation within a Web site is related to both terminology of the labels and scope notes for those labels, then clarifying the terminology and including precise scope notes would result in more accurate and efficient navigation. We predicted that our IV #3 prototype, which included both of those variables, would result in increased accuracy and efficiency as well as more positive attitude assessment scores.

Procedure

The duration of the experiment consisted of one session for each of the three groups tested, which lasted from twenty-five to thirty-five minutes. After an introduction and walk-through of an example task, students worked at their own pace until the last task was completed. The testers provided answers to questions about procedures for completing the tasks but gave no answers or hints.

Scoring and Data Measurement

Three measures were scored and recorded for each task. The first involved the accuracy of the task completion. Responses were scored with a 1 for accurate, a .05 for partial and 0 for incorrect. The second measure rated the efficiency of task completion. By numbering each step taken as the subject progressed through the task completion, we could record the final number of steps to the desired access point. The third measure recorded was the numerical rating on the ease of task completion. A rating scale of from one to five was used, with five being the easiest. A space was provided for anecdotal remarks for an accounted rating in regards to the third measure.

Certain confounding variables need to be considered. Our subject population, by its very enrollment in a Library and Information Science course, would have more exposure to the library gateway pages through instruction. We were able to test one group before they had undergone thorough instruction, which was ideal for our study. Due to delayed schedules in the semester, our other two test groups had spent more time with the gateway pages and had been given instruction about them before they were tested. This was less than ideal and may have had some impact on our study, especially in terms of the outcome for the Control Group, which used the current library gateway pages. In addition by the end of the rather tumultuous semester, there might have been less motivation to participate in such a study, thus affecting the effort by the last two groups of subjects. Though the results of the study might have been slightly different had it been conducted on the Web as opposed to using paper prototypes, our research indicates that leading usability engineers prefer the paper format (UIE, 1998). We also felt it best to use the paper prototype because we wanted to maintain consistency for the efficiency component of the study. Had we relied on the Web environment, we could not have gauged the loading time for pages or foreseen problems with the technology.

Results and Discussion

Each task for each subject was given three ratings as outlined earlier (element a=accuracy, b=efficiency, c=attitude assessment). These results were entered into a spreadsheet with the central tendency of the distribution by calculating the mean average. Appendix C is the total results for all groups and all tasks. Appendix D shows the mean scores for elements "a" through "c", grouped by the two tasks that targeted each of the five main access areas on the gateway page. Table 3 summarizes the mean scores for accuracy, Table 4 for efficiency, and Table 5 for attitude.

Table #3 Average accuracy scores for groups of tasks ranging from 0 (inaccurate) to 1 (accurate)

As is evidenced by the data, there is no definitive support to substantiate our hypothesis. We should have seen the more desirable scores (as highlighted in color) fall mostly in the Group C column for each of the above tables. However, those scores appear to fall randomly throughout the various groups.

We can comment on certain aspects of the study if we narrow our vision somewhat. For example in Table #3, it would appear that either changing the terminology or adding a scope note or both would be the best way to portray "Electronic Journals, Newspapers & E-Books" on the gateway page as all three of these scored higher than the Control Group. The same appears to be true for "Hawaii Voyager Library Catalog". An interesting point is that the terminology "Reference Tools" remained the same for all groups, yet there was a spread of .29 points between groups for accuracy. This could be due to some subjects having an introduction to reference tools during the course.

It should be noted that Group B for the "Digital Archives Collections" and Group A for the "Hawaii Voyager Library Catalog" include mean scores that are not quite accurate. Each of these has one subject who felt so lost that they could not proceed beyond the gateway page, thus giving an efficiency score of one. This lowers the efficiency score in both instances and the mean is not reflective of the actual test. Once again the two groups working with the prototypes designed with the independent variables for the "Hawaii Voyager Library Catalog" appear to be more efficient than the current terminology.

Finally for the third element, that of attitude rating for the ease or difficulty in completing the task, generally Group C scored a higher rating, indicating that it was generally easier to navigate.

An ANOVA was run for each of the three elements tested in order to compare the difference between the four mean scores. (See Appendix E). As can be seen in the results for "accuracy" in Table #6, the p-value shows that the data is not significantly different. Reasons for the lack of significance are discussed in the next section.

Table #6. Mean Value, Standard Deviation, and P-Value for Accuracy

Conclusion and Recommendations

Though it was a disappointment not to see the data substantiate our hypothesis, there is still an indication that for both the "Databases and Indexes" and the "Hawaii Voyager Library Catalog" links we need to look at making some changes on the gateway page.

A number of points will need to be considered before the next study takes place. There was some ambiguity with one of the tasks, and it could be answered via several access points. In order to more thoroughly test the terminology, the task must be focused more closely. All testing needs to be completed at about the same time, especially if it involves LIS 100 classes. We will need to try and have more control over the exposure the subjects have with the gateway pages prior to testing. In addition one must consider that each of these five main access points are not all frequented with the same regularity. I would assume that the LIS 100 students had spent more time with the online catalog and the databases and indexes links. Since they would already know where those links lead, they will not be looking at the terminology and the scope notes with fresh eyes. This supports once again, testing subjects with little or no exposure to the gateway page. Finally, as Nahl points out, using more subjects increases the likelihood of producing more significant results (Nahl, 2001). A larger sample for each of our groups might have revealed more of a significance.