Looking for Answers: A Usability Study of Online Finding Aid Navigation

The traditional archival finding aid was a physical document crafted by an archivist that expressed the structure and content of a collection of materials only accessible from the controlled environment of a supervised reading room. However, in the last few decades, the archival finding aid has transitioned from static document to online interface. Online archival description represents a major step forward in that it facilitates enhanced discovery through remote interaction with collections and allows for wider and easier access to previously sequestered archival materials. However, current user expectations increasingly demand that the online finding aid act as the only point of access to archival collections in today's digital age—this has turned out to be a very tall order for archivists to fill.

The uniqueness and diversity of archival collections, their complicated provenance and context, and their often intricate hierarchical structure all make effective presentation of archival information on the Web a challenge. In the past, archivists have been accused of developing and implementing online archival description without considering user needs.¹ Arguably, the profession is still operating outside the user-centered systems movement when it comes to tools and interfaces for the online presentation of archival materials.² The last two decades of professional discourse about online archival description reveal that, while many extolled the merits of Encoded Archival Description (EAD) for online finding aids early on,³ significant room remains for improvement for online finding aids, especially in the realms of usability, navigation, and user interface design.⁴

To date, a few dozen usability studies have focused on online archival interfaces, and while most of these were relatively small in scope and scale,⁵ some were more extensive.⁶ Taken together, these studies point to several predominant and widespread usability issues, including but not limited to confusing profession-specific jargon, lengthy blocks of unstructured text, long lists of folders and subfolders, and numerous links embedded throughout extensive descriptive hierarchies.⁷ Suggested solutions to these challenges include simplified labeling terminologies,⁸ advanced keyword search options,⁹ and “quick links” for topical searching.¹⁰ As a profession, we are just beginning to understand what the ideal user interface might look like for online archival content, and certainly no model specific to finding aid navigation has been proposed yet. In more recent years, several studies have called for further progress and rigor in archival research investigations of online user behavior and information-seeking.¹¹ However, even as many institutions transition to newer archival information management systems and user interfaces, relatively few have considered the added value that improved navigational features could offer online researchers.

In response to this lacuna, this study asks the central research question: what navigational features are effective, efficient, and user-valued components within an academic archives' online finding aid interface? Discovering the answer requires understanding the needs and expectations of users, testing vetted navigational models, and marrying two fields that, until recently, have been siloed in their respective disciplines—online archival description and Web usability.

Now nearly two decades old, online finding aids have a complicated history within the archives profession. When EAD and online finding aids were new to the scene, they received a wealth of scholarly support and attention. The American Archivist dedicated its entire fall and summer editions of 1997 to a discussion of EAD and its implementation.¹² These issues heralded EAD as a potentially groundbreaking technology that the archival community should support and contribute to. Early proponents of EAD were confident in the schema's features, optimistic about its incorporation into professional practices, and even went so far as to imply that EAD finding aids were the logical next step for archival description. Overall, the sense existed that it was never too soon to begin adopting EAD and putting archival content online, at any institution, as its merits were obvious and significant.

While EAD's reception was undeniably positive in these initial moments, Dennis Meissner could see that online finding aids would need substantial reengineering in terms of look, feel, and structure before they could be effective as online collection descriptions.¹³ He stressed the need “to create finding aids that contain sufficient wayfinding tools to enable users to understand them and the materials they describe without the mediation of archivists” in the context of the virtual environment.¹⁴ In the following decade, online archival description and its EAD schema would come under a significant amount of fire as practitioners began to question the functionality, display, and effectiveness of finding aids in the context of the World Wide Web and its increasingly demanding users.

Just a year after the release of EAD1, Wendy Duff and Penka Stoyanova asked users what information about archival materials they would like to see online and how they would prefer it to be displayed.¹⁵ In the first usability study of its kind, these researchers used focus group feedback to critique existing finding aid interfaces. Their results indicated that users had trouble with abbreviations and specialized terminology like “linear extent” and “fonds,” and preferred archival information presented on the page according to bibliographic display guidelines and not current archival practice.¹⁶ The authors recognized that more research was needed on multilevel description, but suggested that archivists consult current research on system designs and conduct more usability studies to provide better interfaces for users.¹⁷ Luckily, others heard their call for more usability testing.

In 2001, Burt Altman and John Nemmers conducted research that pointed to navigation as a central concern for online finding aid functionality, because users needed to be aware of “where they were” in the collection at all times.¹⁸ They also discovered a need for both basic and advanced search interfaces to allow for different types of searching within a collection.¹⁹ Elizabeth Yakel's usability study a few years later revealed similar findings: the structure of the finding aid proved difficult for study participants, and many stated that they had “gotten lost” within the descriptive hierarchy.²⁰ In addition, Yakel's subjects had trouble understanding archival terminology and how to best search for information within archival websites.²¹

Another study by Jihyun Kim determined that because of significant element inconsistencies across institutions, users did not understand the meaning of labels when moving from one website to another.²² Kim also discovered that data elements in the EAD tag library were not being sufficiently utilized, meaning finding aids did not provide users diverse or granular access points. Finally, and importantly, Kim determined that EAD finding aids tended to contain narrative forms of information and long container lists without appropriate navigational elements, making it difficult for users to identify critical information and determine its location within the finding aid hierarchy.²³ Because of this, browsing within and across collections was proven to be a time-consuming and inefficient activity that did not assist in information retrieval.²⁴

Responding to Kim's note that “search functions are a growing necessity on EAD sites,”²⁵ Xiaomu Zhou analyzed fifty-eight EAD websites and their search capabilities, revealing that search functions supported a disappointingly low number of EAD finding aids. Those finding aids that did allow searching did not arrange search results for users in a structured way.²⁶ Zhou lamented that “It is unfortunate that archivists' focus has been on the issue of encoding finding aids rather than the subsequent process of delivery of archival information via a web interface.”

After a decade of implementation, a consensus was growing within the archival community that unresolved interface issues—particularly overall usability and navigational functionality—represented significant barriers to access and use of online archival description. Summing up the literature and taking into account their professional experiences during a website redesign effort in 2008, J. Gordon Daines and Cory Nimer cited four major problems with online finding aids to date: 1) unintuitive, profession-specific jargon and inconsistently implemented labeling practices; 2) long narratives, big blocks of text, and difficult-to-browse container lists; 3) poor access to item-level content due to ineffective or nonexistent search functionalities; and 4) confusing hierarchical organization and display of content that resulted in users feeling “lost.”²⁷

That same year, Richard Cox declared that despite our having entered the “golden age of archival description, . . . EAD's goal of easy access has been more dream than realization.”²⁸ Cox continued his critique by stating that archivists have been creating their online description “in violation of system analysis . . . and carrying out their descriptive work apart from and with little knowledge of how researchers find and use archival sources.”²⁹

Despite Cox's accusation, since the advent of EAD, several researchers employing usability and other types of studies have made an effort to understand who the target audience is for online archival content and what their information needs might be.

In a 2004 effort to inform developers about user requirements for new online services, Anna Sexton and the other members of the LEADERS Project asked the important question: “Who uses archival repositories' online description?” The LEADERS team's research identified several types of end users for online archival content, including “personal leisure” users, “individuals using archives as part of their professional occupation,” and “those using archives to support an educational or training program.”³⁰ Sexton's team also determined that a majority of archives users approach online finding aids through “an interest of individuals, families, or organizations,” while the remainder of searchers tend to frame their research topically and temporally.³¹ Finally, the project's research revealed that most users enter the online archival context already knowing what they are looking for and with some kind of subject area knowledge, yet the majority are inexperienced and uncomfortable with online finding aids as a research tool.

Rosalie Lack's research at the California Digital Library seems to concur; her focus groups, questionnaires, interviews, and usability testing indicate that, for most novice users, the concept of finding aids is extremely difficult to comprehend because new users don't immediately understand the usefulness of a list of physical objects without direct access to the objects via a digital interface.³² Echoing this finding, Christopher Prom also noted that inexperienced searchers expect finding aids to include digitized materials and not just serve as a guide to physical collections.³³ Wendy Scheir's writing tends to confirm this; she explained that interactions with online finding aids are sometimes “confounding and frustrating for novice users” who are often unfamiliar with both the subject matter of the content and the inherent structure of archival description.³⁴

Gretchen Gueguen at East Carolina University investigated the typical users of digitized special collection materials in an attempt to support multiple access interfaces and suit the needs of two distinct user groups—undergraduate students and humanities researchers. Her results showed that humanities scholars prefer to first search more broadly across archival materials, and, therefore, benefit from browsing a large and diverse set of resources.³⁵ In contrast, undergraduate students, despite having a higher competency in online library tools, have little to no familiarity with online finding aids and do not find them an effective searching platform. Rather, the students she interacted with prefer to engage with a curated, online exhibit interface that directs their focus and provides item-level descriptions for already digitized materials.³⁶

Daines and Nimer later confirmed that their primary user group—college students and casual researchers—reacted positively to the item-level display feature of their new interface and were able to find the information that they wanted more quickly within that context.³⁷ However, the site's secondary audience—advanced researchers—tended to select the expandable tree menu feature within the new interface believing that it provides greater context for the materials being displayed.³⁸ Wendy Duff and Catherine Johnson offered a thoughtful explanation for these tendencies. They argued that historians represent a separate, distinct, and advanced group of archives users, because while historians' research methods may seem “haphazard” and their discovery path almost “accidental,” in actuality they are “systematic and purposeful in the way they go about building contextual knowledge” from broad queries across a massive amount of archival material.³⁹

In summation, most studies to date identify at most three categories of users (casual researchers, college students, and professional researchers) and at least two levels of users (advanced and novice) who tend to interact with online archival description in very different ways. These distinct user groups have divergent information needs and use different search strategies to accomplish their research goals. Such distinctions are crucial to remember when evaluating the effectiveness of a chosen navigational model for online finding aids.

This research study focuses exclusively on Princeton University's finding aid website as it existed between September 2014 and May 2015.⁴⁰ This particular website was chosen because of the range of possible user interactions it encourages and supports.⁴¹ The finding aids can be navigated and searched in several distinct ways: 1) a treelike menu of contents on the left can be browsed by clicking on the nested tabs under “Contents and Arrangement”; 2) the contents of a collection can be viewed at the item level by clicking on the hyperlinks for each series, subseries, or item in a central content area on the page; 3) a single collection can be searched by using the search box at the top of the page; and 4) the items within each collection can be reordered by date or title using a special sorting feature located in the item listings' column header. In addition, the interface provides unique Web 2.0 features and plentiful help documentation. Furthermore, Shaun Ellis and Maureen Callahan, both of whom were involved in creating the interface in question, documented and articulated the logic, purpose, and process behind the site's creation.⁴² The study also benefited from communication with the team that built the website's interface.

Website usability studies represent an effort to evaluate a website's interface by testing it with a group of representative users.⁴³ In this case, the testing group was composed of ten⁴⁴ English-speaking undergraduate student volunteers without vision, speaking, or motor impairments at a large state university, all of whom received a small amount of financial compensation for their time and effort. ⁴⁵ Undergraduate students represent a critical population of users that archives attempt to reach with online finding aids, and, therefore, testing the usability of these interfaces with this particular population was both appropriate and essential. The demographics of this study's user group can be seen in Table 1.

All participants were asked to complete typical tasks often attempted by finding aid users by utilizing the existing navigational features on the Princeton University Library's finding aid website. Each participant was given the same set of ten common tasks, with guiding questions corresponding to each one, to be completed solely within the confines of the website within a period of thirty minutes or less. Table 2 shows a generic version of each task and explains the navigational decision that each task required users to make to be successful.

The following usability metrics were collected from each participant's effort to complete a given task: 1) total time spent; 2) the degree of success based on time-sensitive benchmarks; and 3) the number of “clicks” used before completion. In addition to these tasks, participants were also asked to comment on their experiences in brief written pre- and posttest surveys (free-response questions), a reflective interview with the researcher using think-aloud interview protocols (“think-alouds”), and, finally, a Likert-scale user satisfaction survey based on industry standards and best practices (System Usability Scale).

By reviewing written participant responses to the pre- and posttest questionnaires and looking at the System Usability Scale (SUS) survey results, this study shows what participants liked and disliked about the finding aid website interface, how they felt about its design and organization, and what aspects of the interface they found straightforward or confusing. Usability data points and trends in verbal user feedback collected from think-aloud style interviews also indicate the level of effectiveness and satisfaction users experienced within the chosen interface. Taken collectively, these results can suggest more generalizable usability guidelines, not just in the context of Princeton University Archives, but also for the broader community of stakeholders, be they academic archives, cultural heritage institutions, consortia, or developers.

Before being asked to complete tasks within a specific collection on Princeton's finding aid website, participants were given two minutes to explore the website on their own. Participants were encouraged to navigate around a simple, small collection and the website freely. Afterward, each was asked to write about his or her experience on the website for a full five minutes in a pretest questionnaire, with particular attention to liked features, disliked features, aesthetics, and points of confusion. Table 3 synthesizes participants' initial responses to the website.

Participants then completed their ten assigned tasks within a single and well-developed collection finding aid on the website. Afterward, they were again given five minutes to respond in writing about their experiences on a posttest questionnaire. Table 4 shows participants' responses to the posttest questionnaires, after they had become more familiar with the website and its functions.

As Tables 3 and 4 make clear, at least half of the study participants enjoyed the conciseness of the website and its text, the simple and uncluttered layout of the finding aids as well as the color scheme, and the hierarchically informed viewing it enabled. However, half of respondents indicated that the “Comments Box” at the bottom of every page was more confusing than helpful. Though nearly half of all participants expressed appreciation for an easy-to-find search bar, the same number of participants was disappointed in the lack of visual icons or images available in the finding aids. In addition, some participants found the labels attached to the left-hand tabs unintuitive and the subject terms applied to each collection overly vague.

While a few of the above questionnaire comments are undeniably negative, the results of the SUS survey (see Figure 1), on the whole, reveal a high level of satisfaction with the website, with an average SUS score of 84.5. Since a combined SUS score of over 70 is considered to be above average,⁴⁶ it seems that all participants rated the website “above average” in terms of usability.

A closer look at specific usability metrics yields even more fruitful data about exactly how users navigated the archival description on Princeton's finding aid website and whether or not that navigation should be considered easy and effective.

One of the most basic ways of determining which tasks might be more difficult to navigate than others is to consider “time on task” data, or the amount of time a participant needs to successfully complete a given task. The average “time on task” for each of the ten tasks presented to participants in this study is shown in Figure 2. These averages indicate that while tasks 8 and 9 were the most time consuming (requiring an average of almost one full minute to complete), tasks 2, 3, 6, and 7 were typically accomplished more quickly (in less than thirty seconds on average), suggesting that they were easier to achieve than the others.

Another way to determine the level of success for each task is to compare each participant's completion time to a set of benchmark completion times. In this case, the benchmarks selected by the researcher were 1) the larger group's average completion time for each task; and 2) twice that value. Any participant who completed a task at or before the first benchmark is classified in Figure 3 as having completed that task “with ease.” Similarly, any participant who took longer to complete his or her task than the first benchmark, but was successful at or before the second benchmark is classified in the chart as having completed that task “with difficulty.” Any participant who took longer to complete the task than did the second benchmark was considered unsuccessful.

By classifying the data in this way, we can see that at least 50% of participants were able to complete all tasks “with ease,” and, in most cases, only one in ten participants was not able to complete a given task as defined; these data, on the whole, represent an overwhelmingly positive group success rate. However, less than ideal results are also presented here. A large percentage (40%–50%) of participants could not complete half of the ten tasks—tasks 3, 4, 8, 9, and 10—“with ease.” The navigational decisions relating to each of these include where to find citation information, where to locate the creator's biographical information, how to find a subseries in the collection hierarchy, how to reorder collection contents, and how to find a single item within the collection. The fact that a large percentage of participants only completed these tasks “with difficulty” raises the question of whether or not navigational inefficiencies are to blame. Efficiency measures like the total number of mouse clicks per task can be helpful indicators of whether participants typically made more navigational errors during certain tasks.

Figure 4 shows two sets of data: 1) the optimal number of mouse clicks for each task—that is, the number of mouse clicks necessary to complete a task in the most efficient way—and 2) the average number of mouse clicks used by all participants for each task in the study.⁴⁷ The data are overlaid here to show which tasks the participant group performed most efficiently and which it typically performed inefficiently, that is, with far more than the necessary mouse clicks.

These results indicate that the least efficiently executed task, by far, was task 4—finding the creator's biography within the collection's finding aid. Users seemed to make navigational errors frequently when trying to complete this task, which could indicate that the preferred or intended navigational path to the creator's biography is confusing, unintuitive, or simply not apparent to end users. Other tasks that revealed high inefficiencies (those that averaged double or greater mouse clicks than optimal) included tasks 1, 5, 8, 9, and 10. These tasks included performing a global search across all collections, looking for similar items on the same subject as the current collection using subject terms, finding subseries information within the collection hierarchy, determining how to reorder collection contents, and finding a single item of interest within the collection. This implies that the most efficient pathway for completing common tasks on the website is not apparent to end users. Click inefficiencies can be key indicators of “lostness” on the part of the user—when he or she makes navigational errors by going down inefficient paths during task-oriented movements because of experiencing some degree of disorientation.⁴⁸

During the researcher-led interviews when participants were encouraged to think aloud about their experiences with the finding aid, verbal data were collected to confirm how “lost” or confused users felt. In addition, participants were asked which navigational features they preferred to use to complete their tasks and why. Tables 5 and 6 represent common responses from the participant group during these think-alouds.

These usability data and written survey responses seem to correlate with some of the navigational breakdowns (see Table 5) participants expressed during the verbal response portion of testing. For example, four participants specifically mentioned labeling as a “dislike” in their posttest questionnaire, and the issue came up again as a major navigational failure during the think-aloud. As previously mentioned, task 4, wherein users had to locate the content creator's biography by finding the correct tab label, was the least efficiently executed task. Similarly, the task completion rate for task 4, as well as for task 3, which required users to locate the preferred citation for the collection using tab labels, showed that 50% of users could not complete the task “with ease.” User comments on the first row of Table 5 support this: tab labels confused rather than clarified the proper navigational path for end users in several cases.

One potential, but still unvetted, solution to this vocabulary dilemma is to keep label titles as they are and provide guidance and context for them by inserting hover captions over each label, which would pop up anytime the mouse moved over them. These hover captions, which have been met with positive results in past experiments,⁴⁹ could briefly note what kinds of information each tab housed and therefore prevent confusion.

The other navigational failure many study participants mentioned is that the series-level tabs located in the left-hand menu bar under the “Contents and Arrangement” tab are not clearly related or connected to that tab in any visual way except by proximity. This confusion may help to explain why 50% of users did not complete tasks 9 and 10 “with ease,” which required interacting with collection contents, and why high levels of click inefficiency characterized these same tasks. Finally, as the last few comments on Table 5 hint, task 8, which required users to interact with the collection contents by reordering items, showed equally high levels of click inefficiency, and only half of all study participants completed it with ease.

According to the participant feedback given in the think-aloud interviews, these navigational failures are not the result of inappropriate navigational components, but rather the result of insufficient user-friendly visual cues. The reorderable item columns have no visual indication of “clickability” until a mouse scrolls over the column header. In the same way, the “Contents and Arrangement” tab and lower level series tabs share no visual indicators that might signal to users that they relate to the same content.

Connecting users, especially inexperienced or first-time users, to specific interface features requires clear and obvious visual cues. Responding to this very issue, one study participant made a practical suggestion that could potentially clarify the less-than-clear relationship between the “Contents and Arrangement” tab and the lower-level series tabs: simply hide the series tabs until the “Contents and Arrangement” tab is selected, making it clear that the information in all these tabs is related and connected. In the case of the too-subtle reordering feature—a small, hidden up or down arrow in the column header that appears only when the mouse rolls over it—it might be more logical to present the component in an explicit (set of) button(s) labeled “Reorder Contents.” This would highlight the feature's functionality and draw attention to its usefulness for the end user.

It may seem surprising that most participants in the study, instead of working exclusively within one of the navigational systems supported by the finding aid's interface, tended to split their efforts between several navigational systems, depending on the tasks they needed to perform. In fact, several participants explained their use of the two collection navigational systems as cooperative rather than mutually exclusive. For example, one participant noted that “At the highest level of the collection, the nested tabs on the left were useful, but to explore sub-series and items I preferred to work directly in the central contents box with the hyperlinks.” This, of course, is in line with data collected from both the pre- and posttest questionnaire, wherein half of all study participants mentioned the benefit of having a hierarchical contents list in the menu, and nearly as many commented on the navigational affordances of a readily accessible search box at the collection level, in addition to centralized content hyperlinks and a visible breadcrumb menu at the top of the page.

This usability study of Princeton University's finding aid website offers critical information about how end users of online archival content interact with and navigate around the online finding aids of academic archives. In an effort to translate these results into practical guidelines for archivists, the major findings of this study have been synthesized into a working model for online finding aid navigation. The recommendations presented below represent ten critical pieces of this functional model of the still-to-be-solved usability puzzle for online archival description. The hope is that archivists and developers alike can use these guidelines to make iterative, if small, steps toward improving online finding aid interfaces. While usability considerations and user-interface changes can be labor intensive and challenging to implement, it is important to know that even slight adjustments can yield significantly better user experiences. Furthermore, simply being aware of and vocal about the problems that users face in online finding aids are critical and foundational to moving our profession and finding aid technologies forward.

1. Use words and select titles that make sense to users; that is, make labels inclusive and intuitive.

2. Provide context for end users by maintaining collection hierarchy in the presentation of archival contents such as series, subseries, and container lists.

3. Give users a way to visually explore and browse through collection contents without “losing their place.”

4. Provide easy and quick access to individual items within a collection by minimizing the number of clicks needed to view item-level content.

5. Implement a navigational system that can present content at varying degrees of granularity to avoid information overload for users; in other words, allow users to hide lower-level detail when they don't want to see it.

6. Allow for keyword searching at the collection level and at the global level across the entire finding aid website.

7. Provide sufficient visual cues for special navigational features, such as drop-down menus, sorting buttons, clickable lists, and so on.

8. When possible, supply users with collection-specific visual content in the form of related images, icons, or graphics.

9. Keep the interface uncluttered and concise to support clarity and ease of use.

10. Do not add Web 2.0 features without cause or a consideration of user preferences.

Several of the above recommendations align with the “do's and don't's” of user-friendly finding aids outlined by Joyce Chapman of North Carolina State University.⁵⁰ Like many researchers before her, Chapman noted that archival terminology is often confusing to users and therefore should be avoided or explained wherever possible.⁵¹ In addition, she suggested that navigational menus mimicking a table of contents with links to specific sections of the finding aid can prove useful, as can the “ctrl-F” in-page search function when a collection-level search box is not available.⁵² Furthermore, Chapman argued that clear and easy-to-find help documentation is another important way to support users.⁵³ While none of the test participants in this study used Princeton's online help documentation, the interface did provide multiple routes and opportunities for them to access such information. Help documentation can act as a security blanket for novice users who are altogether unfamiliar with finding aids, and certainly further research is needed on the best way to provide help documentation within the online finding aid environment.

Many other aspects of finding aid usability remain unexplored. This study uncovered very little data about how to best facilitate global, repository-wide searching, yet users undeniably value this navigational feature. Princeton's finding aid website uses faceted search categories for site-level queries so that searchers can narrow their results by date, subject, language, and so on. However, it remains to be seen whether users value faceted search within online archival finding aids.⁵⁴ In addition, this research study focused on participants who self-identified as either beginner or intermediate finding aid users. It would be logical to test whether more experienced finding aid users—professional researchers, historians, and genealogists—would reveal the same navigational preferences as participants in a similar study. Finally, much more needs to be understood about the way the Web 2.0 features can be appropriately implemented to enhance the user experience in online finding aid interfaces. Though the “Comments Box” feature on Princeton's finding aid website seemed to generate more confusion than praise from test participants, recent studies point to moderate amounts of user interest in what have been called “participatory” finding aids—those that allow for user annotations and contributions.⁵⁵ Other Web 2.0 features that remain underresearched in the context of finding aids include tagging, word clouds, hover captions, and even saving and starring features to allow users to revisit their favorite results or queries later.⁵⁶ To date, little has been determined about the potential effectiveness or efficiency of these kinds of interactive features for online archival description. Future research should explore these new opportunities with the same verve that the past two decades of researchers exhibited in their pursuit and refinement of EAD.