Evaluating How Well an Archival Website Allows a Researcher to Prepare for an On-Site Visit

As archival websites became more common in the late 1990s, Jean-Stéphen Piché argued, “This technology greatly facilitates a research-focused agenda for archives, i.e., to make archival knowledge products on websites the manifestation or representation of a broader research process in which archivists (should) engage.”¹ This statement is as true today as it was in 1998.

Given current widespread digitization of archival collections, many researchers might assume that the collections they need to use are available online. The reality is quite different. Wayne Bivens-Tatum notes, “Libraries all over the world are digitizing portions of their archives, for example, but the world's archives will most likely never be completely digitized. It's just too expensive for most libraries to do.”² With much archival material only available for use in the reading room, an on-site visit to the archives will be necessary for most researchers.³

Researchers depend on archives and manuscript materials as vital resources. In an era when connection to the Web is pervasive, a robust archival website⁴ can facilitate researchers' preparations. The website can enable researchers to plan a visit prior to traveling to an archives, allowing time to work through issues that, in the past, had to be resolved at the archives within the time constraints of the visit. Archivists can use their websites to meet the expectations and needs of researchers.⁵ The ability to evaluate their websites empowers archivists to understand, design, and advocate websites that help researchers prepare online for their on-site research visits.

The Web, then, provides an opportunity for archivists to advance the work of researchers in a way that previously was impossible. However, the upkeep of a website adds to the traditional, core responsibilities of archivists.⁶ Archivists need tools to help them answer the challenge of this additional responsibility.

One of the fundamental questions archivists must answer about their websites is, what should an archival website contain? While the literature includes examples of methods for evaluating groups of archival websites, I found none that present a method designed to guide individual archivists in assessing the content of their own websites. This article describes criteria for determining how well a website assists researchers preparing for an on-site visit included in an Archival Research Preparation Online (ARPO) Index.

My presentation on my analysis of academic archives using the ARPO Index to archivists at the Chicago Area Archivists (CAA) Archives Salon: Authors Series⁷ led to requests for access to the assessment tool so archivists could review their own websites. These requests prompted opening the online tool for use by archivists and the writing of this article detailing how the ARPO Index works.⁸

The website components described in this tool enable archivists to design sites that help researchers to enter the reading room aware of local policies and with the relevant holdings paged in advance of their visits. Archivists can evaluate their website components as content that allows researchers to perform tasks while using the website. This article discusses the ARPO Index and its companion assessment tool in detail, so that archivists can understand and use them to evaluate their own websites. Relevant literature supported the formulation of the ARPO Index.

The literature review begins with studies about how historians and genealogists use archival websites. It then covers archival website user studies to document what website features could assist researchers planning research visits. The section concludes with an examination of the content analysis method and models that can be modified for individuals studying their own archival websites. Conclusions from the literature inform the components of the ARPO Index.

Because historians and genealogists are heavy users of archival materials, studies of their use of archives can inform the conceptualization and design of archival websites. In 2003, Helen Tibbo undertook an international project to establish a baseline for academic historians' information-seeking behavior. She found that historians were more likely to visit an archival website (63%) than use a general search engine (44%) when looking for historical materials. Accessing an archival website for information can be compared to the historians' traditional method of contacting the archives directly to determine if a visit would be worthwhile. Tibbo argues that electronic finding aids and archival database tools are important for historians.⁹

Based on a survey, Wendy Duff, Barbara Craig, and Joan Cherry in 2003 found that historians rate finding aids the highest among archival resources. Ninety-three percent of respondents rated finding aids either very important or somewhat important. The authors stated that providing “detailed digital finding aids and expert virtual archivists” was a necessity for archives. Half of the respondents noted that one barrier to access was the lack of a finding aid.¹⁰

Ian Anderson determined it is possible to convert finding aids from print to digital in a way that supports historians' information-seeking behavior. He recommends creating online finding aids for those sources historians find most significant, including minutes, reports, and correspondence.¹¹ While Anderson's guidance helps set priorities for digitizing finding aids, the long-term goal must be to make all finding aid information available online.

In addition to historians, genealogists make up a sizable portion of researchers. Wendy Duff and Catherine Johnson found that genealogists want to search archival information by name, place, document type, and event. The searching interface should support Boolean searching and allow researchers to combine multiple search elements and limit searches by date.¹² These searching abilities could also assist other types of researchers using an archival website.

Several studies have investigated how researchers use archival websites. Burt Altman and John Nemmers surveyed researchers to determine their needs as part of the Pepper Online Archival Retrieval and Information System (POLARIS) Project. Most respondents could search and find the desired information easily through the Pepper Collection finding aid and found that “information, such as the number of folders in a series, was very detailed.”¹³

Christopher Prom's usability study of archival search systems found that both “keyword search options and alphabetical lists facilitate efficient searches for those who use backward chaining (footnote chasing) or name search techniques to gather basic information about a collection.”¹⁴ These findings are useful, as many archival websites provide alphabetical lists or keyword searching options but not always both. Providing browsing or searching capabilities while excluding one or the other leaves a large swath of the audience without their preferred method for finding information.

Rachel Walton culled ten recommendations for archivists from her recent usability study of Princeton University's online finding aids. Among the most important are the ability to search across all finding aids on a website and to browse collection contents without becoming disoriented.¹⁵

Altman and Nemmers, Prom, and Walton all focused on retrieval, search systems, and/or finding aids, but did not investigate other possible functions of archival websites to support planning for on-site visits. As early as 2002, Duff suggested that “virtual archivists” would be needed in the digital world to assist researchers as much or more than archivists are needed in a paper-based setting.¹⁶ Indeed, virtual reference through linked email, an online form, or instant message (IM) has become a vital part of many archivists' reference toolkits.

Respondents to a study by Duff and Penka Stoyanova in 1998 suggested that items are missing from online descriptive systems for archives, including glossaries, online help functions, and indexes. Additionally, respondents identified archival terminology on the websites as a problem. The authors felt more research was needed to allow archivists to create a website that uses common language and makes researchers feel comfortable.¹⁷ Twenty years later, I could find no follow-up research or organized efforts to address researchers' problems with archival terminology.

Studies of archival website users, including historians and genealogists, find they benefit from preparing for on-site visits by using finding aids on archival websites. Researchers prefer to browse lists and perform keyword searches to identify relevant materials. Online communication with archivists is also important to help researchers understand website terminology and other matters unfamiliar to them.

Because an archival website is a designed and ordered collection of content, content analysis provides a framework for developing and applying an instrument to measure whether a website contains the elements that researchers need. Elizabeth Yakel and Jihyun Kim note that this methodology may be used to study websites as manifest content, defined as “the surface meaning or the presence of specific identifiable elements in a text.”¹⁸ Kimberly Neuendorf defines content analysis as “a summarizing, quantitative analysis of messages that follows the standards of the scientific method . . . and is not limited as to the types of variables that may be measured or the context in which the messages are created or presented.”¹⁹

Using the perspective of content analysis focuses the study on the form and content of the website rather than on the interactions of users. The investigator must clearly define the message source being studied through content analysis.²⁰ Content analysis remains a systematic and reliable approach to measuring a website's surface meaning or specifically identifiable elements.²¹ The content encompasses the information displayed regardless of how well it is expressed. This content may be comprehended by researchers, and the evaluation ignores the presentation aesthetics of the design.²²

The online content of archival websites has been analyzed for more than two decades. A January 1995 question posted to the Archives and Archivists Listserv²³ and an article by William Landis²⁴ that same year led Terry Abraham to delineate criteria for evaluating archival websites. The criteria included Contact Information, Finding Aids (with keyword searching), Hours of Operation, Introductions,²⁵ Lists of Manuscript Collections, Regulations for Use, and Scope and Content Notes.²⁶ Using these criteria, Abraham made a very early attempt to measure the state of archival websites, evaluating 100 of the more than 1,000 academic archives website URLs he collected. In 1995, 70% provided a general description of their collections, nearly 50% included links to additional collection descriptions, 30% provided hours information, and 12% linked to rules and/or procedures. More websites used graphic elements (80%), including logos or colored bullets, than had links to additional collection descriptions.²⁷

Richard Cox observed that the types of archival web pages appearing in 1998 suggested that archivists were not up to the challenge of seizing opportunities that the Web provided, such as presenting a mission statement, searchable finding aid indexes, electronic records systems, digital collections, and virtual reference services.²⁸ Cox reported findings from a study by J. Christian Savine that examined online public service features of 43 US state government archives, 59 US academic archives, and 28 US state historical society websites. While most websites included a clear mission statement (58%), the prevalence of other important features was low, including searchable indexes (27%), email reference capabilities (17%), direct access to digitized collections (15%), and electronic records systems (2%).²⁹ With the Web still in its infancy, archival websites had not yet begun to fulfill their potential.

Ian Anderson developed the Model for Archive Web Development in 2008, which draws upon features from digital libraries and online museums. Through iterative development and testing, the model identified six types of websites that transcend the variety of archives, location, and organization. Each type builds upon the previous type, and they include Poster, Brochure, Interactive Brochure, Interactive Finding Aid, Transaction Service, and Interactive User Community. The content and function features collected for the website evaluation included Hours and Physical Location, Contact Email Address, Reference Form, User Services, Collection Description, Electronic Finding Aids (including the number and format), Tips, and Aids and Resources.³⁰ Anderson noted that including all potential archival functions or researcher needs in this model may be impossible and went on to postulate that using this generic example as a model for archival websites may be flexible enough to allow this approach to be incorporated into archival practice. While it suggests a great advancement in the decade after the Cox paper, this model has not been widely adopted or used.

Cox's and Anderson's contributions, while relevant, do not specifically address preparing online for an in-person research visit. While some of the content studied in the past provides a foundation for the ARPO Index, the continuous expansion of web capabilities leads to features not covered by the literature that must be considered and included in a current website.

This combination of previously documented and undocumented features are content elements that may exist on today's archival website. A content analysis approach is a useful method to determine if these elements to guide online researchers to the reading room exist on a website.

Similar to Inhwa Kim and Jasna Kuljis, who proposed a content analysis method to study web-based content,³¹ this article suggests the ARPO Index to guide the evaluation of archival websites. Because it can be used to review an individual website, any archivist can employ the ARPO Index to assess how well online visitors can use an archival website to prepare for an on-site visit.

The ARPO Index's nine measured components represent specific activities to facilitate researcher preparation for an in-person visit. Five components came directly from the literature: Ask Questions,³² Browse Holdings Information,³³ Search Holdings Information,³⁴ View Search Results Information,³⁵ and Review Information about Planning a Research Visit.³⁶ The other four components have not yet been discussed in the relevant literature: Accumulate Selected Information, Save Selected Information, Schedule a Research Appointment, and Request Materials for an Appointment. The details of the coding scheme are explained in the following ARPO Index section and incorporated into the online assessment tool.

This section reviews the details of the index, beginning with the iterative approach used to develop it. Then, each component is dissected to show what it measures, its type, any prerequisite components, and how it scores. Discussion ensues regarding how to use the assessment tool. An explanation of the overall scoring follows, and the section ends with limitations of the ARPO Index.

I used an iterative process in developing the ARPO Index. This process began with the index's initial formulation followed by multiple rounds of revision and the review of 463 academic archives and manuscript websites. Initially, all the components used simple binary choices between “yes” and “no”: yes, if the component was included on the website, or no, if it was not. It became clear during the first pass through the websites that the binary choice often was not granular enough to capture the realities of the existing content.

The ARPO Index was revised based upon colleagues' comments before the next round of content analysis. Some binary components remained, but many were adjusted to accommodate a more refined evaluation.

The next iteration removed a component not focused on those parts of the archival website directly related to planning an on-site visit. The Responsive Web Design component reviewed whether the website automatically adjusted to display well on a desktop computer, laptop, tablet, or cellphone.

Before I posted the tool online, the components and subcomponents were adjusted to work with any archives that allows public use of its collections. This ensures the index does not focus solely on academic archives.

The resulting tool more clearly focuses on preparing for an in-person visit and is usable by a wider range of archives types. The index, however, does not measure the effectiveness of the website components. Separate usability testing is recommended to determine how effectively the components were implemented.³⁷ The importance of conducting usability testing cannot be emphasized enough. A content analysis study of the website using the ARPO Index, however, will highlight the components that should be enhanced before undertaking a usability test.

The components of the ARPO Index are of three types: binary, faceted, and preferred state. One binary component scores 10 points if present or 0 points if not present. Seven faceted components include related, but individual, aspects the website could include. Each aspect, or facet, available on the website scores points, with the total possible value for a faceted component equaling 10 points. One preferred state component includes an ideal state worth 10 points and a lesser state worth 3 points.

To be included in a website's ARPO Index score, each component must meet the requirements described here.

The Ask Questions component evaluates whether the website provides a mechanism that allows researchers to query the archives staff. The “online form” facet scores more points than the other two facets because it is designed to prompt the researcher to provide a structured request designed to elicit responses that assist archivists in replying. The form and “linked email” also allow researchers to ask questions whenever they arise regardless of the hours the archives is open. An IM interaction with archives staff provides a direct connection for researchers, but only when staff is available to monitor this service.

Holdings information may be presented either as one comprehensive list or as a list segmented into several sublists arranged alphabetically by title or topic. Providing a list or lists of collection titles linked to finding aids with container lists is the preferred state for this component. A collection list or lists without links to finding aids provide only limited holdings information to researchers and requires follow-up with the archival staff.

This component requires that a website provide a search box where researchers can query holdings information using keywords or subject terms related to their topics. The search box function on the main page receives more points than a search box on a subpage because researchers have immediate access to it. Given that research shows that searching holdings information is the primary activity of researchers, placing the search box on subpages scores points as well.

The ability to review meaningful information quickly on the search results page enables researchers to determine both relevancy and which results merit further review. Links to parts of the finding aid lead to additional information. The search results page could be laid out much like the compact results of a search engine like Google Scholar.

The example in Figure 1 includes links directly to standard parts of the finding aid plus a linked listing of specific items found. The items link to the appropriate places in the finding aid to make sure the context is not lost. Figure 2 shows sample text using “site selection” as the search term.

Note that arrowheads in Figure 2 next to the listing allow it to be collapsed. The top arrowhead collapses the entire result to just the top line. The bottom arrowhead collapses the list of matched items because this section may be lengthy for some results.

Much like an online shopping cart, this component provides the ability to temporarily gather specific information, such as container list information about boxes or folders. This information can be found in search results or from finding aids and temporarily gathered into an archives-provided space online. This allows researchers to mark items of interest during their current search session.

This component requires the website to provide options for researchers to save box and/or folder information found when browsing or searching. These options assist researchers in future use of the collection information. The “create an account to store selected information” facet scores double the other three facets because the stored information is available for use over multiple sessions. This method does not require researchers to keep separate notes about the materials they have found, although they may choose to print, email, or download a list of archival items stored in their account.

The “hours open” facet provides the times a researcher may visit the archives. The “complete address” facet is the official postal address for the archives. The “location” is guiding information like the name of a building and the room number. The “location” facet aids researchers in finding the archives when the address has little apparent relationship to where the building physically stands or where the archives may be found within the building.

When “hours open,” “complete address,” or “location” appear on the main page, each scores one bonus point because the main page gives immediate access to such vital information. The “policies/procedures” facet may include information like use and access, reading room rules, camera or phone use, or duplication services. The “directions/map” facet guides researchers to the archives and may include basic public transportation information. The “parking instructions” facet assists researchers traveling by automobile with the options for parking near the archives. The “fee schedules” facet includes information about the costs of duplicating archival materials, other fee-based services, and acceptable forms of payment. As described above, these facets represent information types and not necessarily the exact designations an individual archives might use for each.

The Schedule a Research Appointment component measures researchers' ability to schedule a visit date and time, or just a date for those archives that do not require a specific time, through the website. The ability of researchers to “schedule an appointment online with automatic confirmation” on-screen and/or through email scores double the other three facets because researchers do not need to wait for authorization. Additionally, these automatic appointments may be restricted to scheduling with the appropriate lag time built in to allow archivists enough time to review the request, contact researchers with questions or issues, and pull the requested boxes before the appointment.

The “ability to request specific boxes and folders directly from search results, a finding aid, or a list saved in the researcher's account” scores double the other three facets because this method allows researchers to make direct requests using the information found on the website. The other facets require researchers to keep track of the boxes and folders outside the archival website and then enter that information into one of the other request methods.

The ARPO Index Assessment Tool (http://arpoindex.org) guides the archivist through the evaluation. The archivist's selections are automatically coded for the content evaluated for each component on the archives' website. The tool provides a final summary report page that includes the website's ARPO Index as a percentage of the possible total 90 points. Additionally, the report lists, by component, which choices the archivist selected and the raw score for that component. The report may be printed or, depending on the device used for the assessment, saved as a PDF.

Archivists and their web design teams may use the index via the web-based assessment tool³⁸ to evaluate their existing or aspirational websites and assess their redesign iterations. In an organizational context, archivists can also use the ARPO Index Assessment Tool to advocate for website upgrades by providing evidence of the need for such action.

To use the ARPO Index Assessment Tool, archivists should open both the tool and the website to be evaluated either side-by-side on the same screen or on two separate screens or devices. Using responsive web design, the tool automatically reformats the page to match a device's screen size to accommodate cellular phones, tablets, and laptop and desktop computers.

Each component may score up to 10 points for a potential total raw score of 90 points. The selection of 10 points came from the maximum number of facets in any one component. With the details of the nine components quantified based upon the criteria, the raw scores for the ARPO Index may be calculated.

The fact that each component potentially has the same maximum score suggests that all the components have equal weight. However, the dependence of some components on previous components has the potential to compound scores. Those gatekeeping components are more valuable because they may preclude scoring on subsequent components. For example, the View Search Results Information component depends upon the Search Holdings Information component. A score of 0 on Search Holdings Information results in no score for View Search Results Information.

Converting a website's raw score into a percentage of the total possible points results in the ARPO Index score. The initial ARPO Index score serves as a baseline for an individual website. Measuring future iterations using the ARPO Index Assessment Tool provides the means to gauge the improvement of each new design and guards against a design that reduces researchers' opportunities to prepare online. For example, an update to my previous institution's website caused its ARPO Index to drop slightly. The baseline can be used to generate support for a website redesign by highlighting current weaknesses and the path to their improvement, but it also can be used to guard against the erosion of ARPO Index components that already exist on a website.

Note that the assessment tool retains the data it generates for usage statistics only, collecting no identifying information about the archivist performing the assessment. Optional fields are included to provide the name and URL of the website reviewed to enhance the final summary report, but the ARPO Index website cannot indicate if archivists are evaluating their own websites or those of other institutions. This configuration of the tool ensures anonymity during the evaluation process.

The default decisions made while creating the index, though grounded in the literature and empirical data about archival websites, are not relevant for all archival websites. Adjustments can be made to the components, facets, and points awarded to fit specific needs. The ARPO Index runs on the Qualtrics platform, but the criteria can be downloaded³⁹ and adjusted to meet the need for a tailored, manual website review. This will ensure the ARPO Index is reusable.

Plans include maintaining the ARPO Index website and the assessment tool for the long term. Evaluating the state of the art of archival websites on a regular schedule will determine if the ARPO Index's components remain useful indicators of the content an archival website should include. As new types of content are added to archival websites that impact researchers' use of archives, future versions of the ARPO Index will be adjusted to stay current with archival website capabilities. This will help the ARPO Index remain sustainable.

Any archives open to the public with a public-facing website can be evaluated using the ARPO Index. Archivists may determine that one or more of the ARPO Index components are not important for their approach to providing access to their collections, or they may have other goals for their websites than preparing for on-site visits. Perhaps the aim of supporting the transition from online research to the reading room conflicts with other goals of a particular archives. Determining whether an ARPO Index component applies within local constraints and practices is up to each archivist. Clearly, archives vary in the way they provide access to their collections. For example, archives that can page materials quickly may decide not to use criteria in the ARPO Index related to requesting materials online.

Some institutions may focus on providing digital surrogates from their collections, thus reducing the need for researchers to visit in person. In this case, the current ARPO Index is not as useful. Future iterations of the index may include components tailored toward using digitized services if the need to visit an archives becomes unnecessary. However, the cost and time required to comprehensively digitize collections makes this prospect unattainable for most archives, as substantiated by Bivens-Tatum.⁴⁰ Additionally, most archives have items in their collections whose low frequency of use would not justify the cost of digitization.

Archival websites have great potential for improving archival research. As Tibbo,⁴¹ Duff,⁴² and Duff, Craig, and Cherry⁴³ argue, the profession needs to take advantage of that potential. The ARPO Index provides a way to measure whether an archival website's components allow researchers to fully prepare online for their in-person research visits. Researchers' online preparation increases the likelihood they will use their on-site time more efficiently and consult more effectively with the reference archivist.

The ARPO Index suggests other data gathering that will help researchers and the profession. Investigators could use the ARPO Index to assess the state of the websites of different types of archives such as academic or governmental. Periodically repeating such assessments would measure change over time.

Improvement of archival websites depends on the profession adopting a culture of regularly evaluating and upgrading them, especially using assessment tools and rubrics. Evidence-based redesigns not only serve users better but can also attract financial support for effective implementation of new designs.