Jodi Berkowitz 2014-05-27 13:15:31
Between April 2013 and March 2014, the North Carolina State University Libraries Special Collections Research Center (SCRC) conducted a series of three user studies to assess the feasibility of conducting meaningful research using materials at varying levels of processing. In the first study, participants were asked to complete research tasks using a large, unprocessed collection for which only minimal description was available. In the second study, an initial finding aid for the same collection was presented. A more complete finding aid was provided for the third study. The content of each of the finding aids reflected input of subjects from the previous studies. These studies were conducted as part of “Acting for Animals: Revealing the Records of Animal Rights and Animal Welfare Movements,” a two-year project funded by a Council on Library and Information Resources Hidden Collections grant. Methodology Ten participants were recruited for each study. The subjects recruited for the first study had previous experience using special collections materials. Such subjects were desired because the first study asked participants to use a spreadsheet to conduct the research tasks, as opposed to a more familiar, traditional finding aid. It was expected that experienced participants would be better able to mitigate the difficulty of working with a different descriptive tool. Seven of the ten subjects in the first study participated in the second and third studies. Three additional subjects were recruited for the second study, and two of these subjects participated in the third study as well. A single additional subject was recruited to complete the third study participant group. Graduate students and a faculty member in the history, education, and veterinary medicine departments composed the participant groups. Of the first study group, five participants (50 percent) rated their level of experience using archival materials for research as intermediate. Four participants (40 percent) rated themselves beginners and one participant (10 percent) rated herself advanced. The experience levels of participants in the second and third user groups were identical, with five beginners, four intermediate users, and one advanced user. Testing took place in the Usability Lab at D. H. Hill Library at NC State. The first study was conducted in April 2013, the second in November 2013, and the third in March 2014. Each study consisted of four parts: a background questionnaire, questions regarding initial impressions of the descriptive tool being studied, a series of tasks to be completed by interacting with the descriptive tool, and a post-test questionnaire. Each test took approximately one hour and all participants were tested individually. The studies were completely identical with the exception of the descriptive tool being investigated: the first study presented participants with an initial box list for an unprocessed collection in the form of a spreadsheet, the second study presented an initial finding aid for the same collection, and the third study presented a more complete finding aid for this collection. The focus of the studies was the tasks section, which consisted of ten research scenarios in which the participants were asked to use the spreadsheet or finding aid to determine if the collection would be useful to them in their research, and why they made that determination. The facilitator took notes and marked task success according to a rubric of task difficulty levels: easy, medium, hard, or fail. Results Findings based on task performance scores show that researchers will work successfully with the descriptive tool they are given, be it a rudimentary box list, a preliminary finding aid, or a refined finding aid reflecting a processed collection. Study participants were positive and confident about their research activities—strikingly so—when working with only a spreadsheet containing box-level information and with no collection-level or contextual information. However, participants had difficulty accessing search results in context in study two and moreso in study three. As the descriptive tool used by participants was refined over successive studies, task performance improved for four tasks (40 percent), remained the same for three tasks (30 percent), and decreased for three tasks (30 percent). Task performance improvement correlated directly with two factors: finding aid design that includes a series arrangement note in a prominent position along with links to each series, and the availability of a higher amount of description, allowing researchers to conduct successful keyword searches more easily. In tasks that were straightforward and could be completed with an in-document keyword search (one, six, and eight), there were also entire series matching each of these task objectives. The final finding aid presented to subjects contained a series arrangement section with links to each series, and this section appeared above the fold on the default page of the finding aid. In study two a series arrangement note was present but the series were not linked and many subjects glossed over this note. In study three, most participants began their searches by consulting this series list and were able to meet their needs solely through this method. In tasks that involved multiple search variables (seven, nine), many participants commented that a Boolean or advanced search feature would facilitate complex searches. One participant stated a preference for results to be presented like those in a library catalog search, with facets to help narrow results, and another noted that larger topics are harder to search without being able to refine the results. Query construction affected success in tasks with no obvious keyword phrase (task ten). In this task, participants were successful, but some required multiple attempts with different search terms or had difficulty determining related search terms, while others chose a better synonym on their first or second attempt. Summary and Discussion This study observed subjects attempting to conduct research with three different descriptive tools offering varying amounts of description of a large and initially highly disorganized archival collection. Overall, research using this collection became more feasible as a more detailed finding aid became available, but challenges remain. Most participants did not stop to read scope and content notes, even when their search term was highlighted within that note and they were having difficulty locating materials, as in task four. Several participants commented that they typically skimmed such notes or would not read them if they were more than a few sentences long. Study participants were successful at completing the tasks using only an initial box list for the collection; however, despite participants’ impressions, they did not realize that the collection they were considering was unfoldered and highly disorganized. Similarly, many participants in studies two and three expressed a desire for context. Participants who conducted searches using the browser page find feature would locate a result and scroll up to assess what series and subseries they were in and further evaluate that result. This method also led to numerous requests for additional design features, including back to top links, subseries links in the top series arrangement section, and clearer delineation in the finding aid of where in the document the user is at any time. Throughout the studies researchers were satisfied with the amount of description presented but expressed improved satisfaction as more detail became available. Exclusive use of in-document search in study one, even with the knowledge that the description was minimal and not encompassing of the whole collection, indicates the find feature is an asset but also a limitation. In study three, one participant noted that in spite of her two-phrase search failing, she thought there was likely material in the collection that met her research need. Another subject in this study commented on a less successful search, noting that perhaps “the way [the folder titles] are written didn’t come up under this search.” Conclusion These studies were conducted to assess the feasibility of conducting meaningful research with large archival collections with varying amounts of description available. Participant feedback informed further iterations of a finding aid for the collection that served as a use case. In particular, scope and content notes at every level were edited for length and content in an attempt to capture the attention of subjects, as were many folder titles. The SCRC can incorporate these data as we refine our finding aid design and processing practices. While processing archivists will continue to strive to create ideal finding aids, we can take solace in the knowledge that researchers will find success with the descriptive tools we make available. As one subject phrased it, “It’s impossible to anticipate everything everyone is going to search.”
Published by Society of American Archivists. View All Articles.