Article Note: On reference services impact assessment

Citation for the article:

Jacoby, JoAnn and Nancy P. O'Brien, "Assessing the Impact of Reference Services Provided to Undergraduate Students." College and Research Libraries 66.4 (July 2005): 324-340.

Read via Library Lit. and Information Science (WilsonWeb)


I wrote previously that I was looking at articles on reference assessment because my director wanted me to look into the topic. Since that last post, we have implemented some changes in how statistics are collected at the reference desk. Since we did start a new procedure, it may be a while before we have enough data to actually assess how things are working out. My hope is that the new procedures will help us answer some specific questions we have. I also hope the powers that be will actually give us time to collect the data. This is the second change in statistics collection procedure we have done in a year. At any rate, the article that I am noting today deals with reference services to undergraduates. Specifically, it looks at how services have an impact on undergraduate students by looking at three things: ". . . perception of staff approachability, awareness of library resources, and confidence in the ability to find information independently" (324-325). The approachability issue is covered by our use of the LibQual+ survey instrument to an extent. It may be interesting at some point to take some of the results from LibQual+ and see how they relate to a more specific survey like the one the Jacoby and O'Brien article describe. The concern I have is survey fatigue. For some reason, anytime the campus needs to do assessment, they figure a student survey is the way to go. Our students get surveyed on just about anything, and the library is just as guilty at times. So there is some concern of backlash. I don't have the answer to the dilemma, but it is something I think about every so often. For now, let's just make some notes from the article and go from there.

• The criteria libraries used at one time; compare to what Jacoby and O'Brien suggest now: "Matthew Saxton and John V. Richardson identified the 'three desirable outcomes of the reference process: utility, user satisfaction, and accuracy'" (qtd. in 325). The reference here is to the book Understanding Reference Transactions: Transforming an Art into a Science. By the way, user satisfaction is also something that LibQual+ measures.
  
• Some significance: "In a college and university environment, working with students to build skills for independent information discovery is paramount, and awareness and confidence are as important as accuracy, utility and satisfaction" (325). If we are doing our job in an academic reference desk, we are teaching the students how to discover information and deal with it. We are not just giving them the answer and sending them away, but we also show them how to find it and how to become better researchers and learners. Thus we would be interested in seeing how confident the students are in their ability to find and use information after they interact with us. This is also of interest to instruction librarians after they conduct an instructional session.
  
• A reason why we emphasize being accessible at the reference desk, why we want to give the appearance that we can be approached: "As Lynda M. Baker and Judith J. Field found in their 1999 study, the demeanor of reference personnel is a critical factor in the interview process, students' perception of the library, and the success of the reference interaction" (qtd. in 325). The reference goes to Public Libraries 39 (Jan/Feb 2000): 23-30.
• The article goes on to describe their methodology. They implemented a survey of undergraduate students scheduled to take place at specific times throughout a semester. "The surveys were administered immediately following a reference interaction, when the specifics of the encounter were still fresh in the respondents' minds" (327). This is a form of a critical incident survey. However, it is not as simple as it sounds. There are two parts. Part one was the survey itself, which could be done online or in print. From those who answered the survey, some respondents were then selected for a follow-up interview. The authors also detail the challenges they faced, and this is worth reading if you have interest in replicating this experience. The authors did provide a link to the first survey instrument, but as of this writing, the link is a dead-end.
  
• The authors designed their assessment to address three questions, which are:

1. "Do undergraduate students perceive the reference staff as being friendly and approachable?
2. Do the they learn something during the course of the reference interaction?
3. Do they feel more confident about their ability to independently find the information they are seeking following the reference interaction?" (328).
   

• The sample total for their survey was 69 undergrads (49 in print, and 20 online). However, the interview sample was extremely small: out of 12 people that had agreed to be interviewed, only 5 followed through. While the interviews do support some of the findings from the LIS literature, in the end, it is still a very low sample.
  

The authors then go on to discuss their findings. Some of the things they learned or reaffirmed:

• They noticed something we often notice as well: ". . .general difficulty undergraduates have in transitioning from full-text databases (one-stop shopping) to navigating a hybrid and distributed information environment" (332). I know that saying what I am about to say will probably give a couple of the cheery librarians an aneurysm, but I will say it anyways. We need to work in teaching students how to go from a database to other resources. Once you get them to use a database, the next challenge is "how do I find an item when it is not full-text?" Given the various vendors, their interfaces, and their ability (or lack thereof) to play nice with each other, this is something that we always need to teach our students.
  
• "This suggests [the results of their analysis] that friendly service may help bolster students' confidence in their ability to find the answers to their questions on their own, but the effect is rather slight" (332). That I just found interesting. It is important, but not a big deal.
  
• Again, we need to be approachable: "Working to create a climate of approachability helps ensure that students feel comfortable enough to approach staff" (337).
  
• From the conclusion: "This suggests that the reference interaction in college and university libraries can be an effective means of teaching students not only about specific library resources, but also about the process of finding, evaluating, and using information" (338). It's what we strive to do on a regular basis here. It's why I always say that instruction and reference are linked and should be working together.