Development and psychometric evaluation of an information literacy self-efficacy survey and an information literacy knowledge test^*

In recent decades, the health care professions have evolved increasingly toward evidence-based practice (EBP) models. The most commonly used definition of EBP was proposed by Dr. David Sackett in 1996¹ as, “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of the individual patient. It means integrating individual clinical expertise with the best available external clinical evidence from systematic research.” An alternative term, evidence-informed practice, sometimes is used to specifically include patients' preferences along with providers' expertise/experience, and use of best available evidence.

While the initial implementation of EBP was in clinical medicine, EBP is increasingly influential in chiropractic profession, particularly within the last 10 to 15 years.²  Establishing chiropractic as an evidence-based health care profession necessarily requires evidence-based education (EBE); that is, chiropractic colleges and universities must implement curricula that provide students with the skills and knowledge necessary to be evidence-based practitioners.

The Council on Chiropractic Education (CCE) meta-competency 6 mandates chiropractic educational institutions to graduate students capable of, “A. Demonstrating knowledge of relevant research methodologies and ability to critically appraise and apply the literature to clinical cases, and B. Using health informatics to access information.”³  The trend toward chiropractic EBP and EBE was accelerated in 2005 by the National Center for Complementary and Alternative Medicine (NCCAM), which awarded R25 grants to 4 chiropractic colleges for the specific purpose of developing curricula that improve students' EBP skills. The principles shared by the grantee institutions are:

A common element in every aspect of EBE and EBP is information literacy (IL), which is defined by the Association of College & Research Libraries (ACRL) as, “...the set of skills needed to find, retrieve, analyze, and use information.”⁵  The ACRL has established 5 IL standards that are widely accepted throughout higher education:

Various aspects of IL have been studied for medical,⁶  nursing,⁷  rehabilitation therapy,⁷  dental,⁸  physical therapy,⁹  and occupational therapy¹⁰  students. In chiropractic studies, Weinert and Palmer¹¹  investigated the effects of an IL course on chiropractic students' use of peer-reviewed literature and found that, “…students having the information literacy course (Group 2) had increased number and percentage use of peer-reviewed references.” Leo et al¹²  developed a questionnaire to evaluate elements of EPB knowledge in a sample of chiropractic students, concluding that, “Psychometric testing of the EBP knowledge component provided some initial evidence for acceptable reliability and validity.”

We could find no published literature that specifically evaluated IL self-efficacy and/or IL knowledge in chiropractic education. Given that IL is a necessary component of EBE and EBP, psychometrically sound instruments would be useful in evaluating chiropractic students' IL self-perceived abilities, such as self-efficacy and IL knowledge at various points in chiropractic college curricula. Accordingly, this paper presents the development and psychometric evaluation of two IL instruments: an IL self-efficacy survey and an IL knowledge test.

This was an instrument development study designed to be consistent with best practices for developing psychological test instruments.¹³  A 25-item IL self-efficacy survey and a 50-item IL knowledge test were administered in a test–retest reliability design 4 days apart. Item analyses were performed including: test–retest correlation, κ, Cronbach's α, point-biserial correlation, and KR20.

Participants were a convenience sample of 53 students in a 7th trimester research methods course. Students were informed that their participation was voluntary and confidential. As participation was voluntary and anonymous, the study received an exemption from the Logan University institutional review board.

A total of 25 IL self-efficacy survey items, 5 items for each of the 5 ACRL information literacy standards, was developed by the authors. Self-efficacy is “the belief in one's capabilities to organize and execute the courses of action required to manage prospective situations.”¹⁴  In other words, a person's belief in their ability to succeed in performing a task or accomplishing a goal. Each question asked the participants to rate their confidence and competence to perform an information literacy task on a 1 (almost never true) to 5 (almost always true) Likert scale.

A total of 50 test items, 10 items for each of the 5 ACRL information literacy standards, was developed by the authors. The questions were modified (with permission) from The Standardized Assessment of Information Literacy Skills (SAILS)¹⁵  project, which was developed at Kent State University (Kent, OH) and consists of a large test bank of items appropriate for general undergraduate education. Questions for the current knowledge test were modified from SAILS items and tailored for use by chiropractic students with all questions related to chiropractic specifically or health care in general. Each question had 5 alternatives with only one correct answer, three distracters, and one item, “Don't Know.” Instructions were, “If you are fairly confident that you are choosing a correct answer, circle A, B, C, or D — but please don't guess. If you really don't know the correct answer, circle ‘E. Don't Know'.” The purpose of alternative “Don't Know” was to serve as a guide to where additional IL instruction is needed. The IL knowledge test was found to have acceptable face and content validity by an expert panel.

The 50-item test was reduced to a 25-item test by choosing 5 items from each of the 5 ACRL standards with the best combination of psychometric characteristics.

Version 1 of the IL self-efficacy survey was analyzed for reliability (test–retest correlation) and internal consistency (κ and Cronbach's α). Versions 1 and 2 of the IL knowledge test were analyzed for reliability (test–retest correlation), item difficulty (% correct), item discrimination (point–biserial correlation), and internal consistency (κ and KR20). Comparisons between the 5 sections of the IL self-efficacy and knowledge tests were by ANOVA with Bonferroni post hoc. A linear trend test was used to compare self-efficacy tertiles to knowledge scores. Descriptive statistics also were reported. Significance was set to p < .05 and all analyses were performed using SAS 9.3 (SAS Inc, Cary, NC).

The IL self-efficacy survey was found to have acceptable face and content validity by an expert panel of librarians. Version 1 of the IL self-efficacy survey showed good or very good psychometric characteristics (test–retest reliability, mean κ, and Cronbach's α) and was considered to have acceptable reliability and validity for continued testing without further modification.

Version 2 of the IL knowledge test showed good or very good test–retest reliability, item difficulty, internal consistency, and item discrimination, and was considered to have acceptable reliability and validity for continued testing without further modification.

Item mean rating, item range, test–retest correlation, κ and Cronbach's α for the IL self-efficacy test are shown in Table 1. Mean score, score range, test–retest correlation, κ, KR20, and point-biserial mean and range for Versions 1 and 2 of the IL knowledge test are shown in Table 2. Significant differences were found between the 5 sections of the IL self-efficacy survey and the IL knowledge test as shown in Table 3.

For all items of the 5 ACRL standards: The mean self-efficacy rating was 3.49 ± 0.54 (range, 2.4–4.4); the highest rating (4.2) was for item 1.1 “recognize when I need additional information,” the lowest rating (3.17) was for item 2.5 “create a system for organizing retrieved information,” the test–retest correlation was 0.81, the average κ was 0.56 (range, 0.41–0.75), and Cronbach's α was 0.92.

Combining the test–retest administrations for all items for the 5 ACRL standards: the mean score was 40.9 ± 14.9 (range, 2%–72%), the highest item difficulty (1.9% correct) was for question 1.4 “According to the database record below, is the full text article available immediately?,” the lowest item difficulty (90.6% correct) was for question 1.3 “What part of the library record shown below indicates whether you could obtain this book immediately?,” the test–retest correlation was 0.83, the mean κ was 0.66 (range, 0.24–1.0), KR20 was 0.86, and the mean point-biserial was .36 (range, 0.23–0.66). All 50-items received E. “Don't Know” responses, ranging from 5.8% on item 11 (Which of the following search statements would retrieve the most records?) to 77.4% on item 50 (The code of federal regulations, 45 CFR 46, stipulates that the primary purpose of an institutional board (IRB) is:), with a mean of 29.35 for all items.

We chose 25 questions from the 50-item version of the IL knowledge test based on the best combination of item difficulty, point-biserial correlation, and κ item analyses (Table 2). Combining the test–retest administrations for all items for the 5 ACRL standards: the mean score was 53.1 ± 21.7 (range, 0%–84%), the highest item difficulty (18.9% correct) was for question 1.6. “Results from ‘Meade TW, et al. Randomized comparison of chiropractic and hospital outpatient management for low back pain: results from extended follow up. BMJ, 5:349–351.' would be listed as what type of source?,” the lowest item difficulty (90.6% correct) was for question 1.3 “What part of the library record shown below indicates whether you could obtain this book immediately?,” the test–retest correlation was 0.86, the mean κ was 0.69 (range, 0.24–1.0), KR20 was 0.85, and the mean point-biserial was 0.48 (range, 0.33–0.70; Table 3). All 25-items received “Don't Know” responses ranging from 9.4% on item 3 (What part of the library catalog record shown below indicates whether you could obtain this book immediately?) to 56.6% on item 49 (According to the standards established by International Committee of Medical Journal Editors, in order to be an author on a publication submitted to a peer-reviewed journal, you must:), with a mean of 26.66 for all items.

Significant differences (p = .0029) were found between ACRL category 1 vs category 5 and between category 1 vs category 3.

Significant differences (p < .0001) were found between ACRL categories 1 vs 4 and 5, and between categories 2 vs 3, 4, and 5.

The low and middle IL self-efficacy tertiles had significantly lower IL knowledge test scores than the high IL self-efficacy tertile (trend test p = .002).

Considering the importance of IL in chiropractic EBE and EBP, a valid, reliable method of measuring self-perceived IL competence and confidence (self-efficacy) and IL knowledge could be of value in the continuing study of chiropractic EBE and EPB. In our opinion, the results of the psychometric evaluations and expert panel endorsement of the face and content validity of the IL self-efficacy test and IL knowledge tests provide acceptable preliminary evidence to recommend both instruments for continuing study. The publications by Weinert and Palmer¹¹  studied the effects of an IL course on students' information gathering, but did not study IL knowledge directly. Leo et al¹²  developed and tested a questionnaire that included EBP knowledge components in content domains that are related to IL, but are not the same as in our IL knowledge test. To the best of our knowledge the IL self-efficacy and IL knowledge tests in this study are the first such instruments developed and evaluated specifically for chiropractic students.

Standard reference works for item analysis^16,17  consider test–retest correlation >0.8 to be good, κ > .4 to be good and >.6 to be very good, Cronbach's α and KR20 >0.7 to be good and >0.8 to be very good (for smaller samples) and point-biserial >.40 to be good. Good item difficulty is considered to be between 20% and 80% correct, and 20/25 of the 25-item IL knowledge test questions are in this range. In summary, the item analyses for the IL self-efficacy test and the 25 question IL knowledge test were all in the good or very good range with the exception of 5 questions on the 25-item IL knowledge test with correctness under 20% or over 80%.

The “Don't Know” item seems to have been useful as an indicator of IL questions students were unsure how to answer, which may help direct educational efforts to those areas. However, it also seems likely that the “Don't Know” item had considerable impact on the item analyses of the individual items receiving this response and the psychometrics of the IL knowledge test overall. We plan to administer the IL knowledge test again in the same design, but without the “E. Don't Know” item and compare the relative merits of the instrument with and without the item.

In future studies, these instruments, or their improved successors, could be used to study the IL self-efficacy and IL knowledge of chiropractic students, educators, and field practitioners and to evaluate the effects of curricula offering specific IL coursework and other training. Reliable and valid IL instruments also may have value as outcome measures for CCE Meta-Competency 6.

Limitations include that the current sample was modest (n = 53), consisting of students in one trimester at one chiropractic college and may have been subject to selection bias as is possible in a convenience sample. Therefore, generalizability is limited, and continuing study with larger and more diverse samples is needed. The inclusion of a “Don't Know” item in the IL knowledge test may have altered students' responses, and, therefore, the psychometric analyses.

This study resulted in the development of three instruments: a 25-item IL self-efficacy survey, a 50-item IL knowledge test, and a 25-item IL knowledge test. The information literacy self-efficacy survey and the 25-item version of the information literacy knowledge test have shown preliminary evidence of adequate reliability and validity to justify continuing study with these instruments.

This work was funded internally. The authors have no conflicts of interest to declare relevant to this work.