Objective

To develop and psychometrically evaluate an information literacy (IL) self-efficacy survey and an IL knowledge test.

Methods

In this test–retest reliability study, a 25-item IL self-efficacy survey and a 50-item IL knowledge test were developed and administered to a convenience sample of 53 chiropractic students. Item analyses were performed on all questions.

Results

The IL self-efficacy survey demonstrated good reliability (test–retest correlation = 0.81) and good/very good internal consistency (mean κ = .56 and Cronbach's α = .92). A total of 25 questions with the best item analysis characteristics were chosen from the 50-item IL knowledge test, resulting in a 25-item IL knowledge test that demonstrated good reliability (test–retest correlation = 0.87), very good internal consistency (mean κ = .69, KR20 = 0.85), and good item discrimination (mean point-biserial = 0.48).

Conclusions

This study resulted in the development of three instruments: a 25-item IL self-efficacy survey, a 50-item IL knowledge test, and a 25-item IL knowledge test. The information literacy self-efficacy survey and the 25-item version of the information literacy knowledge test have shown preliminary evidence of adequate reliability and validity to justify continuing study with these instruments.

INTRODUCTION

In recent decades, the health care professions have evolved increasingly toward evidence-based practice (EBP) models. The most commonly used definition of EBP was proposed by Dr. David Sackett in 19961 as, “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of the individual patient. It means integrating individual clinical expertise with the best available external clinical evidence from systematic research.” An alternative term, evidence-informed practice, sometimes is used to specifically include patients' preferences along with providers' expertise/experience, and use of best available evidence.

While the initial implementation of EBP was in clinical medicine, EBP is increasingly influential in chiropractic profession, particularly within the last 10 to 15 years.2  Establishing chiropractic as an evidence-based health care profession necessarily requires evidence-based education (EBE); that is, chiropractic colleges and universities must implement curricula that provide students with the skills and knowledge necessary to be evidence-based practitioners.

The Council on Chiropractic Education (CCE) meta-competency 6 mandates chiropractic educational institutions to graduate students capable of, “A. Demonstrating knowledge of relevant research methodologies and ability to critically appraise and apply the literature to clinical cases, and B. Using health informatics to access information.”3  The trend toward chiropractic EBP and EBE was accelerated in 2005 by the National Center for Complementary and Alternative Medicine (NCCAM), which awarded R25 grants to 4 chiropractic colleges for the specific purpose of developing curricula that improve students' EBP skills. The principles shared by the grantee institutions are:

“(a) Course work should incorporate journal club formats, checklist reviews of current studies, and student construction of critical appraised topics. (b) Informational literacy assignments should span all 4 years, be relevant, and relate to other course content. (c) The language and concepts of evidence-based practice must permeate all diagnosis and management courses and, where feasible, basic science courses as well. (d) Focused and ongoing training must target a large proportion of classroom and clinical faculty across the entire school curriculum. (e) Application of these skills must be patient based and become part of the clinic culture as opposed to an endeavor segregated to a journal club activity.”4 

A common element in every aspect of EBE and EBP is information literacy (IL), which is defined by the Association of College & Research Libraries (ACRL) as, “...the set of skills needed to find, retrieve, analyze, and use information.”5  The ACRL has established 5 IL standards that are widely accepted throughout higher education:

“1. The information literate student determines the nature and extent of the information needed. 2. The information literate student accesses needed information effectively and efficiently. 3. The information literate student evaluates information and its sources critically and incorporates selected information into his or her knowledge base and value system. 4. The information literate student, individually or as a member of a group, uses information effectively to accomplish a specific purpose. 5. The information literate student understands many of the economic, legal, and social issues surrounding the use of information and accesses and uses information ethically and legally.”5 

Various aspects of IL have been studied for medical,6  nursing,7  rehabilitation therapy,7  dental,8  physical therapy,9  and occupational therapy10  students. In chiropractic studies, Weinert and Palmer11  investigated the effects of an IL course on chiropractic students' use of peer-reviewed literature and found that, “…students having the information literacy course (Group 2) had increased number and percentage use of peer-reviewed references.” Leo et al12  developed a questionnaire to evaluate elements of EPB knowledge in a sample of chiropractic students, concluding that, “Psychometric testing of the EBP knowledge component provided some initial evidence for acceptable reliability and validity.”

We could find no published literature that specifically evaluated IL self-efficacy and/or IL knowledge in chiropractic education. Given that IL is a necessary component of EBE and EBP, psychometrically sound instruments would be useful in evaluating chiropractic students' IL self-perceived abilities, such as self-efficacy and IL knowledge at various points in chiropractic college curricula. Accordingly, this paper presents the development and psychometric evaluation of two IL instruments: an IL self-efficacy survey and an IL knowledge test.

METHODS

Design

This was an instrument development study designed to be consistent with best practices for developing psychological test instruments.13  A 25-item IL self-efficacy survey and a 50-item IL knowledge test were administered in a test–retest reliability design 4 days apart. Item analyses were performed including: test–retest correlation, κ, Cronbach's α, point-biserial correlation, and KR20.

Subjects

Participants were a convenience sample of 53 students in a 7th trimester research methods course. Students were informed that their participation was voluntary and confidential. As participation was voluntary and anonymous, the study received an exemption from the Logan University institutional review board.

Instrumentation

The IL Self-Efficacy Survey – Version 1: 25 Questions

A total of 25 IL self-efficacy survey items, 5 items for each of the 5 ACRL information literacy standards, was developed by the authors. Self-efficacy is “the belief in one's capabilities to organize and execute the courses of action required to manage prospective situations.”14  In other words, a person's belief in their ability to succeed in performing a task or accomplishing a goal. Each question asked the participants to rate their confidence and competence to perform an information literacy task on a 1 (almost never true) to 5 (almost always true) Likert scale.

The IL Knowledge Test – Version 1: 50 Questions

A total of 50 test items, 10 items for each of the 5 ACRL information literacy standards, was developed by the authors. The questions were modified (with permission) from The Standardized Assessment of Information Literacy Skills (SAILS)15  project, which was developed at Kent State University (Kent, OH) and consists of a large test bank of items appropriate for general undergraduate education. Questions for the current knowledge test were modified from SAILS items and tailored for use by chiropractic students with all questions related to chiropractic specifically or health care in general. Each question had 5 alternatives with only one correct answer, three distracters, and one item, “Don't Know.” Instructions were, “If you are fairly confident that you are choosing a correct answer, circle A, B, C, or D — but please don't guess. If you really don't know the correct answer, circle ‘E. Don't Know'.” The purpose of alternative “Don't Know” was to serve as a guide to where additional IL instruction is needed. The IL knowledge test was found to have acceptable face and content validity by an expert panel.

IL Knowledge Test – Version 2: 25 Questions

The 50-item test was reduced to a 25-item test by choosing 5 items from each of the 5 ACRL standards with the best combination of psychometric characteristics.

Data Analysis

Version 1 of the IL self-efficacy survey was analyzed for reliability (test–retest correlation) and internal consistency (κ and Cronbach's α). Versions 1 and 2 of the IL knowledge test were analyzed for reliability (test–retest correlation), item difficulty (% correct), item discrimination (point–biserial correlation), and internal consistency (κ and KR20). Comparisons between the 5 sections of the IL self-efficacy and knowledge tests were by ANOVA with Bonferroni post hoc. A linear trend test was used to compare self-efficacy tertiles to knowledge scores. Descriptive statistics also were reported. Significance was set to p < .05 and all analyses were performed using SAS 9.3 (SAS Inc, Cary, NC).

RESULTS

The IL self-efficacy survey was found to have acceptable face and content validity by an expert panel of librarians. Version 1 of the IL self-efficacy survey showed good or very good psychometric characteristics (test–retest reliability, mean κ, and Cronbach's α) and was considered to have acceptable reliability and validity for continued testing without further modification.

Version 2 of the IL knowledge test showed good or very good test–retest reliability, item difficulty, internal consistency, and item discrimination, and was considered to have acceptable reliability and validity for continued testing without further modification.

Item mean rating, item range, test–retest correlation, κ and Cronbach's α for the IL self-efficacy test are shown in Table 1. Mean score, score range, test–retest correlation, κ, KR20, and point-biserial mean and range for Versions 1 and 2 of the IL knowledge test are shown in Table 2. Significant differences were found between the 5 sections of the IL self-efficacy survey and the IL knowledge test as shown in Table 3.

Table 1.

Descriptive Statistics, Item Rating, Test–Retest Correlation, κ Mean and Range, and Cronbach's α for Version 1 (25 Items) for the IL Self-Efficacy Survey

Descriptive Statistics, Item Rating, Test–Retest Correlation, κ Mean and Range, and Cronbach's α for Version 1 (25 Items) for the IL Self-Efficacy Survey
Descriptive Statistics, Item Rating, Test–Retest Correlation, κ Mean and Range, and Cronbach's α for Version 1 (25 Items) for the IL Self-Efficacy Survey
Table 2.

Descriptive Statistics, Item Difficulty, Test–Retest Correlation, Internal Consistency, Item Discrimination, and KR20 for Version 1 (50 Items) and Version 2 (25 Items) for the IL Knowledge Tests

Descriptive Statistics, Item Difficulty, Test–Retest Correlation, Internal Consistency, Item Discrimination, and KR20 for Version 1 (50 Items) and Version 2 (25 Items) for the IL Knowledge Tests
Descriptive Statistics, Item Difficulty, Test–Retest Correlation, Internal Consistency, Item Discrimination, and KR20 for Version 1 (50 Items) and Version 2 (25 Items) for the IL Knowledge Tests
Table 3.

Scores by Association of College & Research Libraries Category

Scores by Association of College & Research Libraries Category
Scores by Association of College & Research Libraries Category

IL Self-Efficacy Survey – Version 1: 25 Questions

For all items of the 5 ACRL standards: The mean self-efficacy rating was 3.49 ± 0.54 (range, 2.4–4.4); the highest rating (4.2) was for item 1.1 “recognize when I need additional information,” the lowest rating (3.17) was for item 2.5 “create a system for organizing retrieved information,” the test–retest correlation was 0.81, the average κ was 0.56 (range, 0.41–0.75), and Cronbach's α was 0.92.

IL Knowledge Test – Version 1: 50 Questions

Combining the test–retest administrations for all items for the 5 ACRL standards: the mean score was 40.9 ± 14.9 (range, 2%–72%), the highest item difficulty (1.9% correct) was for question 1.4 “According to the database record below, is the full text article available immediately?,” the lowest item difficulty (90.6% correct) was for question 1.3 “What part of the library record shown below indicates whether you could obtain this book immediately?,” the test–retest correlation was 0.83, the mean κ was 0.66 (range, 0.24–1.0), KR20 was 0.86, and the mean point-biserial was .36 (range, 0.23–0.66). All 50-items received E. “Don't Know” responses, ranging from 5.8% on item 11 (Which of the following search statements would retrieve the most records?) to 77.4% on item 50 (The code of federal regulations, 45 CFR 46, stipulates that the primary purpose of an institutional board (IRB) is:), with a mean of 29.35 for all items.

IL Knowledge Test – Version 2: 25 Questions

We chose 25 questions from the 50-item version of the IL knowledge test based on the best combination of item difficulty, point-biserial correlation, and κ item analyses (Table 2). Combining the test–retest administrations for all items for the 5 ACRL standards: the mean score was 53.1 ± 21.7 (range, 0%–84%), the highest item difficulty (18.9% correct) was for question 1.6. “Results from ‘Meade TW, et al. Randomized comparison of chiropractic and hospital outpatient management for low back pain: results from extended follow up. BMJ, 5:349–351.' would be listed as what type of source?,” the lowest item difficulty (90.6% correct) was for question 1.3 “What part of the library record shown below indicates whether you could obtain this book immediately?,” the test–retest correlation was 0.86, the mean κ was 0.69 (range, 0.24–1.0), KR20 was 0.85, and the mean point-biserial was 0.48 (range, 0.33–0.70; Table 3). All 25-items received “Don't Know” responses ranging from 9.4% on item 3 (What part of the library catalog record shown below indicates whether you could obtain this book immediately?) to 56.6% on item 49 (According to the standards established by International Committee of Medical Journal Editors, in order to be an author on a publication submitted to a peer-reviewed journal, you must:), with a mean of 26.66 for all items.

IL Self-Efficacy Scores by ACRL Categories

Significant differences (p = .0029) were found between ACRL category 1 vs category 5 and between category 1 vs category 3.

IL Test Scores by ACRL Categories

Significant differences (p < .0001) were found between ACRL categories 1 vs 4 and 5, and between categories 2 vs 3, 4, and 5.

IL Knowledge Test Scores by Self-Efficacy Tertiles

The low and middle IL self-efficacy tertiles had significantly lower IL knowledge test scores than the high IL self-efficacy tertile (trend test p = .002).

DISCUSSION

Considering the importance of IL in chiropractic EBE and EBP, a valid, reliable method of measuring self-perceived IL competence and confidence (self-efficacy) and IL knowledge could be of value in the continuing study of chiropractic EBE and EPB. In our opinion, the results of the psychometric evaluations and expert panel endorsement of the face and content validity of the IL self-efficacy test and IL knowledge tests provide acceptable preliminary evidence to recommend both instruments for continuing study. The publications by Weinert and Palmer11  studied the effects of an IL course on students' information gathering, but did not study IL knowledge directly. Leo et al12  developed and tested a questionnaire that included EBP knowledge components in content domains that are related to IL, but are not the same as in our IL knowledge test. To the best of our knowledge the IL self-efficacy and IL knowledge tests in this study are the first such instruments developed and evaluated specifically for chiropractic students.

Standard reference works for item analysis16,17  consider test–retest correlation >0.8 to be good, κ > .4 to be good and >.6 to be very good, Cronbach's α and KR20 >0.7 to be good and >0.8 to be very good (for smaller samples) and point-biserial >.40 to be good. Good item difficulty is considered to be between 20% and 80% correct, and 20/25 of the 25-item IL knowledge test questions are in this range. In summary, the item analyses for the IL self-efficacy test and the 25 question IL knowledge test were all in the good or very good range with the exception of 5 questions on the 25-item IL knowledge test with correctness under 20% or over 80%.

The “Don't Know” item seems to have been useful as an indicator of IL questions students were unsure how to answer, which may help direct educational efforts to those areas. However, it also seems likely that the “Don't Know” item had considerable impact on the item analyses of the individual items receiving this response and the psychometrics of the IL knowledge test overall. We plan to administer the IL knowledge test again in the same design, but without the “E. Don't Know” item and compare the relative merits of the instrument with and without the item.

In future studies, these instruments, or their improved successors, could be used to study the IL self-efficacy and IL knowledge of chiropractic students, educators, and field practitioners and to evaluate the effects of curricula offering specific IL coursework and other training. Reliable and valid IL instruments also may have value as outcome measures for CCE Meta-Competency 6.

Limitations include that the current sample was modest (n = 53), consisting of students in one trimester at one chiropractic college and may have been subject to selection bias as is possible in a convenience sample. Therefore, generalizability is limited, and continuing study with larger and more diverse samples is needed. The inclusion of a “Don't Know” item in the IL knowledge test may have altered students' responses, and, therefore, the psychometric analyses.

CONCLUSION

This study resulted in the development of three instruments: a 25-item IL self-efficacy survey, a 50-item IL knowledge test, and a 25-item IL knowledge test. The information literacy self-efficacy survey and the 25-item version of the information literacy knowledge test have shown preliminary evidence of adequate reliability and validity to justify continuing study with these instruments.

FUNDING AND CONFLICTS OF INTEREST

This work was funded internally. The authors have no conflicts of interest to declare relevant to this work.

REFERENCES

REFERENCES
1
Sackett
DL
,
Rosenberg
WM
,
Gray
JA
,
et al
.
Evidence based medicine: what it is and what it isn't. 1996
.
Clin Orthop Relat Res
.
2007
;
455
:
3
5
.
2
Delaney
PM
,
Fernandez
CE
.
Toward an evidence-based model for chiropractic education and practice
.
J Manipulative Physiol Ther
.
1999
;
22
:
114
118
.
3
Council on Chiropractic Education
.
Principles, Process and Requirements for Accreditation
. .
Published January 2013. Cited September 2014
.
4
Lefebvre
R
,
Peterson
D
,
Haas
M
.
Evidence-based practice and chiropractic care
.
J Evid Based Complementary Altern Med
.
2012
;
18
:
75
79
.
5
American College Research Libraries
.
Chicago: The Association
;
c
2014
.
Introduction to Information Literacy
. .
Cited September 1, 2014
.
6
Eeva-Liisa
E
.
Information literacy of medical students studying in the problem-based and traditional curriculum
.
Info Res
.
2005
;
10
:
paper
221
. .
7
Durando
P
,
Oakley
P
.
Developing information literacy skills in nursing and rehabilitation therapy students
.
J Can Health Libr Assoc
.
2005
;
26
:
7
11
.
8
Virtanen
JI
,
Nieminen
P
.
Information and communication technology among undergraduate dental students in Finland
.
Eur J Dent Educ
.
2002
;
6
:
147
152
.
9
Skov
ASH
.
Fighting an uphill battle: teaching information literacy in Danish institutions of higher education
.
Libr Rev
.
2003
;
52
:
326
332
.
10
Kipnis
DG
,
Frisby
AJ
.
Information literacy and library attitudes of occupational therapy students
.
Med Ref Serv Q
.
2006
;
25
:
11
20
.
11
Weinert
DJ
,
Palmer
EM
.
Influence of an information literacy course on students' information search behavior
.
J Allied Health
.
2007
;
36
:
e1
e12
.
12
Leo
MC
,
Peterson
D
,
Haas
M
,
LeFebvre
R
,
Bhalerao
S
.
Development and psychometric evaluation of an evidence-based practice questionnaire for a chiropractic curriculum
.
J Manipulative Physiol Ther
.
35
:
692
700
.
13
American Educational Research Association, American Psychological Association, National Council on Measurement in Education, Joint Committee on Standards for Educational and Psychological Testing (U.S.)
.
Standards for Educational and Psychological Testing
.
Washington, DC
:
American Educational Research Association
;
1999
.
14
Bandura
A
.
Self-efficacy
.
In
:
V.S
.
Ramachandran
,
ed
.
Encyclopedia of Human Behavior
.
New York
:
Academic Press;
1994
:
71
81
.
15
Project SAILS.org
.
Kent
:
Kent State University
; .
c 2014. Cited September 1, 2014
.
16
DeVellis
RF
.
Scale Development: Theory and Applications
.
Thousand Oaks
:
Sage Publications, Inc;
2003
.
17
Crocker
LM
,
Algina
J
.
Introduction to Classical and Modern Test Theory
.
New York, NY
:
Holt, Rinehart, and Winston;
1986
.

Author notes

Rodger Tepe is an associate professor at Logan University (PO Box 1065, 1851 Schoettler Road, Chesterfield, MO, 63006-1065; rodger.tepe@logan.edu). Chabha Tepe is the senior director of library services at Palmer College of Chiropractic (1000 Brady St, Davenport, IA 52803; chabha.tepe@palmer.edu). Address correspondence to Rodger Tepe, PO Box 1065, 1851 Schoettler Road, Chesterfield, MO, 63006-1065; rodger.tepe@logan.edu. This article was received April 22, 2014, revised August 29, 2014, and accepted September 1, 2014.

*

This paper was selected as a 2014 Association of Chiropractic Colleges – Research Agenda Conference Prize Winning Paper – Award funded by the National Board of Chiropractic Examiners.