Reliable Change Indices for the Serial Administration of the Concussion Clinical Profiles Screening Tool

A prospective, repeated measures research design was used for this study.

Participants included men and women from an undergraduate and graduate college-aged (ages 18–25 years) sample. Any student who self-reported either a current medical diagnosis of a head injury or a head injury between assessments was excluded from participation.

Participants were asked to provide information about personal and health history, including their age, sex, history of medically diagnosed concussion, and self-reported (ie, yes/no) history of migraine, motion sickness, depression, anxiety, attention-deficit/hyperactivity disorder, and/or learning disability. These demographic and medical history questions are a recommended part of a clinical evaluation for concussion.¹⁵ 

The CP Screen is a 29-item self-report symptom inventory used to calculate subscores for 5 clinical profiles and 2 modifiers based on how participants are currently feeling on a 4-point Likert scale ranging from 0 (none) to 3 (severe). The clinical profiles include anxiety/mood (5 items), cognitive/fatigue (3 items), migraine (5 items), vestibular (5 items), and ocular (5 items), and the 2 modifiers include neck (2 items) and sleep (4 items).²  A total score for each clinical profile and modifier was calculated as the sum of all of the CP Screen items that corresponded to that profile. Total scores were used in the current study instead of the average scoring method used in previous research.⁷  Total scoring resulted in whole-number RCI cutoff scores rather than real numbers, which made the RCI values more interpretable (ie, avoided decimals as RCI cutoff values). The CP Screen takes approximately 4 to 6 minutes to complete and is designed for children and adults (12 years and older). Current psychometrics for this measure include high internal consistency in both healthy (Cronbach α = 0.87) and concussed (Cronbach α = 0.92) samples and good predictive validity (partial η² = .49).⁷ 

The study was approved by the university Institutional Review Board. Researchers recruited participants from undergraduate and graduate classes at a large university in the Midwest region of the United States. After the study was explained to students and informed consent was obtained, the demographics and medical health history form and CP Screen were immediately administered to the participants (Time 1) during their class time (ie, group setting). The research team returned to the participating classes 1 week later (Time 2), and participants completed the CP Screen a second time during their class time. Members of the research team supervised all group data collection to ensure a controlled environment for the completion of the questionnaires. If participants did not attend class when Time 2 data were collected, they were excluded from the study and data analysis. All participants who completed measures at Time 1 and Time 2 were entered into a raffle to win gift cards.

Descriptive statistics (eg, mean, median, SD, and range) were used to describe participant demographics, and total CP Screen scores at Times 1 and 2 were calculated for each profile. Normality was assessed for all CP Screen scores, and the Shapiro-Wilks values ranged from 0.51 to 0.90 (all P < .001); therefore, data were decidedly nonnormal. Differences in CP Screen scores between Time 1 and Time 2 for the total sample, men, and women were examined with a series of nonparametric Wilcoxon signed-rank tests. Given the number of these repeated-measures analyses (eg, of the 7 profiles and modifiers), a Bonferroni-corrected α level of P < .007 was used. However, to better gauge the magnitude of difference, Cohen’s d was documented as a measure of effect size, with 0.2 denoting a small effect size, 0.5 denoting a medium effect size, and 0.8 denoting a large effect size.¹⁶ 

Spearman ρ correlations (r_s) were used to examine the relationship of CP Screen scores between the 2 time points for the total sample. Cutoff values of low (≤0.39), moderate (0.40 − 0.59), moderately high (0.60 − 0.79), and high (≥0.80) were used for interpretation.¹⁷  Intraclass correlation coefficients (ICCs) were calculated to distinguish those sets of scores that are merely ranked in the same order from test to retest from those that are not only ranked in the same order but are also in low, moderate, or complete agreement and were used as an indicator of test-retest reliability.¹⁸  The ICC model 2-way mixed type consistency was used, using single measures. Given that systematic error variance may bias or inflate correlation coefficients, ICC analyses also yield an unbiased estimate of reliability (UER), which corrects for any over- or underestimation of the consistency of the measure across 2 time points.¹⁹ 

Reliable change indices were calculated for each profile and modifier using 90% and 95% CIs to assess whether a change between repeated assessments was reliable and meaningful.¹³  Within a normal distribution of difference scores, 10% and 5% of scores are expected to fall outside of the 90% and 95% CIs, respectively. Reliable change index methodology identifies cutoff scores that are an estimate of the probability that a given difference in score would not be obtained as a result of measurement error²⁰  and were calculated for the total sample and separately for men and women. Statistical tests were conducted using Statistical Package for Social Sciences (IBM Corporation) version 28.0 and Microsoft Excel (Excel 2019).

A total of 269 participants enrolled in the study and completed the CP Screen at Time 1. Eighty-seven participants (32%) did not attend class to complete the Time 2 study measures, and 9 participants had missing data, so these 96 participants (36%, 96/269) were excluded from final analyses. A final sample of 173 participants (mean age [M_age] = 20.66 years, SD = 1.43 years) completed the CP Screen at Times 1 and 2 (ie, 64% response rate). There were 82 men (M_age= 20.71 years, SD = 1.38 years) and 91 women (M_age= 20.63 years, SD = 1.47 years) in the study. The 2 study visits were 7 days apart. Due to inclement weather and university closure, 48 participants completed their second study visit online instead of in person. The CP Screen scores for this smaller online sample were compared with the in-person scores with a series of Mann-Whitney U tests, and there were no significant differences between these responses for any of the outcomes (P >. 05). Participant demographics, history of medically diagnosed concussions, and self-reported health histories are presented in Table 1.

The mean, median, and SDs for the total sample, men, and women are presented in Table 2. A series of Wilcoxon signed rank tests revealed that the anxiety/mood (z = −3.46, P < .001, d = 0.15), cognitive/fatigue (z = −4.01, P < .001, d = 0.24), ocular (z = −4.28, P < .001, d = 0.23), and vestibular (z = −3.59, P < .001, d = 0.18) Time 2 CP Screen scores were significantly lower than Time 1 scores for the total sample, but the magnitude of these differences reflected small effect sizes. The total scores for men were significantly lower at Time 2 than at Time 1 for the anxiety/mood (z = −2.80, P =. 005, d = 0.15), cognitive/fatigue (z = −3.66, P <. 001, d = 0.32), ocular (z = −3.87, P <. 001, d = 0.31), and sleep (z = −2.91, P = .004, d = 0.25) profiles and modifier. By contrast, Time 1 and Time 2 scores did not significantly differ for women (P ≥ .01) on any CP Screen outcome. Concerning sex, effect sizes were higher for men than for women on the cognitive/fatigue (d = 0.32 versus 0.20), migraine (d = 0.18 versus 0.01), ocular (d = 0.31 versus 0.17), sleep (d = 0.25 versus 0.02), and neck (d = 0.24 versus −0.04) profiles and modifiers, whereas effect sizes were higher for women on the anxiety/mood (d = 0.16 versus 0.15) and vestibular (d = 0.21 versus 0.15) profiles. Means, medians, and SDs for CP Screen scores across study time points are presented in Table 2.

Spearman correlation coefficients revealed significant, moderate-to-high relationships across the 2 time points for the total sample, ranging from 0.51 (migraine) to 0.79 (anxiety; Table 3). Interclass correction coefficients reflected higher reliability than Spearman r across all measures. Anxiety/mood total scores showed the most stability (single ICC = 0.82; lower and upper 95% CIs = 0.77–0.87), followed by ocular (0.78; 0.71–0.83), vestibular (0.78; 0.71–0.83), migraine (0.71; 0.63–0.78), sleep (0.70; 0.61–0.77), cognitive/fatigue (0.68; 0.60–0.76), and neck (0.65; 0.56–0.73). Unbiased estimates of reliability were consistently higher than ICCs (anxiety/mood = 0.90, ocular = 0.88, vestibular = 0.88, migraine = 0.83, sleep = 0.82, cognitive/fatigue = 0.81, and neck = 0.79; Table 3).

Differences in the test-retest reliability between men and women were observed (Table 3). Both men (r values = 0.48–0.82) and women (r values = 0.51–0.77) had moderate-to-high relationships across the 2 time points. The vestibular profile (ICC = 0.87, UER = 0.94) and neck modifier (ICC = 0.60, UER = 0.76) demonstrated the highest and lowest reliability among men, respectively, whereas the anxiety/mood (ICC = 0.80, UER = 0.89) and cognitive/fatigue (ICC = 0.64, UER = 0.78) profiles demonstrated the highest and lowest reliability among women, respectively.

The percentage of individuals surpassing RCI cutoffs for each profile and modifier is listed in Table 4 and presented at the 90% and 95% CIs for the total sample and for men and women. After 1 week, 1% to 14% of the scores for the total sample and for men and women fell outside the range of normality, which is considerably close to what is expected for a 90% CI (eg, ≤90%; Table 4). However, although the overall percentage of cases demonstrating reliable change was generally within expected CIs (eg, for anxiety/mood, 90% of cases fell within a 90% CI), there was a consistent imbalance between scores decreasing versus increasing (eg, 7% decreasing and 3% increasing). Given that RCI cutoffs are derived for both improvement and decline but assume equal distribution in both directions, and our data revealed significant declines in median scores (Table 2), we adjusted RCI cutoffs to require an additional point for reliable change in decreased scores.²¹  A reference guide for estimating change for each CP Screen profile and modifier total score is presented in Table 5, and additional reliability and RCI outcomes are listed in Table 6. The following RCI cutoffs for estimating clinically significant change (increase/decrease) at the 90% CI for men were used: ocular, vestibular ≥2/≥4; sleep ≥4/≥6; anxiety/mood, cognitive/fatigue, migraine ≥3/≥3; and neck ≥2/≥2. The following RCI cutoffs for estimating clinically significant change (increase/decrease) at the 90% CI for women were used: anxiety/mood ≥2/≥4; cognitive/fatigue, migraine, ocular, vestibular, sleep ≥3/≥3; and neck ≥1/≥1. Overall, women require a greater magnitude of decline than men for anxiety/mood at the 90% and 95% CIs, and men require a greater magnitude of decline than women for ocular, vestibular, and sleep at the 90% and 95% CIs.

The current study documented the 1-week test-retest reliability of the CP Screen in nonconcussed college-aged men and women and established RCI cutoffs for each of the CP Screen profile and modifier scores. Overall, the CP Screen profile and modifier scores have moderate to moderately high reliability over a 1-week administration, as reflected by ICCs and UERs. In addition, although there were significant (P value) reductions in CP Screen scores across 1 week, the magnitude of change was small (effect size). Given that variation in performance is expected across time intervals, RCIs established that scores at the second assessment were generally within expected parameters, using 90% CIs. As such, these results demonstrate that the CP Screen is a stable measure of concussion symptomology over a 1-week retest interval, and these findings can be used to determine meaningful change in symptoms that underlie concussion clinical profiles.

The current study adds important information to a body of literature that has primarily focused on the identification of concussion symptom factors but has lacked study of the reliability and RCIs for these outcomes.^8,10,22–24  To date, only 1 study has established test-retest reliabilities for these symptom factors, and these researchers reported 6-week test-retest reliability for cognitive, physical, affective, and sleep PCSS symptom factors.²⁴  The correlations ranged from 0.44 to 0.80, and ICC values ranged from 0.44 to 0.77.²⁴  Despite a longer retest interval than the one used in the current study (ie, 6 weeks compared with 1 week), the correlations and ICCs across the PCSS symptom clusters and the CP Screen profile scores were comparable (r_s = 0.45 to 0.76, ICC = 0.50 to 0.78).

Concerning sex, differences in baseline and postinjury symptom reporting have suggested the need for separate clinical cutoffs on symptom screens.^14,25–27  The current study reported higher RCI cutoff scores for the anxiety/mood profile at both the 90% and 95% CIs for women and higher RCI cutoff scores for the neck modifier at the 90% CI for men. These differences in RCI cutoffs fill the gaps of previous research by providing evidence of sex being associated with clinically meaningful levels of concussion profiles or subtypes based on symptom reporting. However, the RCI cutoff scores established in this study should not be used as a sole determinant in clinical decisions. Rather, they are intended to assist clinicians in determining what constitutes a meaningful change in CP Screen profile and modifier scores throughout recovery. Given the clinical need for documenting meaningful changes in concussion symptoms across recovery, this study is the first to document RCI values and associated cutoffs for men and women using the CP Screen.

There are several strengths and limitations to the current study. The retest time interval used in the current study was well controlled (ie, all second visits occurred on the seventh day following the first visit) and reflects a weekly retest window that is commonly used for the clinical care of concussion. In addition, the current study included a sufficient sample size for the test-retest design and is comparable to other similar study designs.^7,24,28  However, the current study also had a high dropout rate, which is attributed to participants missing class on the second time that data collection was scheduled. There is no way of knowing if this dropout rate is only attributed to student absence from class or reflects a lack of interest in continued participation in the study. Additionally, the sample age range used in this study is only composed of healthy college-aged students, and these results cannot be applied or generalized to other age groups. Researchers report that adolescent symptom reporting is different than adult symptom reporting, and these differences may result in specific RCI values for these populations.²⁹  Lastly, the time frame used in this study included only 1 week, and these findings may not be applicable to other retest time frames.

There are various directions for future research regarding the clinical use of the CP Screen. More specifically, the retest reliability of this measure should be examined over longer periods (eg, acute, subacute, and chronic) that would better inform improvement and/or deterioration of clinical profile symptoms. In addition, RCI values should be derived for different ages and races as well as other primary risk factors (eg, anxiety and migraine history) to better individualize the clinical use of this tool. Finally, these RCI values should be applied in a clinical sample to determine the percentage of individuals with concussion that exceed RCI values over the course of recovery.

The present study was the first to establish RCI cutoff scores that can be used to determine significant change and document test-retest reliability for the CP Screen. The findings of this study support the continued use of this tool for the individualized management of concussion and to assist clinicians with determining meaningful changes in symptom reporting across recovery. Moreover, the CP Screen is a stable measure of clinical profile symptoms across 1 week.

We would like to thank all the graduate and undergraduate students who contributed to the data collection and helped make this research possible.

A.P.K. and M.W.C. receive funding for research through the University of Pittsburgh from the National Football League and royalties from APA Books. M.W.C. was a codeveloper and shareholder for ImPACT applications (ended December 2019).