King-Devick Sensitivity and Specificity to Concussion in Collegiate Athletes

This study was part of the National Collegiate Athletic Association–Department of Defense Concussion Assessment, Research and Education (CARE) Consortium,²  an ongoing comprehensive study examining the effects of concussion in collegiate athletes and US military service academy members.²⁰  All 30 CARE Consortium sites use a common definition and criteria for concussion.²⁰  The K-D test is used at 6 of these sites as a Level B measure, which is considered an emerging evaluation and is added at the discretion of the clinicians at the performance site.

Athletes were only considered healthy and included in our control group if they had >1 baseline assessment and no concussion documented in the CARE Consortium database. The K-D baseline assessments were administered 1 year apart. Athletes were included in our concussed group if they sustained a concussion and completed K-D longitudinal postconcussion assessments. Athletes with concussions, which were diagnosed by each institution's sports medicine team with guidance from the current consensus guidelines,²¹  were excluded from the control group. Athletes with missing confounding factor variables were excluded from the corresponding analyses. All athletes provided written informed consent before participation, and the US Army Medical Research and Materiel Command Human Research Protection Office and each CARE Consortium site's institutional review board approved the study.

The K-D test consists of a rapid number-naming task that takes <2 minutes to complete and is available on 2 platforms (iPad [various models, although the K-D test and platform were the same for all; Apple Inc] and paper). Athletes complete 3 test cards of rapid number naming, reading from left to right. For baseline assessments, athletes complete 2 error-free trials to supply a baseline score. The athlete's baseline is the faster of the 2 trials. The test is readministered after concussion, but trials are not repeated in cases of error. Total time is defined as the time in seconds to complete the entire test.⁵ 

For participants with concussion, we subtracted their baseline time from their postconcussion time (postconcussion time point − baseline). This was done at each postconcussion time point: 0 to 6 hours postconcussion; 24 to 48 hours postconcussion; asymptomatic, defined as participants beginning the RTP protocol; unrestricted RTP, defined as participants having completed all stages in the gradual RTP protocol; and 6 months postconcussion. For participants without concussion, we subtracted their first baseline K-D test time from their second baseline K-D test time (baseline 2 – baseline 1). Given that repeat assessments at time points corresponding to those of the concussion group were not available in the control group, the difference in baseline times was used for the control group relative to the concussed group's performance across the 4 time points, beginning with the 24- to 48-hour postinjury time point. The mean time between the first and second baseline for the nonconcussed, control group was 398 ± 1.8 days. For the concussed group, the mean time between baseline and 0- to 6-hour postconcussion time points was 286 ± 68.2 days, 0- to 6-hour and 24- to 48-hour postconcussion time points was 1 ± 1.8 days, 24- to 48-hour postconcussion and asymptomatic time points was 12 ± 4.3 days, and asymptomatic and unrestricted RTP time points was 5 ± 4.4 days.

For the second aim, we extracted the following targeted self-reported descriptive and medical history factors from baseline clinical report forms: sex (female, male), sport contact level (contact, limited contact, noncontact),²²  academic year (freshman, sophomore, junior, senior), learning disorder (no, yes), ADHD (no, yes), migraine history (no, yes), concussion history (0, 1, 2, 3+), and administration mode (iPad, spiral-bound cards).

To address our first aim, we used receiver operator characteristic (ROC) curve and area under the curve (AUC) analyses to determine the diagnostic accuracy of the K-D test across the 4 postconcussion time points with the Wilson-Brown method using GraphPad Prism (version 8.1.2). For AUC analyses that were different, we identified cutoff scores and corresponding specificities at both the 80% and 70% sensitivity levels but also included continuous levels of various sensitivity and specificity values in Supplemental Tables 1–5 (available online at http://dx.doi.org/10.4085/1062-6050-0063.21.S1).

To address our second aim, we repeated ROC curve with AUC analyses from the 24- to 48-hour time point segmented by sex (female, male), sport contact level (contact, limited contact, noncontact), academic year (freshman, sophomore, junior, senior), learning disorder (no, yes), ADHD (no, yes), migraine history (no, yes), concussion history (0, 1, 2, 3+), and administration mode (iPad, spiral-bound cards). We used the 24- to 48-hour postconcussion time point because a gain of 38 individuals with concussion compared with 0 to 6 hours postconcussion occurred. To determine whether the characteristics differed across confounding factor segments, we compared the 95% CIs around each AUC outcome. The AUC outcomes with scores of <0.50 were considered poor; 0.51 to 0.69, fair; 0.70 to 0.80, acceptable; 0.80 to 0.90, excellent; and >0.90, outstanding.²³  Values >0.55 were considered different. We evaluated the AUC 95% CI overlap and calculated Z and P values using the following formula: Z = abs(AUC1 − AUC2)/SQRT(⁠\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\bf{\alpha}}\)\(\def\bupbeta{\bf{\beta}}\)\(\def\bupgamma{\bf{\gamma}}\)\(\def\bupdelta{\bf{\delta}}\)\(\def\bupvarepsilon{\bf{\varepsilon}}\)\(\def\bupzeta{\bf{\zeta}}\)\(\def\bupeta{\bf{\eta}}\)\(\def\buptheta{\bf{\theta}}\)\(\def\bupiota{\bf{\iota}}\)\(\def\bupkappa{\bf{\kappa}}\)\(\def\buplambda{\bf{\lambda}}\)\(\def\bupmu{\bf{\mu}}\)\(\def\bupnu{\bf{\nu}}\)\(\def\bupxi{\bf{\xi}}\)\(\def\bupomicron{\bf{\micron}}\)\(\def\buppi{\bf{\pi}}\)\(\def\buprho{\bf{\rho}}\)\(\def\bupsigma{\bf{\sigma}}\)\(\def\buptau{\bf{\tau}}\)\(\def\bupupsilon{\bf{\upsilon}}\)\(\def\bupphi{\bf{\phi}}\)\(\def\bupchi{\bf{\chi}}\)\(\def\buppsy{\bf{\psy}}\)\(\def\bupomega{\bf{\omega}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\({\rm{SE}}_{{\rm{AUC1}}}^{\rm{2}}\) + \({\rm{SE}}_{{\rm{AUC2}}}^{\rm{2}}\)⁠), where abs is absolute and SQRT is square root. We calculated F and χ² values for group comparisons using SPSS (version 24; IBM Corp) and α = .05 (Table). We also conducted a multivariate logistic regression using these same confounding factors to predict false-negatives in the concussed group and a separate multivariate logistic regression to predict false-positives in the control group. To classify false-negatives and false-positives, we used the 80% sensitivity cutoff from the 24- to 48-hour postconcussion time point.

An initial sample of 1719 athletes was examined for eligibility, with 1559 athletes meeting the inclusion criteria. Of these 1559 athletes, 320 were assigned to the concussion group, and 1239 were assigned to the healthy control group (Table).

King-Devick difference scores predicted concussion diagnosis status at the 0- to 6-hour postconcussion (AUC = 0.724, P < .001), 24- to 48-hour postconcussion (AUC = 0.701, P < .001), unrestricted RTP (AUC = 0.640, P < .001), and 6-month postconcussion (AUC = 0.615, P < .001) time point but not at the asymptomatic time point (AUC = 0.513, P = .50). The ROC curves and outcomes across the 5 postconcussion time points are shown in Figure 1. The ROC raw data including sensitivity, specificity, likelihood ratios, and 95% CIs with corresponding K-D cutoffs are provided in Supplemental Tables 1–5 . The 0- to 6-hour and 24- to 48-hour postconcussion time points had acceptable discrimination for sensitivity and specificity, but other time points displayed fair discrimination. A cutoff score of −2.6 seconds (ie, performing 2.6 seconds faster than baseline) yielded 80% sensitivity to athletes with a concussion at the 0- to 6-hour postconcussion time point. However, this resulted in specificity of 46%. A cutoff score of −3.2 seconds (performing 3.2 seconds faster than baseline) yielded 80% sensitivity to athletes with a concussion at the 24- to 48-hour postconcussion time point. Nevertheless, this resulted in specificity of 41%. A cutoff score of −1.5 seconds (performing 1.5 seconds faster than baseline) at 24 to 48 hours postconcussion yielded 70% sensitivity, with a specificity of 57%. The ROC curves for the asymptomatic time point were not different; thus, cutoff scores are not proposed.

A cutoff score of −0.8 seconds (performing 0.8 seconds faster than baseline) at the unrestricted RTP time point yielded 80% sensitivity but specificity of 37%. A cutoff score of −1.9 seconds (performing 1.9 seconds faster than baseline) yielded 70% sensitivity but specificity of 47%.

At the 6-month postconcussion time point, a cutoff score of −0.4 seconds (performing 0.4 seconds faster than baseline) yielded 80% sensitivity but specificity of 34%. A cutoff score of −1.6 seconds (performing 1.6 seconds faster than baseline) yielded 70% sensitivity with specificity of 44%. Individuals with concussion tended to perform better at the 6-month postconcussion time point than at baseline.

Most confounding factors did not demonstrate differences in the AUC (P range = .06–.94; Figure 2), except for administration mode, such that the iPad administration mode (AUC = 0.800; 95% CI = 0.747, 0.854) had a greater AUC than the spiral-bound card system (AUC = 0.646; 95% CI = 0.600, 0.692; Figure 2H). We also observed a trend for learning disorder outcomes (P = .07) such that the K-D test score differences tended to have a better AUC for those with a learning disorder (AUC = 0.819; 95% CI = 0.715, 0.924) relative to those without a learning disorder (AUC = 0.716; 95% CI = 0.680, 0.752; Figure 2D). Similarly, confounding factors did not influence the odds of false-negatives (P = .65) in the concussed group. However, the K-D administration mode did influence the odds of false-positives in the control group (P = .002): iPad administration reduced the odds of false-positives (odds ratio = 0.59; 95% CI = 0.45, 0.76).

Our ROC curves at the 0- to 6-hour and 24- to 48-hour postconcussion time points showed an interesting pattern, whereby sensitivity was lower with slower postconcussion K-D times and specificity was higher. This pattern was reversed (Figure 1D and E) at the unrestricted RTP and 6-month postconcussion time points and followed a pattern in which sensitivity was higher with slower postconcussion K-D times and specificity was lower, which better matched the clinical interpretation. Clinicians should be aware that this pattern does not follow conventional clinical patterns, despite an AUC that suggests overall moderate diagnostic accuracy.

Based on our results, the K-D test contributed acceptable diagnostic accuracy at 0 to 6 hours and 24 to 48 hours postconcussion. This is concurrent with the designer's recommendation that the test be used as a remove-from-play test.⁵  However, the test is not sufficient to stand alone, which supports the use of a multifactorial approach. Additionally, as expected, the K-D test had poor to no diagnostic accuracy at the asymptomatic time point. The diagnostic accuracy was fair at the unrestricted RTP and 6-month postconcussion time points, which was indicative of the K-D test correctly predicting concussion diagnosis and magnitude of practice effects. At these time points, athletes with a concussion have cleared previous time points to return to sport. The RTP timepoint has been associated with modest improvements in K-D test performance due to participating in physical activity and improved cognitive performance.²⁴ 

Practice effects greatly influence repeat K-D test performance and clinical interpretation across repeated administrations. Our results may show that a practice effect (leading to faster completion of the K-D test) has a stronger influence on postconcussion K-D test scores compared with a concussion effect (which in theory is associated with slower completion of the K-D test). Athletes with a concussion take the test 4 times in 6 months, whereas healthy athletes take the K-D test only 2 times in 1 year. Clinicians should expect considerable improvement in completion time (ie, faster) with subsequent K-D test administrations.¹⁵  A faster K-D time is a product of a practice effect, and caution should be used when interpreting the outcomes across time points for individuals with concussion. In addition, our data revealed recommended cutoff scores at each time point to improve the clinical interpretation of the K-D test during concussion recovery.

To achieve 80% sensitivity, clinicians should expect a 1.0- to 2.6-second faster time than the baseline time within 0 to 6 hours postinjury, a 1.0- to 3.2-second faster time at 24 to 48 hours postconcussion, and a 1.0- to 1.9-second faster time at unrestricted RTP. Similar to our results, Dhawan et al²⁵  found that a cutoff time of 2 seconds faster than baseline yielded high sensitivity (90%) and specificity (91%) in adolescent athletes, which is notably higher than our observed level of specificity. However, this study had a smaller sample size, with only 20 participants diagnosed with a concussion and 121 participants without a concussion.²⁵  Sensitivity is calculated as the number of true positives divided by the total number of individuals with a concussion. Therefore, a smaller sample size of those with a concussion could artificially inflate the sensitivity value. Other researchers have shown 0.7- to 2.5-second improvements in K-D times in healthy athletes between 2 baseline measurements.^5,15,26  Breedlove et al¹⁸  determined the K-D test to be reliable between trials and years. Retesting with a 1-year interval revealed a small improvement of 2 seconds among intercollegiate athletes.¹⁸  However, 27% of athletes showed a slower performance from year 1 to year 2.¹⁸  Overall, these previous studies and our results confirm that athletes should perform considerably faster relative to their baseline time with subsequent K-D testing. Repeated tests will cause practice effects and improvement of scores, regardless of injury.¹⁸ 

Confounding factors largely did not influence K-D test diagnostic accuracy postconcussion. The K-D test AUC for those with a learning disorder was slightly higher (AUC = 0.819) relative to that of those without a learning disorder (AUC = 0.716), although the CIs overlapped (Figure 2D). The K-D test was originally developed in 1976 to study eye movement in conjunction with reading ability to screen children for learning disorders, such as dyslexia.^27,28  Our findings suggested that a learning disorder does not influence the diagnostic accuracy of the K-D test much when referenced to a preinjury baseline. Future research is needed to examine whether learning disabilities influence diagnostic accuracy in the absence of a baseline assessment and in a larger sample, as ours included only 15 athletes with concussion who had a diagnosed learning disorder.

The K-D test time and errors decrease (improved performance) as age increases among high school, collegiate, and professional athletes.¹⁴  However, we found no difference in performance by academic year (freshmen to seniors). Investigators¹⁴  have also suggested that sex influences K-D test performance, whereby males tend to score worse or slower. However, our results indicated no differences across sexes. We also did not identify differences in AUC among those with ADHD. A history of concussion did not influence K-D test performance, which had not been studied earlier.

Administration mode did affect AUC outcomes. The iPad version outperformed the card system. A shift has occurred from the card system to the iPad version because the former is no longer commercially available.²⁹  Researchers^29,30  have also recommended using the iPad version over the cards because the former has higher test-retest reliability and improved testing standards to minimize errors from administrators. Institutions should use the iPad version to avoid inappropriately diagnosing individuals with concussion when they do not have one.

Lastly, physical activity such as aerobic fitness may have been another confounding factor contributing to our results at the unrestricted RTP and 6-month postconcussion time points but not at the asymptomatic time point activity, with authors³¹  demonstrating improved cognitive performance after a bout of exercise. The 24- to 48-hour postconcussion time point likely occurs before athletes are asymptomatic, and thus, before physical activity resumes. Our results showed the diagnostic accuracy decreased to poor levels at the asymptomatic time point and then improved to fair levels at the unrestricted RTP time point. Physical inactivity may hinder cognitive performance, which was reflected in our outcomes.

Although our study included a large multisite sample of athletes with concussion and control athletes, it had limitations. These findings were limited to those who were participating only at these sites. In addition, we used the difference in baseline times for the control group across the 4 postconcussion time points of the concussed group because repeat testing was not available for the control group. Sample sizes for the subanalyses were much smaller, which was also a limitation. Future researchers should consider using the K-D test in healthy matched individuals across the same time points as their counterparts with concussion.

We examined a large sample of collegiate athletes from multiple sites to determine the diagnostic accuracy of a test commonly used to evaluate concussions. Based on our findings, the K-D test contributed acceptable diagnostic value at the 24- to 48-hour postconcussion time point, but it was not adequate at the asymptomatic time point and was only fair at the unrestricted RTP time point. Confounding characteristics and medical history factors were not different across the same time points. These results suggest that the K-D test should not be used as a standalone assessment at any time point. Instead, the K-D test should be included as 1 part of a comprehensive assessment battery.

Contributing CARE Consortium Investigators were April Marie (Reed) Hoy, MS, ATC (Azusa Pacific University); Darren Campbell, MD (Brigham Young University); Louise A. Kelly, PhD (California Lutheran University); John DiFiori, MD (Hospital for Special Surgery, National Basketball Association); Justus D. Ortega, PhD (Humboldt State University); Nicholas Port, PhD (Indiana University); Margot Putukian, MD (Major League Soccer); T. Dianne Langford, PhD, and Jane McDevitt, PhD, ATC, CSCS (Temple University); Christopher C. Giza, MD (University of California, Los Angeles); Holly J. Benjamin, MD (University of Chicago); Thomas W. Kaminski, PhD, ATC (University of Delaware); James R. Clugston, MD, MS (University of Florida); Joseph B. Hazzard Jr, ATC (University of Houston-Clear Lake); Patrick G. O'Donnell, MHA (University of Massachusetts Memorial Medical Center); Luis A. Feigenbaum, PT, DPT, ATC (University of Miami); James T. Eckner, MD, MS (University of Michigan); Jason P. Mihalik, PhD, CAT(C), ATC (University of North Carolina at Chapel Hill); Christina L. Master, MD (University of Pennsylvania); Anthony P. Kontos, PhD, and Michael Collins, PhD (University of Pittsburgh Medical Center); Sara P.O. Chrisman, MD, MPH (University of Washington); Alison Brooks, MD, MPH (University of Wisconsin-Madison); Jonathan Jackson, MD, and Gerald McGinty, PT, DPT (United States Air Force Academy); Carlos Estevez, DPT, OCS, ECS (United States Coast Guard Academy); Kenneth L. Cameron, PhD, MPH, ATC (United States Military Academy); Adam Susmarski, DO (United States Naval Academy); Christopher M. Miles, MD (Wake Forest University); and Laura Lintner, DO (Winston-Salem University). We thank all research team members and clinical athletic trainers who helped in the data collection for this study.

This study was made possible, in part, by support from the Grand Alliance CARE Consortium, funded in part by the National Collegiate Athletic Association and the Department of Defense (Dr Kontos and S.P.B.). The US Army Medical Research Acquisition Activity, 820 Chandler Street, Fort Detrick, MD 21702-5014, USA, is the awarding and administering acquisition office. This work was supported by the Office of the Assistant Secretary of Defense for Health Affairs through the Psychological Health and Traumatic Brain Injury Program under award W81XWH-14-2-0151. Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the Department of Defense (Defense Health Program funds).