Minimal Detectable Change for the ImPACT Test Administered Remotely Open Access

A total of 172 NCAA Division I student-athletes (127 women and 45 men) participated in this study (Table 1). Their average age was 19.37 ± 1.24 years at the time of the first remote ImPACT administration and 20.25 ± 1.29 at the time of the second remote test. The time between tests was 302 ± 46.35 days. We recruited participants from the same athletic department if they had completed the NCAA-mandated preseason screening for 2 consecutive years. Successful completion of preseason screening was defined as a valid, uncontrolled remote ImPACT baseline assessment. Although we focused on baseline ImPACT measures, all participants also underwent vestibular and vision testing. Invalid scores were marked by ImPACT, reviewed by a CIC, and excluded from the study. The ImPACT marks results as invalid if test takers between 14 and 59 years of age receive an impulse control composite score of >30, word memory learning percentage correct <69%, design memory learning percentage correct <60%, or a 3 letters total letters correct score of <8. Exclusionary criteria were a self-reported history of attention-deficit/hyperactivity disorder (ADHD) and dyslexia. Originally, 195 participants were identified; however, 19 were excluded due to a self-reported ADHD diagnosis and 1 due to a self-reported dyslexia diagnosis. The exclusionary criteria were designed for consistency with earlier research on this topic, as both ADHD and dyslexia can affect ImPACT performance.^13–15  Although not all authors cited dyslexia in their exclusionary criteria, individuals with a self-reported dyslexia diagnosis scored lower on all ImPACT cognitive measures.¹⁴  Their results also indicated the need for more specificity and diversity among the sample population that ImPACT used to create its norms, which we did not address in our investigation.

The largest cohorts of athletes participated in cheerleading (20.4%), cross-country or track and field (17.7%), soccer (10.3%), and swimming and diving (10.3%), with the remaining athletes participating in baseball, basketball, football, golf, softball, volleyball, and tennis (Table 1). This study was conducted in compliance with the University of Nevada, Reno, Institutional Review Board.

The ImPACT is a computer-based neuropsychological diagnostic tool traditionally used as both a baseline and postinjury concussion assessment and is one of the most widely used neurocognitive assessment batteries in athletics. The NCAA recommends administering a neurocognitive examination before the athletic season and again as a postinjury assessment if a health care provider, such as an athletic trainer, believes that an athlete might have sustained an SRC.

The examination highlights 5 areas of cognitive functioning proven to be impaired during a concussion: verbal memory, visual memory, processing speed, reaction time, and impulse control. The test takes just under 30 minutes and can be administered in both proctored and uncontrolled remote environments.⁵  The clinical report consists of 5 composite scores: verbal memory, visual memory, visual-motor speed, reaction time, and impulse control. In addition to the neurocognitive assessment, 22 self-reported postconcussion symptoms are measured on a 7-point Likert scale, and self-reports of relevant medical history are obtained.

Because of clinicians’ reliance on ImPACT to make an informed SRC diagnosis, the temporal stability of ImPACT has been assessed on numerous occasions with inconsistent results. Some researchers found good test-retest reliability,^9,10,16  some found low to moderate test-retest reliability,^13,17  and others found poor reliability in specific assessment categories.^7,8,18 

Participants were required to complete baseline ImPACT tests before 2 consecutive athletic seasons and before any preseason competition (average of 302 ± 46.35 days between tests). At both times, we provided participants with a unique testing link and instructions to complete the test in an uncontrolled remote environment. Per ImPACT administration guidelines, we also instructed participants to (1) find a quiet location for testing, (2) use a computer with internet access, (3) not have engaged in a strenuous workout in the previous 3 hours, (4) take the test in their native language, (5) maintain silence and stow mobile devices during the examination, (6) not ingest alcoholic beverages before testing, (7) have eaten recently, and (8) have slept ≥6 hours the night before. We asked the athletes to complete the initial demographics, background, and medical history sections before starting ImPACT, but they were not required to complete the additional demographics section.

Although we requested participants complete the test in their native language, some individuals were inconsistent in their language choices, completing 1 test in their native language and 1 in English. The mismatch in testing language was considered during participant recruitment, but ultimately the participants’ scores were included if their ImPACT scores were not marked invalid. The native languages selected at the time of the first test were English (n = 159), French (n = 5), Czech (n = 1), Portuguese (n = 1), Polish (n = 2), Spanish (n = 2), Russian (n = 1), and German (n = 1). For the second test, the native languages selected were English (n = 162), French (n = 4), Czech (n = 1), Polish (n = 1), Spanish (n = 2), Russian (n = 1), and German (n = 1). Increased English testing at time point 2 (T2) could reflect the participants’ attendance at an English-speaking university for ≥1 year, whereas at time point 1 (T1), they may have recently moved from a non–English-speaking country and perhaps did not feel comfortable taking the assessment in English.

Invalid ImPACT results were reviewed by a CIC. Invalid results are often caused by a lack of understanding of the instructions or a lack of effort.¹⁹  A total of 15 participants received an invalid ImPACT result during the data-collection window; all were retested ≥48 hours after their invalid test. Despite results marked valid by ImPACT, 3 participants were excluded from the study by a CIC, as their T1 outcomes reflected a lack of effort and were inconsistent with the results of their second test. Only valid ImPACT results of athletes who were required to retake the test were analyzed.

All statistical analyses were performed using SPSS (IBM Corp). The ImPACT composite scores for verbal memory, visual memory, visual motor speed, reaction time, and impulse control were evaluated for both skewness and kurtosis. Visual memory, verbal memory, and visual motor speed composite scores were parametric and were analyzed using a 1-way repeated-measures multivariate analysis of variance (ANOVA). Reaction time scores were parametric but did not have the same scoring scale as visual memory, verbal memory, and visual motor speed; therefore, a 1-way repeated-measures ANOVA was performed. Impulse control scores were nonparametric and as such, were evaluated separately using a Wilcoxon signed rank test. The α value was set to .05.

The mean values for each of the ImPACT composite scores at each time point were as follows: 90.68 ± 8.14 at T1 and 92.1 ± 8.11 at T2 for verbal memory, 80.47 ± 11.89 at T1 and 91.94 ± 12.02 at T2 for visual memory, 41.18 ± 5.81 at T1 and 41.63 ± 5.86 at T2 for visual-motor speed, 0.60 ± 0.07 at T1 and 0.61 ± 0.08 at T2 for reaction time, and 4.22 ± 2.89 at T1 and 4.54 ± 2.90 at T2 for impulse control. Mean and SD values are shown in Table 2.

The multivariate model for verbal memory, visual memory, and visual motor speed composite scores was not significant (F_3,169 = 2.12, P = .10). This indicates that the composite scores did not differ between the 2 testing time points.

Mean scores for the reaction time composite score were not different (F_1,171 = 1.02, P = .31). This suggests no overall difference between the mean reaction times at the 2 testing time points.

Impulse control scores did not differ across the 2 testing periods (Z = –1.19, P = .23). The median impulse control score was 4.00 at both time points.

For each ImPACT composite score, the MDC was calculated at 80%, 90%, and 95% CIs. The MDC values were MDC₉₅ = 18.6, MDC₉₀ = 15.66, and MDC₈₀ = 12.17 for verbal memory; MDC₉₅ = 24.44, MDC₉₀ = 20.57, and MDC₈₀ = 15.99 for visual memory; MDC₉₅ = 8.76, MDC₉₀ = 7.73, and MDC₈₀ = 5.73 for visual motor memory; MDC₉₅ = 0.14, MDC₉₀ = 0.12, and MDC₈₀ = 0.09 for reaction time; and MDC₉₅ = 6.13, MDC₉₀ = 5.16, and MDC₈₀ = 4.01 for impulse control (Table 3).

The purpose of our study was to determine the MDC of remotely administered ImPACT in Division I student-athletes across 2 consecutive athletic seasons. All 5 ImPACT composite scores were similar to those previously described in the literature at both remote testing time points.^5,7  The MDC values at the 95% CI were similar to those in an earlier investigation; however, some discrepancies could be attributed to differences in population, statistical methods, and test administration.⁷  Minimal detectable change values are meant to serve as a clinician’s guide to relevant changes in performance for repeated tests. Increased understanding of meaningful changes in ImPACT scores will aid clinicians in determining if score changes are due to decreases in neurologic function as a consequence of concussion or natural fluctuations in performance. Our hopes are to offer a clinician’s guide for interpreting remotely administered ImPACT baseline score changes and recommend that the MDC be used for future evaluations of ImPACT test-retest reliability. This research could be furthered by examining the differences in composite scores across various sports. In addition, with the ongoing coronavirus disease 2019 pandemic, remote ImPACT administration has become essential to the functioning of many athletic departments. Therefore, more exploration of the test-retest reliability of remote ImPACT is needed to confirm these findings.

Our ImPACT composite scores and SDs were similar to those in the literature^5,7  and similar to optimal ImPACT scores (Table 2).⁶  The only area in which these values notably differed was the visual memory composite score: our participants’ average score was 80.47 versus previously observed values of 77.67⁵  and 76.⁷  Our participants’ scores were higher and closer to optimal values (80). The differences in composite scores among these studies could be attributed to the larger sample size or differences in test administration.

The 1-way repeated-measures multivariate ANOVA, repeated-measures ANOVA, and Wilcoxon signed rank tests showed no differences in ImPACT composite scores across the 2 testing time points. Earlier authors examined in-person ImPACT administration⁷  or a combination of remote and in-person administration⁵  with no changes across the testing time points. Our findings suggest that the remote ImPACT test can be administered at home with written instructions and result in satisfactory scores between testing time points. Two sets of researchers also examined ImPACT test results from 2 consecutive athletic seasons; therefore, combined with our results, ImPACT appears to be reliable across 2 testing athletic seasons or an approximate calendar year.

The MDC results were similar to those of Mason et al⁷  who used bootstrapping on a smaller cohort (n = 48) of mostly male (64%) NCAA Division I athletes. We found MDC values at the 95% CI to be 18.6 for verbal memory (Mason et al⁷  = 14.19), 24.44 for visual memory (Mason et al⁷  = 17.25), 8.76 for visual motor (Mason et al⁷  = 11.07), 0.14 for reaction time (Mason et al⁷  = 0.17), and 6.13 for impulse control (Mason et al⁷  = 7.38; Table 2). Our participants differed significantly from those in the earlier study with a larger (n = 172) and predominantly female (73.8%) cohort. In addition to differences in populations, we obtained data from unsupervised, remote ImPACT, whereas Mason et al⁷  had certified athletic trainers administer the test. We calculated the MDC using the SEM, matching previous MDC literature.^21,22  Mason et al⁷  did not explicitly state how they calculated their MDC; thus, the methods may not be identical. In addition, the more compact range of composite scores of Mason et al⁷  may have reflected increased overall effort and an ideal, supervised testing environment. We instructed participants to find a quiet, distraction-free environment for their test, but whether they followed those directions is unknown. The differences in population and testing environments most likely account for the discrepancies in MDC values.

Due to clinicians’ reliance on ImPACT to make an informed SRC diagnosis, the temporal stability of the test has been assessed on numerous occasions with inconsistent results. Some researchers found good test-retest reliability,¹⁶  some found low to moderate test-retest reliability,^13,17  and others found poor reliability in specific assessment categories.^7,8,18  We did not aim to evaluate the effectiveness of ImPACT in terms of test-retest reliability. Given the similarity in scores across both time points to similar studies,^5,7  our results seem to suggest that ImPACT is reliable across 2 remote time points. Even so, it is best practice to conduct postinjury ImPACT assessments in person to aid in ensuring an ideal testing environment and allow administrators to monitor the test takers’ symptoms, if needed.

This investigation has limitations related to participants’ demographics. Sports such as football and soccer traditionally present a higher risk for concussion; however, they represented a small portion of our participants (2.3% and 10.5%, respectively). During the study period, the football and soccer teams completed in-person ImPACT evaluations via batch testing and thus could not be included. The sport with the largest representation was cheerleading (23.3%), which is typically classified as high risk for head impact. Also, the cohort was largely female (73.8%); nevertheless, because the composite scores and variability seen here were similar to those reported earlier, sex and ImPACT performance may not be correlated. Men perform slightly better on verbal working memory tasks, but our participants’ average verbal memory composite score was in the optimal range of 90 to 99 across testing time points, suggesting that sex effects were not prominent.²³  We relied on self-reported ADHD and dyslexia diagnoses, and athletes with a diagnosis of ADHD or dyslexia might have been included if they did not disclose their diagnosis when completing ImPACT at either time point. This study did not exclude participants with a concussion history, but data from postinjury ImPACT were removed. An additional limitation was that not all participants used the same testing device across both testing time points. Some individuals used a mouse for their first test and a trackpad for the second test or vice versa, which may have influenced speed scores. As noted previously, whether participants took the examination in a distraction-free environment is unknown. Although invalid scores were excluded, it is possible that participants scored well enough to receive a valid score but did not put all their cognitive effort into taking the test.

Consistent with the existing literature, ImPACT composite scores did not differ between the remote testing time points across 302 ± 46.35 days, indicating that ImPACT results did not differ when testing was self-administered. As the MDC examines the meaningful change in repeated test scores, we suggest that medical professionals should use the MDC as a guideline for evaluating changes in baseline ImPACT scores and making clinical judgments regarding SRCs.