Acute Sport-Related Concussion Screening for Collegiate Athletes Using an Instrumented Balance Assessment

This study was part of a multicenter clinical trial (ClinicalTrials.gov Identifier: NCT02477943) conducted across 10 collegiate sites in the United States by BrainScope. Seven of the 10 sites had contributed data on balance and symptom assessments when the data for this study were compiled. Each site received approval from its university's institutional review board before any individuals were recruited or tested. All participants provided written informed consent, and the ability to provide informed consent was evaluated as necessary. In addition, BrainScope and its contract research organization (Brain Injury Outcomes Division, Johns Hopkins University, Baltimore, MD) monitored the study for compliance with the clinical protocol and applicable regulations and verification of data quality. This was accomplished through frequent communication with each site, biweekly conference calls with all study members, and periodic site visits.

Participants were recruited from 1 or more collegiate or university varsity or club sport teams at each site. Volunteers were included if they were aged 18 years or more and participated in intramural (varsity or club) collegiate athletics. Participants in the SRC group experienced an SRC within 72 hours of study enrollment. Exclusion criteria consisted of loss of consciousness for more than 20 minutes; abnormality visible on computerized tomography of the head related to the traumatic event (neuroimaging was not required for enrollment); hospital admission due to either head injury or collateral injuries for more than 24 hours; evidence of illicit drug usage; inability to speak or read English; use of prescription medications active on the central nervous system, except medications to treat attention-deficit disorder, at the time of the study; skull abnormality (eg, metal plate); or a history of brain surgery or neurologic disease. In addition, categories of vulnerable participants, such as prisoners, wards of the state, minors (age <18 years), and pregnant women, were not included. A history of SRC did not exclude matched control participants if they were fully recovered at the time of the study. When an athlete with SRC was enrolled in the study, a healthy athlete was also enrolled as a control participant. Matching control participants based on age, sex, race, and academic year when possible was a goal. Recruiting an equal number of contact- and noncontact-sport athletes for the control group was also a goal for the larger multicenter study from which our data were derived. Ideally, contact-sport athletes in the control group were further matched to participants in the SRC group by sport and position played.

A total of 93 participants (age = 20.73 ± 1.47 years, age range = 18–26 years) were included in the study based on the available balance data from the larger clinical trial. Participants with SRC (n = 48) were recruited from a variety of teams: football (n = 16), soccer (n = 10), lacrosse (n = 7), rugby (n = 5), ice hockey (n = 4), and other (n = 6). They consisted of 35 (72.92%) men and 13 (27.08%) women, whereas control participants (n = 45) consisted of 28 (62.22%) men and 17 (37.78%) women.

The demographic characteristics by group and sex are shown in Table 1. We observed demographic differences between the SRC and control groups for height (P = .03) and body mass index (P = .03) and between men and women for height (P < .001), mass (P < .001), and body mass index (P = .002). Differences existed between the SRC and control groups for total number of symptoms reported per person (P < .001) and symptom severity score per person (P < .001). We observed a difference between the SRC and control groups for self-reported balance problems at the time of testing (P < .001). A total of 17 of 48 (35.42%) participants with SRC and 2 of 45 (4.44%) control participants self-reported balance problems.

The larger multicenter clinical trial²⁹  included the following measures as part of the BrainScope battery of tests: memory and symptom assessment using the Sport Concussion Assessment Tool-3 test,²⁷  qEEG testing using the Ahead 200id device, cognitive assessment using a tablet computer, and an instrumented balance and sway assessment. However, we focused only on results from the instrumented balance assessment.

Medical professionals at each site diagnosed the patients with SRC as part of their regular duties. Study testing sessions were ideally accomplished within 24 hours of injury or within a maximum of 72 hours postinjury. This testing window was chosen so that we could evaluate test participants with SRC while they were symptomatic but also as a practical time frame for injury-recruitment purposes. Balance and sway were assessed using a commercially available inertial sensor (model Opal; APDM Wearable Technologies, Inc, Portland, OR) placed near the L4-L5 vertebral level and secured with an elastic belt. The inertial sensor calculated measures of postural sway using a 3-axis accelerometer, gyroscope, and magnetometer in addition to a temperature sensor. Mobility Lab software (APDM) wirelessly transferred data to a tablet computer (model 1631 Surface Pro 3; Microsoft Corp, Redmond, WA) equipped with a Bluetooth (Bluetooth SIG, Inc, Kirkland, WA) access point. Testing followed a procedure inspired by the mBESS, which involves 3 tests performed on flat ground with the participant's eyes closed and shoes off. Unlike the standard mBESS, all tests were performed for 30 seconds instead of 20 seconds, following APDM protocols, and the comparisons in the statistical analysis were derived from the methods rather than from practitioner-counted errors. These tests, which were performed in serial order, consisted of standing with 2 feet side by side (double-legged stance), standing on the self-reported nondominant foot (single-legged stance), and standing with the dominant foot forward and heel and toe touching (tandem stance). The double-legged stance was standardized using a 1.5-in (3.8-cm)–thick foam wedge placed between the feet, which created a 17° angle. Tests were administered only once per stance; therefore, the total length of the balance assessment was 1 minute and 30 seconds plus time to rest and reposition for each subsequent balance test. The administrator monitored the test to ensure participant safety.

A total of 26 balance measures were calculated and output using algorithms and Mobility Lab software. We focused our analysis on 2 measures: RMS sway and 95% ellipse sway area. These measures were collected and examined for each of the 3 balance stances, yielding 6 balance measures. All 26 measures were reviewed in our study sample, whereas the 2 selected measures represented general balance and sway measures. These measures also limited some redundancy (eg, RMS sway total and RMS sway in each cardinal direction), so that our findings could be used to suggest future analyses with other measures from the 26 not included. Descriptions of each measure and its calculation have been provided in previous publications.^30,31  The measures are summarized as follows: RMS sway is the RMS of the sway trajectory, and the 95% ellipse measure is derived from the smallest ellipse that will cover 95% of a participant's posturogram. Clinically, poor balance causes both of these measures to increase. When individuals have poor balance, their center-of-mass movement increases, causing the RMS sway measurement to increase. Similarly, more center-of-mass movement covers a larger amount of space, resulting in larger 95% ellipse measurements. The opposite is true for people with good balance, causing RMS sway and 95% ellipse measurements to remain low.

Data were compiled and analyzed to evaluate the normality, distributions, and variance from the graphical presentation. To compare groups across the 3 balance tests, a mixed-effects analysis of variance was used with a within-subject factor of balance test (3 levels: double-legged, single-legged, and tandem stances) and between-subjects factor of group (2 levels: SRC, control). This was repeated for the 2 dependent variables that were examined during the instrumented balance test. A Greenhouse-Geisser correction was used to correct for violations of sphericity. To examine the relative sizes of differences between groups for these understudied metrics, the partial η²  effect size was computed for each metric. Effect sizes were generally considered small (0.01), medium (0.06), or large (>0.14).³²  If we found an interaction, we ignored the main effects and conducted the pairwise comparisons between groups and tests using a Bonferroni correction for repeated comparisons. The α level was set initially at .05 for all comparisons. To gain further insight into the differences between groups, we calculated the sensitivity and specificity of an acute SRC diagnosis using an inertial balance sensor. Cutoff values for determining false-positives and false-negatives were set at the healthy mean for each metric plus 0.5 SD, 1 SD, 1.5 SD, and 2 SD. Multiple cutoff values were explored because the balance metrics derived from the inertial sensor across these 3 balance tests have not been explored and no known cutoff has been accepted for interpreting clinical sensitivity or specificity for this population with SRC. Lower scores indicate better balance performance for both measures analyzed, so only scores greater than the mean were considered abnormal. Statistical comparisons were completed using SPSS (version 22; IBM Corp, Armonk, NY).

We observed 1 interaction (P = .02) between group and stance after using a Greenhouse-Geisser correction on the 95% ellipse sway area measure. Given this interaction, we ignored the main effects for that measurement and evaluated group differences for each balance test. For the RMS sway measurement, a main effect for group was found (F_1,91 = 11.75, P = .001, partial η² = 0.114). This was evident from the increased balance measures in the SRC group, which indicated decreased balance performance. For the 95% ellipse sway area measurement, we observed a main effect for stance (F_1,91 = 55.8, P < .001), with the higher values indicating greater balance disturbance in single-legged stance, then tandem stance, and then double-legged stance. The between-groups pairwise comparisons for the 95% ellipse sway area measure showed group differences, with decreased balance performance by the SRC group and the double-legged (F_1,91 = 11.59, P = .001), single-legged (F_1,91 = 6.91, P = .01), and tandem (F_1,91 = 7.54, P = .007) stances; however, the size of these differences varied among stances, with the largest difference between groups seen for the double-legged test and the smallest difference for the single-legged test. Balance scores were averaged across balance stances for measures with no interactions (RMS sway) because the effects of between-groups differences were consistent across all 3 stances. All group values are reported in Table 2.

Scatter plots of each measure were generated to gain greater insight into group differences (Figure). We visually inspected them for abnormalities in the distribution of the data and between-groups overlap for each measure. We found an overlap between the SRC and control groups for both measures, suggesting that sensitivity values would be low despite group differences.

Sensitivity and specificity for all 4 cutoff values are reported in Table 3. Increasing the cutoff values resulted in lower sensitivity but higher specificity for each metric. The control group was correctly categorized with more than 90% accuracy for both balance measures when using a 2-SD cutoff value. The highest sensitivity (54%) was found using the RMS sway metric and a 0.5-SD cutoff value. The highest specificity (98%) occurred when using the RMS sway metric with a 2-SD cutoff value.

The primary aim of our study was to determine if group differences in objective measures of balance existed between a sample of collegiate athletes with acute SRC and a healthy control group. We found between-groups differences for both balance measures analyzed, suggesting the clinical utility of inertial-sensor balance testing. Our secondary aim was to use the sensitivity and specificity of an inertial-sensor sway balance assessment to determine its diagnostic utility for assessing acute SRC. A visual representation of the data (scatter plots for each group and measure) suggested overlap between groups for both balance measures. Sensitivity and specificity calculations showed that these balance measures had low sensitivity and high specificity for SRC diagnosis, which was consistent with the overlap observed across groups.

We observed between-groups differences for both balance measures. The increase in average scores of each measure in the SRC group compared with the control group indicated a decrease in balance performance for the former. However, our SRC group demonstrated a smaller increase in sway than the control group in a previous study. King et al,²³  who also used instrumented mBESS scores to measure balance performance, reported a 60.0% increase in average RMS sway scores in an SRC group compared with a matched control group. In our study, RMS sway scores were 37% higher, on average, for the SRC (1.84 ± 0.884) than for the control (1.34 ± 0.431) group. However, the more moderate 37% increase in RMS sway scores that we found might be more representative of testing patients with acute SRC, whereas King et al²³  tested patients with persistent balance problems from an outpatient clinic. In addition, we used a 3-axis inertial sensor, whereas King et al²³  used a 2-axis sensor, which perhaps also contributed to the differences noted.

The interaction between group and stance using the 95% ellipse sway area measure suggested that SRC may lead to balance impairments that can be better identified with one balance test over another. For that measure, the effect size was greatest during the double-legged stance (partial η² = 0.113) and least for the single-legged stance (partial η² = 0.071), suggesting that the double-legged stance is preferred for instrumented balance assessments. Other researchers have also questioned using some of the balance stances included in the full BESS. For example, Hunt et al³²  proposed eliminating the double-legged stance from the BESS due to low variance when using a noninstrumented approach. They reported a 0.17% variance for the double-legged stance compared with a 29.43% variance for the single-legged stance in healthy athletes.³²  Our findings suggest that, during an instrumented balance assessment, the increase in variance during the single-legged stance may hinder the ability to separate patients with SRC from a matched control group. Perhaps the key difference between the instrumented and noninstrumented mBESS tests may be that the instrumented mBESS test captures small balance changes and the noninstrumented mBESS test relies on balance changes visible to the human observer.

Overall, the instrumented balance measures had relatively low sensitivity and high specificity. Our results showed that a lower cutoff value can be used to increase clinical sensitivity during an instrumented balance test in patients with acute SRC, but then specificity decreases. Similarly, a higher cutoff value can be used to increase specificity at the expense of sensitivity during an instrumented balance test in patients with acute SRC. Clinically, high specificity would result in few healthy patients testing positive (false-positives) but more concussed athletes testing negative despite being injured (false-negatives). Both balance measures correctly categorized the healthy population with high accuracy when using a 2-SD cutoff value. However, even the most sensitive measure using a low cutoff value (RMS sway at 0.5 SD) would have falsely identified 45.83% (22/48) of the SRC group as healthy. Furthermore, both measures miscategorized some matched control participants even using a high (2-SD) cutoff value, with a maximum of 4 (8.89%, 95% ellipse sway area) and a minimum of 1 (2.22%, RMS sway) false-positives. Given the known risk of second-impact syndrome⁴  and the greater risk of reinjury shortly after an initial concussion,²  the high false-negative rate for this balance assessment is concerning; however, having only minimal false-positives indicates that the negative effect of an incorrect diagnosis would happen infrequently. Using balance testing as only 1 part of a multicomponent screening that includes neurocognitive testing and symptom reporting has been widely recommended^5,7,11  and suggests that improved sensitivity via broader screening is a goal.

High specificity of instrumented balance assessment using an mBESS-type test agrees with previous findings on noninstrumented balance performance. In a literature review of balance assessment after SRC, Murray et al³³  reported that the BESS test had sensitivity of 34% and specificity of 87%. However, they³³  also noted that the reliability of the noninstrumented BESS test was limited by “rater interpretation.” Instrumented balance assessments may address concerns related to rater interpretation. Researchers have also suggested the possibility of ceiling²³  and floor³²  effects during the BESS. Avoiding these effects during concussion assessment with inertial sensor technology could allow for novel or more precise measurements during SRC recovery. Whereas instrumented balance measures may provide a solution to these test challenges, further research is needed to explore the clinical utility of the instrumented BESS compared with the noninstrumented BESS when assessing patients with acute SRC, as these data were not available in our study.

This finding of high specificity but low sensitivity may be further explained by the varied nature of SRCs. This is in agreement with the Berlin Conference on Concussion in Sport,³  which stated that SRC can include 1 or more of the following clinical domains: symptoms, physical signs, balance impairment, behavioral changes, cognitive impairment, [or] sleep or wake disturbance. We explored only instrumented balance measures, yet the SRC group included participants whose diagnosis was made using the full criteria of SRC assessments. Other assessments, such as neurocognitive testing or qEEG, may add to the balance testing described here for use in an SRC screening battery. Barr and McCrea³⁴  found 94% sensitivity and 74% specificity in diagnosing concussions using the Standardized Assessment of Concussion. Schatz and Sandel³⁵  reported that the online version of the Immediate Post-Concussion Assessment and Cognitive Testing computerized neurocognitive test had 91% sensitivity and 69% specificity. Although instrumented balance testing alone may not have clinical value for screening, using it in conjunction with other highly specific tests may be valuable for decreasing the number of athletes with SRC who are misdiagnosed as healthy.

A number of authors^14,36,37  have emphasized the importance of high-quality SRC research as an advancing field of study. They highlighted increased sample sizes, inclusion of baseline testing, and improved recruitment of control participants as potential areas for improving SRC research designs.^14,36,37 

Our study had several limitations. We used a cross-sectional cohort design in the absence of baseline testing. Preparticipation physical examinations are common in collegiate athletics,³⁸  and researchers may consider including baseline tests while investigating objective measures of balance. We also compared participants with SRC and control participants engaged in contact and noncontact sports; however, future researchers could identify control participants using different criteria, such as matching to sport and position. We also did not exclude control participants based on SRC history. Furthermore, the SRC group comprised only collegiate-level athletes, whereas concussion screening is used across both younger and older athletic populations. Also, the samples were not evenly distributed by sex. Injury presentation and recovery have been shown to be influenced by sex,^39,40  so additional studies are needed to determine the influence of sex on instrumented balance measures. Our study was limited in scope because we only investigated acute SRC among collegiate athletes with 1 commercially available inertial sensor. Other inertial sensors may produce different results, particularly those using alternative methods or algorithms to calculate balance metrics. Furthermore, as mentioned, subsequent authors should be aware of the 10-second difference in test length of the inertial-sensor protocol that we used compared with the standard mBESS procedure. Larger longitudinal studies with more diverse populations are needed to fully understand the utility of this inertial-sensor technology.

Whereas instrumented balance assessments resulted in group differences, further analysis revealed that balance assessment using an inertial sensor was limited as a sole screening tool for identifying patients with acute SRC based on sensitivity and specificity. Our data showed that instrumented balance assessments had low sensitivity and high specificity to identify acute SRC injury. Moreover, we highlighted the varied nature of SRCs and illustrated the need for clinicians (eg, physicians, athletic trainers, physician assistants) to use multiple tools, such as neurocognitive, balance, symptom, and newer dual-task testing protocols, to clinically evaluate patients with acute SRC. Overall, if used in isolation, the 3 instrumented static-balance tests in this study offered limited value for correctly identifying patients with acute SRCs.

This study was funded in part through a contract with BrainScope Company, Inc, from the US Navy (Naval Health Research Center, San Diego, CA; contract W911QY-14-C-0098). We thank the BrainScope Company, Inc, team for organization and oversight of the clinical trial that made this study possible. We also thank the following individuals and sites for their assistance with data collection: Robert J. Elbin, PhD, at the University of Arkansas and Kenneth Podell, PhD, at Houston Methodist.