Experiences of Adversity and Validity of Baseline Concussion Testing

As approved by the institutional review board at University Hospitals Medical Center (UH) in Cleveland, we identified 8477 student-athletes born between 1995 and 2005 who completed a baseline ImPACT test when they were between 10 and 18 years old at a middle or high school in northeast Ohio whose athletic department was affiliated with the UH Concussion Management Program. The tests were administered between January 1, 2013, and December 31, 2017. Version 2 was used for 81% of the sample and version 3 for the remaining 19%. All testing was completed under the supervision of an AT who received annual training in ImPACT administration from a board-certified neuropsychologist. Individual schools followed unique protocols for baseline testing, though testing was generally administered in a controlled environment with small groups of athletes simultaneously completing baseline tests (≤10 athletes per AT or informed proctor per site).

From this population, we linked 6495 individuals to the Child-Household Integrated Longitudinal Data (CHILD) system via a probabilistic matching process. The CHILD system is a continuously updated integrated and comprehensive data system that captures detailed individual-level demographics, social service usage, and educational outcomes of children born or living in Cuyahoga County, Ohio from 1989 to the present.¹⁸ 

To match individuals between the UH Concussion Management Program’s ImPACT testing database and the CHILD system, we used a probabilistic matching process standardized by the CHILD system and required in its protocols for all research linking to the administrative database. In the matching process, no single variable (first name, last name, date of birth, sex, or most recent address) determined match validity; rather, each variable contributed to an overall match likelihood through an iterative process that allowed for variations in name spelling, date-of-birth errors, and address changes. Two primary reasons explained why one-quarter of the eligible youth in the ImPACT data could not be linked to the CHILD system: (1) Some schools in the ImPACT data were located outside Cuyahoga County, and the CHILD system is geographically restricted to Cuyahoga County. Therefore, youth born outside of Cuyahoga County who never resided within the county limits and engaged with a Cuyahoga County data-sharing institution (including public schools, blood lead testing, child welfare, and juvenile courts) were unlikely to have been matched between the 2 systems. (2) The ImPACT data contained a limited set of identifiers (name, date of birth, gender, and address), whereas the CHILD system typically uses a wider array of information, including parent or guardian information and social security number, in the matching process to increase the match likelihood.

Validity of the first baseline test as determined by ImPACT’s built-in validity scoring, age, and home address at the first baseline test along with any concussion history and diagnosis of learning or developmental disorders (ADHD, dyslexia, autism) were extracted from the ImPACT testing records (Table 1).

The mother’s race and address provided on the birth certificate as well as measures of poverty and abuse or neglect were extracted from the CHILD system. We used the mother’s race because the infant’s race is not captured on Ohio birth certificates. Addresses at birth and at the baseline ImPACT test were geocoded and linked to the 2015 Area Deprivation Index (ADI) by census block.^19,20  Two binary variables for residence in the highest decile of deprivation at birth and baseline were created using the state-level ADI value, which ranks census blocks by deciles of deprivation within each state. Just over one-third of the study population (37.1%) could not be linked to the ADI in ≥1 of the time periods because of out-of-county birth certificates or errors in addresses that prevented geocoding.

Poverty was defined as enrollment in the Supplemental Nutrition Assistance Program or Temporary Assistance for Needy Families Program. A binary variable was created for ≥1 month of enrollment in either public assistance program during early childhood (0–5 years old), and another binary variable was created for enrollment in either program during adolescence (within 5 years of the baseline test). Experiences of abuse or neglect were identified as cases of child maltreatment (physical, sexual, or emotional abuse perpetrated by a parent, guardian, or custodian or a lack of adequate care, supervision, or both) between birth and the baseline ImPACT test that were found to be credible (ie, substantiated) through investigation of the allegations by the Cuyahoga County Division of Child and Family Services. A binary variable was created for ≥1 substantiated maltreatment incident in early childhood and another binary variable for ≥1 incident in adolescence.

Two categorical variables reflecting life-course experiences of poverty and residence in a highly deprived neighborhood across the 2 time periods were created (no experiences in early childhood or adolescence, only experienced in early childhood, only experienced in adolescence, experienced in both early childhood and adolescence). Few study participants had experiences of abuse and neglect in adolescence; therefore, a similar life-course variable was not created for abuse and neglect.

Descriptive analysis included calculating means, SDs, and percentages in the total sample as well as in subgroups based on baseline test validity. Crude associations between each predictor and baseline validity were calculated. Four multivariable logistic regression models were conducted with baseline validity as the outcome (1 = valid test, 0 = invalid test). The demographic model included demographic covariates of sex, age, ADHD, dyslexia, autism, and history of concussion. The early childhood adversity model included the measures of adversity (poverty, abuse or neglect, highest decile of neighborhood deprivation) restricted to the first 5 years of life as well as the demographic covariates as independent variables. The adolescent adversity model included the measures of adversity restricted to the 5 years before baseline as well as the other demographic covariates as independent variables. The life-course adversity model considered experiences of adversity across the 2 time periods and included the life-course variables of poverty and neighborhood deprivation in addition to the other demographic covariates. Individuals with missing neighborhood deprivation flags were excluded from the relevant models.

Standard errors were clustered by school to account for possible correlations in baseline administration or performance or similar experiences of neighborhood deprivation within schools. All models were also adjusted for the year of the baseline test to correct for potential temporal trends in testing. Post hoc complete-case hierarchical logistic regression was conducted to identify the additional benefit provided by the inclusion of adversity measures along with demographic factors in modeling ImPACT validity. Mother’s race was included as an additional demographic predictor in an extension analysis to explore whether race, as a proxy for experiences of discrimination and a legacy of disinvestment, affected the observed associations with baseline testing validity and adversity. Adjusted odds ratios (ORs) and 95% CIs were generated for each model. All analyses were conducted in RStudio (version 1.3.1056; The R Project for Statistical Computing) and SAS (version 9.4; SAS Institute). A 2-sided α of .05 was considered statistically significant.

Of the 6495 youth in the total study population, 6060 (93.3%) had a valid baseline ImPACT test, whereas 435 (6.7%) had an invalid test (Table 2). The average age of the study population was 14.9 ± 1.3 years, and most were male (59.4%). A history of concussion or ADHD was fairly common (16.2% and 10.6%, respectively), although dyslexia and autism were rare (1.5% and 0.5%, respectively). Many student-athletes experienced poverty in early childhood (0–5 years old; 19.2%) or adolescence (within 5 years of baseline; 21.3%). Abuse or neglect at either time point was uncommon, although a higher percentage of the study population had an abuse or neglect incident in early childhood (2.1%) than in adolescence (0.6%). Of those who had a nonmissing neighborhood deprivation value at birth (N = 5455), 10.3% resided in the highest decile of deprivation during early childhood, whereas only 1.5% of those with a nonmissing value at the baseline test (N = 4843) resided in a highly deprived neighborhood during adolescence.

The majority of the study population did not experience poverty or a high level of neighborhood deprivation in either time period (72.5% never experienced poverty; 89.3% never experienced high neighborhood deprivation). However, of those who experienced poverty in either time period, a higher percentage experienced poverty in both time periods (12.9%) than only during early childhood (6.3%) or adolescence (8.3%). Conversely, of those who lived in the highest decile of neighborhood deprivation in either time period, a larger percentage experienced a high level of neighborhood deprivation only in early childhood (9.3%) than only in adolescence (0.6%) or in both time periods (0.8%).

The multivariable models are presented in Table 3. The demographic model showed that each year increase in age was associated with 26% increased odds of a valid baseline test (OR = 1.26 [95% CI = 1.12, 1.43], P < .001), and a history of concussion or diagnosis of ADHD or autism was associated with significantly reduced odds of a valid test (history of concussion, OR = 0.75 [95% CI = 0.57, 0.98], P = .03; ADHD, OR = 0.57 [95% CI = 0.42, 0.78], P < .001; autism, OR = 0.42 [95% CI = 0.21, 0.85], P = .02).

In the early childhood adversity model (ie, adversity experienced from 0–5 years old), higher odds of baseline validity were associated with each year increase in age at the baseline test (OR = 1.23 [95% CI = 1.06, 1.41], P = .005). No experiences of adversity in early childhood were significantly associated with greater odds of valid test performance. In the adolescent adversity model (ie, adversity experienced in the 5 years before the baseline test), age was associated with increased odds of valid test performance (OR = 1.15 [95% CI = 1.02, 1.29], P = .03), and ADHD with significantly lower odds (OR = 0.58 [95% CI = 0.40, 0.84], P = .004). A stronger relationship between poverty and test validity was observed in the adolescent adversity model than in the early childhood adversity model. Even when adjusting for learning disabilities, a history of concussion, and other experiences of adversity, youth who experienced poverty in adolescence had 45% lower odds of a valid performance (OR = 0.55 [95% CI = 0.39, 0.78], P < .001) than those who did not experience poverty in that time period.

The relationship between poverty and test validity changed when considered across both time periods in the life-course adversity model (adversity in early childhood and adolescence, the best-performing model; Supplemental Table 1). In this model, both ADHD and autism were associated with significantly lower odds of a valid test performance (ADHD, OR = 0.66 [95% CI = 0.45, 0.96], P = .03; autism, OR = 0.35 [95% CI = 0.17, 0.72], P = .005). Compared with youth who did not experience poverty across the life course, those who experienced poverty only in early childhood and not in adolescence (upward social mobility) did not differ with respect to test performance validity (OR = 0.91 [95% CI = 0.53, 1.58], P = .74). Youth who experienced downward social mobility, experiencing poverty in adolescence and not in early childhood, had significantly lower odds of a valid performance (OR = 0.50 [95% CI = 0.31, 0.80], P = .004) compared with those who did not experience poverty. Youth who experienced persistent poverty (poverty in both early childhood and adolescence) also had significantly lower odds of a valid performance (OR = 0.59 [95% CI = 0.35, 0.998], P = .05).

Similar patterns were observed for residence in the highest decile of neighborhood deprivation. Youth who lived in the highest decile of neighborhood deprivation only at birth and did not live in a similarly deprived neighborhood at the time of their baseline test were no different from those who had never lived in a highly deprived neighborhood (OR = 0.85 [95% CI = 0.65, 1.13], P = .26). Those who had lived in a highly deprived neighborhood only at the time of their baseline test had significantly lower odds of a valid test performance (OR = 0.28 [95% CI = 0.17, 0.48], P < .001). The number of youth who lived in the highest decile of deprivation in both time periods was too low to be accurately modeled.

The addition of mother’s race as a demographic variable did not significantly change associations between experiences of poverty or neighborhood deprivation and baseline testing validity in any of the multivariable models (Supplemental Table 2). Having a Black mother was independently associated with 34% to 39% lower odds of achieving a valid test compared with having a White mother in the demographic and early childhood adversity models (P < .05). Mother’s race was not a significant predictor of validity in the adolescent adversity or the life-course adversity models.

We examined the effect of adversity experienced during early childhood and adolescence on baseline neurocognitive testing validity in middle and high school–aged student-athletes. When only the demographic variables were considered, age, ADHD, and autism were significant predictors of baseline test validity (Table 3); however, these associations did not persist reliably with the subsequent addition of adversity measures. It is possible that the variable significance observed for ADHD and autism after the addition of adversity in different life stages was due to correlations between experiences of adversity and neurodevelopmental and behavioral health conditions. For example, Zarei et al²¹  found that adverse childhood experiences, including abuse, neglect, and household poverty, were associated with increased risks for autism, ADHD, and learning disabilities. The addition of mother’s race did not significantly alter the relationship between experiences of adversity and baseline testing validity. This suggests that experiences of poverty and neighborhood deprivation have effects on neurocognitive testing that are independent of adversities related to experiences of systemic oppression typical for racialized minorities.

When adversities were considered under sensitive period and social-mobility frameworks, the experience of poverty in adolescence was associated with a significant reduction in the odds of achieving a valid baseline test. Across the 2 time periods, it was apparent that social mobility played an important role. Upward social mobility mitigated the effect of adversity on baseline validity as these youth did not have different validity patterns than those who never experienced poverty or neighborhood deprivation in our study periods. Youth who experienced downward social mobility were more likely to have an invalid baseline test performance. Persistent poverty had a similarly severe reduction in the odds of a valid baseline test. Although data are scarce on the effect of poverty or neighborhood deprivation on cognitive testing validity specifically, individual and neighborhood poverty have been widely associated with poor cognitive function outcomes, including deficits in attention, memory, and inhibition.^13–15,22  Also, individual and neighborhood poverty were associated with higher exposure to environmental toxins, such as lead in peeling paint or contaminated water, which are known contributors to low cognitive performances and academic challenges.²³ 

Although we were unable to differentiate whether the invalid baseline tests were due to sandbagging (ie, intentionally suboptimal effort) or legitimately poor performance, the outcome of an invalid baseline score is the same. Regardless of the underlying mechanism(s), many youth with an invalid baseline must still rely on that test because additional testing opportunities can be rare. In a survey of ATs who administered ImPACT to their student-athletes, only 51.9% reported reviewing the validity criteria on baseline tests.²⁴  Offering repeat testing after an invalid test drastically lowers the number of students without a valid baseline; multiple researchers have found that more than 85% of test takers with an initial invalid baseline test will achieve a valid performance after readministration.^3,25,26  In our sample, 95% of those who took a repeat baseline test within the same year as their invalid test achieved a valid score. Based on our findings and previous studies,^25,26  we strongly encourage all ATs and ImPACT administrators to prioritize repeat testing after an invalid initial baseline test. We also recommend baseline testing be conducted in a controlled environment, with low group numbers whenever possible (particularly for those individuals who require repeated baseline testing because of an initially invalid performance). Additionally, we advise a brief check-in with an AT or another informed health care provider when an invalid score is obtained to ensure that other clinically relevant factors are not present that might have affected testing and may require follow-up (eg, psychological distress, learning disorders or ADHD, poor sleep). Accurate neurocognitive testing is a useful tool for concussion management, though its utility as an objective measure may be meaningfully reduced if baseline validity is not examined and addressed when concerns regarding invalidity are raised.²⁷ 

A major strength of our work was our use of linked, longitudinal administrative data for a large study population, which allowed us to consider specific, individual-level and community-level measures of adversity typically unavailable in neurocognitive testing research. For example, we were able to identify individual enrollment in federal poverty-support programs along with residence in economically deprived communities, which provides a more detailed measure of socioeconomic status than relying on only regional or school-based markers of poverty. The success of our administrative and testing data linkage suggests that social service data may provide valuable insights into how the social determinants of health intersect with heavily used diagnostic tools in sports medicine.

This research has important limitations worth noting. We used child maltreatment data gathered through the child welfare system, and children with minoritized racial identities are more likely to have their families referred or investigated for maltreatment than White children.²⁸  In addition to oversurveillance by the child welfare system, caseworkers disproportionately substantiate cases of neglect for minoritized children (compared with physical and sexual abuse for White children). Although we were not able to fully account for the effect of institutional racism in the child welfare data, we intentionally chose to include only substantiated cases (rather than referrals or investigations), and we did not differentiate between maltreatment classifications.

Relying solely on Supplemental Nutrition Assistance Program and Temporary Assistance for Needy Families Program enrollment likely provided us with a limited understanding of poverty in our population. Some families in the study population may have experienced functional poverty despite having an income above the public assistance programs’ income thresholds during the time periods considered in our analysis. Additionally, the CHILD system is geographically restricted to Cuyahoga County, the second most populous county in Ohio. Another limitation was that the mother’s race served as a proxy for the child’s race in our analysis due to availability of racial information. If our study population had different experiences of race than their biological mothers, the resulting nondifferential misclassification may have led to an underestimation of the effect of race. Future research is needed to disentangle the underlying social determinants driving this racial disparity in testing validity.

The disparities in baseline validity we revealed offer further rationale that athletic programs using ImPACT must devote the necessary resources to retesting youth with an invalid baseline test performance or optimizing the testing process to enable disadvantaged youth, specifically those experiencing current or recent socioeconomic adversity, to achieve valid baselines. Time and resource concerns may discourage school-based athletic programs from adopting more robust testing protocols, yet without a change in ImPACT administration practices, youth with particular patterns of adversity may face compounded disadvantage, both from the adversities themselves and by disproportionate reliance on inaccurate testing in sport-related concussion management.

We thank Stephen Steh at the Center on Poverty and Community Development for his work maintaining the CHILD system, which allowed us to do this research. We also thank all the athletic trainers and medical providers who work with the UH Concussion Management Program and were instrumental in creating the ImPACT database.

Supplemental Figure. Conceptual model.

Supplemental Table 1. Post Hoc Hierarchical Regression Summary Statistics (N = 4082).

Supplemental Table 2. Associations From the Multivariable Logistic Regression Models for Each Time Period With Race as a Predictor.

Found at DOI: https://doi.org/10.4085/1062-6050-0502.22.S1