Context: Few validation studies of sport injury-surveillance systems are available.
Objective: To determine the validity of a Web-based system for surveillance of collegiate sport injuries, the Injury Surveillance System (ISS) of the National Collegiate Athletic Association's (NCAA).
Design: Validation study comparing NCAA ISS data from 2 fall collegiate sports (men's and women's soccer) with other types of clinical records maintained by certified athletic trainers.
Setting: A purposive sample of 15 NCAA colleges and universities that provided NCAA ISS data on both men's and women's soccer for at least 2 years during 2005–2007, stratified by playing division.
Patients or Other Participants: A total of 737 men's and women's soccer athletes and 37 athletic trainers at these 15 institutions.
Main Outcome Measure(s): The proportion of injuries captured by the NCAA ISS (capture rate) was estimated by comparing NCAA ISS data with the other clinical records on the same athletes maintained by the athletic trainers. We reviewed all athletic injury events resulting from participation in NCAA collegiate sports that resulted in 1 day or more of restricted activity in games or practices and necessitated medical care. A capture-recapture analysis estimated the proportion of injury events captured by the NCAA ISS. Agreement for key data fields was also measured.
Results: We analyzed 664 injury events. The NCAA ISS captured 88.3% (95% confidence interval = 85.9%, 90.8%) of all time-lost medical-attention injury events. The proportion of injury events captured by the NCAA ISS was higher in Division I (93.8%) and Division II (89.6%) than in Division III (82.3%) schools. Agreement between the NCAA ISS data and the non–NCAA ISS data was good for the majority of data fields but low for date of full return and days lost from sport participation.
Conclusions: The overall capture rate of the NCAA ISS was very good (88%) in men's and women's soccer for this period.
Overall, the capture rate of the National Collegiate Athletic Association Injury Surveillance System was 88% for men's and women's soccer injuries during the study period.
Thus, this injury-surveillance system is capable of providing reliable and valid injury statistics, at least for men's and women's soccer.
Surveillance consists of “ongoing and systematic collection, analysis and interpretation of data.”1,(p164) Accurate and timely surveillance of sports injuries is important for monitoring trends in sport injuries.1 Surveillance data can also be used to guide and evaluate injury-prevention efforts.2 Currently, several surveillance systems in the United States are used to examine sport injuries. Systems collecting information on severe sport injuries in the general population include the emergency department–based system operated by the U.S. Consumer Products Safety Commission, known as the National Electronic Injury Surveillance System–All Injury Program (NEISS-AIP),3 and the catastrophic injury registry operated by the National Center for Catastrophic Sports Injury Research.4 For less severe injuries, attention has typically been focused on specific settings (eg, high school or collegiate), and injury data are often collected by certified athletic trainers (ATs). Examples of surveillance systems using data from ATs include the High School Reporting Information Online (RIO; Nationwide Children's Hospital, Columbus, OH)5 and the Big Ten Conference Sports Injury Surveillance System database (B10-ISS).6
The National Collegiate Athletic Association (NCAA) Injury Surveillance System (ISS) is another setting-specific system that calls on ATs to prospectively collect data on the incidence of injury in NCAA collegiate sports.7 Detailed data on mechanism of injury, activity at time of injury, injury diagnosis, and the number of team exposures (games and practices) are collected. The NCAA ISS began operation in 1988 with pen-and-paper forms that were faxed or mailed to the NCAA and then entered by hand into a database.7 In the mid-1990s, electronic scanning of the forms was introduced. Conversion to a Web-based system was completed for all sports in 2004–2005. These data are used by NCAA committees to make decisions relevant to student–athlete welfare.7,8 The data are also used by sports medicine researchers around the world to identify and monitor important descriptive aspects of collegiate sports injuries, such as the increased incidence of anterior cruciate ligament injuries in female athletes.9,10 The NCAA ISS data have facilitated the implementation of measures designed to decrease the incidence of certain injuries, such as protective goggles to reduce eye injuries in women's lacrosse players.8,11
All surveillance systems should be evaluated on a routine basis so that their performance can be assessed.2,12,13 The Centers for Disease Control and Prevention (CDC)12 has published criteria for the evaluation of surveillance systems. An important element in evaluating a surveillance system is estimating the proportion of true cases detected by the surveillance system. The typical method for estimating this attribute is to match the cases detected by the surveillance system with another source (or sources) of injury data external to the surveillance system.13 If the external data source completely enumerates all true cases and contains no false-positives (ie, injuries are measured completely and without error), then it is referred to as a gold standard; however, it is very unusual to have access to a true gold standard in most evaluations of surveillance systems. More often, researchers use an external data source that is less than perfect in its detection of true cases, such as athletes' self-reports, coaches' reports, or clinical reports. In this situation, capture-recapture analysis of the data is appropriate.14
Despite the fact that all surveillance systems should be evaluated on a routine basis, few validation studies of sport injury surveillance systems have been conducted. The purpose of our study was to examine the validity of NCAA ISS data from 2 fall collegiate sports (men's and women's soccer) by comparing NCAA ISS data from a purposive sample of NCAA schools with the information recorded by other data collection systems (other software or paper records) on those same injuries.
We performed a validation study using record abstraction in a purposive sample of NCAA ISS colleges and universities. The validation study was limited to 15 schools reporting NCAA ISS data for men's and women's soccer. Additionally, schools were required to have contributed data to the NCAA ISS for at least 2 years to be included. A gold standard against which to validate the NCAA ISS data does not exist because both paper-based and other software-based systems probably occasionally undercount the true number of injuries. In the absence of a gold standard, capture-recapture methods were used.14 Capture-recapture analysis assumes some undercounting in both systems and estimates the true number of cases or injury events based on the completeness of each data source relative to the total number of injury events captured by both data sources combined.15–17 Capture-recapture methods have been applied to cancer,18 infectious diseases,19 cardiovascular disease,20 and occupational injuries21,22; however, they have not been not widely used in sport injury analyses. The analysis is described further in the “Statistical Analysis: Capture-Recapture Methods” section.
In 2005–2006, 149 of an estimated 1075 NCAA colleges and universities voluntarily participated in the NCAA Web-based ISS by providing usable data for at least 1 sport. We performed a validation study using record abstraction in a purposive sample of NCAA ISS colleges and universities that provided data on both men's and women's soccer to the NCAA ISS for at least 2 years during the 2005–2006, 2006–2007, and 2007–2008 academic years. Participants included men's and women's soccer athletes and ATs at these NCAA ISS institutions.
We restricted the study to schools with at least 2 years' experience with the Web-based NCAA ISS at the time of enrollment in the validation study to ensure that schools were reasonably proficient in its use and that any discrepancies in the data due to learning effects would be minimized. In the interest of feasibility, the project was limited to 2 sports (men's soccer and women's soccer), and we included only schools that provided data to the NCAA ISS for both sports.
NCAA ISS Injury Definition
During this study, a reportable injury in the NCAA ISS was defined as one that occurred as a result of participation in an organized intercollegiate practice or contest, necessitated medical attention by ATs or physicians, and resulted in restriction of the student-athlete's participation for 1 or more days beyond the day of injury.7 Such injuries are hereafter referred to as time-lost medical-attention injuries.
All procedures for this study were approved by the Duke University Medical Center Institutional Review Board. We were also required to obtain approval from institutional review boards at 11 of the 15 participating schools. The 4 remaining schools did not have boards of their own; therefore, individual investigator agreements under our institution's federal-wide assurance were obtained from the school's ATs to ensure adequate oversight of human subject ethics and principles. It is important to note that this study involved abstraction from existing electronic or paper clinical records, and no athletes were directly interviewed as part of our data collection process. Nevertheless, we were required to obtain written informed consent specifically for this study from all participating athletes before any record abstraction could occur. This consent form was in addition to the NCAA form governing access by the NCAA to student–athlete injury data. All ATs completed training in human participants research ethics, and informed consent was obtained by the AT at the beginning of the fall season from current men's and women's soccer athletes. Injury data for athletes who had graduated were not abstracted because there was no ready means of obtaining consent from graduates. We also obtained consent from all the participating ATs so that they could complete a short questionnaire about their school's experience with the NCAA ISS.
School eligibility, recruitment, and participation are illustrated in the Figure. The NCAA provided a list of the 80 schools that entered data into the NCAA ISS for men's or women's soccer for the 2006–2007 season; of these, 45 schools entered data for both sports. In March 2007, the NCAA sent an e-mail informing the ATs who maintained NCAA ISS records for men's and women's soccer at those schools about this study. The e-mail stated that they would receive a call from a researcher in the next few weeks. Members of the research team then contacted these ATs by phone and recruited them into the study.
All 45 schools were contacted. Of the 45 NCAA ISS schools that entered both men's and women's soccer data for 2006–2007 season, we determined that 15 (33%) were ineligible for the study because the NCAA ISS was their sole injury record-keeping system (ie, they had no other record system against which we could validate their NCAA ISS data). Reasons for ineligibility of the remaining 3 schools (7%) were not using the NCAA ISS in the coming year, not entering data for both sports, and using another data source that was insufficient for comparison. A total of 27 schools were therefore eligible for the study, and 21 were enrolled (78% school-level initial response rate). After initial recruitment, 6 schools withdrew, leaving 15 schools (56% school-level final response rate) and 37 ATs in this study. All ATs completed the AT questionnaire.
Non–NCAA ISS Data Sources
The other data sources maintained by the ATs were used to validate the NCAA ISS data. These included hard-copy AT injury-assessment forms and rehabilitation and progress notes, coaches' reports, notes from other clinicians (eg, physicians, physical therapists), and non–NCAA ISS electronic databases (eg, Sportsware Injury Tracking Software; Presagia Corporation, Montreal, QC, Canada). Seven schools used a non–NCAA ISS electronic database, and 8 schools used hard-copy records (in addition to their NCAA ISS reporting).
Injury Data Abstraction
From February 2008 to December 2008, 5 researchers (K.L.K., D.R.B., M.J.D., C.P.G., S.O.) traveled to each of the 15 study schools that agreed to participate in the study and abstracted data onsite from each school's non–NCAA ISS data source. All abstractors were ATs with prior work experience in the collegiate setting who participated in a half-day training session before data collection began.
Injury data reported to the NCAA ISS for the 2005–2006, 2006–2007, and 2007–2008 soccer seasons were compared with injury data from the ATs' paper files or some other (non–NCAA ISS) electronic injury-tracking database. Researchers reviewed injury data only for athletes who consented to participate in the study. All time-lost medical-attention sport injuries were matched by sport, athlete name or identification, and injury date. Close misspellings of the name or identification and date of injury within 1 week were considered a match. Data fields abstracted for each sport injury from the non–NCAA ISS data source included sport, date of injury, sport relatedness, time of season, event type, injury mechanism, activity before injury, whether the injury was incident or recurrent, whether the injury was chronic or acute, side of the body, whether the athlete had surgery, injured body part, type of injury, diagnosis, outcome, date of full return, and total days out due to injury. Detailed categories for all key variables are included in the Appendix. We made extensive efforts to verify that all the injury-events included in the study resulted in at least 1 day of lost time. This included verification against written (eg, AT injury-assessment forms, rehabilitation and progress notes, coach's report) or electronic sources (eg, injury-tracking software data) and personal conversations with the AT for each team. As an example, consider a scenario in which the non–NCAA ISS data source indicated a missed practice or game, but the actual date of return and the number of days lost were missing. In this case, it was clear that the athlete missed 1 day for this injury, and therefore the injury was included. If the non–NCAA ISS data source did not indicate a missed practice or game, we consulted the AT to determine whether this event was a time-lost event. The data fields for date of return and days lost were still considered missing for these injuries.
We asked each AT to complete a short questionnaire to help us characterize the AT and the school in terms of their support for injury-surveillance activities. The background data collected included demographic data about the AT and his or her training, basic information about the institution (eg, AT staffing levels), AT knowledge and beliefs about injury surveillance, how and whether they used surveillance data, and experience using the NCAA ISS.
Statistical Analysis: Capture-Recapture Methods
Because neither data source (NCAA ISS or non–NCAA ISS data system) could serve as a gold standard that was without error, we used capture-recapture methods15–17 to estimate the total number of injury events and the proportion of injury events captured by the NCAA ISS for men's and women's soccer participants during the study period. After matching injury events during data abstraction, we derived values for 3 cells for a 2 × 2 table comparing capture in the NCAA ISS with capture in the non–NCAA ISS data systems (Table 1). These 3 cell counts quantified injury events recorded in both systems (cell a), injury events recorded in the NCAA ISS but missing from the non–NCAA ISS source (cell b), and injury events in the non–NCAA ISS source but missing from the NCAA ISS (cell c). Based on the degree of overlap between the data systems, the hypothetical number of injury events missed or unobserved by either system was estimated (x = bc/a).15
The overall proportion captured by the NCAA ISS was defined as the number of time-lost medical-attention injury events reported to the NCAA ISS divided by the estimated total number of such events: (a + b)/N, where N = a + b + c + x (Table 1). We also assessed whether the proportion captured by the NCAA ISS varied by 6 key variables: calendar year, sport (men's versus women's soccer), NCAA division, use of electronic database versus paper records for the non–NCAA ISS system, presence of an undergraduate athletic training program at the school, and whether data were entered into the NCAA ISS by a supervised athletic training student versus solely by the ATs. Chi-square tests were used to determine statistical differences between categories of these variables. All injuries, incident and recurrent, were included in capture-recapture analyses.
Statistical Analysis: Agreement for Injury Data Fields
In addition to the NCAA ISS capture rate, we estimated the percentage of agreement for key injury-surveillance data fields: date of injury, sport relatedness, time of season, event type, injury mechanism, activity before injury, whether the injury was incident or recurrent, whether the injury was chronic or acute, side of the body, whether the athlete had surgery, injured body part, type of injury, diagnosis, outcome, date of full return, and total days lost due to injury. Percentage of agreement between the data sources was assessed in 2 ways. First, the overall or effective percentage of agreement23 was calculated. This was simply the proportion of values for each data field for which the NCAA ISS and non–NCAA ISS data sources were in agreement: that is, both sources recorded the same value. Second, the ϰ percentage of agreement was calculated. Kappa analysis has the advantage of accounting for agreement that occurs purely by chance23,24; however, ϰ has some limitations (see “Discussion”). Given the large number of possible category combinations for number of days lost, we created a severity variable (0, 1–7, 8–14, 15–30, or 31+ days lost) and calculated both effective agreement and ϰ. If the information for any data field was missing in either data source, we considered “missing” a valid category in both the effective agreement and ϰ analyses. This decision was relevant mainly to injury mechanism (n = 57 missing), activity before injury (n = 143 missing), outcome (n = 88 missing), date of full return (n = 254 missing), and total days lost due to injury (n = 248 missing).
Statistical Analysis: AT Questionnaire
Quantitative analyses of data from the AT questionnaire included descriptive frequencies. Narrative responses to questions were examined for patterns. The AT sport coverage and workload were quantified in 2 ways: using a sport-to-AT coverage index (# NCAA sports/# ATs, including staff and graduate assistants) and using the National Athletic Trainers' Association (NATA) unadjusted base Health Coverage Index (HCI).25 The NATA's College/University Athletic Trainers' Committee developed the HCI as a means of accounting for the variability in injury rates and AT treatment among 41 collegiate sports. The HCI is composed of the estimated injury rate and treatment by sport and ranges from a minimum of 0.5 for bowling, men's golf, and rifle to a maximum of 4.0 for women's basketball and gymnastics and men's volleyball. The sum of all sport HCIs represents the overall potential AT sport workload accounting for variability by sport. A school sponsoring all 41 collegiate sports would have a maximum HCI sum of 86.2.
Characteristics of Schools and ATs
The final group of 15 schools included 6 Division I schools, 3 Division II schools, and 6 Division III schools. Undergraduate and graduate athletic training programs were offered by 8 and 2 schools, respectively. Schools offered a mean of 19 NCAA-sponsored sports (SD = 5.4 sports; range, 10 to 31 sports) and had a mean HCI of 42.6 (SD = 11.8; range, 25.3 to 70.2). The sport-to-AT coverage index ranged from 3 to 19 sports covered per AT, with a mean of 9 sports (SD = 4.7 sports) per AT.
Participating ATs had a mean of 3.1 (SD = 1.7; range, 0.5 to 7) years' experience using the Web-based NCAA ISS, and 49% (18/37) had participated in the paper version of the NCAA ISS. Schools entered NCAA ISS data for an average of 7 sports (SD = 5.1 sports; range, 1 to 19 sports). The top 3 reasons given for participating in the NCAA ISS were the ability to compare school incidence rates with division and national totals (24%, n = 9), providing data to the NCAA rule committees to make policy decisions (22%, n = 8), and providing data for school decisions on health and safety of the athlete (19%, n = 7). In 14 of the 15 schools, the responding AT supervised people (eg, staff, graduate assistant AT, athletic training student) who entered NCAA ISS data; in 7 of 14 schools, the person entering the data was a supervised athletic training student.
Roughly half of the 37 ATs surveyed were female (51%, n = 19). Most ATs were between 20 and 29 (51%, n = 19) or 30 and 39 (41%, n = 15) years of age, and 8% (n = 3) were 40 to 49 years of age. Participants reported a mean of 8.0 total years of experience as an AT (SD = 6.0 years; range, 0.5 to 26 years) and a mean of 4.4 years working for their current employer (SD = 3.6 years; range, 0.5 to 14 years) as a head AT (32%, n = 12), assistant or staff AT (51%, n = 19), graduate assistant AT (14%, n = 5), or program director (3%, n = 1). In addition to their athletic training certification, 76% (n = 28) had acquired a master's degree and 8% (n = 3) were also physical therapists. The majority reported attending the most recent NATA meeting (65%, n = 24).
Of 824 men's and women's soccer athletes, 737 (89.4%) consented to provide access to their clinical records for the purposes of this study (Table 2). Athlete participation rates did not vary by either sport (χ1 = 1.3, P = .25) or division (χ2 = 3.5, P = .17).
We abstracted 712 injury events at 15 schools. For a variety of reasons, 48 events did not generate usable data (Table 3), leaving at total of 664 injury events for analysis.
Capture-recapture analysis estimated that the NCAA ISS captured 88.3% (95% confidence interval = 85.9%, 90.8%) of an estimated total of 677 injury events (Table 4). The proportion of injury events captured (hereafter termed the capture rate) by the NCAA ISS varied by division (χ22 = 19.2, P < .01), with greater capture in Division I and less capture in Division III. The capture rate tended to be slightly lower when the non–NCAA ISS data source included an electronic database (χ12 = 2.6, P = .11) and was similar for both men's and women's soccer (χ12 = 1.5, P = .22), academic year (χ22 = 4.1, P = .13), and presence of an undergraduate program in athletic training (χ12 = 0.09, P = .76). The NCAA ISS capture rate per school ranged from 67.9% to 100.0%.
A higher capture rate was observed at schools where a supervised athletic training student entered data in the NCAA ISS compared with schools where other staff entered data (χ12= 9.1, P < .01). The proportion of events captured by both data sources increased over the 3-season period (χ12 = 25.6, P < .01), possibly indicating a learning curve associated with use of the NCAA ISS, dual injury-tracking systems, or both.
Agreement for Key Data Fields
For injury events captured in both data sources (n = 500), the effective agreement between the NCAA ISS and the non–NCAA ISS data source for specific data fields ranged from 32.4% for number of days lost from sport to 99.2% for season (preseason, regular season, postseason, or other) (Table 5). Kappa percentages of agreement ranged from 35.0% for severity to 94.6% for body part and tended to be lower than the effective agreement values. For both statistics, agreement was highest for all the injury detail fields (incident or recurrent, chronic or acute, side of the body, surgery needed, body part, injury type, and diagnosis) and was substantial for injury mechanism, activity at time of injury, and outcome. Of the 124 that did not agree on the injury mechanism, 46.0% (n = 57) were unknown or missing; 36.3% (n = 45) were coded as acute noncontact; 10.5% (n = 13) were coded as contact with player, surface, or apparatus; and 7.3% (n = 9) were coded as other in the non–NCAA ISS data source. These events were coded in the NCAA ISS as acute noncontact (19.4%, n = 24); contact with player, surface, or apparatus (60.5%, n = 75); and other (20.1%, n = 25). Only 24 records did not agree for body part; of these, 29.2% (n = 7) were coded as thigh, 25.0% (n = 6) were coded as hip, 12.5% (n = 3) were coded as ankle, and 8.3% (n = 2) were coded as knee in the non–NCAA ISS data source versus 25.0% coded as thigh (n = 6) or knee (n = 6) and 20.8% (n = 5) coded as lower leg in the NCAA ISS.
Irrespective of the statistic used, agreement was lowest for 2 of the return-to-play data fields: date of full return and days lost. Considering only the records with days lost not missing on both the NCAA ISS and non–NCAA ISS, 128 of 252 injury events did not agree; of these 128 events, 60.2% (n = 77) of the injury events were ± 1 to 2 days of each other, 28.1% (n = 36) were ± 3 to 7 days of each other, and 11.7% (n = 15) were ± 8 or more days of each other. The correlation between the data sources for days lost was 0.62. Effective agreement was higher for less restrictive return-to-play variables: outcome (82.2% coded as full return to play, medical disqualification, athlete no longer with team, or other) and severity (51.8% coded as 0, 1–7, 8–14, 15–30, or 31+ days lost).
Characteristics of Injuries Not Captured by the NCAA ISS
Injuries not captured in the NCAA ISS (n = 66) were predominantly sprains, contusions, and strains to the ankle, thigh, and knee (Table 6). Time-lost and event outcome information was missing for 20 events; for the remaining 46 events, all athletes returned to play with a median of 5 days lost from sport (mean = 10; SD = 16.8; range, 1 to 96 days).
To our knowledge, this 2-sport validation study is the first to be conducted for a national Web-based sports injury-surveillance system. We found that the NCAA ISS captured 88% of time-lost medical-attention injury events in the 15 collegiate men's and women's soccer study schools during the 3-season period, a very good capture rate. The capture rate was highest in Division I and lowest in Division III, which may reflect the increased resources available to ATs in Division I. Effective agreement between the NCAA ISS and the non–NCAA ISS data sources was also high, close to or above 90% for most of the data fields examined. However, 2 interrelated data fields, date of full return and days lost, had low agreement. In view of this finding, administrators of collegiate sports injury-surveillance systems and end users of the data from these systems should carefully consider the implications of using time-loss criteria to define injury severity. Similar variability with time-loss data was noted in a previous NCAA ISS study26 and is consistent with these findings. Other authors27 have more globally discussed the limitations of using time-loss criteria in this manner.
The only previous validation study of sports injury did not address capture rate but rather focused on reliability of self-reported injury details.28 These authors noted moderate levels of agreement for injured body part and treatment but low agreement for injury severity. Therefore, to place these findings in context, we need to go beyond the sports medicine literature to the literature on general injury surveillance. The most comparable study is a validation study of the CDC's emergency department surveillance system for all injuries (including non-sports injuries), NEISS-AIP, which had a capture rate of 83% (490/593 injury events).29 This capture rate was considered very good for surveillance purposes, and NEISS-AIP is widely regarded as the most reliable system for monitoring emergency department injury visits.
Determinants of Capture Rate
We found that the capture rate was more than 10 percentage points higher in Division I schools than in Division III schools, probably reflecting the greater resources available to Division I ATs. The capture rate was 8 percentage points higher when data were entered into the NCAA ISS by a supervised athletic training student rather than entered solely by the ATs. The increase by year in the capture rates for the NCAA ISS and non–NCAA ISS systems combined may reflect a learning curve as ATs became more experienced using the new Web-based NCAA ISS and developed better methods for managing 2 data systems during the study period. In addition, increased NCAA resources and staff were directed toward training, follow-up, and monitoring of ATs and schools from 2005 to 2008 (J. Corlette, oral communication, January 15, 2010).
Data Agreement and Days-Lost Data Field
Agreement for all data fields was good except for mechanism of injury, activity before injury, date of full return, and number of days out. Date of full return and the number of days out were not available or missing more often in the non–NCAA ISS data source (n = 230 missing) than in the NCAA ISS (n = 71 missing). Maintaining accurate information on days lost was not the primary intent of the non–NCAA ISS data systems accessed in this study, and the missing information illustrates the different purposes of sports injury-surveillance systems and clinical record keeping. Also, the non–NCAA ISS data source in this study represents a variety of different methods, none of which was designed for injury surveillance. Therefore, our results concerning the low agreement for days lost and date of return should be interpreted with caution, because we acknowledge the possibility that these data fields may be more commonly and accurately recorded in the NCAA ISS than in the non–NCAA ISS data source.
Assuming it is valid, the low level of agreement for days lost suggests that using time loss as the only marker of injury severity may be unwise, at least in this population. Previous consensus statements30,31 have made similar recommendations for specific sports and discussed more globally the distinction between incidence of injury and severity of injury.32 Although widely used as a marker of injury severity, time loss is only one method of quantifying severity.18 Other markers of severity include cost, treatment, and disability. Time loss for any particular injury is not a constant but varies among athletes, clinicians, and sports. For example, a hand injury might be severely debilitating to a tennis player, but the same injury might not result in any time loss for a runner. Additionally, advancements in prevention, treatment, and rehabilitation mean that injury severity measures based solely on days lost have an interpretation that changes over time. Observed shifts in the distribution of time loss among schools may merely reflect differences in treatment patterns rather than a true change in the underlying severity of injury. These are all important points to consider when developing and using severity measures in sports injury-surveillance systems.
Although all researchers received training to ensure standardization and consistency in data abstraction, no formal statistical assessment of interabstractor or intra-abstractor reliability was conducted. Therefore, we do not know the degree of variability in data coding attributable to differences in abstractor coding. Most of the data coding was conducted by pairs of researchers to ensure consensus in data abstraction.
In this study, there was no gold standard. Thus, comparisons were made between the data sources with capture-recapture analysis, which accounts for assumed error in both sources. This is common practice in public health surveillance studies when neither source can be considered complete.21,22,33 However, the 88% NCAA ISS capture rate may be an underestimate if the non–NCAA ISS data source includes many injury events that were false-positives (events recorded as time loss that were, in fact, not time-loss injuries). With this possibility in mind, we went to considerable lengths to establish that all the injury events reported in the non–NCAA ISS data were true time-lost events. When we were unable to determine whether the event had resulted in time loss based on the non–NCAA ISS data source, we queried the AT to verify that the event did result in time loss. Time loss was verified with the AT regarding 12 events for which the non–NCAA ISS data source did not indicate a missed practice or game. These events were included in the study, but values for date of full return and number of days remained missing. In 2 cases in which time loss could not be confirmed in the non–NCAA ISS source or with the AT, we excluded the events from our analysis (Table 3). If the injury existed only in the NCAA ISS and we found no written record of the injury anywhere else, we sought additional information from the AT to confirm that the injury was indeed a time-lost medical-attention injury. However, verbal information from memory in the absence of a written record was not considered an acceptable comparison data source; events based only on the AT's recollection without documentation could not be qualified as time loss and were excluded from our analyses (n = 24, Table 3). As a limited sensitivity analysis, if all 26 events that we could not confirm as time loss had been included in our analysis, the estimated capture rate of the NCAA ISS would not have materially changed: (500 + 124)/(500 + 124 + 66 + 16.4) = 88.1%. The daily injury report to the coach provided a good source of information regarding lost days and date of return. However, not all schools in this study provided a formal daily report to the coach. Some relied heavily on e-mail to communicate with their coaches, and others gave verbal reports to the coach, neither of which were available or considered valid sources for comparison.
Effective agreement is a valuable tool for assessing agreement. Yet because effective agreement does not account for agreement due to chance alone, we also calculated the ϰ percentage of agreement (Table 5). Kappa has several limitations, including the fact that it is unduly conservative in some settings and tends to be low when the number of response categories is small or the prevalence of the attribute is high.23,34 The latter factor (it tends to be low when the prevalence of the attribute is high) may particularly have affected the ϰ values for sport relatedness and outcome. Missing values for data fields were considered valid entries in both agreement analyses. This was not an unreasonable assumption if the person entering the data into both sources left it missing because he or she truly did not know the value (eg, activity before the injury event was unknown). However, this assumption might not be reasonable when the value was missing in only a single data source. Restricting the analysis to nonmissing values for both data sources would have affected mainly the results for mechanism (effective agreement in nonmissing of 84.9%), activity before the injury (84.3%), outcome (98.5%), and injury severity (81.8%). Similarly, when restricted to nonmissing values, effective agreement was still low for date of full return (54.9%) and days lost (49.2%).
At the time of enrollment, schools had to have at least 2 years of experience with the NCAA ISS to be eligible for this study. Although some ATs did not have 2 years of experience with the NCAA ISS, all 15 site administrators had at least 2 years of experience (mean = 4.3; range, 2 to 7 years). The capture rate of the NCAA ISS was similar when we compared ATs with less than 2 years of experience (90.9%) with those who had 2 or more years of experience (87.8%) using the NCAA ISS.
This validation study was performed for only 2 collegiate sports: men's and women's soccer. Therefore, these results may not be generalizable to other sports or to the winter and spring seasons, in which ATs' workloads and coverage might be different. Comparing participating schools (n = 15) with nonparticipating schools (n = 30), our study group included a greater proportion of Division I (40% versus 27%) and Division II (20% versus 10%) schools and a lesser proportion of Division III schools (40% versus 63%). We also acknowledge that schools that use the NCAA ISS as the sole record-keeping system may report more accurate information; it was impossible to examine this suggestion using our study design. Finally, abstraction of practice and game exposure data was beyond the scope of this investigation. Future researchers should examine the validity and reliability of injury data in other sports and address exposure data.
The overall rate of capture of the NCAA ISS for the 15 schools and 2 sports in this study (88%) was very good. This finding indicates that the NCAA ISS can yield reliable and valid injury statistics. Yet the low level of agreement for days lost (32%) raises concerns about the validity of using time loss as the only marker of injury severity in surveillance systems. However, it should be noted that maintaining accurate information on days lost was not the primary intent of the non–NCAA ISS data systems accessed in this study and, therefore, our results regarding time loss should be interpreted with caution.
This study was supported by a grant from the National Collegiate Athletic Association (NCAA). Drs Kucera and Marshall have served as consultants to the NCAA on the Web-based Injury Surveillance System. Dr Marshall is part of the Datalys Center for Sports Injury and Prevention, a private not-for-profit company that currently operates the Injury Surveillance System under contract to the NCAA.
Our thanks to the student-athletes and, especially, to the certified athletic trainers who participated in this study. We also thank NCAA staff members Randy Dick, Jill Corlette, and David Beery, and we are grateful to this study's advisory board for their many helpful comments: Julie Agel, MA, ATC, of the University of Minnesota; Julie Gilchrist, MD, and Jennifer M. Hootman, PhD, ATC, FNATA, FACSM, of the Centers for Disease Control and Prevention; and Troy Hege of the Datalys Center. In addition, we thank Karen Richmond for data entry and Kristie Wicker for study management at Duke University.
Conclusions drawn from or recommendations based on the data are those of the authors and do not represent the views of the officers, staff, or membership of the NCAA or the Datalys Center for Sports Injury and Prevention.