Since 1982, the National Collegiate Athletic Association has used the Injury Surveillance System (ISS) to collect injury and athlete-exposure data from a representative sample of collegiate institutions and sports. At the start of the 2004–2005 academic year, a Web-based ISS replaced the paper-based platform previously used for reporting injuries and exposures.
To describe the methods of the Web-based National Collegiate Athletic Association ISS for data collection as implemented from the 2004–2005 to 2013–2014 academic years.
The Web-based ISS monitored National Collegiate Athletic Association–sanctioned practices and competitions, the number of participating student–athletes, and time-loss injuries during the preseason, regular season, and postseason in 25 collegiate sports. Starting in the 2009–2010 academic year, non–time-loss injuries were also tracked. Efforts were made to better integrate ISS data collection into the workflow of collegiate athletic trainers. Data for the 2004–2005 to 2013–2014 academic years are available to researchers through a standardized application process available at the Datalys Center Web site.
As of February 2014, more than 1 dozen data sets have been provided to researchers. The Datalys Center encourages applications for access to the data.
The National Collegiate Athletic Association (NCAA) has partnered with the Datalys Center for Sports Injury Research and Prevention, Inc, an independent, nonprofit research organization, to provide researchers access to the historical data beginning in 2004–2005 through 2008–2009 collected from its Web-based Injury Surveillance System (ISS). Subsets of the 5-year ISS data are available to researchers upon completion and review of the appropriate data-request forms and data-licensing agreement by an external independent review committee (IRC). The Datalys Center will release a new data set every 5 years for access by external researchers.
We describe the methods used for the Web-based NCAA ISS during the 2004–2005 through 2013–2014 academic years and the process for gaining access to the data.
BACKGROUND: INJURY SURVEILLANCE IN THE NCAA
With more than 450 000 student–athletes annually and steadily increasing participation numbers,1 the NCAA unifies a large and diverse group of athletes under a single administrative national governing body. The NCAA student–athletes are an important population to monitor for athlete health, safety, and well-being given the high demands placed on them.
The NCAA's genesis is related to a series of serious injuries that occurred in collegiate football in the early 20th century, which nearly caused the sport to be dismantled in the college setting.2 One of the primary missions of the NCAA was and continues to be the protection of the health and well-being of collegiate student–athletes. It took nearly 50 years for the NCAA to emerge as the national regulatory authority in collegiate sport.2 However, the NCAA has participated in numerous sport-injury–prevention initiatives, such as the National Survey of Catastrophic Football Injuries and the National Center for Catastrophic Sports Injury Research, which now monitors all catastrophic injuries at the high school, collegiate, and professional levels of multiple sports. Injury-surveillance information has helped inform the development of prevention strategies to reduce the incidence and severity of sports injuries.3,4
THE NCAA ISS
Development of the ISS (1982–1983 Through 2002–2003)
The NCAA ISS was created in 1982 as a pen-and-paper data-collection surveillance program with the aim of collecting injury and exposure data from a sample of NCAA institutions in a variety of sports.5 Data collection was performed by athletic trainers (ATs) and included any injury occurring during an organized intercollegiate practice or competition. When student–athletes from participating teams were injured, the AT completed a 2-page injury form, which was then mailed or faxed to the NCAA. In addition, on a weekly basis, the AT provided the NCAA with reports summarizing specific participation information per participating team, including the number of practices and competitions, the season of data collection, and the average number of participants. This weekly exposure report was sent to the NCAA regardless of the number of injuries reported that week.
The ISS data were shared with the appropriate NCAA sport and policy committees to provide a foundation for evidence-based decision making with regard to health and safety concerns. In addition, the ISS data provided individual institutions with injury information to assist them in their risk-management decision making, such as delegating ATs to the activities that placed athletes at highest risk and comparing institutional injury rates with divisional and national rates.5 In addition, data pertaining to injuries in 15 sports during the 1988–1989 through 2003–2004 academic years were published in a special issue of the Journal of Athletic Training (2007:42).
The advent of the Internet raised a number of concerns with the paper-based system. In the paper-based system, the ATs were required to enter data twice: first for their own records and second for the ISS. Having to mail or fax injury and exposure data took large amounts of time and resources. The paper-based format also required substantial human and technical resources and hindered data quality.
First Web-Based ISS Platform (2002–2003 and 2003–2004 Through 2008–2009)
The limitations inherent in paper-based data collection led the NCAA to transition to a Web-based ISS during the 2002–2003 and 2003–2004 academic years. After a thorough redevelopment process, the NCAA launched a Web-based platform to track injury and exposure data at the beginning of the 2004–2005 academic year (Table 1).
Among the benefits of the Web-based platform over the previous paper-based data-collection methods was that the former provided secure and streamlined delivery of data from the AT to the NCAA. To incentivize participation, the Web-based ISS integrated some of the functional components of an electronic medical record (EMR), such as athlete demographic information and preseason injury information. Although the ISS did not have all the functional capabilities of an EMR, an AT could consider using the surveillance system as an EMR, thereby eliminating the need to enter data twice.
Second Web-Based ISS Platform (2009–2010 Onward)
When the Datalys Center introduced new components to the ISS to improve process flow in 2009, a common data element (CDE) standard was implemented (Table 1). The program was renamed the Injury Surveillance Program to demarcate when the CDE standard began. The CDE standard allows data to be gathered from different EMR and injury-documentation applications, including the Athletic Trainer System (Keffer Development Services, Grove City, PA), Injury Surveillance Tool (Datalys Center, Indianapolis, IN), and the Sports Injury Monitoring System (FlanTech Computer Services, Iowa City, IA). The CDE export standard allows ATs to document injuries as they normally would as part of their daily clinical practice instead of asking them to report injuries for the purposes of participation in an injury-surveillance program. All injury-documentation applications must successfully complete a data-validation process to be certified. Certification involves having data quality-control staff practice data collection: data are entered into the injury-documentation application in the same manner as a participating AT would enter data. These data must successfully pass through an automated verification engine (VE) and then successfully land in the research database with the expected values.
Although this approach significantly reduces the burden of reporting injuries, each application is slightly different in terms of how many additional variables have to be specifically entered by the AT versus those that were easily mapped during the certification process. For example, if an application has a free-text field for a required variable, the vendor would have to create a categorical option coded to the Datalys CDEs required of that variable. The frequency of export or submission of data also varies slightly among vendors.
OPERATIONAL DEFINITIONS (2003–2004 ONWARD)
A reportable injury in the ISS was defined as an injury that (1) occurred as a result of participation in an organized intercollegiate practice or competition, (2) required attention from an AT or physician, and (3) resulted in restriction of the student–athlete's participation for 1 or more days beyond the day of injury. Multiple injuries occurring from 1 injury event could be included. If an off day followed the injury event, the AT was asked to assess whether the injured athlete would have been able to participate. In addition, ATs were asked to include any dental injury that occurred in an organized practice or game, regardless of time loss. Beginning in the 2009–2010 academic year, non–time-loss injuries were also monitored. A non–time-loss injury was any injury that was evaluated or treated (or both) by an AT or physician but did not result in restriction from participation for more than 1 day.
The academic year was defined as beginning July 1 and ending June 30. As a result, an academic year of data collection spanned 2 calendar years.
Sport participation was subdivided into the 3 categories defined by the NCAA. Preseason refers to all formal team practices and exhibition games conducted before the first regular season contest. Regular season describes all practices and competitions from the first regular season competition through the last regular season competition. Postseason includes all practices and competitions after the last regular season competition through the last postseason competition. Any injuries and exposures that did not occur during the preseason, regular season, or postseason (eg, summer conditioning, individual workout or training) were excluded.
Exposure was defined as the organized team practice or competition in which student–athletes were participating. Within the ISS, this was limited to NCAA-sanctioned practices and competitions occurring in the preseason, regular season, or postseason. Preseason scrimmages were considered practice exposures, not competition exposures. Only varsity-level practice and competition events were included; junior varsity programs were excluded. Additionally, a few championship sports were not included because the number of ATs reporting data was too few to provide usable data for meaningful analysis.
A reportable athlete-exposure (AE) was defined as 1 student–athlete participating in 1 NCAA-sanctioned practice or competition in which he or she was exposed to the possibility of athletic injury, regardless of the time associated with that participation. Only athletes with actual playing time in a competition were included in competition exposures.
Time loss was defined as the time between the original injury and return to play at a level that would allow competition participation.
SAMPLING AND DATA COLLECTION (2003–2004 ONWARD)
The ISS depends on a convenience sample of teams with ATs voluntarily reporting injury and exposure data. Web-based ISS data collection for the 2004–2005 academic year included the 15 core sports from the paper-based ISS. Additional sports were included in the 2005–2006 and 2006–2007 academic years (Table 2).
Individual institutional sport sponsorship varies significantly within the NCAA. For example, in the 2013–2014 academic year, there were 1070 men's basketball programs, 535 women's swimming and diving programs, and 88 women's gymnastics programs. Compared with data collection from the 2004–2005 through 2008–2009 academic years (Table 3), participation for the 2009–2010 through 2013–2014 academic years dropped (Table 4). This was attributable to the transition from the NCAA to the Datalys Center; ATs' unfamiliarity with the Datalys Center, which led to declining participation; increases in AT workloads; and increases in the number of sports programs switching to other commercial electronic health record systems. However, since the 2009–2010 academic year, participation has steadily increased.
The current process flow of the NCAA ISS is illustrated in the Figure. The AT from each participating team logs into the electronic health record system weekly throughout the academic year to report injury incidence and exposure information. In addition to injuries, the surveillance system captures other sport-related adverse health events, such as heat-related conditions, illnesses, and skin infections. For each event, the AT completes a detailed event report on the injury or condition (eg, site, diagnosis, severity) and the circumstances (eg, activity, mechanism, event type [ie, competition or practice], playing surface). The AT is able to view and update previously submitted information as needed during the course of a season. To obtain AE data, the AT also provides the number of student–athletes participating in each practice and competition. For exposure data from the first Web-based ISS platform (2002–2003 and 2003–2004 through 2008–2009), ATs created master rosters of all student–athletes and then for each practice or competition indicated with a check box whether an athlete played. For the Injury Surveillance Program (2009–2010 through 2013–2014), how exposure data were entered depended on the EMR or injury-documentation application used. For example, the Sports Injury Monitoring System automatically excludes athletes who were marked as not being currently active because of injury. However, the Injury Surveillance Tool does not automatically account for inactive players.
Participating ATs received training materials via mail and training sessions at the National Athletic Trainers' Association Clinical Symposia & AT Expo. In addition, data quality-control staff were available to provide ATs with phone and e-mail support and to remind them to complete injury and exposure data.
DATA QUALITY CONTROL (2003–2004 ONWARD)
The data that are reported to the Datalys Center contain school and athlete identifiers. However, these identifiers are assigned by the system and do not contain any elements derived from the athlete's name, birthdate, Social Security number, etc. These data identifiers are stored in separate databases that only the users with access privileges (eg, the AT from the institution) can see, retrieve, or manipulate. The Datalys Center is not able to access these records. Datalys removes all such identifiers when it creates data files for external researchers. Thus, the data sets used by external researchers do not include any identifying information or the date of injury. These processes are compliant with the Health Insurance Portability and Accountability Act.
As data are exported, they pass through a verification process, in which data quality-control staff review them, flag invalid values, and resolve pertinent issues. Starting in the 2009–2010 academic year, the ISS used an automated VE system that examines entered data for valid values. Invalid data are flagged, and the VE notifies data quality-control staff and participating ATs, who work together to correct the error before data enter the research database. Specific concerns are explained below.
Exposure data are considered valid if values per exposure are neither zero nor missing. Unacceptable exposure data (1.6% of all exposure data) are replaced with mean imputation values. These mean values are based on all other valid AE data from the same year, division, sport, and exposure (ie, practice or competition). For example, if AE data were missing for a practice occurring in the 2005–2006 academic year in Division III softball, the mean value computed from all other AE data for Division III softball practices occurring that same year would be used.
All injury-data submissions are considered final 30 days after the last postseason competition. The AT can modify injury data after the 30-day postseason deadline for his or her personal records, but these modifications are not sent for inclusion in the ISS data sets.
Each injury event is given a unique identifying injury-event number. Data quality-control staff assess injury events with multiple injuries reported. These injuries are retained in the data set if they all have (1) separate specific injury definitions (eg, an anterior cruciate ligament rupture and a medial meniscus injury) or (2) different body parts (eg, ankle and knee injured in the same event) or both. Otherwise, duplicate injuries are removed.
After the injury-event record is cleaned of duplicate data, its recorded date and season are compared with the date and season of its associated exposure record. If discrepancies are noted between the records, the exposure record is considered correct, and the date and season associated with the injury-event record are changed.
Time loss is also assessed. For data from the first Web-based ISS platform (2002–2003 and 2003–2004 through 2008–2009), the return-to-play date and time loss were entered manually by participating ATs. The time-loss data were then checked by the data quality-control staff to ensure that the value equaled the difference between the time of injury and the return date. For the Injury Surveillance Program (2009–2010 through 2013–2014), how time-loss data were entered depended on the EMR or injury-documentation application used. For example, the Injury Surveillance Tool required manual entry of return date and time loss; however, the Sports Injury Monitoring System calculated time loss based upon injury and return dates. Nevertheless, the VE system ensured that time-loss and return-date data were congruent.
Reported time-loss injuries of more than 1 week were reviewed individually to ensure that the data appeared valid based on the information provided. For example, an anterior cruciate ligament rupture with associated fractures and surgery may require more than 1 year of rehabilitation. However, a contusion would likely not require as much recovery time. If the time-loss value for an injury was negative, the return date was evaluated, and an assessment was made as to whether or not the return date could be logically changed based on the month and day entered (eg, an AT might have accidentally entered the incorrect year). If no consensus could be achieved regarding an appropriate date, the return date was set to missing.
DATA-INCLUSION CRITERIA (QUALIFYING; 2003–2004 ONWARD)
Two criteria were used to qualify teams for inclusion in the analysis. A minimum of 8 weeks of exposure activity was required for a team's data to be included in the analysis. The 8 weeks must have included both preseason and regular-season activities; postseason was not required for inclusion. The 8-week cutoff for inclusion was identified after divisional manuals for the first allowed date of practice and the conclusion of the divisional championship were examined. For sports teams that conclude their season within approximately 12 weeks, 8 weeks represents 70% of the competition season. Because not all schools continue into the postseason competition segment (with the divisional championship as the conclusion) and not all divisions have the same length of seasons, 8 weeks was considered the fewest number of weeks that indicated 70% capture of a regular competition schedule without any postseason competition. The 8 weeks of activity had to occur during the championship season as determined by NCAA legislation.
A second criterion for inclusion was that within those 8 weeks of activities, at least 80% of the minimum required number of NCAA competitions must have been recorded. The number of required competitions for each sport in both Web-based platforms was determined using the 2007 Division I, II, and III manuals. If the 80% minimum number of competitions was different across divisions, the smallest number was used to qualify all 3 divisions.
SAMPLING WEIGHTS AND ADJUSTING FOR UNDERREPORTING (2003–2004 ONWARD)
All NCAA ISS data sets include weights that can be applied to data to generate national estimates that adjust for potential underreporting of injuries.
Poststratification sample weights, based upon sport and division, help ISS data provide national estimates of injury events occurring in collegiate sports based upon the sampled teams. In addition, because of year-to-year variations in the reporting sample, poststratification sample weights are modified every academic year. Poststratification sample weights are calculated using the formula
where weightijk is the weight for sport i in division j in year k.
Adjustment for Underreporting of Injuries
Underreporting has been found in other injury-surveillance systems.6–8 In the ISS, injuries may be underreported by ATs because of competing demands on ATs' time and the dynamic nature of the athletic training facility environment. A validation study matched and compared ISS data with data abstracted from other types of clinical records maintained by ISS ATs.9 A sample of 15 universities that provided data on men's and women's soccer to the ISS for at least 2 years from 2005–2007 was used. The validation study estimated that the ISS captured 88.3% (95% confidence interval = 85.8%, 90.6%) of all time-loss medical-care injury events. This high level of agreement between the ISS and clinical medical records suggests that the Web-based ISS provides valid injury estimates. Based on these findings, weights were further adjusted to correct for underreporting by scaling weighted counts up by a factor of 0.883−1. Although Kucera et al9 used ISS data only for soccer, the weighting was applied to all ISS data under the assumption that underreporting does not vary by sport, year, school, or division.
NCAA POLICY AND RULE CHANGES
Aggregate data, in the form of sport-specific reports containing text and tables, are provided to the NCAA annually and used by committees such as the Committee on Competitive Safeguards and Medical Aspects of Sports to develop health and safety policies and monitor ongoing injury trends. The ISS is particularly valuable given its ability to generate analyses related to the effects of policy and rules changes on injury rates. Examples of policy and rules changes are provided in Table 5.
GAINING ACCESS TO THE 2004–2005 THROUGH 2013–2014 ACADEMIC YEAR DATA SETS
The comprehensive sport-injury data collected by the Web-based ISS provide exposure and injury data sets. The variables collected for the 2004–2005 through 2013–2014 academic year data set that are available for research requests can be divided into 2 categories: exposure variables and injury variables (Table 6).
The 2004–2005 through 2008–2009 academic year data sets are available free of charge to external independent researchers. Interested researchers can apply at the Datalys Injury Statistics Clearinghouse Web site (http://www.disc.datalyscenter.org). Data for the 2009–2010 through 2013–2014 academic years will soon be available to external researchers. Because of the methodologic changes between the 2004–2005 through 2008–2009 and 2009–2010 through 2013–2014 data sets (eg, inclusion of non–time-loss injuries), we recommend that such data be reported separately.
To date, the Datalys Center has supplied these files at no cost to the applicants, provided the data requests do not impose an excessive data-management or data-analysis burden. Documentation on these data files is available on request from the Datalys Center. Numerous researchers have applied for subsets of ISS data, and a majority of data requests are approved. Applications are reviewed by the Datalys Center's IRC, which comprises an external board of scientists and epidemiologists unaffiliated with the Datalys Center or the NCAA. The IRC sometimes requests that applications be revised to offer additional information or improve clarity regarding proposed research questions. Rejections typically occur because the ISS data are not appropriate (eg, insufficiently detailed) to answer the research questions proposed by applicants. Upon IRC and NCAA approval, researchers must sign a data-release agreement before receiving the deidentified data sets. As of February 2014, more than 1 dozen applicants had been approved and provided with data sets. Several successful applications have resulted in peer-reviewed journal articles, although most have focused on injuries in collegiate football players.10–13
The NCAA and ATs have collected collegiate student–athlete injury data for 30 years using the ISS. The annual review of these data by the NCAA provides a unique opportunity to advance the safety and health of student–athletes and to implement policy and rules changes based on surveillance data.
More information about the Datalys Center and its oversight of the ISS is available at www.datalyscenter.org. Those NCAA-affiliated schools interested in becoming part of the ISS can contact Datalys Center staff at (866) 807-0075. Sport-injury fact sheets describing ISS data can be found at www.datalyscenter.org and http://www.ncaa.org.
We thank the many ATs who have volunteered their time and efforts to submit data to the NCAA ISS. Their efforts are greatly appreciated and have had a tremendously positive effect on the safety of collegiate athletes.