Development of a Heat-Illness Screening Instrument Using the Delphi Panel Technique

Exertional heat stroke is the third leading cause of death in the United States among high school athletes,1 and, with effort to reduce risk factors, many heat illnesses can be prevented.2–4 Screening athletes during preparticipation physical examinations (PPEs) can help health care professionals identify predisposing factors of exertional heat illnesses (EHIs). Using the PPE to identify patients at risk for EHI can provide the athletic trainer (AT) with information about predisposing conditions that might not otherwise be disclosed.5,6 Subsequent action to reduce these risks is an essential component of the prevention process. Typically, the PPE includes an evaluation of general medical considerations and orthopaedic injuries; however, ATs would be better equipped to prevent injury and illness with more information about any previous history of cardiovascular, respiratory, and heat illnesses.5–7 

The recognition of inherent risk factors can help practitioners make sound clinical decisions when extrinsic risk factors can inhibit safe participation. Extrinsic risk factors include exercising in warm or hot, humid environmental conditions; wearing protective equipment; having inappropriate work-to-rest ratios; or having insufficient access to water and shade.2 The intrinsic risk factors for EHI include history of EHI; poor cardiovascular and physical fitness (and accompanying obesity); inadequate heat acclimatization; dehydration or electrolyte imbalance; recent febrile illness; sleep deprivation; a “never give up” or “warrior” mentality; a high level of motivation or zealousness; and use of questionable drugs, herbs, or supplements.3,4 These intrinsic risk factors of EHI can be identified during the PPE, but most examinations are inadequate to obtain enough information to identify individuals at risk. Current research supports extending the length of the PPE to include more indicators for cardiovascular, respiratory, and general medical conditions, including EHI.5–11 Expanding the PPE would allow practitioners to identify at-risk athletes and likely would prevent undue injury or illness. Using a preparticipation screening instrument to identify intrinsic risks for EHI would allow ATs to determine which individuals might be susceptible to heat illnesses.8 Therefore, the purpose of our investigation was to determine content validity of a heat-illness screening instrument, the Heat Illness Index Score (HIIS), designed to be used by the AT as part of the PPE.

We used the Delphi panel technique to estimate content validity of the HIIS. The Delphi panel technique is a research design using several rounds (3–5) of communication among experts to establish consensus for the content.12–19 The technique uses the opinions of expert panelists while maintaining anonymity among them.12–19 This is the preferred technique for determining content validity because some researchers have suggested that focus groups and consensus conference techniques often force participants into consensus or that one or a few experts might dominate the consensus process.17 Selection of panelists or experts has been questioned throughout the literature because of investigator bias13; however, choosing panelists who provide a balance of investment in the topic and impartiality helps to develop a qualified panel.18 We used the Delphi panel technique to establish consensus on the content and quality of the HIIS instrument by sampling and interviewing individuals across diverse locations and with expertise in EHI.17 Although we were not blinded to each panelist, we requested that they keep their participation confidential in an effort to maintain anonymity among panelists.

We recruited potential panelists via telephone and provided a brief overview of the investigation. We selected a panel of 6 experts (5 researchers, 1 clinical AT) using the following criteria: certified AT; environmental illness researcher or team physician; publications (total = 236, mean = 39 ± 60) and presentations in scholarly journals or at clinical symposia related to environmental illness; advanced degree in the area of kinesiology, exercise physiology, or exercise science; and/or clinical experience with frequent exposure to the prevention, recognition, and treatment of EHI. Upon agreement to engage in the investigation, we scheduled individual semistructured interviews at the annual meeting of the National Athletic Trainers' Association in 2006 or by telephone. During the interview session, we explained the objectives, procedures, risks, and benefits of the study. Panelists provided written informed consent, and the institutional review board of Florida International University approved the study.

The HIIS instrument was developed as a screening tool to identify the 10 major risk factors for EHI as outlined in the “National Athletic Trainers' Association Position Statement: Exertional Heat Illnesses”3 and the “Inter-Association Task Force on Exertional Heat Illness Consensus Statement”4 (Table 1). The instrument was designed to be administered by an AT during the PPE using questions and clinical information available in the athletic training clinical setting. We created objective and measurable items and subitems from the intrinsic risk factors of EHI.3,4 Each risk factor was attached to a 5-point Likert scale, with 0 indicating lowest risk and 4 indicating highest risk. The rating for the risk factor was summed at the end of the instrument. A rating of high risk was associated with a total score ranging from 30 to 44 or a score of 4 on 3 or more questions, a rating of moderate risk was associated with a total score ranging from 15 to 29, and a rating of low risk was associated with a total score ranging from 0 to 14. Areas to include additional descriptive information were also available for several items. In addition, we included the maximal oxygen consumption (V˙o_2max) run test as a physiologic measure of overall fitness because it is strongly correlated with direct measurement of V˙o_2max on a treadmill.20 

The Delphi panel technique commonly uses 3 rounds of review but can use up to 5 rounds until consensus is achieved. Our investigation required 3 rounds of review.

Although an interview is not a required procedure within the Delphi panel technique, some researchers have suggested that the personal effect of face-to-face initial contact with the researchers influences panelists to maintain participation through subsequent rounds.15 Therefore, we conducted semistructured interview sessions with the participants. We allowed panelists time to review the instrument and then instructed each panelist to answer a series of questions (Table 2). Immediately after the interviews, data were transcribed and coded with the feedback used to revise the HIIS instrument. Approximately 2 weeks were required to analyze and organize round 1 data.

In round 2, we sent panelists the revised instrument, a summary of data gathered in round 1, reference documents,3,4 a detailed list of revisions, and a questionnaire. We instructed panelists to complete the questionnaire by marking their levels of agreement with an X on a visual analog scale (11.4 cm), with the extreme indicators of strongly disagree and strongly agree, for each of the 43 revisions. After providing feedback for the suggested revisions, we instructed the panelists to rate their overall agreement with each revised item and their overall agreement with the scoring criteria for each item. The panelists marked their levels of agreement with an X on the visual analog scale. We analyzed the data using the mean and SD. We accepted revisions if the mean was greater than agree and the SD still permitted a positive response (≥midline). We also instructed the panelists to provide additional comments and suggestions if they did not agree with any of the items.

In round 3, we sent panelists the revised HIIS and a detailed list of revisions with another questionnaire. Because we were approaching consensus, we instructed participants to rate their levels of agreement with each item and to rate their overall levels of agreement with the instrument on a 4-point Likert scale, with anchors of 1 (strongly disagree) and 4 (strongly agree). Participants also had space to provide additional comments or suggestions for the final instrument. We calculated the mean and SD for each question in the questionnaire. The HIIS items were accepted for the final instrument if panelists demonstrated consensus greater than or equal to 3. We calculated frequency of responses for the overall level of agreement with the instrument, and the HIIS was accepted if panelists demonstrated a consensus of responses greater than or equal to 3.

In round 1, panelists supported the inclusion of 7 of the 10 questions in the instrument. Panelists supported the other 3 questions but requested revisions. We gathered the panelists' suggestions and revised the instrument accordingly.

Based upon the feedback from the panelists in the first round of interviews, we developed 43 revisions for the instrument. When we asked the panelists to rate their overall agreement with the 43 revisions, they rejected 7 (16.3%) and suggested further modification to 2 of the revisions. We used the panelists' quantitative and qualitative feedback from the questionnaire to further develop the instrument.

Because the means were greater than or equal to 3 (Table 3), all items were accepted in the HIIS. Furthermore, 100% of the panelists agreed (n = 3 of 6) or strongly agreed (n = 3 of 6) with the content of the final version of the instrument ( Appendix).

The purpose of our investigation was to determine content validity of a heat-illness screening instrument. We established content validity with 3 rounds of panelist consensus. We believe that the instrument requires further revision and implementation to establish diagnostic accuracy and clinical usefulness.

The risk factors associated with EHI are well established in the literature, and the development of a screening instrument is a logical step toward identifying individuals at inherent risk of EHI. Recently, Cooper et al21 investigated the presence of heat illness at 5 southeastern US universities. In addition to gathering environmental-condition data, the researchers also instructed ATs to report the occurrence of EHI throughout 3 months of football training and competition.21 They found 139 EHIs, which primarily included heat cramps, heat exhaustion, and heat syncope, were reported over approximately 33 000 exposures. Furthermore, professional position stands and consensus statements have identified the best methods for prevention, recognition, and treatment of EHI.2–4 Prevention includes appropriately identifying at-risk athletes and educating athletes to reduce risk factors that they can control. Moreover, it is the role of the AT to restrict or modify participation when the risk is too great. Although we were unable to implement the instrument to identify its ability to attenuate these occurrences of heat illnesses, we believe we were able to establish the appropriate content to do so.

Screening instruments, particularly PPEs, have been used for more than 30 years to identify potentially harmful illnesses or conditions that might limit participation.9 The medical-history aspect of a PPE has the potential to identify almost 75% of the conditions that prohibit participation,11,22 yet more and more conditions are causing concern and should be included.11 The American Academy of Pediatrics considers a history of heat illness to be a potentially disqualifying condition and recommends individual evaluation to determine the risk for participation.9,11 When we can reveal previous medical history of conditions, such as recurrent heat stroke or rhabdomyolysis, practitioners can make the appropriate adjustments to restrict or modify activity in extreme environmental conditions. These recommendations can be followed only when health care professionals are able to access this information before participation by using a valid screening instrument. Through the consensus of experts, we identified the appropriate criteria for expansion of the history-collecting capabilities of a PPE to include risk factors for EHI. Implementation will help to further evaluate the criteria and identify the variable importance of each risk factor in future investigations.

The identifiable risk factors for EHI, both intrinsic and extrinsic, should serve as a means of awareness for ATs responsible for preventing EHI. In this investigation, experts agreed that failure to train in warm or hot, humid environmental conditions while wearing protective equipment; having a history of EHI; poor cardiovascular and physical fitness (and accompanying obesity); dehydration, electrolyte imbalance, or inadequate heat acclimatization; recent febrile illness; sleep deprivation; a “never give up” or “warrior” mentality; a high level of motivation or zealousness; and using questionable drugs, herbs, or supplements were important data to collect via the HIIS. All criteria achieved a level of agree (Table 3), and the item with the most contention and variability concerned motivation (item 4). All panelists remarked (via the additional feedback sections on the questionnaires) that, although extremely crucial, this information would be difficult to measure objectively. These remarks likely accounted for the lower score and variability among the panelists on this item.

Although the general consensus supports the use of the PPE, evidence has suggested that the PPE does not effectively screen patients for a variety of preventable, catastrophic conditions.23 We should work to establish accuracy (the ability to detect the target condition) and effectiveness (detection that improves the likelihood of favorable outcomes)23 within these screening instruments. Future implementations of the HIIS should include a comparison group and rates of participation among all athletes screened that will allow us to determine sensitivity and false-negative rates (diagnostic accuracy). In addition, a linear regression model should be used to determine the predictive capabilities of each item because some items might not be necessary or might not have the same weight in the final score. Until the instrument can be implemented on a large scale to determine diagnostic accuracy, we suggest that practitioners include questions related to EHI on the typical PPE (Table 4)24 to identify risk factors. Finally, an instrument that does not require implementation by an AT and can be completed by the athlete might be the most efficient means of screening athletes.

We used a Delphi panel technique with the consensus of experts to estimate content validity of a heat-illness screening instrument, the HIIS. Future research is necessary to refine a user-friendly and effective instrument for screening athletes. Although a valid instrument is not finalized yet, we suggest that practitioners ask questions related to EHI risk factors during the PPE until an instrument with strong diagnostic accuracy is available. Furthermore, we invite feedback from those using the HIIS in clinical practice.

Abbreviations: ATC, certified athletic trainer; EMT, emergency medical technician; USG, urine specific gravity; V˙o_2max, maximal oxygen consumption.

Reference for Borg scale: Borg.25 

Manufacturers: Ritalin (Novartis International AG, Basel, Switzerland); Red Bull (Red Bull North America, Santa Monica, CA).