Athletic Training Students’ Mental Health Recognition and Referral Skills, Part 1: A Randomized Controlled Trial

• Mental Health First Aid (MHFA) training improved athletic training students’ knowledge and confidence with mental health recognition and referral.

• The use of standardized patient encounters and case-based learning interventions following the MHFA standardized curriculum did not further enhance knowledge or self-reported confidence with mental health recognition and referral.

• The use of simulation offered participants the opportunity for continued practice and feedback in a safe environment that mimics clinical practice.

The growing prevalence of mental health conditions in the United States^1–3  underscores the importance of early recognition, support, and referral for those in need. Athletic trainers (ATs) are in a unique position to address this need. Despite the importance of ATs recognizing the signs and symptoms of mental health challenges and appropriately referring patients, professional confidence and competence in this area need improvement.^4–6  Previous literature reported that during educational preparation, ATs lack experience interacting with patients undergoing mental health challenges.⁴  Additionally, credentialed clinicians’ confidence in this area has been identified as being deficient by their supervisors⁵  and through self-reports.⁶  Expanding mental health research within athletic training practice aligns with and supports the priorities of the Strategic Alliance’s research agenda,⁷  the Board of Certification (BOC) practice analysis,⁸  and the Commission on Accreditation of Athletic Training Education (CAATE) standards.⁹ 

To adequately equip athletic training students with the skills needed to assist patients with mental health concerns, educators need to evaluate approaches for content delivery and application. Mental Health First Aid (MHFA), a standardized curriculum offered by the National Council for Behavioral Health,¹⁰  is an interactive option that has been incorporated into professional and continuing education by a variety of health care professions.^11–13  As a public education tool, MHFA training introduces signs and symptoms associated with mental health challenges and provides referral resources, which can increase literacy and reduce stigma.¹⁰  While MHFA does not go into depth about all mental health disorders, it does highlight some conditions, provides participants with opportunities to identify warning signs, and allows them to practice determining the appropriate next steps using the ALGEE action plan (Figure 1). Thus far, ATs have investigated MHFA as a continuing education offering¹⁴  and as part of an institution’s comprehensive mental health curriculum.¹³ 

While MHFA provides a solid foundation with mental health content, athletic training-specific application is also vital for developing students’ skills. One approach to skill development is simulation, an experiential learning technique that involves the representation or replication of real-world scenarios.¹⁵  The risk-free environment created by simulation allows for practice along with reflection and debriefing, with a focus on skill correction and self-awareness.¹⁶ 

Standardized patients (SPs) and case-based learning (CBL) are types of simulation that can provide athletic training-specific application of content. Standardized patients are trained individuals portraying a specific case. The benefits of SPs include assessment of clinical skills in a controlled environment¹⁷  and improved confidence in transition to practice.¹⁸  Unlike SPs, CBL examines a realistic case of patient care through a written narrative rather than live patient interaction. Students have reported that CBL is enjoyable and has a positive impact on their learning.¹⁹  An interrupted case format progressively presents information, with pauses for student reflection, and has been shown to increase critical thinking, learning, and retention.²⁰ 

Previous studies within athletic training education have compared CBL and SPs; however, these examinations have lacked a control group.^21,22  A study examining lower-extremity evaluation skills compared group SP encounters and individual CBL with 1 cohort of students. While both techniques increased student confidence and reflection on specific actions, the CBL improved internal organization, and the SP encounter highlighted the benefit of peer learning.²¹  An exertional heat stroke study found that hybrid simulation and CBL led to similar results on a knowledge assessment.²²  Both of these explorations incorporated some small-group simulation techniques but did not include a control group.

Previous explorations of mental and behavioral health learning strategies have included only single institutions. A recent study reported positive student reflections on incorporating exploratory counseling, MHFA, and SP encounters as a way to meet accreditation standards for mental health recognition and referral. Another institution implemented a scaffolded suicide prevention curriculum that incorporated didactic training, experiential exercises, and the creation of a mental health emergency action plan.²³  For greater generalizability, an examination of educational interventions focused on behavioral and mental health with students from multiple institutions is needed.

The purpose of this study was 2-fold: (1) to investigate the effect of MHFA training on student knowledge and confidence in mental health recognition and referral and (2) to compare the impacts of small-group CBL and SP encounters after MHFA training on these same measures.

A sequential, explanatory mixed-methods design was used.²⁴  The study was divided into 2 phases. This paper focuses on phase I, a randomized controlled trial involving pretest and posttest assessments of knowledge and confidence. We used the Consolidated Standards of Reporting Trials (CONSORT) checklist to guide the study’s rigor and reporting. Phase II involved follow-up interviews with participants and is presented as a separate work.

A convenience sample of students from CAATE-accredited, graduate-level professional athletic training programs was recruited between October 2021 and May 2022. Program personnel were invited to incorporate the educational interventions into their curriculum. For a program to be invited, its students needed to have some experience with simulation (eg, task trainers, SPs, mock patients). Students from 14 cohorts (students from the same program at equivalent academic progression) representing 11 institutions completed the educational interventions. While recruitment and allocation occurred programmatically, individual students volunteered to participate in the study by completing electronic informed consent before accessing the baseline assessment. Participants were excluded if they had previously completed MHFA training. Participants’ previous experience with other behavioral health instruction or virtual interventions did not influence their eligibility.

Using an online randomization tool, the primary investigator (ASA) assigned institutional cohorts to the intervention groups (MHFA only, MHFA + SP, and MHFA + CBL). To achieve sufficient responses, each intervention group had 4 or 5 cohorts assigned, totaling 54, 52, and 47 potential students, respectively.

The target sample size was 48 as determined by G*Power based on a power of 0.95, an alpha value of 0.05, and an effect size of 0.3. While 121 students were enrolled in the study, only 70 also completed the posttest. The overall attrition rate was 42.14%.

Institutional review board approval was obtained from Rocky Mountain University of Health Professions. All students in participating cohorts completed the MHFA training and subsequent interventions as a programmatic requirement. The MHFA training was provided free of charge and resulted in a national certification upon individual completion. However, students volunteered to complete the study’s knowledge and confidence assessments.

To maintain separation between assessment data and participant identity while also allowing for comparison of pre- and posttest results, each participant provided an identification code for both assessments. This code included their institutional cohort abbreviation followed by the last 4 digits of their phone number. Since group allocation was done via cohorts, the institutional cohort abbreviation allowed for appropriate coding and comparison.

Shortly after the completion of the baseline assessment, all participants took part in the MHFA training. The National Council for Mental Wellbeing’s nationally standardized MHFA curriculum involved online individual prework followed by live, synchronous training. The primary investigator facilitated two 3-hour training sessions within 1 week via Zoom for each cohort.

Three to four weeks after their MHFA training, participants engaged in SP or CBL encounters. After their assigned encounter, participants were immediately sent the posttests. Since the MHFA-only group had no follow-up encounter, they were emailed the posttest link 3–4 weeks after their initial MHFA training to maintain consistency with the other groups. The CONSORT flow diagram (Figure 2) details how many individuals were involved in each phase of the study.

Qualtrics XM (Qualtrics) was used to distribute the data collection instruments, including a demographic survey, knowledge assessment, and confidence scale. The demographic survey screened for the exclusion criteria of previous MHFA training and then collected basic categorical information about participants, including semesters completed in the program, age, NATA district, and gender.

The baseline knowledge assessment incorporated MHFA content using multiple-choice, multiselect, and true/false questions. The assessment was validated with a content validity index (CVI) whereby a panel of 3 individuals rated item relevance and clarity on a 4-point scale.²⁵  The individual ratings were averaged to create a CVI score; the minimally acceptable threshold for CVI scores was set at 0.75.²⁵  The grand mean for relevance was 3.75; the grand mean for clarity was 3.42 (Table 1).

The CVI panel consisted of 2 athletic training educators with MHFA instructor certification and 1 MHFA instructor trainer. Of the 15 items on the initial draft, all items met the minimally acceptable threshold for relevance and clarity. However, based on some of the reviewers’ comments, 3 items were eliminated due to possible incongruence with the MHFA training.

The final knowledge assessment had 4 true/false items, 4 multiple-choice items, and 4 multiselect items. Multiselect items had 2 correct responses; each was scored as half a point. As part of the study’s outcomes, each item on the assessment was scored as 1 point for a possible total of 12 points. Since the knowledge assessment included multiselect questions, it was not compatible with performing Cronbach’s alpha analysis to measure reliability.

A previously published and validated clinical performance confidence scale²⁶  was adapted to include mental health-specific items (Figure 3). The confidence scale was assessed for relevance and clarity using the above-mentioned CVI procedures. Of the 13 initial items, 9 met the minimally acceptable threshold for relevance. Of these 9 items, 2 did not meet the acceptable level for clarity. These 2 items along with 1 other were revised based on reviewer comments and returned to the panel for evaluation of clarity. The panel confirmed that all 3 revised items met the acceptable threshold for clarity (Table 2). The grand means for relevance and clarity were 3.77.

The finalized instrument consisted of 9 items using a Likert scale, with a score of 1 representing strongly disagree and 5 representing strongly agree. Therefore, confidence scores had the potential to total 45 points. Cronbach’s alpha analysis was performed, with a resulting coefficient of 0.88, which indicates that the confidence scale had good reliability.

The SP and CBL encounters involved the same “case” of a patient who was experiencing grief, stress, and related psychosomatic symptoms. The primary investigator created the case using an existing SP template that was adapted with permission (Table 3). The SP case was then converted to an interrupted CBL format (Table 4). Both documents were peer reviewed by a panel of 3 athletic training educators familiar with CBL and/or SPs to ensure overall quality and uniform presentation between the formats. The panel confirmed that the CBL and SP cases were similar and clear. Specific feedback from reviewers was integrated into the final draft to improve consistency and clarity. To allow for similarity between allocations, both SP and CBL encounters occurred in small groups (2–5 participants from the institutional cohort).

Individuals assigned to the SP encounter interacted with a trained individual during a 15–30-minute video conference with peers from their institutional cohort (2–5 participants), followed by debriefing with the primary investigator. Due to cohort size variability, 18 small-group sessions occurred with 52 students.

The SP’s training consisted of 3 parts: a written case guide, a recorded orientation, and a meeting with the primary investigator. Initially, the SP received a written case guide that provided background about the scenario, their opening statement for the encounter, and pertinent patient history information. The SP then reviewed a prerecorded introduction to the case before meeting with the primary investigator. This recording provided guidance on how to prepare for the encounter in terms of tone, interpersonal skills, and the case context. During the meeting with the primary investigator, the SP asked questions about the case and honed their portrayal through role-play.¹³ 

Individuals assigned to the CBL activity cooperatively examined the same case through an interrupted format, meaning that the case was introduced in stages. The primary investigator facilitated the participants’ discussion of pertinent details and brainstorming of appropriate decision-making. In addition, each participant had the opportunity to role-play appropriate responses. The CBL sessions were 30–45 minutes via Zoom. Participants were in small groups with members of their institutional cohort (2–5 people); based on cohort size, there were 47 students across 11 group sessions.

Statistical analyses were run with Intellectus Statistics (version 1.01). Participant demographic information produced descriptive statistics. The influence of the pedagogical strategies on knowledge and confidence was analyzed via a mixed-model analysis of variance (ANOVA) with the a priori value set at a P value of < .05. Before running a mixed-model ANOVA, it was determined that the assumption of normality was met using a Q-Q scatterplot.²⁷ 

Students were recruited from institutions located in 6 NATA districts. The participants were aged 23.3 ± 2.27 years old with a gender breakdown of 62.86% women, 35.71% men, and 1.43% nonbinary. Participants had completed an average of 2.87 ± 1.76 didactic semesters and 2.3 ± 2.27 semesters of clinical education. Full participant demographics can be found in Table 5.

Participants showed improvement in their knowledge assessment scores (max score = 12, pretest average = 8.65, posttest average = 10.28). Using a mixed-method ANOVA, the main effect for the within-subjects factor was significant [F_(1,67) = 70.31, P < .001], indicating that there were significant differences between the pretest and posttest values for knowledge. The reported effect size (η² = 0.51) was large.

A similar result was found for confidence. Participant scores increased from 29.65 on the pretest to 39.21 on the posttest. The main effect for the within-subjects factor revealed a significant difference [F_(1,67) = 206.41, P < .001] between the values of pretest and posttest confidence scores. In addition, the η² value of 0.75 is considered a large effect size.

The knowledge and confidence score averages are presented in Table 6. Full mixed-method ANOVA results are shown in Table 7.

Pretest and posttest scores were similar among intervention groups as indicated by an examination of the main effect for allocation [knowledge: F_(2,67) = 0.21, P = .807; confidence: F_(2,67) = 1.29, P = .281].

The mixed-model ANOVA showed no significant difference between the control, SP, and CBL groups in terms of knowledge or confidence improvement. For knowledge, the interaction effect of the within-subjects factor and allocation was not significant [F_(2,67) = 0.26, P = .771], indicating a similar relationship between pretest and posttest. The relationships between pretest and posttest confidence scores were similar between levels of allocation, with an interaction effect of an F_(2,67) value of 0.08 and a P value of .919.

As athletic training students and professionals increasingly encounter patients with mental and behavioral health concerns, MHFA provides an opportunity to deliver updated information about signs and symptoms, distinguish crisis from noncrisis situations, and provide basic intervention and referral strategies.¹⁰  Our results showed a significant improvement in participants’ knowledge and confidence related to managing mental and behavioral health issues after their MHFA training. While all participants’ knowledge and confidence improved, there were no differences in improvement between the 3 intervention groups.

MHFA is evidence based, standardized, and developed by individuals with content expertise in mental and behavioral health. While MHFA does not cover each component of the CAATE standards for behavioral health pertaining to policy creation, action planning, examination, diagnosis, and intervention, it does addresses core content related to anxiety disorders, depression, mood disorders, psychosis, trauma, and substance use disorders. Thus, incorporating MHFA into athletic training programs adheres to high standards of content delivery and may reduce the preparation burden on athletic training educators.

Our study’s increased scores for knowledge and confidence align with the findings of other examinations of MHFA training within health care education. Incorporating MHFA training into pharmacy education improved the identification of mental illness, decreased stigmatizing attitudes, and increased confidence in providing pharmaceutical support.¹²  A study of nursing students indicated significant improvements in knowledge, confidence in helping, destigmatization, and mental health first aid intentions both immediately after MHFA training and at follow-up.²⁸  Within athletic training education, Ostrowski et al¹³  found that MHFA followed by an SP encounter boosted student confidence and the ability to recognize, intervene, and refer patients experiencing mental health challenges. MHFA training seems to be an appropriate way to improve health care students’ knowledge and confidence with mental health recognition and referral.

The increased confidence reported by our study’s athletic training students mirrors the findings of an exploration of certified ATs in which self-efficacy scores were significantly higher after MHFA training than pretest.¹⁴  Self-efficacy and confidence are closely related as descriptors of participants’ perceptions of their abilities. Given that ATs have been reported to lack confidence in providing care and referral to patients with mental health concerns,^4–6  MHFA training incorporated into professional education or provided as a continuing education offering may help bridge this gap.

We found no difference in knowledge and confidence between CBL and SP encounters following a didactic learning experience. In examining the literature, we found a range of results in similar studies. Schwartz and colleagues, for example, found no difference between CBL and human patient simulation (HPS) on students’ clinical examination performance.²⁹  In their study, medical students completed required didactic sessions on the emergency management of acute chest pain and were then randomly assigned to either the CBL or HPS group; their end-of-clerkship objective structured clinical examinations (OSCEs) revealed similar overall and subscale scores between the intervention groups.²⁹ 

While Schwartz and colleagues studied clinical performance, others have found differences between intervention modes when assessing knowledge acquisition. For example, Aluisio et al conducted a randomized controlled trial examining the effectiveness of CBL or SPs in addition to didactic instruction in disaster triage preparedness among nursing students.³⁰  They found that CBL outperformed SP experiences in terms of knowledge acquisition.³⁰  In contrast, an evaluation by Lee Chin et al³¹  of HPS and CBL in pharmacy students found that students in the HPS group had greater changes from pre- to posttest than the CBL group. Given the variability between these studies and our results, there does not seem to be an intervention that consistently performs better.

The use of simulation for high-risk, low-repetition skills has shown benefits for student knowledge and confidence in athletic training education. Kinslow et al²²  compared hybrid simulation to CBL for the assessment and treatment of exertional heat stroke and found that both strategies similarly improved student scores on the Knowledge, Preferences, and Practices of Certified Athletic Trainers Regarding Recognition and Treatment of Exertional Heat Stroke (KPP-EHS) survey. Similarly, a study focused on high-fidelity simulations for cardiopulmonary resuscitation (CPR) found an increase in student knowledge acquisition and confidence from pretest to posttest,³²  and a study of facemask removal using an SP found increased performance and confidence.³³  While the types of simulation used in each of these studies varied slightly from the CBL activity and SP encounters that we incorporated, similar impacts on knowledge and confidence were observed. Unlike those previous studies, our examination also incorporated a control group receiving only the MHFA training.

Our control group showed improvement similar to that of the simulation groups. This differs from other findings in the literature. Therefore, variables surrounding the simulation strategies and the standardized education used for original content delivery should be investigated further. A medical laboratory technology program compared the incorporation of SPs and CBL into a cell biology course to a control group that experienced traditional methods of lecture and experiments; the final scores in the course were significantly higher for the students who engaged in SP and CBL experiences.³⁴  For our study, the didactic exposure was quite comprehensive as the MHFA training involved both self-paced and synchronous work totaling 8 hours. Perhaps this intensive exposure may surpass typical didactic instruction before CBL or SP intervention, thereby leading to less of an impact of the simulated experiences. The CBL and SP interventions in our study were 30–45 minutes. It is also possible that the difference in contact times between the didactic instruction and intervention strategies also contributed to the lack of a difference in scores between the allocation groups. Perhaps multiple follow-up simulations would have revealed significant gains in knowledge and confidence compared to a control group.

As ATs evaluate professional and continuing education offerings, it is helpful to know how adding types of simulation compares to just completing an interactive training curriculum. MHFA training has demonstrated the ability to improve knowledge and confidence with recognition and referral for mental and behavioral health concerns. Incorporating MHFA training into professional athletic training programs as well as continuing education has the potential to improve clinicians’ skills and abilities in caring for patients experiencing mental health challenges.

Educators who look to engage participants in simulated sessions to enhance skill development following didactic-based lessons have many options. Given the required resources associated with SP encounters, the use of CBL offers an economical approach to reinforcing knowledge and skills while allowing for practice in a low-risk environment.

Since MHFA is geared toward a lay audience, providing students and professionals with opportunities to apply their knowledge and skills to athletic training scenarios may help transfer content to practice. While there was not a statistically significant difference between those participants who did only the MHFA training and those who performed the CBL activity or SP encounter, participant interviews (found in Part 2) support increased feelings of comfort, confidence, and capability from discipline-specific practice.

Although didactic contact with MHFA was uniform, the students’ previous and concurrent didactic and clinical experiences and exposure to the study’s pedagogical techniques were not controlled. The convenience sample limits generalizability, and the randomization of cohorts rather than individuals leaves room for confounding.

In addition, our participant attrition rate was 42.14%. While 121 students took the pretest, only 70 posttests were recorded. Either the remaining 51 students failed to complete the posttest or their identification codes could not be linked to a pretest. Additionally, some individuals did not accurately provide their institutional abbreviation or unique 4-digit code. A few factors that may have influenced attrition include the timing of the posttest near the end of a term, time elapsed between training and the posttest, and failure to use the correct identification code.

This study focused on knowledge acquisition and self-perceived confidence and did not include skill-based assessments, such as a practical examination, of recognition and referral. Future research should examine the effects of mental health simulation experiences on skill performance.

This study used only 1, short, small-group encounter with CBL or SPs. Future research could incorporate an encounter of greater length, multiple encounters, or an individual experience rather than a small-group encounter compared to a control group. Using an individual evaluative SP experience to assess students’ clinical application of knowledge as an additional outcome measure may allow for better differentiation between the interventions’ benefits. In addition, exploring differences in retention of knowledge and confidence between those in SP, CBL, or MHFA-only groups is another potential avenue.

Due to the increasing commonness of hybrid and online instruction in athletic training professional preparation and continuing education, virtual presentation of MHFA training, SP encounters, and CBL activities was used for this study. It would be interesting to compare modes of instruction, both virtual and face to face, to determine if the delivery method influences participant knowledge and confidence with mental health recognition and referral.

While this study focused on mental and behavioral health, future comparisons of CBL activities and SP encounters with a control group could be done with other content areas pertinent to athletic training. Mental and behavioral health topics may be considered a novel content area within athletic training; the lack of distinction between allocation groups may be tied to the steeper learning curve of this material. Future research using a less complex or more common topic may allow a better comparison of the intervention strategies.

As athletic training professionals seek continuing education to enhance knowledge and skills transferable to their clinical practice, MHFA training is a promising offering. Likewise, MHFA training provides professional athletic training programs with an effective instructional tool to present content relevant to the CAATE curricular standards for mental and behavioral health. This standardized curriculum and the virtual delivery option offer a comprehensive, practical, and flexible approach. While a difference in scores between the allocations (CBL, SP, and control) was not found, the use of simulation offered participants the opportunity for continued practice and feedback in a safe environment that mimics clinical practice.

This study was made possible in part by the financial support of the Rocky Mountain University of Health Profession’s Graduate Research Support Fund and the Illinois Athletic Trainers’ Association Research Grant.