Context

There have been multiple education reforms aimed at incorporating patient-reported outcome measures (PROMs) into routine clinical care. However, many factors can impact PROM use, including employment setting/policies or access to literature.

Objective

To explore the influence of educational background and employment setting/policies on athletic trainer (AT) attitudes and practices related to PROM use in the secondary and collegiate settings.

Design

Cross-sectional survey.

Setting

Secondary schools; National Collegiate Athletic Association Division I, II, and III; National Association of Intercollegiate Athletics; and junior colleges.

Patients or Other Participants

Five hundred and seven ATs working in secondary and collegiate settings.

Data Collection and Analysis

An established scale evaluating attitudes toward PROMs was distributed to a purposeful sample of 4000 ATs. Also included were questions concerning use of PROMs, employer policies related to PROMs, access to non–National Athletic Trainers' Association (NATA) journals, and educational background. Scores on each scale were compared based on educational background, employment setting/policies, and access to non-NATA scientific journals using Kruskal-Wallis tests with a Bonferroni correction. Frequency of PROM use was evaluated using chi-square tests (α = .05).

Results

PROMs were viewed positively across all participants, with more positive attitudes toward PROMs observed among those with clinical doctorates (P = .029) compared to those with master's or bachelor's degrees and by those with an athletic training-–related postprofessional master's degree (P = .030) compared to a non–athletic training master's degree. There were no differences in attitudes based on timing of professional degree completion. There were no differences in the usage of PROMs based on educational background or employment setting, with only 10% of respondents reporting routine PROM use. However, access to non-NATA scientific journals (P = .016) and employer policies related to the use of PROMs were associated with increased use of PROMs (P < .001).

Conclusion

Educational reforms have enhanced the attitudes of practicing ATs toward PROMs; however, environmental facilitators, such as expanded access to scientific literature and policies regarding the use of PROMs, are necessary to increase the use of PROMs.

KEY POINTS

  • Those impacted by educational changes intended to increase evidence-based practice, including the use of patient-reported outcome measures (PROMs), demonstrated more positive attitudes and beliefs toward PROM use.

  • Specifically, those with more advanced clinical degrees, such as those with athletic training–related postprofessional degrees and clinical doctorates, indicated more positive attitudes and beliefs toward PROMs.

  • Despite these more positive attitudes, there were no differences in the routine use of PROMs based on educational background or employment setting. However, environmental factors, such as access to scientific journals beyond those associated with the NATA and employer policies related to the use of PROMs, did significantly increase the use of PROMs.

INTRODUCTION

Evidence-based practice (EBP) involves integrating the best available evidence and clinical expertise with patients' individual predicaments and preferences to provide optimal, patient-centered care.1  As the discussion of EBP and resources surrounding proper implementation of EBP in athletic training have increased, so has the focus and attention on the inclusion of patient-reported outcome measures (PROMs) in clinical practice.24  The inclusion of PROMs in clinical practice is a critical component of EBP, as they can be a mechanism with which to incorporate the patient's perspective when developing and executing the treatment plan.3,5  In addition, PROMs are also necessary to improve the documentation of clinical outcomes and to demonstrate the effectiveness of treatments or interventions in athletic training clinical practice.68 

Previous research911  has demonstrated that the actual clinical implementation of PROMs into athletic training practice has been limited, with only 15% to 26% of athletic trainers (ATs) using PROMs in clinical practice There are many barriers to the use of PROMs in practice, including patient completion time, patient difficulty, patient confusion, time to complete and analyze PROMs, and lack of support.911  These relatively low numbers and identified barriers may be a function of practice setting. In traditional athletic training settings, such as secondary schools, colleges, and universities, reimbursement or compensation is rarely associated with documentation of outcomes, policies regarding the collection of outcomes are rare,12  and access to scientific literature or other continuing education resources to provide guidance regarding PROMs may be varied.13  However, at this time it is unclear if employment setting and policies or access to scientific literature have an impact on implementation of EBP and use of PROMs in clinical practice.

The relatively low numbers of ATs that use PROMs in clinical practice may also be due to gaps in professional education. Since at least 2011, there has been a sustained, formal effort to include PROM-related information at all levels of education. At the professional level, the EBP content area was introduced in the 5th edition of the National Athletic Trainers' Association (NATA) “Athletic Training Education Competencies”14  in 2011 and subsequent Commission on Accreditation of Athletic Training Education (CAATE) 2012 “Standards for the Accreditation of Professional Athletic Training Programs.”15  Evidence-based practice was also included in the CAATE 2013 “Standards for Accreditation of Post-Professional Degree Programs.”16  Finally, the addition of the EBP category requirement to fulfill the Board of Certification Inc continuing education requirements went into effect in 2014.17  However, there remain many practicing ATs who completed their professional education before the implementation of most of these educational reforms. Furthermore, the influence of these educational reforms on PROM use has not been thoroughly investigated.

An additional barrier to the use of PROMs, and EBP more broadly, may be a lack of access to the best available evidence.6  It has previously been reported18  that in the field of athletic training multiple databases must be used to perform comprehensive literature searches encompassing those journals considered key to athletic training. Access to literature beyond that provided by the Journal of Athletic Training has been proposed19  to lead to enhanced EBP and better clinical outcomes. ATs with broad access to professional journals have been reported13  to have strong beliefs regarding EBP as well as higher rates of EBP implementation. If clinicians do not have access to the databases and scientific repositories that house current evidence, it may prohibit their ability to conduct EBP and may undermine the objectives of recent educational reforms.

If educational reforms have been successful in modifying athletic training practice to increase the use of PROMs, it would be anticipated that positive attitudes and beliefs and overall use of PROMs would be higher among those who have been most impacted by these educational reforms and that PROMs would be used similarly across practice settings. Therefore, the purpose of this study was to document the current attitudes, beliefs, and practices of ATs related to the use of PROMs during the provision of health care services in the secondary and collegiate settings. Additionally, we sought to document the impact of educational background and work setting on these attitudes, beliefs, and practices. It was hypothesized that those entering the profession after the inclusion of EBP concepts into the educational standards and those with advanced clinical degrees would demonstrate more favorable attitudes toward and greater use of PROMs. Furthermore, it was hypothesized those with greater access to scientific literature would also demonstrate greater use of PROMs.

METHODS

A cross-sectional survey design was implemented. The dependent variables of interest were attitudes related to acquiring and using PROMs, attitudes regarding benefits of PROMs, overall attitude toward PROMs, and actual clinical use of PROMs. The independent variables were highest degree completed, primary employment setting (secondary school; National Collegiate Athletic Association [NCAA] Division I, II, or III; National Association of Intercollegiate Athletics [NAIA]; or National Junior College Athletic Association Junior College [JUCO]), access to scientific literature beyond sources published by the NATA, and completion of professional degree pre-2014 versus 2014 or later. The year 2014 was chosen as this was 3 years after the February 2011 release of the 5th edition of the NATA's Education Competencies14  and the release of the CAATE 2012 “Standards for the Accreditation of Professional Athletic Training Programs,”15  which went into effect in July of 2013. At this time there was clear evidence that concepts relating to EBP should be a part of the knowledge, skills, and abilities of an entry-level AT.4,6,20,21  Additionally, descriptive data and questions regarding what information sources were frequently used to learn about new advances in practice or health care as it relates to athletic training were collected. This research underwent review by the Appalachian State University Institutional Review Board and was approved as exempt. All participants viewed an institutional review board–approved consent document before completing the electronic survey. Manuscript development and data reporting followed the “Strengthening the Reporting of Observational Studies in Epidemiology” (STROBE-Cross-sectional) guidelines and checklist.22 

Participants

Electronic surveys were distributed in April of 2019 to a unique, purposeful sample of 4000 individuals believed to be clinically practicing ATs at either the collegiate or secondary level. This number was chosen a priori to ensure a large and diverse sample of responses from clinically practicing ATs. Based on previously published survey-based research in athletic training10,11,13  it was estimated that a minimum response rate of approximately 10% was feasible, which would give a minimum sample of 400 participants.

Email addresses from collegiate athletic department websites were culled using a randomized list of NCAA member institutions at the Division I, II, or III levels (N = 2000 emails). The list was developed by copying NCAA member lists from each division23  into a spreadsheet. Random numbers were then generated for each row of the spreadsheet, and the spreadsheet was then sorted numerically based on the generated random numbers. Athletic department websites were visited in the order of the composed list until a list of 2000 emails was generated. The same process was completed using a list of JUCO member institutions (N = 250 emails) and a list of NAIA member institutions (N = 250 emails). The final 1500 emails for secondary school ATs were identified from the NATA Athletic Training Locations and Services Database (ATLAS), maintained by the Korey Stringer Institute at the University of Connecticut.24  The number of ATs identified per state were weighted based on US Census population data, such that a total of 1500 emails were identified. For each state, the state map was selected and filtered to identify secondary schools with part-time or full-time ATs. Secondary schools were then clicked on at random until the appropriate number of unique emails were obtained for a given state.

Survey Development

The developed survey consisted of 3 main parts, which were presented in the following order: (1) Attitudes, Beliefs, and Practices relating to PROMs (23 items); (2) Access to Evidence-Based Practice (EBP) Information (5 items); and (3) Demographics and Educational Background (8 items) (see Supplemental Figure, available online at www.nataej.org).

Attitudes, Beliefs, and Practices Instrument

The questions regarding attitudes and beliefs regarding PROMs were modified for athletic training from a previously published instrument25  containing 20 Likert-type items used with nurses and general practitioners. Changes included slight rewording for cultural differences in terminology and broadening the wording of the questions to encompass patients with many different health conditions. The scoring for this portion of the survey was completed as described by Meadows et al,25  with 9 items being reverse scored such that a higher score indicated an overall more favorable view of PROMs. Scores greater than 3 were considered to represent a positive attitude toward PROMs, while scores less than 3 were considered to indicate a negative attitude toward PROMs.25  Instrument responses were averaged to calculate scores across 2 subscales (Acquiring and Using subscale and Benefits subscale) and one Total Attitude score. An additional 3 items regarding existence of employer policies related to PROMs and actual use of PROMs were included on this instrument. As a result of numerous cells containing counts of zero for the question “Do you use or collect Patient-Reported Outcome Measures in your practice?” we dichotomized responses to less than half the time (responses of never or sometimes) or half the time or more (responses of about half the time, most of the time, or always). We chose to dichotomize the data between these categories as there was a clear break in the data, with 30% reporting sometimes using PROMs and only 4% reporting using PROMs about half the time. Additionally, it was felt that use of PROMs in 50% of patients was an appropriate clinical threshold to consider as routine use of PROMs.

Access to Evidence-Based Practice Resources and Demographics, Including Educational Background

Questions regarding access to EBP resources (5 items) were developed by an expert panel of 3 ATs with over 25 years of combined experience teaching EBP in athletic training academic curricula. These questions were designed to examine access to scientific literature and other sources of knowledge related to advances in health care for EBP implementation. In addition to these questions, we developed 8 questions to collect demographic and educational background information that would possibly be influential in understanding the attitudes, beliefs, and current practices of practicing ATs.

Validity and Reliability Assessment of Instrumentation

To establish content validity, the survey was then reviewed by 4 additional certified ATs with content expertise in EBP, PROMs, and survey research. The study purpose and proposed survey items were provided to each reviewer. The survey was then revised based on feedback and returned to the reviewers for further clarification, if needed. A sample of convenience with snowball recruitment methods (participants were encouraged to share the survey link with other ATs) was used to examine the reliability of the overall instrument. A total of 32 participants were used to evaluate the reliability of the survey by completing it twice a minimum of 1 week apart. Further qualitative feedback was solicited from a subset of the reliability participants to establish face validity. Overall, the reliability and internal consistency for the Attitude and Beliefs scale were acceptable (intraclass correlation coefficient [ICC] = 0.85 and Cronbach's α = 0.92).26  Test-retest reliability for the remaining questions all exceeded the minimum threshold of ICC > 0.70, with one exception. The question that did not meet this threshold pertained to access to scientific journals and was revised for clarity based on feedback.

Survey Distribution

Using commercially available survey software (Qualtrics XM, Qualtrics LLC), we distributed survey invitations via email with an anonymous link to the survey instrument and a requested completion date of 4 weeks after receipt. For surveys that were not completed in the first 2 weeks, 2 additional reminders requesting completion were sent approximately 1 week apart. Emails were scheduled such that initial invites and reminders all were sent on varying weekdays to optimize response rate. Email addresses that the system identified as “undeliverable” were replaced by additional email addresses from the same classification (NCAA, NAIA, JUCO, or secondary school) as the original addresses, using the same methods of random selection previously presented.

Data Analysis

Descriptive statistics (means/medians, standard deviations/ranges, and frequency counts) were calculated. Because of nonnormative distributions of responses (as indicated by Shapiro-Wilks tests), Kruskal-Wallis tests were used to examine attitude scores based on clinical setting, educational background (highest level of degree achieved, type of professional program completed), and access to scientific journals beyond those affiliated with the NATA. Pairwise comparisons were only examined if the overall test statistic was significant (P ≤ .05). Chi-square tests were used to compare bivariate responses regarding use of PROMS, with z-tests for independent proportions for post hoc comparisons of significant results (P ≤ .05), and where applicable, odds ratios with 95% confidence intervals (CIs) were calculated. Chi-square tests were also used to compare proportions of respondents with access to scientific journals based on traditional clinical setting.

RESULTS

Of the original 4000 email invitations sent, 287 were identified as undeliverable and were subsequently replaced, until 4000 emails were confirmed as sent. Overall, 801 individuals clicked on the link for the survey, resulting in an access rate of 20%. A total of 507 individuals (12.6%) responded to the Attitudes, Beliefs, and Practices portion of the survey, and 499 participants (12.5%) completed the survey through the Access to Resources for Evidence-Based Practice section. The survey was fully completed through the Demographics and Educational Background section by 484 individuals, for an overall response rate of 12.1%, and a completion rate of 95% by those who initiated the survey. Descriptive statistics for those responding can be seen in Table 1.

Table 1

Descriptive Data for Participants

Descriptive Data for Participants
Descriptive Data for Participants

Attitudes, Beliefs, and Practices Related to PROMs Use

Each participant who started the Attitudes and Beliefs section of the survey completed all 20 Likert-type items; therefore, all available data (N = 507) were used to calculate the Acquiring and Using subscale, Benefits subscale, and Total Attitude scores. Scores for the overall sample and by degree level/type, access to scientific journals, and year of graduation are presented in Table 2. Overall, respondents viewed PROMs positively, with median subscale scores and Total Attitude scores exceeding 3.0. There was no effect for highest degree level for Acquiring and Using (P = .168) or Total Attitude (P = .110). There was a main effect for highest degree level for the Benefits subscale (P = .029), with those having completed a clinical doctorate demonstrating a significantly more positive attitude toward the benefits of using PROMs when compared to individuals with a bachelor's degree (P = .032) or a master's degree (P < .001). Within degree types, a main effect for bachelor's degree type was observed for the Benefits subscale (P = .001), with individuals who completed curriculum-based bachelor's programs reporting more positive attitudes toward the benefits of using PROMs when compared to individuals who completed an internship-based bachelor's program (P = .002).

Table 2

Attitudes and Beliefs Regarding the Use of Patient-Reported Outcome Measures (Rating Scale 1–5, with 5 Representing Most Positive Views)

Attitudes and Beliefs Regarding the Use of Patient-Reported Outcome Measures (Rating Scale 1–5, with 5 Representing Most Positive Views)
Attitudes and Beliefs Regarding the Use of Patient-Reported Outcome Measures (Rating Scale 1–5, with 5 Representing Most Positive Views)

Among those with a master's degree, there were main effects for Acquiring and Using (P = .002) and Total Attitude (P = .030), but not for Benefits (P = .262). Those with athletic training–related postprofessional master's degrees reported more positive attitudes toward Acquiring and Using PROMs (P = .021) and overall Total Attitudes toward PROMs (P = .035) than did those with a non–athletic training–related master's degree. There were no differences on any of the attitude and belief scales based on employment setting (P = .821 to .878). Similarly, there were no differences on any of the attitude and belief scales between those completing their professional degree before 2014 and those completing their professional degree in 2014 or after (P = .098 to .137). Finally, those reporting access to scientific journals beyond those associated with the NATA reported significantly higher scores on all 3 scales: Acquiring and Using (P = .002), Benefits (P = .036), and Total Attitudes (P = .002).

Use of PROMs

Overall, 60% (n = 303) of ATs reported never using PROMs in their clinical practice, 30% (n = 153) reported using them sometimes, 4% (n = 21) reported using them about half the time, 4% (n = 22) reported using them most of the time, and 2% (n = 8) reported using them always. Once these responses were dichotomized for comparisons, only 10% (n = 51 out of 507) of respondents reported using PROMs half the time or more (Table 3). There were no significant differences in PROM use based on highest level of education (P = .104), bachelor's degree type (P = .707), master's degree type (P = .818), year of professional degree completion (P = .825), or employment setting (P = .629). However, individuals who reported having access to professional and scientific journals beyond those associated with the NATA were significantly more likely to routinely use PROMs (13% [n = 39] versus 6% [n = 11], P = .016). The odds ratio for those with access compared to those without using PROMs was 2.3 (95% CI = 1.2–4.6).

Table 3

Percentage of Responders Who Reported Using Patient-Reported Outcomes Half the Time or More

Percentage of Responders Who Reported Using Patient-Reported Outcomes Half the Time or More
Percentage of Responders Who Reported Using Patient-Reported Outcomes Half the Time or More

Participants were also asked if their employer/enterprise had a policy concerning the collection of PROMs or other health outcomes. Those responding Yes (6%, n = 29) were significantly more likely to routinely use PROMS compared to those with no policy (81%, n = 409) or those who were unsure about a policy (14%, n = 69) (59% versus 7% and 7%, P < .001). Specifically, the odds ratio for those with a policy compared to those without was 13.6 (95% CI = 6.0–31.4).

Access to Resources for Evidence-Based Practice

Access to scientific journals beyond those associated with the NATA is summarized in the Figure. Access to journals differed significantly based on traditional employment setting, with a greater proportion of individuals employed in NCAA Division I settings having access to additional journals than did those employed at NAIA schools or secondary schools (P < .001). Participants also indicated up to 5 resources they used for learning about health care advancements and then ranked these recourses based on how commonly they used each indicated resource. These results are presented in Table 4. The most frequently indicated resource was NATA position statements, followed by local, state, or regional meetings and workshops; the NATA Clinical Symposia & AT Expo; and colleagues, friends, or coworkers.

Figure

Access to scientific journals beyond those published by the National Athletic Trainers' Association by traditional employment setting (n = 445). *Statistically different from NCAA Division I College (P < .001).

Figure

Access to scientific journals beyond those published by the National Athletic Trainers' Association by traditional employment setting (n = 445). *Statistically different from NCAA Division I College (P < .001).

Table 4

Sources for Learning About New Advances in Health Care as they Relate to Athletic Training (Respondents Could Select Up to 5 Choices)

Sources for Learning About New Advances in Health Care as they Relate to Athletic Training (Respondents Could Select Up to 5 Choices)
Sources for Learning About New Advances in Health Care as they Relate to Athletic Training (Respondents Could Select Up to 5 Choices)

DISCUSSION

Our results demonstrate more positive attitudes and beliefs regarding PROMs among those ATs who reported being impacted by revised curriculum requirements and evolving opportunities for advanced clinical degrees, particularly clinical doctorates. However, it appears that these educational changes have had limited impact on the actual use of PROMs in traditional practice settings. Instead, the use of PROMs was observed to be dependent on the environmental factors of policy related to the use of PROMs and access to scientific journals beyond those published by the NATA. These results are largely consistent with previous research13  identifying more positive beliefs regarding EBP and higher levels of implementation of EBP among those required to perform documentation for third-party reimbursement and those with broad journal access.

In the present study, only 10% of respondents reported routinely using PROMs in clinical practice. These results are below the levels document in previous research, 911  which has reported that 15% to 26% of respondents routinely used PROMs in clinical practice, and well below the nearly 50% usage reported among physical therapists a decade ago.27  These lower numbers in use may represent variations in sample populations. Valier et al9  and Lam et al11  targeted a wider range of clinical settings, including clinical, hospital, industrial, and military settings, in addition to collegiate and secondary schools, whereas Coulombe et al10  (whose finding most closely resembled our finding with 15% use) only examined ATs working in the secondary school setting.

In other ways, our findings were similar to those reported by Lam et al,11  who observed that those holding doctoral degrees reported using PROMs more frequently and that organizational structure and support facilitated the use of PROMs. The role of these environmental factors may also explain our lower observed use of PROMs. Nontraditional settings and those secondary school positions affiliated with larger health care systems may have greater environmental facilitators of PROM use, such as policies or documentation requirements.

Impact of Education on Attitudes and Beliefs Toward PROMs

Advanced clinical training that is focused on athletic training was associated with increased awareness of and more positive attitudes and beliefs toward PROMs. This was evidenced by the higher Acquiring and Using and Total Attitude scores among individuals completing an athletic training–related postprofessional master's degree compared to those completing a non–athletic training, postprofessional degree. The impact of advanced, formalized clinical training was also supported by the higher Benefits scores observed among individuals who had completed a clinical doctorate. In 2013, CAATE released the “Standards for Accreditation of Post-Professional Athletic Training Programs,” which included the core competences of “Evidence-Based Practice,” “Quality Improvement,” and “Patient-Centered Care,” among others.11  These core competencies emphasize the importance of linking current research, didactic instruction, clinical practice, and the unique personal and contextual factors of every patient. PROMs are central to many of these efforts, as they represent a method by which to evaluate outcomes, both within a single patient and across one's practice, or even compared to those described in other published literature.

Furthermore, PROMs create an opportunity to engage patients in the process and to solicit their individual feedback on their health. These core competencies have since been instilled in many athletic training master's and doctoral degree programs (both accredited and nonaccredited). The installation of the competencies may explain why those with a clinical doctorate demonstrated the highest Attitude and Belief scores and why those with an athletic training–related postprofessional master's degree demonstrated higher scores than those completing a non–athletic training–related master's degree. These results suggest that not all advanced degrees are created equal, and athletic training–related postprofessional degrees, including clinical doctorates, may better expose students to PROMs and their potential use. This observation is particularly important as the profession transitions to requiring a professional master's degree and as opportunities for formal postprofessional education and transition to practice may become limited.

There were no differences on any of the Attitude and Belief scales based on graduating before or after 2014, when the CAATE standards15  began requiring the inclusion of EBP and PROM-related knowledge and skills. Keeley et al13  and Hankemeier et al28  similarly observed no effect for years of experience on beliefs or perceptions related to EBP. This lack of an effect for graduation year may reflect the multilayered educational strategy that was implemented to expand awareness of EBP and PROMs. Concurrent, formalized initiatives, aimed at expanding AT knowledge regarding EBP, were ongoing in 2013 to 2014 as new professional and postprofessional program standards went into effect and as EBP-specific continuing education requirements were initiated.1517  These broad and overlapping educational efforts may explain the overall positive attitudes and beliefs that we observed regardless of timing of professional degree completion.

Impact of Education on PROMs Use

Despite improving attitudes regarding PROMs, educational initiatives appear to have had little impact on the actual use of PROMs. As previously stated, the rate of PROM usage we observed does not represent an increase in adoption when compared to the results of previous literature. Furthermore, no differences in use were observed based on degree level, degree type, or completion year. This limited impact of educational interventions alone, particularly didactic interventions, is consistent with that noted in other health professions.2931  A systematic review29  observed that traditional continuing medical education activities, such as conferences, were ineffective in altering physicians' professional practices, and more in-depth, practice-based interventions were necessary to meaningfully change behavior. Specifically within the rehabilitation sciences, educational interventions have been observed to have some short-term effects, particularly on knowledge and attitudes, but they fail to consistently result in the meaningful use of outcome measures in clinical practice.31 

Within athletic training, previous research13,32  has emphasized the importance of workplace support and clinical mentorship for the use of EBP. Our results suggest that until PROM education is occurring both in the classroom and in the clinic (both at entry level and postcertification), actual transfer of knowledge and awareness to practice may be limited. Experienced practitioners are necessary for impactful clinical mentorship32  to occur. Developing evidence-based practitioners should be a point of emphasis as both professional and postprofessional education continue to evolve. The implementation of immersive clinical experiences into professional education may present an opportunity through which practice-based educational experiences, under the guidance of strong clinical mentors, can help transition PROM knowledge to PROM use.33  However, for this to be an effective path to increase PROM use, mentors who are actively using PROMs will need to be identified and/or fostered.

Barriers to PROMs Use

Previous studies34  have established numerous barriers to the use of PROMs in allied health care. Although knowledge regarding PROMs has been reported as a barrier across professions,34  it is not reported to be one of the top perceived barriers in athletic training.9,11  More prominent barriers to the use of PROMs by ATs include the following reported perceptions: a lack of relevance to the AT's patient, negative time consumption for the clinician and/or patient, sense that information gained is not worth the effort, feeling that instruments are only useful for research, and sense that instruments do not improve clinical practice or treatment planning.911  The combination of these perceptions of PROMs as a low-value undertaking and the lack of clear clinical standards for achieving patient engagement in a manner which documents the patient's perspective5  create strong barriers that must be addressed if we are to gain widespread use of PROMs in athletic training.

Attempts to adopt PROMs methodology from other clinical health professions may heighten barriers to widespread PROM use. For example, most existing PROMs that are common among physicians, physical therapists, and occupational therapists are primarily based on the performance of activities of daily living (ADLs). Athletes typically have less difficulty with ADLs, as demonstrated by the moderate to high ceiling effects often reported among ADL-focused questions on various PROMs.35,36  Lam et al11  reported that the most common PROMs being used by ATs were the Lower Extremity Functional Scale and the Disability of the Arm Shoulder and Hand (DASH). Both of these scales are heavily focused on ADLs, with minimal questions for high-functioning, physically active individuals who participate in sport. Therefore, these instruments may not provide unique, useful, and meaningful information to patients and clinicians in traditional athletic training settings. Although some instruments do include items addressing performance of higher level activities, these items are often nonspecific and may represent only a small sample of the total items contained within an instrument. For example, on the DASH only 2 of the 30 questions inquire about “recreational activities,” with listed examples including golf, hammering, tennis, frisbee, and badminton.37  Second, differences in clinical models between athletic training and the fields of medicine, physical therapy, and occupational therapy should also be considered. Patients in non–athletic training fields are seen with less frequency and often during scheduled times. Less frequent visits allow time for changes to occur, which, in combination with requirements for reimbursement, likely contributes to the use of PROMs in these clinical settings being routine. The requirements for reimbursement create an environmental facilitator to the collection of PROMs. As evidenced in the present study, the greatest use of PROMs occurred among those with employers who had policies related to the collection of PROMs.

An additional environmental barrier that appears to restrict the use PROMs, and EBP in general, is a lack of access to scientific literature.13  Although use of PROMs was universally low, we observed that those with access to journals beyond those published by the NATA had double the odds of routinely using PROMs. While NATA publications are open access, many journals are not. It may be difficult to incorporate “the best available evidence” into clinical practice when access to it is restricted, as was reported by nearly 40% of respondents to our survey. It is particularly concerning that those working with some of the highest risk patients for long-term effects of injury, high school athletes, have significantly less access to journals beyond the NATA library than do those working in Division I college settings. NATA position statements (see https://www.nata.org/news-publications/pressroom/statements/position) were the most commonly indicated source for learning about new advances further demonstrating the importance of the coordinated provision of open-access information. While it is fortunate that these resources are used, we should also observe caution. Many position statements are years in the making and may go a decade or more without significant update. The continued growth of timely, open-access publishing and the building of partnerships between professional organizations to allow members reasonable access to high-level evidence from a variety of sources are paramount to ensuring disparities in care do not exist.

Future Directions

The results of the present study support that previous educational advancements have been effective in introducing ATs in general, and especially those with advanced clinical training, to the use of PROMs. However, there continues to be a gap between knowledge and practice. Strong clinical mentorship and structured transitions to practice may help bridge this gap. Furthermore, supporting preceptors by incorporating information related to PROM use and the availability of institutional scientific databases into preceptor training or working collaboratively to identify electronic medical records that incorporate PROMs may help bridge the knowledge-to-practice divide. The use of varied clinical placements,4  including placements in which the use of PROMs is a part of institutional policy, may also be beneficial. In particular, immersive clinical sites where PROMs are routinely used may provide in-depth, real-world opportunities for athletic training students to transfer knowledge to practice in a manner that persists into independent clinical practice.

Overall, our results, combined with existing literature, suggest that we cannot “educate our way out” of the lack of use of PROMs in athletic training clinical practice. Therefore, future research should look closer at developing strategies to address the barriers reported by practicing clinicians. First, efforts must continue to develop and validate outcome measures that are highly relevant to both clinicians and high-functioning, physically active populations. Second, policy and/or structural changes may be needed to aide and support the use of PROMs and the overall documentation of outcomes in athletic training clinical practice. Future interventions should look at more practice-based interventions, including policy changes and ongoing access to evidence-based resources. Other examples of potential structural or policy changes that might increase PROM use include, but are not limited to, the development of national databases, the incorporation of PROMs more broadly into common athletic training electronic medical record systems, or the linking of PROMs and other outcomes data to reimbursement or other incentives.

Limitations

As with any survey study, our results are limited by the interpretation and accuracy of our respondents. There is the potential that some respondents may have had access to additional journals beyond those associated with the NATA and simply not been aware of this access. Similarly, we did not ask the source of access to additional journals. For many, it likely related to employment based on institutional subscriptions, but it may also have been related to personal subscriptions, which could represent internal, individual differences and not environmental differences. We also did not differentiate between postprofessional accredited master's programs and other programs that respondents deemed “related to athletic training.” Given the voluntary nature of postprofessional CAATE accreditation (which is now being discontinued), we felt that such a division might be somewhat arbitrary. Instead, we left it up to respondents to indicate whether their master's degrees were related to athletic training or not. Likewise, engagement in advanced professional training may be a function of underlying personal factors that may also predispose an individual to engage in evidence-based practice and the use of PROMs. Finally, although we met our overall sample size target, we did not conduct an a priori power analysis. However, we conducted a purposeful sampling strategy intended to recruit ATs practicing in traditional secondary school and collegiate settings at all levels. The sample of participants with a clinical doctorate was small, representing just 2% (n = 10) of all respondents. However, given that in the 2018 NATA Salary Survey38  only 0.8% of respondents working in secondary school or collegiate clinical settings reported having a clinical doctorate, we believe the sample is proportionally representative of those with a doctorate degree working in these settings. Nonetheless, these results should be interpreted with caution, given the small sample of this subgroup. It is likely a larger sample would yield additional significant findings, particularly related to PROM use, in this subgroup compared to others.

CONCLUSIONS

Overall, PROMs were viewed positively across all participants, regardless of subgroup. We observed no differences in attitudes related to PROMs or usage of PROMs based on clinical setting. PROMs were viewed significantly more positively by those with advanced clinical degrees, such as those with athletic training–related postprofessional degrees and clinical doctorates. Despite these more positive attitudes, there were no differences in routine usage of PROMs based on educational background or employment setting. However, environmental factors, such as access to scientific journals beyond those associated with the NATA and employer policies related to the use of PROMs, did significantly increase the use of PROMs. To encourage the incorporation of PROMs into routine clinical practice, future investigations should look beyond typical educational interventions (eg, didactic training, conference presentations) and examine the impact of clinical-based mentorship and other practice-based interventions related to policy and environmental factors. Furthermore, for professional athletic training programs seeking to link didactic education regarding PROMs with clinical application it may be advantageous to seek out preceptors with advanced clinical training and clinical sites with specific policies related to the collection of PROMs.

REFERENCES

1. 
Sackett
DL,
Rosenberg
WM,
Gray
JA,
Haynes
RB,
Richardson
WS.
Evidence based medicine: what it is and what it isn't
.
BMJ
.
1996
;
312
(7023)
:
71
72
.
2. 
Snyder
AR,
Parsons
JT,
Valovich McLeod
TC,
Curtis Bay
R,
Michener
LA,
Sauers
EL.
Using disablement models and clinical outcomes assessment to enable evidence-based athletic training practice, part I: disablement models
.
J Athl Train
.
2008
;
43
(4)
:
428
436
.
3. 
Valovich McLeod
TC,
Snyder
AR,
Parsons
JT,
Curtis Bay
R,
Michener
LA,
Sauers
EL.
Using disablement models and clinical outcomes assessment to enable evidence-based athletic training practice, part II: clinical outcomes assessment
.
J Athl Train
.
2008
;
43
(4)
:
437
445
.
4. 
Snyder
AR,
Valovich McLeod
TC,
Sauers
EL.
Defining, valuing, and teaching clinical outcomes assessment in professional and post-professional athletic training education programs
.
Athl Train Educ J
.
2007
;
2
(2)
:
31
41
.
5. 
Howard
JS,
Sciascia
A,
Hoch
JM.
Using patient evidence to guide clinical care: consulting the other expert in the room
.
Int J Athl Ther Train
.
2018
;
23
(2)
:
53
56
.
6. 
Steves
R,
Hootman
JM.
Evidence-based medicine: what is it and how does it apply to athletic training?
J Athl Train
.
2004
;
39
(1)
:
83
87
.
7. 
Sauers
EL,
Snyder
AR.
A team approach: demonstrating sport rehabilitation's effectiveness and enhancing patient care through clinical outcomes assessment
.
J Sport Rehab
.
2011
;
20
(1)
:
3
7
.
8. 
Denegar
CR.
Advancing patient care: everyone wins
.
J Athl Train
.
2008
;
43
(4)
:
341
.
9. 
Valier
ARS,
Jennings
AL,
Parsons
JT,
Vela
LI.
Benefits of and barriers to using patient-rated outcome measures in athletic training
.
J Athl Train
.
2014
;
49
(5)
:
674
683
.
10. 
Coulombe
BJ,
Games
KE,
Eberman
LE.
The use of patient-reported outcomes measures: secondary school athletic trainers' perceptions, practices, and barriers
.
J Athl Train
.
2019
;
54
(2)
:
142
151
.
11. 
Lam
KC,
Harrington
KM,
Cameron
KL,
Valier
ARS.
Use of patient-reported outcome measures in athletic training: common measures, selection considerations, and practical barriers
.
J Athl Train
.
2019
;
54
(4)
:
449
458
.
12. 
Bacon
CEW,
Eppelheimer
BL,
Kasamatsu
TM,
Lam
KC,
Nottingham
SL.
Athletic trainers' perceptions of and barriers to patient care documentation: a report from the athletic training practice-based research network
.
J Athl Train
.
2017
;
52
(7)
:
667
675
.
13. 
Keeley
K,
Walker
SE,
Hankemeier
DA,
Martin
M,
Cappaert
TA.
Athletic trainers' beliefs about and implementation of evidence-based practice
.
J Athl Train
.
2016
;
51
(1)
:
35
46
.
14. 
Athletic training education competencies
.
National Athletic Trainers' Association Website
.
Accessed July 12,
2021
.
15. 
Standards for the accreditation of professional athletic training programs.
Commission on Accreditation of Athletic Training Education Web site
.
Updated February 16, 2018. Accessed July 12, 2021.
16. 
Standards for the accreditation of post-professional athletic training degree programs.
Commission on Accreditation of Athletic Training Education Web site
.
Updated January 16, 2017. Accessed July 12, 2021.
17. 
Evidence based practice to enhance clinicians' skills
.
Board of Certification for the Athletic Trainer Web site
.
Accessed July 12,
2021
.
18. 
Delwiche
FA,
Hall
EF.
Mapping the literature of athletic training
.
J Med Libr Assoc
.
2007
;
95
(2)
:
195
201
.
19. 
Hankemeier
DA,
Van Lunen
BL.
Perceptions of approved clinical instructors: barriers in the implementation of evidence-based practice
.
J Athl Train
.
2013
;
48
(3)
:
382
393
.
20. 
Hertel
J.
Research training for clinicians: the crucial link between evidence-based practice and third-party reimbursement
.
J Athl Train
.
2005
;
40
(2)
:
69
70
.
21. 
Manspeaker
S,
Van Lunen
B.
Overcoming barriers to implementation of evidence-based practice concepts in athletic training education: perceptions of select educators
.
J Athl Train
.
2011
;
46
(5)
:
514
522
.
22. 
von Elm
E,
Altman
DG,
Egger
M,
Pocock
SJ,
Gøtzsche
PC,
Vandenbroucke
JP.
The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies
.
Ann Intern Med
.
2007
;
147
(8)
:
573
577
.
23. 
NCAA directory
.
National Collegiate Athletic Association Web site
.
Accessed July 12,
2021
.
24. 
ATLAS athletic training locations and services
.
Korey Stringer Institute Web site
.
Accessed July 12,
2021
.
25. 
Meadows
KA,
Rogers
D,
Greene
T.
Attitudes to the use of health outcome questionnaires in the routine care of patients with diabetes: a survey of general practitioners and practice nurses
.
Br J Gen Pract
.
1998
;
48
(434)
:
1555
1559
.
26. 
Portney
LG,
Watkins
MP.
Foundations of Clinical Research: Applications to Practice. 2nd ed.
Prentice Hall Health
;
2000
.
27. 
Jette
DU,
Halbert
J,
Iverson
C,
Miceli
E,
Shah
P.
Use of standardized outcome measures in physical therapist practice: perceptions and applications
.
Phys Ther
.
2009
;
89
(2)
:
125
135
.
28. 
Hankemeier
DA,
Walter
JM,
McCarty
CW,
et al
Use of evidence-based practice among athletic training educators, clinicians, and students, part 1: perceived importance, knowledge, and confidence
.
J Athl Train
.
2013
;
48
(3)
:
394
404
.
29. 
Davis
DA,
Thomson
MA,
Oxman
AD,
Haynes
RB.
Changing physician performance: a systematic review of the effect of continuing medical education strategies
.
JAMA
.
1995
;
274
(9)
:
700
705
.
30. 
Forsetlund
L,
Bjørndal
A,
Rashidian
A,
et al
Continuing education meetings and workshops: effects on professional practice and health care outcomes
.
Cochrane Database Syst Rev.
2009
(2)
;
CD003030.
31. 
Colquhoun
HL,
Lamontagne
ME,
Duncan
EA,
Fiander
M,
Champagne
C,
Grimshaw
JM.
A systematic review of interventions to increase the use of standardized outcome measures by rehabilitation professionals
.
Clin Rehabil
.
2017
;
31
(3)
:
299
309
.
32. 
Allard
BV.
Exploring the Decisions Involved in Evidence-Based Practice Adoption by Athletic Trainers
.
Dissertation.
Seton Hall University
;
2019
.
33. 
Bowman
TG,
Mazerolle
SM,
Barrett
JL.
Professional master's athletic training programs use clinical education to facilitate transition to practice
.
Athl Train Educ J
.
2017
;
12
(2)
:
146
151
.
34. 
Duncan
EA,
Murray
J.
The barriers and facilitators to routine outcome measurement by allied health professionals in practice: a systematic review
.
BMC Health Serv Res
.
2012
;
12
(1)
:
96
.
35. 
Tyser
AR,
Allen
CM,
Presson
AP,
et al
Evaluating the performance of PROMIS and QuickDASH instruments in an intercollegiate Division 1 athlete population
.
J Shoulder Elbow Surg
.
2012
;
30
(1)
:
158
164
.
36. 
Frobell
RB,
Roos
EM,
Roos
HP,
Ranstam
J,
Lohmander
LS.
A randomized trial of treatment for acute anterior cruciate ligament tears
.
N Engl J Med
.
2010
;
363
(4)
:
331
342
.
37. 
Kennedy
CA,
Beaton
DE,
Solway
S,
McConnell
S,
Bombardier
C.
Disabilities of the Arm, Shoulder and Hand (DASH). The DASH and QuickDASH Outcome Measure User's Manual
.
3rd ed.
Institute for Work & Health
;
2011
.
38. 
NATA salary survey results
.
National Athletic Trainers' Association Web site
.
Accessed July 12,
2021
.