Comparing the Primary Concerns of Injured Collegiate Athletes With the Content of Patient-Reported Outcome Measures

This observational cross-sectional study used the Measure Yourself Medical Outcome Profile (MYMOP-2)²³  to identify the items most relevant to injured collegiate student-athletes and compare those symptoms and activities with those represented on established PROMs. This study was approved by the Appalachian State University Institutional Review Board.

Participants consisted of a convenience sample of student-athletes experiencing an injury at 4 collegiate institutions in 2 states in the mid-Atlantic region of the United States, including National Collegiate Athletic Association Division I (Football Bowl Subdivision), Division II (2 schools), and Division III schools. Any student-athletes over the age of 18 receiving care for any injury, fluent in English, who were identified by their treating ATs as having modified or restricted their sport participation in games or practices because of injury and were willing to sign a Health Insurance Portability and Accountability Act (HIPAA) release for their ATs to share information regarding their injuries were eligible for the study. Participants were not enrolled more than once for a given body region.

Participants were asked to complete the MYMOP-2 and a demographics form requesting their treating AT, age, year in school, sport, date of injury, and location and description of injury. Data collection occurred 1 time per participant per region of injury. At the time of enrollment, the treating ATs were asked for their assessment of the injury phase of each participant (acute, subacute, or chronic).

Each participant's responses to the MYMOP-2 and each item on the included established PROMs were linked to an ICF code via the established ICF linking rules described by Cieza et al.¹¹  In brief, 3 raters reviewed all MYMOP-2 responses independently and assigned a code in the deepest relevant level of the ICF taxonomy. The raters were 2 licensed ATs and an individual licensed as both an AT and a physical therapist. Before rating responses, reviewers underwent additional training in the ICF model via the ICF e-learning tool; reviewed recommendations from the literature outlining and examining the ICF model, the World Health Organization ICF manuals, and previously conducted studies with similar methods; and completed pilot linking sessions and discussed and compared results.^9–11,13,15  The ICF model is a flexible, multitiered, hierarchical framework split into 2 parts. Part 1 contains the domains of body functions (b), body structures (s), and activity and participation (d). Part 2 contains contextual factors called environmental factors (e) and personal factors (pf). Each contains up to 4 levels. The first level is referred to as the chapter, and each chapter contains up to 3 nested levels (second, third, and fourth) increasing in specificity.^9,10  This hierarchical structure was used to link each patient concern to an ICF code, guided by the ICF browser definitions, inclusions, and exclusions available at the second and third levels.²⁴  For example, if a response was “pitching a baseball,” it would be coded as “d4454 Throwing.” The definition for this code states: “Using fingers, hands and arms to lift something and propel it with some force through the air, such as when tossing a ball.”²⁴  The process to arrive at this code is as follows:

• d Activities and participation (domain)

• d4 Mobility (chapter)

• d430–d449 Lifting and carrying objects (component)

• d445 Hand and arm use (level 2)

• d4454 Throwing (level 3)

Raters were permitted to assign multiple ICF codes to a single MYMOP-2 response in order to fully represent the participant's concern. For example, if a participant listed “pain when I run,” separate codes were assigned to represent pain (b280) and running (d4552). Similarly, multiple codes could be assigned as needed for PROM items. Once the individual rating was complete, the raters met to review the identified codes. When disagreement existed among raters, they discussed the code(s) until a consensus was reached. Fleiss κ statistics indicated moderate to very good (0.433 to 0.846) interrater agreement for coding of the MYMOP-2 responses at both levels 2 and 3 before the consensus process that was used to arrive at final codes for each participant response and each PROM item.²⁵  The linking process resulted in a final single set of agreed-upon codes for each participant response and PROM item. Throughout the review process, previously agreed-upon codes were reviewed to ensure consistency.

The MYMOP-2 is a patient-generated outcome measure. It asks the patient to “choose one or two symptoms (physical or mental) that bother you the most” and “choose one activity (physical, social, or mental) that is important to you, and that your problem makes difficult or prevents you from doing.”²³  The MYMOP-2 has been examined in various acute and chronic settings and has been reported to detect changes in several populations, including the military^23,26,27 ; therefore, its application to an athletic population is not unreasonable. As an open-ended tool, it allows for a patient-centered evaluation of symptoms across multiple phases of the injury, disease, or condition and across multiple pathologies.^23,26,27  Although not completed by participants, all possible items of the investigated PROMs were linked to ICF codes. These instruments were selected because the LEFS and DASH have been identified as the most common upper and lower extremity-specific PROMs used by practicing ATs⁶  and the PROMIS has been widely proposed as a generic HRQOL instrument that can be applied across populations.¹⁶  The mDPAS, KOOS, IKDC, FAAM (including Sports subscale), FAST, and KJOC were included because of their use in athletic populations. The mDPAS is a general HRQOL PROM designed to evaluate both mental and physical components of disability in physically active populations.¹⁸  The KOOS is a knee-specific PROM that assesses pain, symptoms, activities of daily living, sport and recreation function, and knee-related quality of life.¹⁹  The IKDC is a region-specific measure of symptoms, function, and sports activity for those experiencing knee-related problems.²¹  The FAAM is a region-specific PROM assessing concerns related to activities of daily living and sport-specific concerns for those experiencing injuries of the lower leg, ankle, and foot.²²  The FAST is an upper extremity, region-specific PROM targeting overhead-throwing athletes experiencing multiple domains of disablement.²⁰  Finally, the KJOC is a region-specific PROM that evaluates function and performance for overhead athletes experiencing upper extremity injuries.¹⁷ 

To reduce the data to those codes representing the items most important to the majority of participants, we performed frequency counts to identify those codes cumulatively representing 80% of patient-generated ICF codes overall, by phase of injury, and for the upper and lower extremities. Those codes in the cumulative 80% were considered the primary concerns and were used for our analyses. Primary concerns were examined by phase of injury and region for substantial content differences. To explore various levels of specificity, codes were examined at both levels 2 and 3 of the ICF taxonomy. To address the relevance of the established PROMs, we determined that a minimum of 70% of the primary participant-generated codes must be encompassed in a PROM if it was to be considered representative of patient concerns (ie, 70% of the cumulative 80% of generated codes were addressed by the established PROM). The a priori 70% threshold was based on established logic that indicates if a special test is considered clinically useful.²⁸  Additionally, an approximately 70% threshold has been applied in recent Delphi analyses examining both return-to-sport decisions (70%)²⁹  and PROM content validity among both researchers and practicing clinicians (67%).³⁰  Collectively, this past use of a 70% threshold supports its application as an established and acceptable standard for clinical decision-making and PROM-related research. To evaluate the efficiency of commonly used PROMs and determine the amount of extraneous content present, we examined if 70% of the established PROM codes were among the primary participant-generated codes. Chi-square, 1-sample goodness-of-fit tests were performed to test the a priori selected 70% thresholds. If less than 70% (P < .05) of the relevant most common participant-generated codes were represented in the established PROMs, they were considered nonrepresentative of the participant-generated primary concerns. If less than 70% of the established PROM-generated codes were encompassed by the most common participant-generated codes, the PROM was considered to have a significant amount of extraneous content.

Participants were 149 collegiate athletes (74 women, 75 men, age = 19.6 ± 1.3 years) and represented 150 injuries. Lower extremity injuries accounted for 77% of our sample and upper extremity injuries, 15%. Back, head, and neck injuries constituted 6%, with 2% missing the region of injury. Most participants were classified by their treating ATs as being in the acute phase of injury at the time of collection (50%), followed by chronic (35%), subacute (7%), and missing (7%). Participant sport distributions are shown in Table 1. Forty participants chose to list only a single symptom on the MYMOP-2 in response to the question prompting the listing of 1 or 2 symptoms. The final set of agreed-upon ICF codes for participant responses yielded 594 total codes, containing 90 unique codes. The majority were represented in the body function (56%) and activities and participation (41%) domains. The body structure domain represented 1%, and 2% were not codable. Code frequency distributions for the commonly occurring codes overall are displayed in Tables 2 and 3. Code frequency distributions by extremity and phase of injury overlapped substantially (see Supplemental Tables 1–6, available online at http://dx.doi.org/10.4085/1062-6050-0516.21.S1). Frequency counts of participant-generated ICF codes overall, by extremity, and by phase of injury are provided in Table 4. A summary of shared codes between the primary concerns of participants and established PROMs can be seen in Table 5.

Comparisons between all participant-generated common codes and the PROMIS were mixed, with PROMIS content relevant at level 2 (P = .061) but falling significantly below the 70% threshold for relevance at level 3 (P < .001). In contrast, the PROMIS efficiently evaluated primary concerns at level 3 (P = .081) but not at level 2 (P = .011). When all participant-generated codes were compared with codes generated from the mDPAS, level 2 codes appeared relevant to the primary concerns of participants (P = .506). However, this relevance was lost for level 3 codes (P = .004). Additionally, based on codes generated from the mDPAS, it was not efficient in evaluating the primary concerns of injured student-athletes at either level (P < .001, P < .001).

Among participants with lower extremity injuries, analyses of level 2 and level 3 codes revealed that the LEFS was not relevant to the primary concerns of participants (P < .001). Similarly, the LEFS was not efficient in evaluating participant concerns at either level 2 or level 3 (P < .001). Comparison of participant-generated concerns for lower extremity injuries with the IKDC indicated relevance for both level 2 and level 3 codes (P = .801, P = .143). Additionally, the IKDC was efficient in evaluating primary concerns at both levels 2 (P = .159) and 3 (P = .078). Comparison of codes for those experiencing lower extremity injuries with the KOOS demonstrated relevancy at both level 2 (P = .705) and 3 (P = .143) analyses. However, the KOOS was not efficient at either level (P = .003, P < .001). Comparison of participant-generated codes with the FAAM displayed a lack of relevance at both level 2 (P = .001) and level 3 (P = .001) analyses. Similarly, the FAAM was not efficient at either level (P = .001, P < .001).

Among those with upper extremity injuries, level 2 ICF code analysis indicated that 7 of the 8 primary codes (88%) were represented in the DASH, reflecting PROM relevance (P = .28). However, this relevance was not maintained at level 3 (P < .001). Furthermore, examination of the ICF codes generated from the DASH revealed that it did not efficiently evaluate the primary concerns of participants at either level 2 or level 3 (P < .001). The FAST was relevant in capturing participant concerns at level 2 (P = .758) but not at level 3 (P = .006). Additionally, the FAST was not efficient at either level 2 (P = .027) or level 3 (P < .001). Comparison of participant codes for upper extremity injuries with KJOC-generated codes at level 2 supported KJOC relevance and efficiency (P = .758, P = .076). However, analysis of the more specific level 3 ICF codes did not support continuation of this relevance or efficiency (P = .006, P < .001).

The objective of our study was to identify the primary concerns held by injured collegiate athletes and compare those concerns, as measured by the MYMOP-2, with the content of established PROMs. Sensations of pain and sport participation were the most commonly reported participant concerns, representing 39% of the total. Codes related to running or moving around, emotional function (stress, confidence, frustration, anxiety, etc), mobility (range of motion), swelling, and strength or muscle power were also consistently among the primary participant concerns overall. It is important to note that regardless of the level of analysis (level 2 or 3 of the ICF taxonomy) or region of injury, “not codable” was consistently present as well. These primary areas of concerns were in keeping with previous works linking participant concerns to ICF codes.¹⁵  Pain, sport, and movement function were among the most common concerns reported across all phases of injury. Notably fewer participants were in the subacute and chronic phases; however, their primary concerns were consistent with those of participants in the acute phase, regardless of the region of injury. These results may suggest that use of the same PROM may be appropriate throughout the rehabilitation process. However, it is critical that selected PROMs reflect the patient's concerns and be sensitive enough to detect change at various stages of rehabilitation.

Both the PROMIS and the mDPAS are global PROMs intended to evaluate HRQOL; nonetheless, the mDPAS was specifically developed for use with physically active populations. For both global measures, the percentage of agreement between PROM content and primary concerns at ICF level 2 was not statistically different from 70%. Yet the mDPAS addressed more level 2 participant concerns (62%) than the PROMIS (46%). This greater representation in the mDPAS likely occurred because it was specifically developed for active populations. Significant content differences were present at level 3 for both measures. Uniquely for the PROMIS, we observed significant superfluous content at level 2 but not at level 3. These results indicate that both measures have potential utility in assessing more general level 2 concerns (eg, muscle power functions, recreation and leisure, hand and arm use)²⁴  but not more specific and descriptive level 3 concerns (eg, sports, throwing, catching, running, jumping).²⁴  However, the PROMIS was the more efficient instrument, with 40% to 50% of its content aligning with the primary concerns of patients. These findings are counter to our hypothesis that PROMs designed for highly active individuals would be more relevant and efficient than those designed for use with the general population. A previous analysis¹⁴  of the mDPAS demonstrated a strong focus on the physiological domains of health and the activity domain of the ICF, with less representation in the psychological or participation areas. Conversely, our participants' concerns aligned more with their lived experiences, as exemplified by consistent reports of emotional functions and sport participation concerns. These differences in primary focus may explain the lack of alignment in the specific level 3 analysis.

The LEFS and FAAM content failed to represent the primary concerns of participants as measured by the MYMOP-2. Perhaps most notable was that pain, the most commonly occurring participant concern, was not represented in the LEFS or FAAM content. Our results also indicated that the LEFS and FAAM contain a large amount of extraneous content. It should be noted that we evaluated the full FAAM and the sport-specific subscale, so the findings could differ if only the sport-specific subscale of the FAAM was examined. Still, these results are consistent with earlier concerns regarding the FAAM's ability to address concerns of an injured athletic population.³¹  Lam et al⁶  previously classified both instruments as being heavily focused on physical health and almost exclusively on the activity domain of the ICF. Given the large number of our patient-generated responses that were related to pain, sport performance, and emotional function, it is not surprising that the LEFS and FAAM appear to have limited relevance to the primary concerns of injured collegiate athletes.

The KOOS and IKDC demonstrated the strongest agreement between content and the primary concerns of collegiate athletes experiencing lower extremity injury as assessed by the MYMOP-2. Both PROMs adequately represented the concerns of participants at both ICF level 2 and level 3. The results endorse the hypothesis that these instruments, designed to address a wide range of activity levels, accurately reflect both broad and specific concerns of participants. Furthermore, the percentage of ICF codes generated from the IKDC at level 2 (53%, 8 of 15) and 3 (52%, 11 of 21) was not different than the 70% criterion value, demonstrating that the IKDC was the most efficient of the investigated PROMs in evaluating the concerns of participants experiencing lower extremity injuries. Conversely, the KOOS provided significant extraneous content at both levels 2 and 3, suggesting that the IKDC is preferable for use with highly active patients with lower extremity conditions. The relevance and efficiency of the IKDC may be products of its development. The IKDC was specifically designed to represent knee impairment–related symptoms and limitations in function and sports activity.²¹  Moreover, the IKDC was iteratively tested with large samples of patients representing the target population and revised through a formal item-reduction process.²¹  The use of patient engagement in PROM development has been proposed to improve relevance to the patient.³²  Creation of the KOOS similarly engaged patients, but the developers purposefully included a subgroup with osteoarthritis.¹⁹  It is possible that items relevant to patients with osteoarthritis may not align with the primary concerns of the college-age participants in our study, resulting in the lack of efficiency for the KOOS. This observation highlights a challenge of PROM selection and development. An instrument that is broadly generalizable may lack specificity, efficiency, or both for a target population. Although generalizability can be beneficial, a lack of perceived relevance by both clinicians and patients can be a barrier to PROM adoption.^2,5,6  Overall, the IKDC was the most relevant and efficient PROM in this analysis for addressing the primary concerns of student-athletes experiencing injury.

Contrary to our hypothesis, for the upper extremity, relevance to patient concerns was similar among the more general DASH and the more activity-focused FAST and KJOC. In support of our hypothesis, a trend toward greater efficiency with those PROMs designed for highly active individuals, particularly the KJOC, was noted. All investigated upper extremity PROMs were relevant to patient concerns at level 2, but none were relevant at level 3. Therefore, we conclude that the DASH, FAST, and KJOC may be sensitive to the general concerns of participants but are not specific to the precise concerns of participants as assessed by the MYMOP-2. Only the KJOC at level 2 achieved efficiency, with the DASH and FAST containing a significant number of superfluous codes in both the level 2 and level 3 analyses. An earlier analysis¹⁴  of the DASH showed that the physiological, social, physical, and psychological domains of health and the ICF domains of body structure and function, activity, and participation are all encompassed within the DASH. This broad spectrum of health assessment likely contributed to the DASH's addressing the highest percentage of participant concerns (80%) but also having a significant amount of extraneous material (73%–85%). Both the FAST²⁰  and the KJOC¹⁷  were specifically developed with input from baseball and softball players, whereas our sample included those with upper extremity injuries from a variety of sports in addition to baseball and softball. The heterogeneity of our sample with upper extremity conditions may have contributed to the limited agreement between FAST and KJOC content and the primary concerns of our respondents. In particular, this may explain why the broad concerns of level 2 were adequately addressed but not the more specific level 3 concerns.

The content differences we observed between the evaluated PROMs and patient concerns as assessed by the MYMOP-2 were consistent with the perceived barriers and concerns identified by surveyed clinicians.^2,5  They were also consistent with the previously reported tendency of PROMs to focus on clinician-oriented items, such as strength and range of motion, but lacking coverage of items of importance to patients, such as psychological and social factors.³³  Not only did the majority of measures we examined represent the patient-generated concerns only superficially, they also contained a large amount of extraneous content. This extraneous content places additional demands on the patient and adds unnecessary time for clinicians to score PROMs. Furthermore, perceived barriers and concerns regarding PROM implementation may be reinforced by the PROMs being used in practice.⁶  Lam et al⁶  identified the LEFS, FAAM, and DASH as the most common extremity measures being selected by ATs using PROMs for clinical practice. None of these instruments were both relevant and efficient for evaluating the primary concerns of student-athletes experiencing injury. To combat this, we believe that the development and use of PROMs targeting athletic populations, informed by both clinician and patient concerns, are essential to facilitate the clinical utility of PROMs in athletic training.¹  Patient-reported outcome measures must be purposefully developed and selected with their end use in mind, and clinicians and researchers should not assume that a form is useful or appropriate simply because it has been used frequently in the past.

Additionally, we believe these results support the use of a patient-generated outcome measure, such as the MYMOP-2 or the Patient-Specific Functional Scale.³⁴  Such open-ended measures may be beneficial, as patient concerns were highly individualistic (Table 4) and frequently included items beyond standard ICF codes, as evidenced by the consistent presence of “not codable” items in our findings. Patient-generated items may be particularly useful if paired with the commonly used⁶  single-item Numeric Pain Rating Scale,³⁵  given that nearly 30% of the identified codes were related to “sensation of pain.” Similarly, it may be relevant to consider using psychosocial measures if a PROM addresses only physical impairments or limitations. Finally, the limited ability of the investigated PROMs to address concerns at the more specific level 3 of the ICF highlights the importance of using PROMs not only as an endpoint but as another piece of the clinical evaluation that can prompt follow-up questions and lead to deeper and more engaging conversations with patients regarding their health.¹ 

For both participant responses and PROM items, many of the final codes were those listed as other specified or unspecified²⁴  or “not codable.” Coding language limitations were evident when individuals reported emotional experiences or symptoms such as swelling. Concerns related to a level of function surpassing activities of daily living were also constrained by the existing ICF taxonomy. For example, although codes are available for concerns such as sport, swimming, and lifting, the definitions provided by the ICF do not necessarily encompass the participants' concerns. For example, the ICF code for lifting is defined as “raising up an object in order to move it from a lower to a higher level, such as when lifting a glass from the table.”²⁴  This definition can be applied to both lifting weights (frequently listed by our participants) and lifting or carrying a shopping bag (as stated on the DASH),⁷  yet these tasks are not clinically equivalent. Such limitations in the ICF taxonomy may have contributed to some of the overlap in content between the PROM-generated codes and the participant-generated codes.

These concerns regarding the inclusivity of the ICF taxonomy are not unique. Mitra and Shakespeare³⁶  proposed that a revised ICF model would need to consider an individual's ability to participate in components of life deemed personally meaningful. Our data confirm this need, especially related to functional and psychosocial concerns. Specifically, although a small set of codes is related to handling stress, the inclusion criteria do not lend themselves to experiences frequently described by participants, such as anxiety, confidence, or frustration. Therefore, raters linked these concerns to the emotional functions component, defined as

However, this definition is more related to pathologic disorders than to emotional responses to injury or external stimuli. The frequency of emotional experiences among participant concerns highlights the need for improved recognition and evaluation of patient emotional states throughout care.

Data were collected from 4 collegiate campuses from 2019 to 2020, and the data collection period was shortened because of the COVID-19 pandemic. Although we assumed that the responses provided by participants are generalizable to other collegiate athletes, further investigation is needed to verify this, particularly among individuals with upper extremity injuries, because of our limited sample. For both participant-generated responses and PROM questions, we chose to link every component of a response or question. This generated multiple codes for most responses and questions. If we had linked only the questions themselves, less extraneous content might have been identified; however, common concerns might also have been eliminated. Additionally, we classified injuries by body region and did not separate them by individual joints. We also operationally defined the primary concerns as those cumulatively representing 80% of all patient-generated codes and used 70% content agreement as the threshold by which to evaluate PROMs for relevance and efficiency.

We aimed to identify the primary concerns of collegiate athletes experiencing injuries and determine if those PROMs commonly used or recommended for use in athletic training adequately and efficiently encompass those primary concerns. These results validate the concerns expressed by clinicians regarding PROM content while also identifying the primary concerns of an athletic population as pain and sport participation ability when assessed by the MYMOP-2. Most PROMs investigated presented with notable differences in content compared with the primary concerns of participants, particularly at the more descriptive ICF level 3. Similarly, most PROMs analyzed revealed significant amounts of extraneous content. Of the studied PROMs, the IKDC was the most relevant and efficient for collegiate athletes experiencing lower extremity injuries. Additionally, the development or use of measures designed for highly active populations based on the identified primary concerns of participants may be beneficial, as the primary codes were highly consistent (pain, sport, mobility or muscle function of the involved limb, emotional functions or experiences), regardless of the injury phase or location. Finally, improved, patient-centered evaluation of emotional experiences is needed, as this factor was largely inadequately encapsulated by both the evaluated PROMs and the ICF taxonomy. The implementation of patient-generated evidence is imperative to facilitating successful well-rounded practice, as well as the Athletic Training Research Agenda (https://www.natafoundation.org/research/atresearchagenda) surrounding PROMs. To continue to further these endeavors, future researchers and clinicians must focus on developing and implementing PROMs targeting athletic populations with a focus on relevance and efficiency.

Found at DOI: http://dx.doi.org/10.4085/1062-6050-0516.21.S1