There is a lack of current evidence to help athletic training programs identify students poised for first-time Board of Certification (BOC) exam success as a means to ensure compliance with contemporary accreditation standards. Various academic variables have been previously identified as predictors of first-time success; however, these investigations reflect prior versions of the credentialing examination based on what are now obsolete editions of the professional practice analysis.
To determine the accuracy of undergraduate athletic training student performance factors as indicators of successfully passing the current version of the BOC exam on a first attempt.
Casual-comparative (ex–post facto) quasi-experimental.
An accredited undergraduate athletic training program.
A recent sample of athletic training program graduates (n = 43).
Archived data were collected from student academic records and analyzed.
Receiver operating characteristic curve analysis computed the area under the curve (AUC), which was used to determine accuracy of variables. Optimal thresholds for passing the BOC exam on a first attempt were computed for each variable by the index of union method.
The AUC and associated thresholds for factors were as follows: cumulative grade point average (GPA) upon entry to the major (AUC = 0.67, threshold = 3.20); preprofessional-phase course GPA (AUC = 0.81, threshold = 3.48); averaged admission interview score (AUC = 0.49, threshold = 8.46); averaged preceptor evaluation of applicant score (AUC = 0.53, threshold = 93.88%); unique composite score (AUC = 0.80, threshold = 7.34); cumulative GPA upon graduation (AUC = 0.68, threshold = 3.19); combined math and reading SAT score (AUC = 0.90, threshold = 960); clinical education hours (AUC = 0.61, threshold = 1131.08).
Preprofessional-phase course GPA provided the greatest accuracy of the secondary admissions criteria; however, the combined math and reading SAT score provided the greatest accuracy of all examined variables. Furthermore, the unique composite score may represent an accurate, all-inclusive indicator of success. Program administrators and athletic training educators may use these outcomes to shape their related programmatic operations.
Exposure to professional competencies, and thus associated credentialing exam content, in preprofessional-phase courses may lend to the accuracy of this secondary admission criterion in predicting first-time Board of Certification (BOC) exam success.
SAT outcomes may provide an early indication of student acuity and aptitude for other standardized tests, such as the BOC exam.
A unique composite score that emphasizes athletic training-specific course GPA, while accounting for other intangibles, represented a good indicator of BOC exam success.
Personnel tasked with delivering the BOC exam may find the outcomes of this study beneficial in their charge to continually shape the evaluation to be a valid reflection of candidate clinical competency.
The Commission on Accreditation of Athletic Training Education (CAATE) implemented Standard 11 of the 2012 CAATE standards in 2013; this standard requires athletic training programs to maintain a 3-year aggregate first-time pass rate of 70% on the Board of Certification (BOC) exam as a measure of curricular effectiveness.1 Between 2016 and 2018, approximately 24% of professional bachelor's degree programs were identified as noncompliant with this standard.2 Athletic training programs that fail to meet this accreditation standard are placed on probation until they provide sufficient evidence that demonstrates progress in addressing and correcting this deficiency.1,2 While this contemporary standard may be used to promote a diligent admissions process, there are no uniform entrance-to-major criteria for athletic training programs3,4 as well as a lack of current evidence to help guide faculty and administrators in identifying students poised for related success to ensure accreditation compliance.
Independent academic variables, such as cumulative grade point average (GPA),5,6 athletic training–specific course GPA,6 and academic minor GPA,6 have been identified as single predictors of first-time BOC exam success. Furthermore, Harrelson et al6 found that a combination of these variables, in addition to ACT composite score and number of semesters enrolled in an academic program, significantly predicted first-time success. Contrastingly, while educators perceive time spent in clinical experiences as contributing to BOC exam success, these data have been suggested5,7,8 to be statistically insignificant predictors. Conflicting reports5–8 describing curricular metrics associated with first-time BOC exam success may be attributed to such studies being insufficiently powered as a result of an absence of accompanying sample-size calculations.
Although insightful, the inclusive outcomes from prior investigations reflect preceding iterations of the credentialing examination based on what are now obsolete editions of the professional practice analysis; thus, their impact in terms of shaping present curricular models, policy, and practice may be limited. Therefore, the primary aim of this study was to describe the accuracy of undergraduate athletic training student performance factors and to delineate related thresholds indicative of successfully passing the current version of the BOC exam, as outlined under the BOC Practice Analysis 7th edition,9 on the first attempt. Based on the outcomes of Middlemas et al,5 and per our prior pilot data, we hypothesized that cumulative GPA upon admission to an athletic training program would be the most accurate secondary admissions criterion. A secondary aim entailed determining the practical utility of a unique composite score as an inclusive secondary admissions criterion for identifying students poised for success to help ensure program accreditation compliance.
MATERIALS AND METHODS
Experimental Design and Participants
A retrospective casual comparison (ex–post facto) quasi-experimental design was conducted at a R1-classified university. The university's Office for Research Protections determined that this study did not meet the definition of human subject research and therefore did not require further review or approval by the institutional review board.
A power calculation was performed to determine the sample size needed to assess the accuracy of athletic training student factors as indicators of first-time success on the BOC exam. Per receiver operating characteristic (ROC) curve analysis, using the following parameters—single test method; significance level (α) of 0.05; power (1−β) of 80%; area under the ROC curve (AUC) of 0.80; and allocation ratio of 0.23—a respective minimum sample of 25 and 6 occurrences for passing and failing the exam were required to execute this study.10
An administrative support assistant provided the investigators with deidentified secondary program admissions data and corresponding BOC exam results from a sample of 43 records for students who had recently graduated from the undergraduate athletic training major. Considering the retrospective nature of our study, the records were consecutively included for analysis to prevent selection bias. Records that were missing data points for the variables of interest were excluded from the study. A brief accounting of the sample population's demographics and related first-time BOC exam results are described in Table 1.
Secondary admissions data consisted of cumulative GPA upon entry to the major, preprofessional-phase course GPA, averaged admission interview scores, averaged preceptor evaluation of applicant scores, and a unique composite score inclusive of the aforementioned variables. The content areas of the 4 courses calculated in the preprofessional-phase course GPA included introduction to athletic training, functional human anatomy, foundations of clinical practice, and acute care and emergency response. Questions included in the admission interview primarily addressed the applicants' strengths, communication skills, interest in the athletic training major, and where they believe the profession fits under the health care umbrella (Figures 1 and 2). Furthermore, the categories of the preceptor evaluations included professional appearance, communication skills, sports health care administration, preventative care, basic clinical skills, injury recognition, work ethic, and personal attributes (Figure 3). The composite score was configured of weighted factors and was developed to provide the program's admissions panel with a simple, single ensemble metric of applicant viability for entrance to the major. Various athletic training and other allied health care programs incorporate such a metric in their admissions process.6
Grade point averages were calculated on a 4-point scale, interviews were scored out of 10 points, averaged preceptor evaluation of applicant scores were out of 100%, BOC exam outcomes were tallied on a pass or fail basis, and the unique composite score ranged from 0 to 10. Descriptive statistics for the secondary program admissions data are provided in Table 2, and Table 3 outlines the composite score format.
We utilized IBM® SPSS Statistics 25 (IBM Corporation, Armonk, NY) software to perform ROC curve analysis for each variable of interest. The accuracy of each factor was determined by the AUC and interpreted according to Carter et al,11 as detailed in Table 4. Using the ROC curve data, optimal thresholds for passing the BOC exam on a first attempt were computed for each variable by the index of union (IU) method. The IU method identifies an optimal threshold from the ROC curve that possesses values for both sensitivity and specificity that are close to that of the AUC while minimizing the difference between sensitivity and specificity.12 Additionally, positive- and negative-likelihood ratios were computed from the sensitivity and specificity of each optimal threshold to aid in translating our outcomes. We interpreted likelihood ratios according to the method of Jaeschke et al,13 as described in Table 5. Data were evaluated for normality, randomness, and variance to ensure statistical conclusion validity.
Preprofessional-phase course GPA (AUC = 0.81; 95% confidence interval [CI] = 0.62, 0.99) and the unique composite score (AUC = 0.80; 95% CI = 0.63, 0.98) provided good accuracy in predicting first-time BOC exam outcomes. Preprofessional-phase course GPA had a higher sensitivity (0.80), suggesting it is better suited to identify students who may fail the BOC exam on their first attempt; moreover, associated likelihood ratios (positive likelihood ratio [+LR] = 2.13; negative likelihood ratio [–LR] = 0.32) suggest that a preprofessional-phase course GPA of 3.48 yields a small shift in the probability of either passing or failing. The unique composite score had a slightly higher sensitivity (0.77), suggesting it is better suited to identify students who may fail the BOC exam on their first attempt; furthermore, related likelihood ratios (+LR = 3.09; –LR = 0.30) suggest a composite score of 7.34 yields a small shift in the probability of either passing or failing.
Cumulative GPA upon entry to the major (AUC = 0.67; 95% CI = 0.47, 0.87) and the averaged preceptor evaluation of applicant score (AUC = 0.53; 95% CI = 0.26, 0.80) provided poor accuracy in predicting first-time BOC exam outcomes. Cumulative GPA upon entry to the major had a balanced sensitivity (0.63) and specificity (0.62), suggesting it is equally suited to identify students who may fail or pass the BOC exam on their first attempt; furthermore, related likelihood ratios (+LR = 1.68; −LR = 0.59) suggest that a cumulative GPA of 3.20 upon entry to the major yields a very small shift in the probability of either passing or failing. The average preceptor evaluation of applicant score had a higher sensitivity (0.57), suggesting it is better suited to identify students who may fail the BOC exam on their first attempt; furthermore, associated likelihood ratios (+LR = 1.14; −LR = 0.86) suggest an averaged preceptor evaluation of applicant score of 93.88% yields a very small shift in the probability of either passing or failing. The average admission interview score was found to be inaccurate (AUC = 0.49; 95% CI = 0.27, 0.71) and, thus, an impractical variable to use for projecting first-time success or failure for passing the BOC exam.
The ensemble outcomes of the ROC curve analysis are found in Table 6.
General observation of the cohort demographics indicated that an equal number of males and females failed the BOC exam on a first attempt. More interestingly, we found that all 8 individuals who failed the BOC exam on a first attempt were in-state students. Anecdotally, this may be a reflection of a heightened awareness for, and emphasis on certification exam preparation resulting from greater financial burden associated with out-of-state tuition; however, this is speculative, and should be interpreted with caution. Nonetheless, it represents an interesting observation, which warrants attention in future related studies. Furthermore, no students of minority status failed the BOC exam on a first attempt. Finally, 6 of the 8 students who failed their first attempt began their education at a branch campus. Historically, students admitted to our institution's main campus have a higher GPA and SAT score when compared with those initially admitted to branch campuses. With SAT score being the strongest indicator of first-time success on the BOC exam, this may reflect the higher SAT demands for the main campus compared with those of branch campuses. Future investigation should examine how similar demographics contribute to first-time BOC exam success. The general outcomes of our analyses can be found in Table 1.
Secondary Admissions Criteria
Preprofessional-phase course GPA was a more accurate secondary admissions criterion for gauging first-time success on the BOC exam than was the hypothesized cumulative GPA upon entry to the major, which may be the product of our pilot data being potentially underpowered. This finding may also reflect the considerable changes made to the BOC exam and educational competencies since a prior investigation was conducted.5 Furthermore, our results complement those of Ennulat et al,14 who found that a specified prerequisite course GPA was a greater estimator of success on the Physician Assistant National Certifying Exam (PANCE) than was cumulative GPA. Exposure to professional competencies, and thus credentialing exam content, provided in the preprofessional (or prerequisite) courses, and absent from general education courses, may lend to this finding. Additionally, students' interest in the profession and health care–related content of the preprofessional phase, in comparison with the differing content areas of required courses included in the cumulative GPA, may underpin our results.
Contrastingly, the averaged admissions interview score and averaged preceptor evaluation of applicant score were found to be inaccurate and poor indicators of first-time success on the BOC exam, respectively. Similarly, Higgins et al15 found interview scores to be a predictor of PANCE success for only 2 of the 6 programs included in their study. Therefore, program personnel may consider prioritizing student performance in prerequisite courses followed by general education courses with regard to athletic training admission decisions to help ensure first-time success on the BOC exam and compliance with Standard 11. Subsequently, our findings suggest that scored interviews and appraisals require thoughtful consideration for their utility in the admissions process, apart from BOC exam outcomes, given the time burden associated with capturing these variables. For instance, interviews and appraisals may be of other benefit, such as in evaluating intangible skills, though programs must delineate their intended function. This is especially applicable given that communication skills, trustworthiness, dependability, and commitment to the field have been identified as characteristic markers of quality athletic trainers.16,17 Therefore, these variables may better reflect a student's penchant for the profession, as opposed to first-time BOC exam success.
Unique Composite Score
Prior evidence6 suggests that a composite set of variables including overall academic GPA, athletic training–specific GPA, academic minor GPA, ACT composite score, and number of semesters predict first-time exam success. However, this was established for a previous iteration of the credentialing exam and only accounted for 42% of the variance in predicting first-time BOC exam success. We found our unique composite score to be 80% accurate. Although our ensemble metric was 38% more accurate than that of Harrelson et al,6 it was on the low end of “good” accuracy. This may be due to the limited accuracy of the interview and preceptor evaluation in determining this outcome. Comparatively, the difference among the preprofessional-phase course GPA, which was the most accurate criterion, and the unique composite score was within 1%. Despite being poor indicators of successfully passing the BOC exam on a first attempt, these factors appear to assess intangibles significant to athletic training professionals, suggesting that the unique composite score can be utilized to comprehensively gauge the professional and academic preparedness of students. The findings of this study suggest that athletic training specific course GPA should be the emphasis of an ensemble metric, as the accuracy of the unique composite score was attributed to this GPA outcome. The results of prior pilot analysis determined the weights we applied to each component of the unique composite score; however, weighting of components should be continually reexamined and reconsidered or rescaled as appropriate. Future research should be conducted to determine weighting of such variables and the potential inclusion of other applicable indices.
Post Hoc Analysis of Additional Academic Variables
We conducted a post hoc analysis for academic variables that have been previously investigated as indicators of first-time BOC exam success. This analysis was conducted in order to account for the current version of the BOC exam and to provide additional variables for programs to consider when devising admission processes. These additional factors included cumulative GPA upon graduation, combined math and reading SAT scores, and clinical education hours. From the original sample, 33 records contained the inclusive data for our post hoc analysis. Descriptive statistics for these additional academic variables are provided in Table 7.
The combined math and reading SAT score provided excellent accuracy. This variable had a higher sensitivity, suggesting that it is better suited to identify students who may fail the BOC exam on their first attempt; furthermore, related likelihood ratios suggest a combined math and reading SAT score of 960 yields a small shift in the probability of passing and a moderate shift in failing. Cumulative GPA upon graduation and clinical education hours provided poor accuracy. Cumulative GPA upon graduation had a higher sensitivity, suggesting that it is better suited to identify students who may fail the BOC exam on their first attempt; moreover, associated likelihood ratios suggest a cumulative GPA of 3.19 upon graduation yields a very small shift in the probability of either passing or failing. Meanwhile, clinical education hours had a higher specificity, suggesting it is better suited to identify students who may pass the BOC exam on their first attempt; furthermore, related likelihood ratios suggest a total of approximately 1131 hours yields a very small shift in the probability of either passing or failing. The ensemble outcomes of the ROC curve analysis for the additional academic variables are found in Table 8.
Our findings suggest that SAT outcomes may provide an early indication of student acuity, as the combined math and reading SAT score delivered the greatest accuracy of all examined variables. Although they vary in content, both the SAT and BOC are standardized exams, potentially lending to this excellent accuracy. Prior investigations in the sister disciplines of physician assistant (PA) and physical therapy (PT) have utilized the standardized Graduate Record Examination (GRE) to predict success on their respective credentialing exams. The outcomes of the GRE have proven18,19 to be a reliable predictor of success on both the PANCE15 and National Physical Therapy Exam (NPTE). Therefore, athletic training programs at the undergraduate level may consider incorporating SAT outcomes in their admission decisions, while programs that already have or intend to transition to the entry-level master's may utilize GRE outcomes. Furthermore, the inclusion of either SAT or GRE outcomes in a composite score may increase the accuracy and utility of such a variable in program admission decisions. Future research should be conducted to determine the appropriate weight such outcomes should be allocated within a composite score.
Although cumulative GPA upon graduation provided poor accuracy, it has been cited20 as a common measure utilized to gauge student quality. Prior investigations have revealed cumulative undergraduate GPA as contributing to and indicating success on the PANCE14,15 and NPTE18,19,21 as well as the BOC exam,22 reflecting its utility in graduate health care programs. Utzman et al19 identified an undergraduate GPA of 3.49 as an optimal threshold indicative of NPTE success. Furthermore, Bruce et al22 identified an undergraduate GPA of 3.18 as an optimal threshold indicating success in an athletic training master's program. Although their study did not identify an undergraduate GPA threshold specific to first-time BOC exam success, the 3.19 threshold identified in our study suggests that graduate-level athletic training programs may consider elevating the minimum GPA to be above the common industry standard of a 3.0.20,23
A concluding post hoc analysis consisted of examining the influence of all variables as predictors of first-time BOC exam success using a stepwise regression approach. The results of this assessment produced the following regression equation: −173.1 + 85.5 preprofessional-phase course GPA + 0.029 clinical education hours + 0.199 combined math and reading SAT scores. The corresponding coefficient of determination (R2) was 76.69%, with preprofessional-phase course GPA accounting for 38.23% of the variance, followed by 37.25% for combined math and reading SAT scores and 1.20% for clinical education hours. Our post hoc regression analysis complements our initial ROC curve outcomes in that preprofessional-phase course GPA and combined math and reading SAT scores were the overall strongest predictors of first-time BOC exam success. Consequently, there was a high degree of collinearity between preprofessional course GPA and the unique composite score (r = 0.842), reinforcing the practical utility of the latter original admissions metric as an all-inclusive or holistic indicator of first-time BOC exam success.
While the results of our study may be employed as evidence with which to consider changing or bolstering admission requirements for athletic training professional programs, the realization of such efforts will likely require review by, and discussion with, administrators at various levels of an institution's hierarchy. It is reasonable to assume that admission requirements are often analyzed in the context of overall student enrollment and the specific mission or philosophical framework of a respective institution.24 Currently, trends suggest that enrollments in American higher education institutions are declining, as noted by a decrease of 1.8% in 2018 and of 1.7% in 2019, with community colleges and 4-year public universities seeing the greatest drop,25 although enrollment in postbaccalaureate degree programs are on the rise.26 Hence, personnel should be cognizant of such variables that may influence their particular admissions model and strategically plan or adapt to help manage challenges inherent to their individual settings as a means to facilitate student and, thus, program success.
This study was descriptive in nature, and the sample was exclusive to our institution; therefore, program personnel are cautioned that applying the exact same strategies and tactics to their framework may not be feasible. In light of this, we are confident that our original work provides educators and administrators with basic concepts and advanced approaches to consider in modeling or revising their curriculum to promote overall program efficiency and quality. These mainly consist of encouraging a balanced, evidence-based, or informed, as opposed to anecdotal, perspective to drive policies and practices. Use of such a strategy will help mitigate forms of bias in what is a very complex process of operating academic programs that are subject to disparate individual administrative, faculty, and staff viewpoints. This may be especially applicable in instances in which contrasts arise in the utilization of quantitative and qualitative metrics to determine admissions.27 Correspondingly, we provide educators with explicit tools with which to weigh quantitative and qualitative data from their cohorts to establish sound minimum thresholds for such variables and to devise an all-inclusive criterion, such as our unique composite score, that intends to reflect the spirit of a holistic process.
We trust our data may also be used to realize additional efforts through which to better prepare students for sitting for the BOC exam. Given the strong association between the SAT and BOC exams, which are standardized high-stakes assessments, educators may elect to identify students who did not perform well on the SAT and explore the elements that underpin this performance, such as test anxiety, and implement strategies to offset such negative influence. Additionally, performance in preprofessional-phase courses may be an advantageous benchmark with which to recognize students who may experience difficulties passing the BOC exam on a first attempt. Thus, remediation plans for those students at risk may be contemplated as a means to improve the likelihood of BOC exam success, as has been suggested28 in other allied health care professional programs. Moreover, personnel tasked with delivering the BOC exam may find the outcomes of our study helpful in assisting in their charge to continually shape the evaluation to be a valid reflection of candidate clinical competency. The intertwining of educator and assessor standpoints in this undertaking may contribute to elucidating the intrapersonal and interpersonal factors that advance competency learning and appropriately evaluating such skills, which are requisite for success in a profession.29
Findings of this study require careful consideration with application to the current transition in entry-level degree to a master's. SAT score and the specific GPA measures analyzed in this study may not be broadly represented among entry-level master's programs' admissions criteria. However, our findings suggest that performance on standardized exams required for entry to institutions, such as the GRE, may be considered a potential influential factor in forecasting student first-time pass success on the BOC exam, which has implications for program compliance with accreditation standards. Accordingly, this particular phenomenon has been observed in related studies specific to PT education,18,19,28,30 in addition to the predictive power of prerequisite GPA18,19,21,30 and post-admission GPA.28 Thus, the findings of this study ultimately lend to developing an evidence-based practice basis for admissions criteria that programs may consider with on-boarding an entry-level master's degree in athletic training.
Some limitations exist for this study. Primarily, the data were collected from a sample of students from a single R1-classified university and retrospectively analyzed. This institutional bias may prevent our findings from being generalized to programs nationwide. Variances in athletic training program and degree requirements across universities may also limit generalization of these study findings.
The admission interview panel and preceptors providing evaluations in our program were inconsistent; therefore, the levels of expertise and years of experience in evaluating students may have varied. While this limits the internal validity of our study, it may in fact bolster external validity. Additionally, the clinical experiences of the preprofessional phase are observational in nature, meaning that preceptors can only speculate upon the success of the applicant as an athletic training student. Although preceptors can gauge the students' interest and foundational knowledge, these evaluations do not reflect the application of knowledge, skills, and abilities that are acquired and demonstrated further along in the athletic training curriculum. Hence, program administrators and faculty should view the outcomes of these assessments as supplementing the academic success of a student before admission decisions are made.
An additional factor potentially influencing our results is the consistency of course instructor. Two of the 4 preprofessional courses are offered solely at the university's main campus and are routinely taught by the same instructors during the semester that immediately precedes admission to the athletic training program, which provides consistency in course content and delivery for each incoming cohort. Contrastingly, general education courses can drastically vary with regard to campus offering, course instructor, and the semester in which the student takes such courses. Therefore, course consistency and content may be factors lending to the greater accuracy we observed for the preprofessional-phase course GPA.
Lastly, the outcomes of this study are primarily focused on bachelor degree programs; thus, the comparisons drawn to PA and PT programs require careful consideration. However, the results of this study may lend insights into athletic training programs as they continue to transition to the entry-level master's degree.
Preprofessional-phase course GPA was the most accurate secondary admission criterion indicative of successfully passing the BOC exam on a first attempt. Furthermore, the combined math and reading SAT score was the most accurate of all examined variables, which may be due to the standardized nature of the exam. The unique composite score represented a good all-inclusive indicator of success. A minimum preprofessional-phase course GPA of 3.48, combined math and reading SAT score of 960, and unique composite score of 7.34 are better suited to identify those students who may fail the BOC exam on their first attempt. The outcomes of this study may be used by educators to shape their related programmatic operations. Further investigation should focus on identifying the best student performance factors indicative of successful BOC exam outcomes.