Detection of Residents With Progress Issues Using a Keyword–Specific Algorithm

Early identification of a resident with progress difficulties who is enrolled in postgraduate medical training is an ongoing challenge. In various studies, between 4.3% and 9.1% of residents show evidence of struggling during training.^1,2  The learning difficulties of residents are frequently identified late in their training,³  as assessors are often reluctant to mark “in difficulty” or “failure” on in-training evaluation reports (ITERs), or to designate “fail” on other end-of-rotation assessment forms.^4,5  However, the length of narratives and percentage of ambiguous or negative comments on rotation assessments, such as ITERs, indicate a potential need for resident remediation.^3,6  Narrative comments in assessing trainees have been shown to be valuable,^7,8  and must be taken into account when determining learners' progress toward achieving competencies.^9–11 

Although the tools of language analytics have been applied in education, very few articles concerning language analytics in medical education have been published, and those that have tend to focus on undergraduate students.¹²  In order to handle all the narrative information becoming available as part of competency-based medical education, we sought to develop a novel computerized semantic analysis, which consists of an algorithm that is able to detect residents with progress issues, based on certain keywords.

A database containing all ITERs (forms indicating whether preset objectives are met, corresponding narratives, overall score [pass/in difficulty/fail], and general comments) from all residents training in accredited programs at Université Laval between 2001 and 2013 was extracted and anonymized to ensure confidentiality of their track records. The ITERs were split into either family medicine or Royal College of Physicians and Surgeons of Canada (RCPSC) programs (provided as online supplemental material), as the ITER format used in family medicine for the period covered in this study differed significantly from that of other residency programs. The databases included the name of the program, residency level, rotation block number, residency beginning and end dates, residency site, CanMEDS¹³  role assessments (ratings and comments), overall rotation evaluation (comments), and number of days of absence during the training period. In general, ITERs were completed by attending faculty within 30 days of rotation completion.

For each database we identified residents with progress issues, defined as having an ITER either rated “in difficulty” (ie, struggling) or “fail.” For the purposes of the study, all ITERs from a given resident were kept until a form with the mention “failure” or “in difficulty” appeared. All ITERs following an ITER with identified progress issues were discarded.

An instructional designer reviewed all ITERs written in French and proposed terms associated with positive feedback and underperformance. Terms were determined from 133 216 words entered in the overall performance comments section and 84 365 words entered in the narrative section of each CanMEDS role of the ITERs. The first half of the database was used to make a list of positive and negative keywords that was checked for consistency against the second half of the database. French words that could have a dual meaning (either positive or negative) and conjunctions were discarded. The practical significance of this list was confirmed by the associate dean of postgraduate medical education and by a nonmedical member of the faculty of medicine. The list of these keywords with an English translation is provided as online supplemental material.

A classification rule based on these keywords was constructed by recursive partitioning using classification and regression tree methods.¹⁴  This methodology was preferred due to its flexibility in automatically selecting variables and cutoff values and its ability to produce relatively simple classification rules. Technical methodologic details are presented as online supplemental material. The classification and regression tree algorithm was applied independently to the family medicine and specialized programs data sets and tuned with the aim of obtaining rules with near 100% sensitivity (proportion of actual positives correctly identified as such) and maximal specificity (proportion of actual negatives correctly identified as such). Sensitivity, specificity, and positive and negative predictive values (proportion of true positives and true negatives, respectively) were computed for each derived classification rule. The algorithm was compared to the stringent standard of “fail” or “in difficulty” overall score, either concurrent to the keyword or anytime thereafter. We present only data of the final rules obtained for each data set. Due to the low prevalence of struggling residents, data sets were not split into training and testing sets to avoid increasing performance variability.¹⁵ 

The Université Laval Ethics Board exempted this project from review.

Statistical analyses were carried out with R 3.2.3 (R Foundation for Statistical Computing, Vienna, Austria).

There was a total of 41 618 ITERs for the 3292 registered residents. The RCPSC database contained 30 073 ITERs from 2163 residents (60% female) training in 36 accredited programs at Université Laval between May 2002 and November 2013. The family medicine database was composed of 11 545 ITERs from 1129 residents (73% female) training between August 2001 and September 2013. There are currently 910 residents registered in the 50 rural and urban, university-based accredited training programs.

Table 1 presents the performance of the chosen rule when classifying residents enrolled in an RCPSC residency program. This classification rule achieves 100% sensitivity while maximizing specificity at 87.7%. In this particular group, progress issues were identified in 78 of the 2163 residents (4%). The classification tree correctly classified these residents as having progress issues. However, the classification tree identified 256 residents as having progress issues, although they did not have an overall score indicating difficulty or failure, resulting in a positive predictive value of 23.4%.

Table 2 presents the performance of the chosen classification rule when classifying residents enrolled in the family medicine program. The classification rule achieves 100% sensitivity while maximizing specificity at 79.2%. In this group, progress issues were identified in 67 of 1129 residents (6%). The classification rule identifies these residents correctly, but it identified 221 residents as having progress issues, although they did not have an overall score indicating difficulty or failure, resulting in a positive predictive value of 23.3%.

Figures 1 and 2, respectively, present the chosen classification rules for RCPSC and family medicine residents in classification tree form. Each node of the tree represents a simple binary criterion, which includes both a keyword and its frequency, with movement down the tree going to the left when the criterion is met.

In this retrospective study, we were able to demonstrate that an algorithm based on keywords associated with a suboptimal performance would help a program director identify a struggling resident. The algorithm correctly ranked all residents who had difficulty progressing, as evidenced by the 100% sensitivity and 100% negative predictive value.

This ability of the algorithm to detect all residents with progress issues is embedded in its design. Specificity was maximized, knowing that this compromise would give a lower positive predictive value. In our opinion, the consequences of delaying the detection of a resident in difficulty are much more important than reviewing the file of an otherwise well-performing resident. A total of 334 residents were identified as having progress issues in the RCPSC data set (figure 1). However, this database includes assessment forms from 36 programs and covers a period of 12 years. Therefore, each year, on average, a program director would have to review less than 1 resident file that was falsely identified by the algorithm as being in difficulty. As for family medicine, the algorithm presented in figure 2 proved to be the most effective in detecting residents in difficulty (100% sensitivity and negative predictive value). Considering the large size of the family medicine program, the 221 false positives over the 12-year period represent only 1 or 2 cases per year per teaching site, among a group of residents already well known to the teaching site director. For the purpose of this study, the standard against which the algorithm was tested is the overall global score. It is likely that some of the false positives represent struggling residents who improved their performance following feedback from their supervisors, and joined the well-performing cohort thereafter. Alternatively, supervisors could describe underperformance in the narratives without assigning the corresponding overall score “in difficulty” or “fail.”^4,5  Therefore, some false positives may include true strugglers.

The algorithms in figures 1 and 2 highlight a series and frequency of keywords needed to detect struggling residents. They also provide some insight into the evaluation practices of assessors. For example, the high frequency of “good” suggests that this word is generally overused by assessors, as it is commonly used in most ITERs, even to describe struggling residents. The keyword “lag” has coincided with several occurrences of the keyword “excellent” (figure 1). Likewise, some positive keywords, such as “interested,” were associated with negative performance, as indicated in figure 2. This might suggest that specific encouragement wording may be preferentially used in ITERs of struggling residents.

The results of this study parallel the findings found in a previous study of 34 internal medicine residents' ITERs, reviewed individually by blinded faculty members, in which the number of words in the comment section and the percentage of ITERs with negative or ambiguous comments were associated with serious progress issues.⁶  Using a similar design, a retrospective study of general surgical residents demonstrated that 84% of struggling residents could be identified in their first year of training.¹⁶ 

Considering that automated essay scoring of the first part of the Canadian Medical Council examinations has been shown to be reliable,¹⁷  the use of an automated computer semantic analysis could facilitate the work of program directors and the office of postgraduate medical education. Given the low prevalence of residents in difficulty, a keyword approach would be a valuable flagging tool for program directors with large resident cohorts and those with little experience, as well as for the postgraduate associate dean. A subanalysis of ITERs for each program would have been an interesting addition, but the low rate of residents who experienced progress difficulties and data confidentiality concerns made this impossible.

While this algorithm made it possible to accurately identify residents in the database who have shown progress issues, it remains unable to determine the lead time between the first use of the negative keywords and the global ITER rating of “in difficulty” or “fail.” Moreover, a computerized algorithm does not understand the subtleties in the diplomatic language sometimes used in assessing trainees.¹¹  The negative predictive value and sensitivity could also vary with a different data set. Another limitation was the separate data set for family medicine residents, since at the time of the study this program used an ITER form that was significantly different from the RCPSC specialties. Moreover, all data were collected at a single university. Thus, some of the linguistic patterns could be a result of a broader institutional culture, potentially limiting its generalizability. Language could be used differently according to the gender¹⁸  or ethnic background of trainees, inducing a bias. If used inappropriately, such an algorithm could lead to false labeling of residents as strugglers. Finally, for publishing purposes, the keywords of the algorithms presented in this article were translated into English, but the statistical analysis was done using ITERs written in French. Using this algorithm in a language other than French would require transcultural validation of the keywords.

Additional prospective analyses are required to provide validity evidence for the use of the keywords of the algorithm in current cohorts. Further study to assess the algorithm's efficacy for earlier detection of underperforming trainees and to determine whether one set of keywords could be used for all programs are key next steps now that all ITERs share the same structure.

Systematic monitoring of resident progress through a prospective computerized semantic analysis using an algorithm derived from a classification with regression trees may be an effective way to identify residents in difficulty, especially given the need to analyze increasing numbers of narrative evaluations as part of competency-based medical education.