Behavioral Evaluations of Anesthesiology Residents and Overuse of the First-Person Pronoun in Personal Statements

The admissions process for anesthesiology residency encompasses an array of objective and subjective measures though which applicants are evaluated. The subjective nature of measuring an applicant's personality and affective traits often makes such measures the most difficult to tease out. The personal interview and prior clinical evaluations are considered the most reliable method to evaluate an applicant's personality to identify unwanted behavioral characteristics such as selfishness, poor work ethic, and lack of compassion, respect, or altruism.1–5 The interview represents a vital aspect of the selection process; this is confirmed by the growing use of “behavior description interviewing” among residency programs.6,7 However, the role of the personal statement and its relationship to behavior is less clear. Handwriting and text analysis among patients with psychiatric illnesses show weak correlations in the psychiatric literature.8,9 Graphology, the study and analysis of handwriting and its relation to human behavior, has largely been disproved and is considered a pseudoscience by most.10–12 Most behavioral research originated in the analysis of verbal speech, with several large studies demonstrating a relationship between distinct behavioral traits and word choices in spontaneous verbal speech.13 From these emerged a rational but intriguing correlation between egocentrism and use of the personal pronoun “I.” This association is unsurprising, as much of our social observations related to personality rely heavily on verbal interactions. However, evaluation of this self-referencing behavioral relationship within a written document, such as a personal statement, has yet to be scientifically explored. There are several individual reports of residency programs using “I” counts from personal statements in an attempt to identify selfish or egocentric behavioral traits among applicants.11 In addition, a multitude of online and printed resources warns applicants of the social dangers associated with liberal use of the “I statement,” demonstrating the pervasive cultural aversion to this pronoun.12–15 The purpose of this retrospective study is to explore the relationship between evaluations of anesthesiology residents and the use of “I” in their personal statements. The “I” counts between residents with high evaluation scores were compared to those with low scores, with the hypothesis that there is no significant difference between these 2 groups in regard to the frequency of the first-person pronoun in their personal statement.

Data were obtained from recent anesthesiology graduates of our residency program, totaling 48 residents for 5 consecutive years (2005–2009). Printed copies of residency application personal statements and former resident evaluations by faculty during the clinical anesthesia–3 (CA-3) year were confidentially collected. The personal statements were scanned into a digital file and converted to searchable text by optical character recognition software. The text was interrogated by word processing software to determine total word count and pronoun “I” count. The iScore was then calculated by determining the frequency of the self-reference pronoun “I” in relation to total word count. Evaluations consisted of former resident evaluations completed by anesthesiology faculty collected from the resident's third quarter CA-3 year. We omitted first- and second-year evaluations because variation in technical skill may influence first-year evaluations and, for the second year, the limited operating room time may limit evaluation by anesthesiology faculty. Faculty scores were tallied and averaged for the CA-3 year and reported as a 2-digit score on a 5-point scale, where a score of 1  =  unacceptable, 2  =  below expectations, 3  =  satisfactory, 4  =  above average, and 5  =  excellent/superior. Of note, faculty had no formal training on how to evaluate residents. The residents were evaluated in 5 of 6 core clinical competencies, as defined by the Accreditation Council for Graduate Medical Education (ACGME), as follows:

Attention to detail, punctual, careful, thorough, reliable, adaptable to new situations, stays cool in stressful situations, appropriately confident, admits faults truthfully, open to criticism, behaves ethically, demonstrates appropriate concern for patients.

Communicates and works effectively with patients and colleagues. Sensitive to issues of race and gender, appropriately exhibits leadership, supports teamwork.

Develops a sound anesthesia plan, independence appropriate for clinical situation and level of training, recognizes and responds appropriately to unexpected events, learns from experience and knows limits.

Fund of knowledge appropriate to level of training. Adequate general knowledge.

The data were analyzed for correlation between iScore and resident evaluations with the use of a scatter plot and the best fit line. R² values were used to determine the significance of any correlation between these variables.

Fifty-seven former residents' personal files from the previous 5 years were evaluated. Four residents were excluded as they did not complete the program and no evaluations for their CA-3 year were available. Five residents were excluded because they transferred into the program and their file did not include a personal statement. For the 48 residents in the study, the number of faculty evaluators and the faculty evaluations varied, with an overall average number of 10 faculty evaluations per resident (see table).

The mean iScore for the 48 residents in the study was 0.0427 (SD, 0.0130). Correlation analysis (expressed as R²) between iScore and evaluations was as follows: 0.06 for clinical judgment (figure 1), 0.05 for medical knowledge (figure 2), 0.08 for overall competency (figure 3), 0.11 for professionalism (figure 4), and 0.14 for interpersonal skills (figure 5).

Of the 48 former residents evaluated, we found low correlations between iScore and evaluations in clinical judgment, medical knowledge, overall competency, professionalism, or interpersonal skills. This suggests that “I” counts are not feasible as predictors of residents, performance in the ACGME core competencies.

One limitation of our study is that its retrospective nature introduces a sampling bias, as those applicants possessing unwanted behavioral traits may have been identified and excluded during the application and personal interview process. In addition, residents possessing these unwanted personality traits may have edited out many occurrences of the pronoun “I,” in response to peer editing and extensive proofreading. It is also likely that the arbitrary exclusion of other personal pronouns such as “me” or “my” may have weakened relationships in this study.

Another limitation is that our sample size is small and is confined to 1 institution. There were variations in the faculty evaluation process as well. For example, the average number of evaluations per resident remained fairly constant for the 2005 to 2008 period (9, 8, 10, and 8, respectively) but increased to 17 for 2009. During this period the department began emphasizing the importance of evaluations, and the faculty responded accordingly. A more consistent (perhaps a prospective) methodology using the same faculty evaluators who have been formally trained in evaluations might have been useful.

Ultimately, it is possible that the personal statement is a weak representation of personality when compared to other behavioral evaluation techniques such as the personal interview, particularly behavioral interviewing. In most cases, the personal statement is a thoughtfully developed document that undergoes extensive editing. This may suppress or minimize writing styles that suggest the presence of unwanted behavioral traits. Personal statements still may offer insight into personality and future behavior, but not through the “I” count. Further examination of personal statements with a larger sample size and data from other institutions and specialties is needed.