Residency Applications in the Era of Generative Artificial Intelligence

The strength of GAI programs lies in their versatility for different writing styles and their extensive knowledge bases. Programs like ChatGPT can generate text in a conversational exchange based on patterns it has learned from vast amounts of text-based data and human feedback. As such, GAI can be used in several ways to support medical students during the application cycle. Most evidently, it can assist in crafting and refining students’ personal statements or other text-based components of their applications. Recent studies have found that reviewers cannot distinguish between personal statements written by GAI and those written by human applicants.^2,3  For example, one study comparing 5 applicant-written personal statements and 5 ChatGPT-written personal statements for plastic surgery residency found that blinded faculty participants rated them similarly for readability, originality, authenticity, and overall quality.³ 

Naturally, however, if students were to use entirely fabricated essays, it would render these statements completely impersonal and useless in the selection process.⁴  Short of ghostwriting a statement in totality, GAI could be used in a range of more acceptable ways to assist or enhance personal writing. The United Kingdom’s shared admissions service for higher education, for instance, suggests GAI could be used to brainstorm ideas, help with structuring essays, and proofread for readability.⁵  For example, in early stages, an applicant could input raw anecdotes and thoughts in the form of a “zero draft” and ask ChatGPT to organize some of the ideas into a statement outline. In later stages of revisions, ChatGPT could improve concision, refine grammar, or adjust tone. Because ChatGPT has been known to occasionally “hallucinate” false information,⁶  writers will still need to verify all output to ensure the integrity of representations of their personal beliefs and experiences.

The core challenge for medical educators involved in the resident selection process revolves around the concern that content generated or modified by ChatGPT may lack authenticity or specificity to the applicant it represents. Aside from relying on honor system statements, there is no accurate method yet to discern the use of GAI in text-based responses. Consequently, some educators may be concerned about the potential homogenization of applications due to the use of ChatGPT, making it challenging to assess for “fit.”⁴ 

However, it is worth noting that there have always been ways for medical students to enhance their applications beyond their own efforts. For example, students are often advised to seek input from mentors who can revise personal statements and other application components.⁷  Additionally, students can use grammar software or even hire specialized application consultants, although the latter has raised objections because it would disadvantage applicants who cannot afford these services.⁸  In this context, GAI is not more objectionable than these existing methods through which students refine their application materials. In fact, freely or cheaply accessible generative AI could level the playing field, aiding those who might not have access to traditional resources. Given these considerations, we believe that medical students should not be prohibited from using GAI such as ChatGPT to improve their applications. At a time when artificial intelligence is poised to streamline a plethora of tasks within medicine, selectively barring its use for residency applications seems out of step with contemporary realities.⁹ 

Moreover, those who fear that ChatGPT will increase the generic nature of students’ answers should acknowledge that even prior to the advent of GAI, most application responses already harbored similar themes. For example, one linguistic analysis of personal statements for an internal medicine program found that 95% could be classified into only 5 themes (memorable patients, research/academic, family inspirations, appeals of the program, and health care as public policy).¹⁰  Similarly, a textual analysis of general surgery personal statements found that 74% could be classified into 8 topics.¹¹ 

To determine whether GAI should be prohibited from residency applications, it is essential to define what exactly is under scrutiny in each component of the application and selection process.¹²  For example, what is the purpose of a personal statement? Some may argue that it might be used as a proxy for the applicant’s own writing ability. If creative nonfiction writing skills could be shown to predict resident performance (in medical writing or patient communication skills, for example), educators might be motivated to employ timed writing assessments in controlled settings since presently there is no limit to how many revisions applicants might make with the help of others. If instead the purpose of the personal statement is to communicate detailed information about applicants that may not be found in the remainder of the application, one should consider the potential for GAI to help facilitate this goal.¹³  The residency application process, with or without the availability of GAI, has always faced challenges in rendering holistic and meaningful portrayals of applicants.¹²  Each year, innovative instruments emerge to improve the way that we gauge resident interest and aptitude, from standardized interviews¹⁴  to technical skill assessments.¹⁵  Interestingly, these novel evaluations may be harder to game with GAI or similar technological aids.

In conclusion, while the use of GAI in residency applications might stir unease among educators involved in resident selection, we advocate against any overarching bans on its use. Instead, we urge applicants and selection committees to responsibly navigate this new era of AI. In the Box, we provide specific recommendations for how each party could approach the generation and evaluation of text-based materials for the application, considering the widespread availability of GAI. Examples of ways in which applicants can use GAI include using programs like ChatGPT to inspire, clarify, or reorganize ideas. Examples of ways in which educators could mitigate the impact of malicious uses of GAI include comparing the ideas conveyed in an applicant’s text answers against what is presented in the rest of the application package and interview. Educators may also reflect on the design of different residency application services as they may vary in their use of text-based questions as compared to other materials.

While research advances may one day allow us to accurately detect AI-generated work,¹⁶  education researchers are encouraged for now to remain curious about the effects of GAI on the residency application and selection processes. Examples of future research projects could include qualitatively examining text-based application materials in the era of GAI, compared to those submitted in prior years or anonymously surveying applicants on their use of GAI in preparing applications.

Disclosure: The views expressed herein are those of the author(s) and do not necessarily reflect the official policy or position of the Defense Health Agency, the Brooke Army Medical Center, the Department of Defense, nor any agencies under the US Government.