Proceed, With Caution: Unconscious Bias in Technical Assessment

In 1964, Supreme Court Justice Potter Stewart made famous the expression, “I know it when I see it,” while admitting difficulty accurately converting the human impression of a visual depiction into written form. Lawyer William T. Goldberg went on to critique Stewart's statement by stating, “I know it when I see it, and someone else will know it when they see it, but what they see and what they know may or may not be what I see and what I know.”¹  The challenges described by these esteemed jurists are not dissimilar to those facing clinical faculty in the operating room who need to assess competency and autonomy in surgical trainees. Unfortunately, the intersection point between impression and competency assessment proves to fluctuate based on gender of the subject, gender of the evaluator, and bias, requiring the implementation of tools and training to reset objectivity.

At the turn of the century, increasing effort on the part of surgical educators went into transitioning the subjective technical performance of a trainee into quantifiable format. While prior technical assessments were largely end-of-rotation summative impressions, operative performance rating systems were developed to quantify performance in a specific operation immediately following the event.²  Over time, technical assessments have been refined and corroborated with performance over a variety of procedures and specialties. In recognition that technical competency and the ability to autonomously complete an operation are distinct though interrelated skills, the Zwisch scale for assessing operating room autotomy was introduced and widely adopted to complement assessments of technique.³  Most, if not all, trainees in accredited, technically based training programs now receive assessments of both competencies using numerical scoring systems. Additionally, the conversion from paper-based assessment forms to digital apps immediately available on smartphones has increased adoption and application of these valuable assessment tools.⁴ 

From the earliest proposals for use of such assessment tools, enthusiasm has grown for a transition from time-based medical education to competency-based completion of training. However, to truly attain this Holy Grail, it is incumbent on medical educators that the tools and selected terminus for completion are proven not only feasible, but also reproducible, correlative with ability, and free of bias. Failure to do so in the transition to a “meritocratic” training paradigm risks further lowering the glass ceiling on women surgeons while opening windows for early exit to men who better align with the ideal mental model of their faculty assessors. Two articles from recent issues of the Journal of Graduate Medical Education demonstrate the need to further optimize this transition to competency assessments. Chen et al report a multicenter review of nearly 95 000 assessments and identified the development of a gap in autonomy ratings for women over the course of surgical trainees, with the largest gap occurring in the most complex cases.⁵  Additionally, women trainees self-rated as having less autonomy and worse performance than men, even after controlling for most other potential confounding elements. Cooney et al's review of 8377 assessments of plastic surgery residents in 3 different programs found women trainees were scored significantly lower than men in the first 4 years of residency when rated by men attending surgeons, but not when rated by women attending surgeons.⁶  Both articles add to the large body of evidence that bias creeps into clinical assessment, and mitigation is required before transition to graduation based on competency assessments alone.

Implicit bias, or the impact of unconscious beliefs, attitudes, or stereotypes on human perception, has increasingly been implicated as negatively impacting the health care environment. While initially emphasizing the impact of such bias on patient care, societal changes are now putting the spotlight on implicit bias in the workplace as well. In 2018, the Association of Women Surgeons convened a task force and published best practices for addressing implicit bias in surgical departments.⁷  Following acknowledgement that eradication is necessary, leaders are encouraged to introduce departmental and individual assessment for bias, education for recognition and understanding, exposure to counter stereotypic individuals for resetting of impressions, and deliberate policy for bias constraint. In a similar vein, Ogunyemi demonstrated increased perception and knowledge around implicit biases in graduate medical education programs through a 90-minute workshop.⁸  Whether through standardized courses or internal efforts, it is critical that faculty and trainees completing both self-assessments and assessments of others develop awareness of the impact of their own biases on scoring. With the rollout of any assessment tool, education on not only how to use the tool but also how to avoid bias in the application of the tool is needed. Following pilot implementations, an intentional search for unconscious bias in the outcomes should be obligatory prior to wide application of the tool for high-stakes utilization, such as assessment of physician competency to practice.

As pointed out by Goldberg when considering the comments of Supreme Court Justice Stewart,¹  the different life experiences of diverse people create different biases and interpretations. As educators, it is incumbent that we work to widen the lens in which we look at others in the face of evidence that there is still work to be done in the identification and elimination of gender bias in training and assessment.