Gender Effects in Assessment of Clinical Teaching: Does Concordance Matter?

Assessments by learners are often the sole measure of teaching effectiveness in medical education¹  and have a significant effect on faculty performance reviews, merit pay, and promotion.^1,2  This is despite evidence in the broader literature that no significant relationship exists between student assessments of teaching effectiveness and actual learning.³  Such assessments often ask students to rate content and teaching methods they are not qualified to judge and are vulnerable to bias.¹  For example, greater involvement with trainees,⁴  charisma and physical attractiveness,⁵  extraversion,⁶  and even the presence of cookies during a course⁷  have been associated with higher teaching effectiveness scores in medicine.

Recently, there has been a greater focus on the effect that gender bias may have on learners' ratings of faculty in medicine. Medical students rated male physicians higher on trustworthiness, competence, and professionalism compared to female physicians,⁸  and being male was also a protective factor in perceptions of medical error.⁸  During an objective structured clinical examination, internal medicine (IM) residents perceived male faculty providers of feedback to be more credible than female faculty providers.⁹  Gender-based linguistic differences exist in how physician trainees describe faculty in their narrative assessments.¹⁰  Morgan et al¹¹  found that medical students rated female faculty lower than male faculty across surgery, obstetrics, pediatrics, and IM. Fasiotto et al¹²  similarly found that residents rated female faculty lower than male faculty across specialties; this was especially pronounced in specialties with low female faculty representation.

Both of these studies^11,12  examined teaching effectiveness ratings of faculty using the overall cohort of teaching assessments, with the latter also controlling for ethnicity, seniority, rank, specialty, and low female representation. However, neither accounted for the gender of the learner in the rating of teaching effectiveness, nor did they look at effects at the dyadic interaction level to determine whether concordance or discordance between learner and faculty influenced ratings. Small but significant effects of concordance of gender and underrepresented minority status on resident ratings of faculty were identified in a study across 18 clinical departments,¹³  but were not analyzed within or between departments that may have had very different proportions of male and female faculty and residents, which may lead to important differences in ratings. Therefore, this study aimed to determine if gender bias exists, and whether gender concordance or discordance between faculty and residents has an effect on residents' assessments of faculty teaching effectiveness across 3 clinical departments at 1 university faculty of medicine.

We conducted this study across the departments of medicine (DOM), surgery (DOS), and family and community medicine (DFCM) at the University of Toronto in Canada using data from a single academic year (July 2016 to June 2017). We selected these clinical departments for their large numbers of learners and faculty and for their differential in gender composition to enable interdepartmental comparisons. In each department, residents assess faculty teaching across a variety of clinical contexts using the Resident Assessment of Teaching Effectiveness (RATE) form at the end of each rotation (at least one faculty member of the resident's choosing must be evaluated per rotation). The RATE form has 7 items on a 5-point scale and each department has slightly different items on their forms (box), but there is commonality in the intent of each item and the same overall global.

We created a database containing all RATE forms for 1 academic year across the 3 departments. In each department, an independent research officer extracted, anonymized, collated, and linked data that included resident and faculty gender, resident relation to department (eg, medicine resident on DOM rotation = “on-service,” psychiatry resident on DOM rotation = “off-service”), and faculty academic rank (lecturer, assistant professor, associate professor, professor). We recognize there is a difference between biological sex and gender, which is how a person feels internally and/or identifies with publicly.¹⁴  We use the term gender in this article as (1) our databases are based on self-report (at time of initial employment with options of male, female, other) and are therefore more representative of the concept of gender, and (2) the term gender is more commonly used in the literature.^{8–12,15–17}  We included faculty academic rank to adjust for supervisory experience and/or perceptions of hierarchy. Given the longitudinal nature of training in DFCM, the duration of contact between resident and faculty was also coded for this cohort (in 3 categories ranging from minimal, moderate, and extensive contact based on resident interpretation and self-report at time of form completion).

We analyzed each department's data separately for 3 reasons: (1) the use and instructions around completing the RATE form varied across the departments; (2) the available covariates were not consistent across departments; and (3) separate analysis allowed a clearer comparison of the effect of gender concordance across departments. The primary outcome for the analysis was the overall global rating. While the average of all the items had greater reliability and range, the overall is the rating used for teaching effectiveness reports and merit decisions in our faculty of medicine. All analyses were replicated with the average of all items on the RATE form as a sensitivity check.

Faculty and resident gender were coded along with resident relation to service, faculty academic rank (where available), and duration of contact (where applicable). The primary analysis of the data was a mixed-effects linear regression using restricted maximum likelihood estimation to account for the nesting of data within each faculty member. Accordingly, we specified a random intercept model for faculty to estimate the variance associated with nesting of scores within each faculty and entered faculty gender and gender pairing with the resident evaluator (ie, gender discord) as fixed effects. Other covariates in the analysis included whether ratings were given by on-service versus off-service residents, faculty rank (where available), and duration of contact (where applicable).

The model building approach was to hierarchically enter potential influencers of teaching assessments, including faculty academic rank, on- vs off-service resident, duration of contact, and faculty gender. Interaction terms between these factors were also entered into the model and retained if significant. Resident gender was also included in some models to better estimate if leniency or severity of rating differed significantly between female and male residents. Lastly, gender discord was built into the model. F tests were used to assess the statistical significance of each effect with the threshold set at 0.05. The final model reported for all analyses includes only variables that reached the alpha threshold and adequate model fit, compared to the model that includes only the covariates. All analyses were done in SPSS 23 (IBM Corp, Armonk, NY) and SAS 9.2 (SAS Institute Inc, Cary, NC). The University of Toronto Research Ethics Board approved this study.

A demographic summary appears in table 1. The DFCM had the greatest percentage of female faculty and residents, followed by DOM, then DOS. Off-service residents accounted for approximately one-third of residents on medical and surgical rotations. Gender discordance in RATE scoring was lowest in DOS and highest in DFCM.

Overall, the RATE scores showed a strong skew to favorable ratings for all faculty (table 2) and a ceiling effect. Thus, these scores were reversed and log transformed prior to being converted to Z-scores to minimize skew for purposes of analysis. Because the final coefficients express effects on transformed variables, we report only the significant tests and mean differences on the original scale where applicable. As the available covariates and overall effects of gender were inconsistent across the departments, we present each's RATE score results individually (table 3). Within each department there was significant covariance associated within individual faculty's ratings (DOM [ICC = 10.1%, Wald Z = 9.6, P < .01]; DOS [ICC = 8.8%; Wald Z = 5.4, P < .0001]; and DFCM [ICC = 16.1%; Wald Z = 7.8, P < .0001], respectively). Across all 3 departments, male residents gave higher scores than female residents (table 2).

There was no significant overall difference on RATE scores given to male and female faculty. We detected a very small effect of gender discordance whereby male residents rated female faculty higher than male faculty. Female residents rated female and male faculty similarly. Off-service residents gave lower scores to faculty than on-service residents at 4.54 vs 4.72 (F(1,7431) = 3.64, P < .040). We could not determine an interaction effect with gender, as not all off-service residents' genders were known. There was no significant effect of faculty academic rank on scores.

On overall RATE scores, male faculty received higher ratings than female faculty. When resident gender was entered into the model, we detected a significant effect of gender discordance. Further exploration showed that this effect was driven by male residents giving higher scores to male versus female faculty. Female residents rated male and female faculty similarly. Off-service residents gave lower scores to faculty than on-service residents at 4.19 vs 4.51 (F(1,2045) = 60.1, P < .0001); however, there was no significant interaction between this factor and the gender of residents or of faculty. Academic rank was unavailable as a covariate for the DOS faculty.

On overall RATE scores, male faculty received higher ratings than female faculty. In the DFCM, both male and female residents rated male faculty higher than female faculty. Increased duration of contact between faculty and residents was also associated with higher ratings of faculty (F(2, 3386) = 61.2, P < .00001), and there was an effect of faculty rank with assistant professors receiving higher ratings than lecturers or associate/full professors (F(3,3386) = 48.2, P < .001). Observed gender differences remained significant after controlling for differences in the distribution of male and female faculty across academic rank and duration of contact.

In this study of the effect of dyadic gender pairing as a construct irrelevant source of variance in resident teaching assessments of IM, FM, and surgical faculty, we found that gender bias and effects of concordance varied significantly by department. While no overall gender bias was detected in the DOM, in DFCM both male and female residents rated male faculty slightly higher, and in the DOS, gender bias was driven by the concordance of slightly higher ratings of male faculty by male residents. This complex pattern of results suggests that gender bias effects require further exploration.

The lack of gender bias in the DOM contrasts with recent studies that have reported bias against female faculty.^8,9,11,12  However, a closer examination of Morgan et al¹¹  and Fassiotto et al¹²  also demonstrates less gender bias in IM. Morgan et al¹¹  observed the least discrepancy in scores in IM, and in Fassiotto et al¹²  after controlling for academic rank and proportional representation of women in specialty (IM being high), gender differences did not persist in IM ratings. Reasons for why less gender bias was detected in our DOM are unclear. Almost 40% of our faculty are female and we have had longstanding female chairs of medicine and IM program directors, so it is possible that role-modeling and female leadership may create a culture of equality and mitigate bias to some extent.

However, if role-modeling fully explained our results, we would not have expected the significant though albeit small difference in DFCM, as at the time of the study they also had a female chair and greater than 50% of faculty are female. Neither Morgan et al¹¹  nor Fassiotto et al¹²  included family medicine in their studies. In DFCM, the duration of the relationship between resident and faculty was important, with higher ratings associated with a greater duration of interaction regardless of gender. This is not entirely unexpected and may be attributed to greater time to develop a coaching relationship with the benefits that may arise from this^18,19 ; conversely, this finding may also reflect another form of bias, the “mere exposure effect,” a heuristic by which individuals favor things which are more familiar to them,²⁰  which has been observed in other studies of faculty teaching effectiveness.²¹ 

The effect of gender in surgery was heavily influenced by gender concordance, with men giving male faculty higher teaching ratings. However, as only 16% of faculty are female, the 65.5% concordance rate observed between resident and faculty was predominantly male resident and male faculty. Although female surgical residents rated male and female faculty similarly, due to the very small concordance rates of female resident–female surgeon dyads, the numbers may have been insufficient to demonstrate a difference. Morgan et al¹¹  observed the greatest difference between faculty in surgery; similarly, the effect of gender bias only persisted in the study by Fassiotto et al¹²  for specialties with low female representation, which were largely surgical. In our study, the combination of low representation at the faculty level and the high concordance with men may have contributed to the largest difference being observed here. This suggests that female surgical faculty are most vulnerable to gender bias influencing their teaching effectiveness ratings and most susceptible to the potential consequences of low ratings.

Although existing effects were small, our findings suggest that gender is a source of variance in teaching effectiveness scores. It is often on these tiny razor-thin edges that decisions regarding merit are made. Gender bias that disadvantages women has also been observed in many other areas of medicine.^22–24  Efforts have recently been directed to better understand gender bias toward women in academic medicine¹⁷  and to increase awareness of and decrease bias toward women.^15,16  However, while specific interventions may diminish gender bias, they do not eliminate it.¹⁵  The phenomena of implicit bias as it affects gender is pervasive and hard to address.

The inability to erase bias entirely suggests that additional approaches should be taken to mitigate their potentially harmful influence. Teaching assessments are often one of the main determinants of teaching awards, academic rewards, and promotion.^1,2  Although effects of gender may be small, differences can appear large, given the ceiling effect on ratings and very narrow standard deviations around means. Additionally, gender is not the only bias that influences teaching assessments: attractiveness, charisma,⁵  extraversion,⁶  and race^13,25  are just a few other examples that may unfairly disadvantage some teachers, both male and female. There have recently been calls to reduce such heavy reliance on learner ratings as the only means to rate teaching effectiveness,²⁶  and instead to include multiple data sources beyond just student assessments.¹  Other sources might include peer assessment, external expert input, learning outcomes, portfolios,¹  and a greater inclusion of narrative assessment.²⁷  None of these alone are a perfect measure of teaching effectiveness, but using collective input from multiple sources should improve the rigor and validity of teaching effectiveness measurement.

Our study is limited by the use of a dichotomous variable for gender, as this was the format of data available to us; we recognize this is no longer acceptable. We were missing data on academic rank for the DOS so could not examine the effect of seniority in this department. Our rating scale used 5 points, and scores were positively skewed. It is possible with a wider response range (eg, a 10-point scale) we may have detected more subtle differences in ratings, as previously documented in a recent study of gender bias.²⁸  Additionally, residents' ratings might have differed among the specialties due to other, not accounted for, reasons. Our study included 3 academic departments over 1 year, which may limit its generalizability to other settings. Finally, our analysis identified correlations but cannot determine causation.

Deeper exploration to understand the effects of bias, including gender, on assessments in education is an area for further study, particularly when somewhat surprising effects are observed as in the DFCM in our study (with a predominance of both female faculty and residents). Additionally, given the limitations to eliminating biases, attention should be turned to improving measurements of teaching that combine multiple methods and sources to both provide meaningful feedback to faculty and to inform robust decisions about teaching awards, merit, and career advancement.

In this study examining the relationship of gender concordance with residents' assessments of faculty teaching in IM, FM, and surgery at one institution, the effects of gender were very small and generally, but not universally, favored male faculty. Gender concordance only mattered for the DOS, whereby male surgical residents rated male faculty; in the DFCM both male and female residents rated male faculty more favorably.