A Document Analysis of Nationally Available Faculty Assessment Forms of Resident Performance

Formative feedback is critical in moving toward self-regulation of an individual. The importance of this feedback in medical education, particularly in competency-based medical education, is well documented.¹  Characteristics of formative, useful feedback¹  have been described, and methods on how to deliver feedback, verbal or written, have also been defined.^2–5  Written feedback is considered valuable to trainees, especially when it includes components based on assessment for learning, a key feature of competency-based assessment.^6,7  High-quality elements of written feedback include information based on direct observation, specific details to assist with recall of events, balanced remarks in providing areas of strength and improvement, and providing suggestions so that residents can change.^2,7,8 

Opportunities to provide written feedback vary depending upon the training program. Faculty assessment forms (also known as end of rotation assessments and in-training evaluation reports) are typically delivered to faculty assessors through the use of residency management software. These forms will frequently have an open textbox, referred to as “Comments.”⁷ 

Found on many different types of questionnaires (eg, patient surveys, performance assessments, etc),⁹  comments textboxes can be considered the epitome of an open-ended question.^10,11  Potential advantages of having open-ended comment textboxes are that respondents are free to express individualized thoughts, opinions, and feelings, and to capture some unique information.^11,12  However, it is also recognized that open-ended comments can be problematic with regard to the analysis of the responses, the lack of specificity of comments, and the fact that respondents do not like them due to the time and effort needed to complete.^12,13  If a textbox prompt at the end of a faculty assessment form of a resident reads “specific feedback” versus “comments,” the comprehension phase (ie, the meaning interpreted by the words) in the cognitive processing of the question¹⁴  may be different.

While an open ended narrative section, such as a textbox, provides flexibility of content,⁷  the phrasing of the instructions, or prompts, directs and influences the assessor in their completion of that section.¹⁵  The placement of such textboxes, usually at the end of instruments, can lead to incompletion due to respondent or survey fatigue.^16,17  Purpose and clarity when using open-ended comment textboxes are needed, just as when using any type of questioning (eg, multiple choice or closed-ended questions).^10,13  Assessment forms should also prompt faculty to provide high-quality feedback, which we defined as being based on direct observation, adding context specific details, balanced with areas for improvement and strengths, and providing suggestions for change.

The purpose of our research is therefore twofold. First, with the knowledge from the survey field, we sought to explore existing assessment forms to determine if the forms follow survey methodology recommendations with regard to open-ended questions. Second, we utilized a theoretical framework guided by self-regulation theory¹⁸  and formative and summative components of competency-based assessment¹⁹  to determine if the forms cue faculty to complete the assessments with the explicit goal of providing high-quality written feedback.

In 2020 a document analysis was performed at a large urban academic medical center. The assessment forms analyzed for this study were created by US residency programs (n = 48), ranging from small community-based programs to large academic/university-affiliated medical centers.

Faculty assessment forms, found on our institution's residency management software (MedHub, Minneapolis, MN) and designated to be shared with other institutions, were eligible for selection in this study. MedHub allows for the creation of assessment forms through the use of a form builder. This builder walks users through all aspects of the form creation, from titling the form through linking of questions to milestones. Creators can add instructions, section headers, and multiple types of question formats in any order deemed appropriate. The system requires forms to be identified as a form “type” (ie, “Resident evaluation of staff” or “Resident self-evaluation”). Once forms are created, residency programs can choose to keep the assessment forms private or allow them to be publicly available. This designation of assessment forms permits other residency programs access and use of these forms, decreasing the burden of creation and development.

To narrow the scope of our research, the form type “Faculty evaluation of a resident,” was chosen, which resulted in 2176 forms (Figure). This designation was selected because it is a common assessment type, as most graduate medical education (GME) programs rely on faculty to assess residents in some manner.²⁰  We then divided these forms based on medical specialty. In order to decrease potential variation in assessments based on specialty, to decrease biased selection, and to obtain enough assessments to provide a robust analysis, the core residency specialties of internal medicine (250 evaluations) and surgery (92 evaluations) as defined by the Accreditation Council for Graduate Medical Education (ACGME) were selected. Based on previously published document analyses, a statistician determined approximately 30 forms would need to be analyzed. We placed the assessment titles on a numbered spreadsheet and used a random number generator to select documents from each specialty separately for analysis. Assessments created by national certifying boards (American Board of Internal Medicine and American Board of Surgery) were excluded because these same forms are used by many programs and would result in data being biased toward those forms' characteristics. Because some programs had multiple assessment forms identified as “Faculty evaluation of a resident,” only one assessment per specialty within each institution was eligible for selection to minimize data skewing based on potential program tendencies.

Document analysis, or content analysis, as a research methodology requires an iterative process of researchers reading and interpreting written or electronic text to provide a voice to the document(s).²¹  The findings can be numerical and/or qualitative depending on the research question(s) being asked.²²  Document analysis was used to describe the prevailing practices in assessment forms created and used by residency programs across the United States.

We developed a document summary form,²²  used to analyze the faculty assessment forms, in Excel (Microsoft Corp, Redmond, WA) and refined it over several iterations to capture the variations in faculty assessment of residents in order to describe prevailing practices with regard to these assessment forms. This type of summary form allowed us to turn non-numerical data (eg, assessment instructions) into numbers. We created several categories not only to help describe the assessments overall, but also to specifically examine the written feedback sections available in the documents. The items of direct observation, discussed with resident box (radio button to indicate if assessment contents were discussed with the resident), and space for comments were chosen based on the framework of self-regulation. Reference to competency, entrustable professional activity (EPA), norm referenced language, milestone-based, and formative or summative were determined to be important to identify characteristics of competency-based assessment. Consensus was reached on the document summary categories found in the Table.

Once the subset of assessment forms was determined, we utilized the document summary form to analyze the assessments one at a time. For example, each form was reviewed to determine if form instructions used to describe the logistics or layout of the form were present. The document summary form was then marked either with 1 (Yes, present) or 2 (No, not present). If the form mentioned any of the 6 ACGME core competencies, the document summary form was marked with a “1” to indicate the presence of a core competency. See the Table for the complete list of items analyzed and how those items were scored. To increase interrater reliability, we analyzed the first 5 assessment forms together until consensus was reached across all document summary categories. Because of the time-consuming nature of partnered analysis, the remaining forms were equally divided between us and individually analyzed. After individual analysis, we combined and reviewed our document summary sheet results and discussed our findings. We further reviewed any assessment form as needed based on our discussion to ensure accurate scoring of form items.

We conducted this research within a post-positivist paradigm, which recognizes that reality is probable and fallible.²³  This paradigm was chosen because multiple measures and observations completed by 2 researchers would reduce bias in order to construct an agreed upon view of assessment form characteristics.

As qualitative researchers, we are mindful that our experiences shape the way we collect and interpret data.²⁴  Both authors have training in qualitative research methodology. J.C.F. is a PhD educator in a general surgery residency program and is vice chair of education for the department. L.C.P. is a clinician-educator, being the associate director of staff educator faculty development programs, actively involved in GME training programs, and is a practicing clinician. We have been involved in the creation of assessment forms and use assessment forms in our roles on Clinical Competency Committees and in training residents. We have access to the residency management software through our roles in GME. The use of these assessment forms in our various roles has forced us to view them from different perspectives, but we are mindful that the purpose of faculty assessments of residents should be to help improve resident performance.

The study variables (document summary categories across all forms) were described using numbers with proportions as appropriate or sample medians with interquartile ranges. The Cleveland Clinic Institutional Review Board deemed this study exempt from oversight.

With the selection criteria in place, this left 48 assessment forms each from a unique residency program (30 internal medicine and 18 surgery) to analyze.

Most assessment forms contained 11 to 20 questions (23 of 48, 48%, range 5–53), and 14 assessments (29%) consisted of more than 21 questions. Nearly half (23 of 48, 48%) contained form instructions on how to complete the form (eg, “Rate resident performance based on the scale provided below”) and instructions specific to assessment, referred to as assessment instructions (52%; eg, “Milestone levels are not grades”).

All 48 forms provided an open textbox for a faculty narrative, and 26 (54%) made this textbox required for assessment form completion. The open textbox was most often found at the end of the assessment (40 of 48, 83%). Sixteen forms (33%) contained a simple prompt (“Comments”) to the narrative section, but many (21 of 48, 44%) provided more specific instructions (eg, “List areas of strength”).

Thirty-nine forms (81%) made reference to direct observation of resident performance. Five assessments (10%) had a box to check if the program director required contact based on the performance, and 7 (15%) had a box to check specifically if the information on the form was discussed with the resident.

Forms were also analyzed for terminology and themes common to competency-based education. Twenty-one (44%) forms made some mention of at least 1 of the 6 ACGME core competencies, and 4 (8%) assessments mentioned EPAs. Fifteen (33%) forms made reference to specific postgraduate year levels, and 9 (19%) used norm referenced language. Nineteen (40%) used milestone levels for the rating scale, and 27 (56%) assessments contained some type of specific criteria (defined as any criteria beyond milestone levels) within the assessment scale. When attempting to determine if the form was supposed to be for formative or summative purposes, 50% of documents were found to contain terminology specific to both and were thus labeled “unclear formative or summative.” Nineteen (40%) forms were identified as being an end of rotation assessment, and 25 (52%) were unclear as to their frequency based on the information on the form.

Our document analysis of available faculty assessment of residents reveals that assessment forms do not always follow survey methodology recommendations for open narrative sections. The assessment forms in our analysis also failed to universally address elements of self-regulation and competency-based assessment characteristics. Thus, many assessments of resident forms miss elements necessary for the provision of quality feedback.

While the ubiquitous inclusion of comment textboxes is promising for feedback to residents, the lack of requirement for completion before submission decreases their potential benefit.⁷  The common end of form placement of the narrative section introduces the possibility of respondent fatigue. This is a common problem in survey research,¹⁶  and the concept applies to assessment forms in medical education.²⁵  The length of an assessment form can affect completion rate, quality of responses, and assessor motivation. Competing factors such as clinical demands, lack of interest, and boredom can decrease faculty responses.²⁶  Survey methodology also highlights the importance of purpose and clarity for open-ended responses^10,13,17,27 ; however, the prompt “Comments” for the narrative textbox is lacking in both. Previous research has shown that the prompt “General Comments” elicited hundreds of comments not related to any medical education competency.²⁸  If programs truly want to obtain written, quality feedback from these forms, the narrative section must be created with attention to survey research and medical education recommendations. The purpose for the narrative feedback section should be clearly stated⁷  with an appropriately worded prompt (eg, “Please list the resident's strengths” or “Suggest a plan for improvement”) needed to elicit rich responses.²⁹  Programs cannot rely on busy faculty clinicians to remember what should be included in written feedback when the only prompt they are provided is “Comments.”

Self-regulation theory highlights the value of external feedback that can aid learners in improving their self-assessments.¹⁸  Competency-based medical education requires the need for meaningful assessments and feedback.¹⁹  Ambiguity about the purposes of the assessment, formative or summative, can cause struggles for the trainees and the faculty as each stakeholder attempts to determine what “hat” they should be wearing at any given time (eg, learner, performer, teacher, or assessor).³⁰  Previous research has highlighted the importance of ensuring clarity of the purpose of the assessment to all involved.³¹  The feedback learners receive, regardless of its purpose, is recommended by medical educators and researchers to be more than just documentation on an assessment form, and conversations with feedback providers may be considered.³²  Since few assessment forms in our analysis required documentation of these conversations, they could be occurring organically outside of the assessment form process. There may be value in asking faculty assessors if these conversations took place, as this could potentially serve as a reminder to faculty to have these important in-person discussions.⁷  The type of assessment, assessment for learning or assessment of learning, is recommended to be clear to all involved to guide the assessment process.³³ 

These findings are limited by being confined to the assessments available only through a share function in a single residency management software system; however, there were more than 2000 “faculty evaluation of resident” forms that could have been analyzed. It is possible that programs obtain formative written feedback via other methods, and they may not rely exclusively on these types of faculty assessment forms. We have no way of knowing what other instructions, regarding form completion or resident assessment, are given to assessors outside of the actual form (eg, an email before faculty completion, verbal communications, etc). We are unable to determine what actual responses were obtained from the written feedback textbox in the assessment forms analyzed in this study as that is outside the scope of our research. The possibility exists that the current formatting and instructions for the narrative textboxes on those forms provide high-yield, high-quality written feedback, but we know from several studies that faculty feedback is not always adequate or useful to trainees.^2,34,35 

Future research should compare how different characteristics of the questions used on assessment forms affect the quantity and quality of faculty response. Special attention should be paid to the prompts used to elicit narrative comments and if the differences then alter faculty feedback quality.

Our document analysis of a random sampling of 48 nationally available internal medicine and surgery faculty assessments of residents revealed that, while assessment forms universally included a narrative box for comments, there was not a requirement for box completion in half of those forms. Prompts to aid faculty in writing narrative feedback were absent in 33% of forms and did not address elements deemed important for promotion of self-regulation and competency-based assessment.

The authors would like to thank all programs for making their assessment and evaluation forms available to others in MedHub. They would also like to thank Xiaofeng Wang, PhD, Department of Quantitative Health Sciences, Cleveland Clinic, for statistical support.