Background

Written feedback by faculty of resident performance is valuable when it includes components based on assessment for learning. However, it is not clear how often assessment forms include these components for summative and formative feedback.

Objective

To analyze prompts used in forms for faculty assessment of resident performance, guided by best practices in survey research methodology, self-regulation theory, and competency-based assessment.

Methods

A document analysis, which is a qualitative approach used to analyze content and structure of texts, was completed on assessment forms nationally available in MedHub. Due to the number of forms available, only internal medicine and surgery specialties were included. A document summary form was created to analyze the assessments. The summary form guided researchers through the analysis.

Results

Forty-eight forms were reviewed, each from a unique residency program. All forms provided a textbox for comments, and 54% made this textbox required for assessment completion. Eighty-three percent of assessments placed the open textbox at the end of the form. One-third of forms contained a simple prompt, “Comments,” for the narrative section. Fifteen percent of forms included a box to check if the information on the form had been discussed with the resident. Fifty percent of the assessments were unclear if they were meant to be formative or summative in nature.

Conclusions

Our document analysis of assessment forms revealed they do not always follow best practices in survey design for narrative sections, nor do they universally address elements deemed important for promotion of self-regulation and competency-based assessment.

Objectives

To analyze prompts used in forms for faculty assessment of resident performance forms as guided by survey research methodology, self-regulation theory, and competency-based assessment.

Findings

Our document analysis of assessment forms revealed they do not always follow survey methodology recommendations for narrative sections, nor do they universally address elements deemed important for promotion of self-regulation and competency-based assessment.

Limitations

The findings are limited by being confined to the blank assessments only available through a share function in a single residency management software system.

Bottom Line

Attention should be paid to the prompts used to elicit narrative comments to see if these differences alter faculty feedback quality.

Formative feedback is critical in moving toward self-regulation of an individual. The importance of this feedback in medical education, particularly in competency-based medical education, is well documented.1  Characteristics of formative, useful feedback1  have been described, and methods on how to deliver feedback, verbal or written, have also been defined.25  Written feedback is considered valuable to trainees, especially when it includes components based on assessment for learning, a key feature of competency-based assessment.6,7  High-quality elements of written feedback include information based on direct observation, specific details to assist with recall of events, balanced remarks in providing areas of strength and improvement, and providing suggestions so that residents can change.2,7,8 

Opportunities to provide written feedback vary depending upon the training program. Faculty assessment forms (also known as end of rotation assessments and in-training evaluation reports) are typically delivered to faculty assessors through the use of residency management software. These forms will frequently have an open textbox, referred to as “Comments.”7 

Found on many different types of questionnaires (eg, patient surveys, performance assessments, etc),9  comments textboxes can be considered the epitome of an open-ended question.10,11  Potential advantages of having open-ended comment textboxes are that respondents are free to express individualized thoughts, opinions, and feelings, and to capture some unique information.11,12  However, it is also recognized that open-ended comments can be problematic with regard to the analysis of the responses, the lack of specificity of comments, and the fact that respondents do not like them due to the time and effort needed to complete.12,13  If a textbox prompt at the end of a faculty assessment form of a resident reads “specific feedback” versus “comments,” the comprehension phase (ie, the meaning interpreted by the words) in the cognitive processing of the question14  may be different.

While an open ended narrative section, such as a textbox, provides flexibility of content,7  the phrasing of the instructions, or prompts, directs and influences the assessor in their completion of that section.15  The placement of such textboxes, usually at the end of instruments, can lead to incompletion due to respondent or survey fatigue.16,17  Purpose and clarity when using open-ended comment textboxes are needed, just as when using any type of questioning (eg, multiple choice or closed-ended questions).10,13  Assessment forms should also prompt faculty to provide high-quality feedback, which we defined as being based on direct observation, adding context specific details, balanced with areas for improvement and strengths, and providing suggestions for change.

The purpose of our research is therefore twofold. First, with the knowledge from the survey field, we sought to explore existing assessment forms to determine if the forms follow survey methodology recommendations with regard to open-ended questions. Second, we utilized a theoretical framework guided by self-regulation theory18  and formative and summative components of competency-based assessment19  to determine if the forms cue faculty to complete the assessments with the explicit goal of providing high-quality written feedback.

In 2020 a document analysis was performed at a large urban academic medical center. The assessment forms analyzed for this study were created by US residency programs (n = 48), ranging from small community-based programs to large academic/university-affiliated medical centers.

Faculty assessment forms, found on our institution's residency management software (MedHub, Minneapolis, MN) and designated to be shared with other institutions, were eligible for selection in this study. MedHub allows for the creation of assessment forms through the use of a form builder. This builder walks users through all aspects of the form creation, from titling the form through linking of questions to milestones. Creators can add instructions, section headers, and multiple types of question formats in any order deemed appropriate. The system requires forms to be identified as a form “type” (ie, “Resident evaluation of staff” or “Resident self-evaluation”). Once forms are created, residency programs can choose to keep the assessment forms private or allow them to be publicly available. This designation of assessment forms permits other residency programs access and use of these forms, decreasing the burden of creation and development.

To narrow the scope of our research, the form type “Faculty evaluation of a resident,” was chosen, which resulted in 2176 forms (Figure). This designation was selected because it is a common assessment type, as most graduate medical education (GME) programs rely on faculty to assess residents in some manner.20  We then divided these forms based on medical specialty. In order to decrease potential variation in assessments based on specialty, to decrease biased selection, and to obtain enough assessments to provide a robust analysis, the core residency specialties of internal medicine (250 evaluations) and surgery (92 evaluations) as defined by the Accreditation Council for Graduate Medical Education (ACGME) were selected. Based on previously published document analyses, a statistician determined approximately 30 forms would need to be analyzed. We placed the assessment titles on a numbered spreadsheet and used a random number generator to select documents from each specialty separately for analysis. Assessments created by national certifying boards (American Board of Internal Medicine and American Board of Surgery) were excluded because these same forms are used by many programs and would result in data being biased toward those forms' characteristics. Because some programs had multiple assessment forms identified as “Faculty evaluation of a resident,” only one assessment per specialty within each institution was eligible for selection to minimize data skewing based on potential program tendencies.

Figure

Assessment Form Selection Flow Diagram

a Only forms created in MedHub (Minneapolis, MN) were eligible for selection.

b Some programs had multiple forms eligible for selection, but only one per program was chosen.

Figure

Assessment Form Selection Flow Diagram

a Only forms created in MedHub (Minneapolis, MN) were eligible for selection.

b Some programs had multiple forms eligible for selection, but only one per program was chosen.

Close modal

Document analysis, or content analysis, as a research methodology requires an iterative process of researchers reading and interpreting written or electronic text to provide a voice to the document(s).21  The findings can be numerical and/or qualitative depending on the research question(s) being asked.22  Document analysis was used to describe the prevailing practices in assessment forms created and used by residency programs across the United States.

We developed a document summary form,22  used to analyze the faculty assessment forms, in Excel (Microsoft Corp, Redmond, WA) and refined it over several iterations to capture the variations in faculty assessment of residents in order to describe prevailing practices with regard to these assessment forms. This type of summary form allowed us to turn non-numerical data (eg, assessment instructions) into numbers. We created several categories not only to help describe the assessments overall, but also to specifically examine the written feedback sections available in the documents. The items of direct observation, discussed with resident box (radio button to indicate if assessment contents were discussed with the resident), and space for comments were chosen based on the framework of self-regulation. Reference to competency, entrustable professional activity (EPA), norm referenced language, milestone-based, and formative or summative were determined to be important to identify characteristics of competency-based assessment. Consensus was reached on the document summary categories found in the Table.

Table

Document Summary Categories for Assessment Forms

Document Summary Categories for Assessment Forms
Document Summary Categories for Assessment Forms

Once the subset of assessment forms was determined, we utilized the document summary form to analyze the assessments one at a time. For example, each form was reviewed to determine if form instructions used to describe the logistics or layout of the form were present. The document summary form was then marked either with 1 (Yes, present) or 2 (No, not present). If the form mentioned any of the 6 ACGME core competencies, the document summary form was marked with a “1” to indicate the presence of a core competency. See the Table for the complete list of items analyzed and how those items were scored. To increase interrater reliability, we analyzed the first 5 assessment forms together until consensus was reached across all document summary categories. Because of the time-consuming nature of partnered analysis, the remaining forms were equally divided between us and individually analyzed. After individual analysis, we combined and reviewed our document summary sheet results and discussed our findings. We further reviewed any assessment form as needed based on our discussion to ensure accurate scoring of form items.

We conducted this research within a post-positivist paradigm, which recognizes that reality is probable and fallible.23  This paradigm was chosen because multiple measures and observations completed by 2 researchers would reduce bias in order to construct an agreed upon view of assessment form characteristics.

As qualitative researchers, we are mindful that our experiences shape the way we collect and interpret data.24  Both authors have training in qualitative research methodology. J.C.F. is a PhD educator in a general surgery residency program and is vice chair of education for the department. L.C.P. is a clinician-educator, being the associate director of staff educator faculty development programs, actively involved in GME training programs, and is a practicing clinician. We have been involved in the creation of assessment forms and use assessment forms in our roles on Clinical Competency Committees and in training residents. We have access to the residency management software through our roles in GME. The use of these assessment forms in our various roles has forced us to view them from different perspectives, but we are mindful that the purpose of faculty assessments of residents should be to help improve resident performance.

The study variables (document summary categories across all forms) were described using numbers with proportions as appropriate or sample medians with interquartile ranges. The Cleveland Clinic Institutional Review Board deemed this study exempt from oversight.

With the selection criteria in place, this left 48 assessment forms each from a unique residency program (30 internal medicine and 18 surgery) to analyze.

General Characteristics

Most assessment forms contained 11 to 20 questions (23 of 48, 48%, range 5–53), and 14 assessments (29%) consisted of more than 21 questions. Nearly half (23 of 48, 48%) contained form instructions on how to complete the form (eg, “Rate resident performance based on the scale provided below”) and instructions specific to assessment, referred to as assessment instructions (52%; eg, “Milestone levels are not grades”).

Narrative Comment Characteristics

All 48 forms provided an open textbox for a faculty narrative, and 26 (54%) made this textbox required for assessment form completion. The open textbox was most often found at the end of the assessment (40 of 48, 83%). Sixteen forms (33%) contained a simple prompt (“Comments”) to the narrative section, but many (21 of 48, 44%) provided more specific instructions (eg, “List areas of strength”).

Self-Regulation Characteristics

Thirty-nine forms (81%) made reference to direct observation of resident performance. Five assessments (10%) had a box to check if the program director required contact based on the performance, and 7 (15%) had a box to check specifically if the information on the form was discussed with the resident.

Competency-Based Assessment Characteristics

Forms were also analyzed for terminology and themes common to competency-based education. Twenty-one (44%) forms made some mention of at least 1 of the 6 ACGME core competencies, and 4 (8%) assessments mentioned EPAs. Fifteen (33%) forms made reference to specific postgraduate year levels, and 9 (19%) used norm referenced language. Nineteen (40%) used milestone levels for the rating scale, and 27 (56%) assessments contained some type of specific criteria (defined as any criteria beyond milestone levels) within the assessment scale. When attempting to determine if the form was supposed to be for formative or summative purposes, 50% of documents were found to contain terminology specific to both and were thus labeled “unclear formative or summative.” Nineteen (40%) forms were identified as being an end of rotation assessment, and 25 (52%) were unclear as to their frequency based on the information on the form.

Our document analysis of available faculty assessment of residents reveals that assessment forms do not always follow survey methodology recommendations for open narrative sections. The assessment forms in our analysis also failed to universally address elements of self-regulation and competency-based assessment characteristics. Thus, many assessments of resident forms miss elements necessary for the provision of quality feedback.

While the ubiquitous inclusion of comment textboxes is promising for feedback to residents, the lack of requirement for completion before submission decreases their potential benefit.7  The common end of form placement of the narrative section introduces the possibility of respondent fatigue. This is a common problem in survey research,16  and the concept applies to assessment forms in medical education.25  The length of an assessment form can affect completion rate, quality of responses, and assessor motivation. Competing factors such as clinical demands, lack of interest, and boredom can decrease faculty responses.26  Survey methodology also highlights the importance of purpose and clarity for open-ended responses10,13,17,27 ; however, the prompt “Comments” for the narrative textbox is lacking in both. Previous research has shown that the prompt “General Comments” elicited hundreds of comments not related to any medical education competency.28  If programs truly want to obtain written, quality feedback from these forms, the narrative section must be created with attention to survey research and medical education recommendations. The purpose for the narrative feedback section should be clearly stated7  with an appropriately worded prompt (eg, “Please list the resident's strengths” or “Suggest a plan for improvement”) needed to elicit rich responses.29  Programs cannot rely on busy faculty clinicians to remember what should be included in written feedback when the only prompt they are provided is “Comments.”

Self-regulation theory highlights the value of external feedback that can aid learners in improving their self-assessments.18  Competency-based medical education requires the need for meaningful assessments and feedback.19  Ambiguity about the purposes of the assessment, formative or summative, can cause struggles for the trainees and the faculty as each stakeholder attempts to determine what “hat” they should be wearing at any given time (eg, learner, performer, teacher, or assessor).30  Previous research has highlighted the importance of ensuring clarity of the purpose of the assessment to all involved.31  The feedback learners receive, regardless of its purpose, is recommended by medical educators and researchers to be more than just documentation on an assessment form, and conversations with feedback providers may be considered.32  Since few assessment forms in our analysis required documentation of these conversations, they could be occurring organically outside of the assessment form process. There may be value in asking faculty assessors if these conversations took place, as this could potentially serve as a reminder to faculty to have these important in-person discussions.7  The type of assessment, assessment for learning or assessment of learning, is recommended to be clear to all involved to guide the assessment process.33 

These findings are limited by being confined to the assessments available only through a share function in a single residency management software system; however, there were more than 2000 “faculty evaluation of resident” forms that could have been analyzed. It is possible that programs obtain formative written feedback via other methods, and they may not rely exclusively on these types of faculty assessment forms. We have no way of knowing what other instructions, regarding form completion or resident assessment, are given to assessors outside of the actual form (eg, an email before faculty completion, verbal communications, etc). We are unable to determine what actual responses were obtained from the written feedback textbox in the assessment forms analyzed in this study as that is outside the scope of our research. The possibility exists that the current formatting and instructions for the narrative textboxes on those forms provide high-yield, high-quality written feedback, but we know from several studies that faculty feedback is not always adequate or useful to trainees.2,34,35 

Future research should compare how different characteristics of the questions used on assessment forms affect the quantity and quality of faculty response. Special attention should be paid to the prompts used to elicit narrative comments and if the differences then alter faculty feedback quality.

Our document analysis of a random sampling of 48 nationally available internal medicine and surgery faculty assessments of residents revealed that, while assessment forms universally included a narrative box for comments, there was not a requirement for box completion in half of those forms. Prompts to aid faculty in writing narrative feedback were absent in 33% of forms and did not address elements deemed important for promotion of self-regulation and competency-based assessment.

The authors would like to thank all programs for making their assessment and evaluation forms available to others in MedHub. They would also like to thank Xiaofeng Wang, PhD, Department of Quantitative Health Sciences, Cleveland Clinic, for statistical support.

1. 
Sargeant
JM,
Holmboe
ES.
Feedback and coaching in clinical teaching and learning
.
In:
Holmboe
ES,
Durning
SJ,
Hawkins
RE,
eds.
Practical Guide to the Evaluation of Clinical Competence. 2nd ed
.
Amsterdam, Netherlands
:
Elsevier;
2018
:
256
269
.
2. 
Dudek
NL,
Marks
MB,
Wood
TJ,
Lee
AC.
Assessing the quality of supervisors' completed clinical evaluation reports
.
Med Educ
.
2008
;
42
(
8
):
816
822
.
3. 
Watling
CJ,
Ginsburg
S.
Assessment, feedback, and the alchemy of learning
.
Med Educ
.
2019
;
53
(
1
):
76
85
.
4. 
French
JC,
Colbert
CY,
Pien
LC,
et al
Targeted feedback in the milestones era: utilization of the ask-tell-ask feedback model to promote reflection and self-assessment
.
J Surg Educ
.
2015
;
72
(
6
):
e274
e279
.
5. 
Sargeant
J,
Lockyer
J,
Mann
K,
et al
Facilitated reflective performance feedback: developing an evidence- and theory-based model that builds relationship, explores reactions and content, and coaches for performance change (R2C2)
.
Acad Med
.
2015
;
90
(
12
):
1698
1706
.
6. 
Brutus
S.
Words versus numbers: a theoretical exploration of giving and receiving narrative comments in performance appraisal
.
Hum Resour Manag Rev
.
2010
;
20
(
2
):
144
157
.
7. 
Dudek
N,
Cook
D.
Narrative assessment
.
In:
Yudkowsky
R,
YS,
Park
Downing
SM,
eds.
Assessment in Health Professions Education. 2nd ed
.
London, United Kingdom
:
Routledge;
2020
:
173
180
.
8. 
Zendejas
B,
Toprak
A,
Harrington
AW,
Lillehei
CW,
Modi
BP.
Quality of dictated feedback associated with SIMPL operative assessments of pediatric surgical trainees
.
Am J Surg
.
2021
;
221
(
2
):
303
308
.
9. 
Lockyer
JM.
Multisource feedback
.
In:
Holmboe
ES,
Durning
SJ,
Hawkins
RE,
eds.
Practical Guide to the Evaluation of Clinical Competence. 2nd ed
.
Amsterdam, Netherlands
:
Elsevier;
2018
:
204
214
.
10. 
Fink
A.
How to Conduct Surveys: A Step-by Step Guide. 4th ed
.
Thousand Oaks, CA
:
SAGE Publications;
2009
.
11. 
Sue
VM,
Ritter
LA.
Conducting Online Surveys
.
Thousand Oaks, CA
:
SAGE Publications;
2007
.
12. 
Fraenkel
J,
Wallen
N,
Hyun
H.
How to Design and Evaluate Research in Education. 10th ed
.
New York, NY
:
McGraw Hill;
2019
.
13. 
O'Cathain
A,
Thomas
KJ.
“Any other comments?” Open questions on questionnaires—a bane or bonus to research?
BMC Med Res Methodol
.
2004
;
4
:
25
.
14. 
Willis
GB,
Artino
AR
Jr.
What do our respondents think we're asking? Using cognitive interviewing to improve medical education surveys
.
J Grad Med Educ
.
2013
;
5
(
3
):
353
356
.
15. 
Fowler
FJ,
Cosenza
C.
Design and evaluation of survey questions
.
In:
Bickman
L,
Debra
J,
eds.
The SAGE Handbook of Applied Social Research Methods. 2nd ed
.
Thousand Oaks, CA
:
SAGE Publications;
2009
:
375
412
.
16. 
O'Reilly-Shah
VN.
Factors influencing healthcare provider respondent fatigue answering a globally administered in-app survey
.
Peer J
.
2017
;
5
:
e3785
.
17. 
Sullivan
GM,
Artino
AR
Jr.
How to create a bad survey instrument
.
J Grad Med Educ
.
2017
;
9
(
4
):
411
415
.
18. 
Sandars
J,
Cleary
TJ.
Self-regulation theory: applications to medical education: AMEE Guide No. 58
.
Med Teach
.
2011
;
33
(
11
):
875
886
.
19. 
Lockyer
J,
Carraccio
C,
Chan
MK,
et al
Core principles of assessment in competency-based medical education
.
Med Teach
.
2017
;
39
(
6
):
609
616
.
20. 
Holt
KD,
Miller
RS,
Nasca
TJ.
Residency programs' evaluations of the competencies: data provided to the ACGME about types of assessments used by programs
.
J Grad Med Educ
.
2010
;
2
(
4
):
649
655
.
21. 
Bowen
GA.
Document analysis as a qualitative research method
.
Qual Res J
.
2009
;
9
(
2
):
27
40
.
22. 
Miles
MB,
Huberman
AM,
Saldana
J.
Qualitative Data Analysis. 4th ed
.
Thousand Oaks, CA
:
SAGE Publications;
2019
.
23. 
Teherani
A,
Martimianakis
T,
Stenfors-Hayes
T,
Wadhwa
A,
Varpio
L.
Choosing a qualitative research approach
.
J Grad Med Educ
.
2015
;
7
(
4
):
669
670
.
24. 
Merriam
SB.
Qualitative Research: A Guide to Design and Implementation
.
Hoboken, NJ
:
Jossey-Bass;
2009
.
25. 
Artino
AR
Jr,
La Rochelle
JS,
Dezee
KJ,
Gehlbach
H.
Developing questionnaires for educational research: AMEE Guide No. 87
.
Med Teach
.
2014
;
36
(
6
):
463
474
.
26. 
Lavrakas
PJ,
ed.
Encyclopedia of Survey Research Methods
.
Thousand Oaks
:
SAGE Publications;
2008
.
27. 
Schwarz
N.
Self-reports: how the questions shape the answers
.
Am Psychol
.
1999
;
54
(
2
):
93
105
.
28. 
Ginsburg
S,
Gold
W,
Cavalcanti
RB,
Kurabi
B,
McDonald-Blumer
H.
Competencies “plus”: the nature of written comments on internal medicine residents' evaluation forms
.
Acad Med
.
2011
;
86
(
suppl 10
):
30
34
.
29. 
LaDonna
KA,
Taylor
T,
Lingard
L.
Why open-ended survey questions are unlikely to support rigorous qualitative insights
.
Acad Med
.
2018
;
93
(
3
):
347
349
.
30. 
Prentice
S,
Benson
J,
Kirkpatrick
E,
Schuwirth
L.
Workplace-based assessments in postgraduate medical education: a hermeneutic review
.
Med Educ
.
2020
;
54
(
11
):
981
992
.
31. 
Hamstra
SJ.
Workplace-based assessment of procedural skills
.
In:
Holmboe
ES,
Durning
SJ,
Hawkins
RE,
eds.
Practical Guide to the Evaluation of Clinical Competence. 2nd ed
.
Amsterdam, Netherlands
:
Elsevier;
2018
:
155
164
.
32. 
Kogan
JR,
Hatala
R,
Hauer
KE,
Holmboe
E.
Guidelines: the do's, don'ts and don't knows of direct observation of clinical skills in medical education
.
Perspect Med Educ
.
2017
;
6
(
5
):
286
305
.
33. 
Pangaro
LN,
Durning
SJ,
Holmboe
ES.
Evaluation frameworks, forms, and global rating scales
.
In:
Holmboe
ES,
Durning
SJ,
Hawkins
RE,
eds.
Practical Guide to the Evaluation of Clinical Competence. 2nd ed
.
Amsterdam, Netherlands
:
Elsevier;
2018
:
37
60
.
34. 
Sokol-Hessner
L,
Shea
JA,
Kogan
JR.
The open-ended comment space for action plans on core clerkship students' encounter cards: what gets written?
Acad Med
.
2010
;
85
(
suppl 10
):
110
114
.
35. 
Jackson
JL,
Kay
C,
Jackson
WC,
Frank
M.
The quality of written feedback by attendings of internal medicine residents
.
J Gen Intern Med
.
2014
;
30
(
7
):
973
978
.

Author notes

Funding: The authors report no external funding source for this study.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.