Use of Portfolios for Assessment of Resident Teaching Skills Open Access

Residents spend 20% of their daily activity in teaching efforts, primarily instructing other residents and medical students on the practical delivery of medical care.1 Fifty-five percent of internal medicine programs have created formal resident-as-teacher curricula to enhance their residents' teaching skills.2 However, assessment of this teaching with commonly used assessment tools has significant limitations. Two common instruments, surveys of learners and direct observations of teaching, are limited by their context specificity and potential for grade inflation and halo effects.3 Objective measures of teaching skills are another tool that can serve to standardize the teaching environment1; however, time and resource constraints limit the ability to perform repeated assessments needed to track growth over time. Professional teachers and their assessors have long recognized these limitations and have moved to the use of portfolios for longitudinal assessment of teaching effectiveness.4 

However, assessment of portfolio content presents its own set of challenges. Reliability is too low for high-stakes decisions (r  =  0.1–0.6).5–7 Think-aloud studies suggest that portfolio assessors apply none of the heuristics they use in vetting data from their field of study and demonstrate no specific, identifiable criteria for their analysis of portfolios.8,9 To create a fair and transparent process for portfolio assessment, it is recommended that assessors specify clear and transparent goals and develop scoring rubrics for assessment.8 For those rubrics to be authentic and valid for users, they should be derived from either existing learning theories or from reviews of existing portfolio assessments.

The question that guided our study was “What evidence of scholarly teaching and longitudinal growth could be identified from reviewing resident portfolios?” Our intent was to identify themes that might serve as the basis of a scoring rubric.

The Reading Hospital and Medical Center piloted a portfolio assessment tool 5 years before this study for its 26 internal medicine resident trainees and moved all portfolio content to an online format 2 and a half years ago. The stated goals of the portfolio are mentor-guided development of personal learning goals. Contents of the portfolio include all of the residents' assessment documents, including sections on scholarly activity (formal presentations, research, abstracts, and publications). Residents are instructed to choose 1 representative document (eg, PowerPoint slides or a handout) from any scholarly activity and reflect on their performance, with or without the help of prompted questions. They have the option to password protect any content from mentor review. Residents meet with faculty advisors to review the contents of their portfolio 3 times per year to develop performance improvement plans. Residents have an annual summative review of the portfolio by a (nonmentor) program director who generates an unstructured letter describing the positive qualities of the learner's work as well as a global score for reflective ability.

Residents receive 6 hours of workshop training on teaching during residency training. Topics include small-group teaching, large-group teaching, techniques for feedback, fostering learning climate, and application of learning theory to practice. Residents typically teach 5 to 15 conferences per year, including morning report, noon conference seminar, morbidity and mortality conference, and journal club. Residents receive immediate oral feedback as well as electronic feedback following each teaching session.

The Reading Hospital Academic Affairs Office approved this project as Quality Improvement, which was confirmed by the Reading Hospital Institutional Review Board. Eleven of the 12 residents' (92%) portfolios were selected because they had graduated, and because during the initiation of the electronic portfolio they had given consent for re-review of their scholarly activity, with 1 resident (8%) not able to be reached. Analyzed portfolio documents include those from the 11 learner portfolios only.

Three purposefully selected faculty members were instructed to review portfolio sections pertaining to scholarly activity for the 11 learner portfolios, searching for evidence of effective teaching and longitudinal growth in teaching skills. This faculty then participated in a group discussion of their findings. A qualitative analysis was performed of the 3 faculty members' independent written assessments of residents' structured portfolios as well as their oral comments made during a group discussion of their findings. Qualitative analysis was performed by the primary investigator (A.A.D.), who was not a portfolio assessor in the study.

The portfolio assessors were selected for their prior experience with mentoring using portfolios. They are experts in internal medicine, with 10, 21, and 28 years' experience in residency education. All 3 had participated as mentors for reviewing resident portfolios for 2 to 5 years, and 2 of the 3 (67%) had given seminars on teaching skills for faculty development and residents. One (33%) had been a summative portfolio reviewer for 3 years. One (33%) had completed a certificate program in medical education.

The 3 faculty assessors reviewed all available documents in the portfolio pertaining to scholarly activity during the 3 years of a resident's time in residency and were asked to independently write comments based on the uploaded documents, the residents' reflections, or the improvements seen when comparing multiple documents uploaded at different times. The faculty were asked to make comments in response to the following 2 questions, in light of reviewing the portfolios: “What evidence do you find of effective teaching?” and “What evidence do you find of responses to feedback about teaching?” Each faculty member independently reviewed 8 of 11 portfolios (73%), so that each portfolio was reviewed by 2 participants. Each faculty member recorded all evidence they obtained about both questions on each portfolio in a Microsoft (Redmond, WA) Word document. The 3 participants were then brought together for a group discussion to discuss their perspectives, during which they responded to focused questions about their written assessments. The group discussion was transcribed.

The constant comparative method of analysis associated with the grounded theory approach10 was used to analyze the individual faculty assessors' portfolio-assessment comments. The unit of analysis was the phrase or sentence reviewed to identify concepts or constructs. All concepts were grouped into distinct themes. The themes were identified by the primary author (A.A.D.) and were labeled with codes for facilitation of analysis. A transcription of the group discussion was similarly reviewed to identify illustrative quotes for each theme to provide a thick description. Quotes were corrected for grammar and changed (indicated by brackets) to improve readability and to make the author and/or resident anonymous. All 3 faculty members reviewed the themes, along with their own and the other faculty members' comments, and the transcript of the group discussion to confirm the analysis as a member check.

There were an average of 21.9 uploaded documents and associated reflections per resident for portfolio assessors to review (range, 7–44; median, 16). All 3 faculty participants were familiar with the work of the residents whose portfolios they reviewed. From analyzing the faculty members' individual assessment comments about these portfolios, 3 main themes were identified: (1) Application of Teaching Skills, (2) Presentation Skills, and (3) Insights as a Teacher.

The theme with the greatest number of comments was Application of Teaching Skills, which included 88 comments. Identification of a specific teaching technique in either the uploaded documents or the accompanying reflection was noted 35 times. Examples of specific techniques identified included use of breakout sessions, linking clinical findings to pathophysiology, and use of standardized test questions. An illustrative quotation is

The development of objectives as the framework for presentations was described 18 times. An illustrative quotation from the group discussion is

Other frequently described examples of teaching techniques are the framing of discussions with introductions and conclusions (described 16 times). One faculty member commented in the group discussion

The subtheme of Appropriateness for Audience included 15 comments. As 1 faculty member commented in the group discussion

The subtheme of Subject Mastery demonstrated in teaching included 4 comments. One illustrative quote from the group discussion was

A distinct theme identified focused on Presenting Skills, and 74 comments were identified. Organization of Slides and Presentation (22 comments), General Speaking Skills (20 comments), and Involving the Audience (16 comments) were frequently identified in faculty assessments of the portfolios. Illustrative quotes are

Faculty comments about Preparation (11 comments) and Time Management (5 comments) related to presentations were also classified under this theme. Illustrative quotes from the group discussion included

Thirty-five comments focused on issues of the feelings of residents about teaching and insights into their learners. Comments about residents' feelings as a teacher were identified 16 times. These included being “excited” or finding “enjoyment” as a teacher, “feeling inadequate” as a teacher, recognizing the “role of anxiety in their performance,” and describing a change in attitude toward teaching. One typical quote was

Comments classified in this theme also related to residents' self-assessments of the quality of their teaching (12 comments). An illustrative quote from the group discussion was

Comments on residents' insights about the learner were identified 6 times. An illustrative quote from the group discussion was

This theme, Insights as a Teacher, also included comments suggesting lack of insight as a teacher, such as the comment

This article reports a qualitative study of assessor observations of resident portfolios about teaching. Analysis identified 3 main themes that were confirmed by member checking with participating faculty assessors. Triangulating these findings with conceptual frameworks in the literature, we found that our observations were consistent with the 6 Glassick et al11 standards for scholarly work, namely, (1) clear goals, (2) adequate preparation, (3) appropriate methods, (4) significant results, (5) effective presentation, and (6) reflective critique. Of the 6 Glassick et al11 standards, “significant results” was the only 1 not represented among our themes and subthemes, confirming that portfolio assessment for teaching and presentations may capture most of the critical aspects of scholarship. Our findings were also consistent with important themes identified in the portfolios of professional teachers, namely, identification of specific learner progress over time, evidence of specific teaching skills, and self-reflection on their skills.8 

This analysis has several limitations. Most important, the findings may be unique to either our faculty assessors or the structure of our electronic portfolio itself. Given that residents were free to choose their best work as content and could restrict review by mentors, the portfolio may represent a positively skewed view of longitudinal performance. In addition, the portfolio was primarily used for formative feedback and was not a prerequisite to graduate, and our findings may not be generalizable to other portfolio systems that might be used for high-stakes decisions. Also, assessors' prior knowledge of residents' performances may have influenced their impressions of the portfolios, overstating what can be learned solely from a review of the portfolios. A strength of this work is the breadth of findings from the portfolio reviews that can be triangulated with the Glassick et al11 standards for scholarship and the review of portfolios of professional teachers.

Medical educators expend significant effort in improving the teaching skills of residents. This work demonstrates that portfolios can be used effectively to assess the longitudinal growth of teaching skills in 1 medical education program. The themes identified by our qualitative analysis might be used as a scoring rubric to frame an assessor's analysis of those portfolios. Whether using a rubric generated from these themes will improve the quality of mentor feedback and the longitudinal growth of mentees and whether this assessment method can be adapted to assess full-time medical teachers are avenues for future research.