The Surgical Learning and Instructional Portfolio: What Residents at a Single Institution Are Learning

The focus on graduate medical education outcomes has intensified during the past several years. The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties have developed guidelines to assess the 6 ACGME competencies.1 One of the proposed assessment tools is the portfolio.

Portfolios are described in the ACGME Toolbox of Assessment Methods as a method to assess competency in practice-based learning and improvement.1 Several studies have reported that learning portfolios can provide evidence of ongoing professional growth and development.2–6 The learning portfolio provides a vehicle for reflective practice; and reflection is an essential component of professional development.7,8 However, few articles describe the key components to creating and implementing a useful learning portfolio for graduate medical training.2,4–6 The literature now includes a Best Evidence Medical Education review of portfolios used in graduate medical education, which ultimately concludes that more research is necessary to demonstrate the utility of portfolios for assessment and learning.9 

In 2001, we began a learning portfolio program within our surgery residency program. The Surgical Learning and Instructional Portfolio (SLIP) was designed to provide a structure for residents to develop their individual portfolio of cases demonstrating evidence of self-reflection and practice-based learning. The program has been previously described,10 including an initial evaluation describing key lessons learned during the implementation of the portfolio program. The basic components of the SLIP include a monthly case chosen by the resident and reported using a template that includes case history, supporting diagnostic studies, differential diagnosis, final diagnosis with International Statistical Classification of Diseases (ICD-9) coding, management options, treatment used, 3 lessons learned, further elaboration and discussion of 1 of the lessons, and 2 articles related to the lessons.10 This template is used to guide learners as they develop a self-directed learning plan while providing documentation of representative lessons learned during their clinical training experience. Residents are encouraged to write about any lessons learned from their clinical experience. Furthermore, the SLIP program is reviewed for the residents each July as a tool to encourage and document practice-based learning.

As resources for educational initiatives are scarce, it is important to use resident and faculty time wisely. New and existing educational programs must be evaluated to determine their educational value and maximize educational impact. As part of an overall evaluation of our learning portfolio program, we elected to analyze the SLIP content in anticipation of quantifying what lessons our residents choose to write about.

The lessons identified through a thematic analysis could be used as a needs assessment for future educational initiatives within the residency program. To align teaching efforts to the needs and learning preferences of our learners, we must first identify the aspects of clinical encounters upon which residents of various levels choose to reflect. In this study, we wanted to identify what lessons were learned by residents as they progressed from novice to competent and proficient surgeons-in-training. To our knowledge, no qualitative study of the lessons formulated in learning portfolios has been published.

This study is a qualitative analysis of descriptions of lessons learned provided in case-based learning portfolio entries of surgery residents (postgraduate year-1 [PGY-1] through PGY-5) at a single academic institution in the Midwest. The Institutional Review Board approval at Medical College of Wisconsin deemed this study exempt because the intent of the study was to evaluate the educational program and the study used deidentified data, so individual resident consent was not obtained. All resident-authored SLIP entries from July 2008 to 2009 (n = 420) were deidentified and randomized using a random number generator. We conducted a thematic content analysis of the first 50 random portfolio entries and their associated 3 lessons learned (n = 150) to identify subject matter chosen. We chose to review only 50 entries because of the large volume of analyzable data and the authors' expectation that theme saturation would occur within this volume of entries.

Each SLIP was analyzed individually to identify the key lessons learned. Two independent raters analyzed the 3 lessons learned portion of the portfolio entries and identified themes and subthemes using the constant comparative method associated with grounded theory.11,12 The raters (T.P.W. and T.R.M.) were trained in qualitative analysis techniques through previous research experience and graduate coursework (T.P.W.) as well as through required reading and practice analysis (T.R.M.) with debriefing to assure a structured approach to coding. As they emerged, themes were identified with words or phrases as units of analysis. After all 150 lessons learned were analyzed, the raters met and conferred to compare, contrast, and negotiate the identified themes. Interrater reliability was confirmed as consensus on the coding schema was developed through an iterative process of reanalysis of all comments and themes. Comments fitting within each theme and subtheme were tabulated and presented as the number of times each lesson learned was indicated by residents and the number of residents citing lessons learned by theme. Exemplar quotes were selected and presented by the authors as illustrative examples of the lessons learned.

The 50-entry, randomized, blinded sample included a similar number of entries across all years of training relative to the total number of SLIP entries per training year (table 1). The number of lessons learned by year of training is presented in table 2. Finally, the tally of resident learning by lesson theme is shown in the figure.

The collaborative coding process between the 2 raters resulted in theme saturation after the identification of 7 themes and their subthemes. Themes with their subthemes included (1) complications (subthemes: risk, risk and decision making, avoidance, identification, management), (2) disease epidemiology (subthemes: incidence, definitions, classification systems, prognosis), (3) disease presentation (subthemes: clinical findings, disease workup, diagnostic tools), (4) surgical management of disease (subtheme: indications), (5) medical management of disease, (6) operative techniques, and (7) pathophysiology. Each theme will be discussed in order from most to least frequently reported, along with representative comments.

Complications comprised the most frequent theme of lessons learned by the residents and the most frequently cited lessons learned by PGY-4 and PGY-5 residents. Twenty-eight residents wrote about complications, with 38 lessons learned. Subthemes were identified as complication risk (10 lessons), risk and decision-making (9 lessons), avoidance (9 lessons), identification (2 lessons), and management (8 lessons).

The complications described were varied and included a broad spectrum of common complications. Examples of lessons learned (with subthemes) related to complications include “complications of balloon angioplasty” (complication risk), “inferior vena cava filter placement is not without risk” (risk and decision making), “risks and benefits of abdominal aortic aneurysm repair” (risk and decision making), “poor communication can directly affect patient care” (avoidance), “symptoms of anastomotic bowel leak” (identification), “treatment of bowel anastomotic leak” (management), and “how to treat hoarseness after thyroidectomy” (management).

The next most commonly identified theme focused on various aspects of disease epidemiology. Twenty-five residents wrote 30 lessons learned describing disease incidence (10 lessons), definitions (6 lessons), classification systems (6 lessons), and prognosis (8 lessons). These themes were commonly associated with rarely encountered surgical disorders. Examples of entries (with subthemes) include “incidence of gastric duplication cysts” (disease incidence), “definition of Amyand hernia” (definitions), “types of biliary atresia” (classification systems), “classification of congenital vascular anomalies” (classification systems), and “prognosis of gastrointestinal stromal tumor” (prognosis).

Twenty-five residents wrote about disease presentation as a lesson learned. Twenty-six lessons learned were included in this theme. Disease presentation was the most frequent theme written by PGY-1 residents. There were 3 subthemes that could be identified: disease workup (11 lessons), clinical findings (9 lessons), and diagnostic tools (6 lessons). Disease workup included the general approach and steps used to evaluate a patient with such problems as “patients with soft signs versus hard signs of vascular injury,” “workup of gastrointestinal bleeding,” “diagnosis of acute respiratory distress syndrome,” and “abdominal compartment syndrome.” Residents also described the physical exam and specific clinical findings associated with 9 different disease processes. Examples of clinical findings included “proper way to examine a patient with a sports hernia,” and “trust your physical exam.” Finally, residents described the diagnostic tools used for evaluation of certain diseases, such as “computed tomography has a sensitivity and specificity of 80% for blunt pancreatic trauma,” “small-bowel obstruction and x-ray versus computed tomographic scan findings,” and “ankle brachial index for rest pain and tissue loss.”

Residents frequently wrote about the overall surgical management of various disease processes. Twenty-five residents wrote 25 lessons learned under the theme of surgical management of diseases. The lessons learned were focused on general aspects of decision making regarding operative therapy of various diseases. A clear subtheme with 6 entries was the identification of indications for surgical intervention.

Residents chose to write about a diverse set of disease processes, such as “signs for immediate operative intervention of stab injury to the abdomen,” “different surgical approaches to perforated peptic ulcer disease,” “operative strategy for pseudomyxoma peritonei,” and “surgical treatment of inflammatory bowel disease.” No specific disease process was identifiable as a theme.

Nineteen residents wrote about the overall medical management of disease processes. They reported 20 lessons learned, with a focus on how to manage patients without operations. Eleven of these lessons learned were from PGY-1 residents. Medical management of common and uncommon disease processes was reported with similar frequency. Examples of lessons learned included “nonoperative treatment of basal cell carcinoma,” “proton pump inhibitors work wonders,” “management of flail chest,” and “most hemodynamically stable patients with retroperitoneal hemorrhage can be managed nonoperatively.” Uncommon diagnoses to the practicing surgeon requiring medical management were listed and included “Bartter syndrome,” “role of antibiotics for cholangitis prophylaxis in biliary atresia,” “treatment of hepatorenal syndrome,” and “treatment of endocarditis.”

Operative techniques were included as lessons learned by 14 residents. The 15 lessons learned were related nearly exclusively to complex or uncommon surgical procedures. Specific lessons learned related to operative techniques included “how to do a laparoscopic common bile duct exploration,” “for duodenal injuries primary repair should suffice,” “operative exposure of zone I thoracic outlet injuries,” “what a Kasai procedure is,” “recommended repair of an Amyand hernia,” and “you fix the aneurysm based on the anatomy.”

Eight residents wrote 8 entries describing learning about the pathophysiology of specific diseases. These descriptions included common diseases, such as “review basic pathophysiology of inflammatory bowel disease,” “pathophysiology of flail chest,” and “small-bowel obstruction is due to adhesive disease in 75% of cases.” However, some less-common diseases were described, including “gastric duplication cysts have been reported to develop into adenocarcinoma,” “pathophysiology of Barter syndrome,” and “mechanism of renal failure caused by hepatorenal syndrome.”

The results of this qualitative analysis of residents' lessons learned in a case-based, surgical learning portfolio provides insight into the perceived needs and preferences of the residents when involved in self-directed learning and reflection. The results of this analysis could serve as a guide for content choice when teaching residents of various levels of experience and knowledge.

Our surgery training program has a rigorous, didactic curriculum with separate, structured learning activities for junior and senior residents. Most sessions are case-based discussions with minimal lecture time. The results of our analysis of learner-centered, lesson-learned themes could be used to guide the instructors in their case-based discussions with the residents. For example, during senior resident sessions, faculty should focus primarily on complications as the basis to begin discussions regarding learning points. It may be that as residents progress through their training, they want to focus on “how to avoid or get out of trouble” when learning about clinical topics.

Residents in our surgical residency program also focus on the overall management of common disease processes, with frequent reference to complications as a source of educational focus. Similar to classic surgical teaching using morbidity and mortality conferences, the residents commonly reflect on complications to drive their learning. Furthermore, there appears to be a progression of topics chosen based on training level. Senior residents are more likely to use reflection about complications as a learning tool. Perhaps this variable could be used as a marker of learner development during their training. Using the Dreyfus model of skill acquisition,13 perhaps competent or proficient residents can be identified by their thematic choices when describing lessons learned in their portfolio. Furthermore, the topics chosen by the residents may be reflecting their sophistication and stage of learning within the Kolb learning cycle. More data need to be gathered to understand this progression and to identify whether most or all individuals make this transition during their training.

The analysis likewise highlights the importance that residents place on the understanding of the overall disease processes, exclusive of specific operative techniques. It is not clear why the residents focus on these areas, but it may be that they feel a portfolio is most conducive to these learning points or perhaps these are the areas that are not taught adequately in other venues.

There are other implications of our findings from the qualitative analysis. Although learning portfolios should remain primarily self-directed and unscripted, as an introduction to the process or during initial implementation, the novice learners might be encouraged to focus on the topics in the context that our residents have found helpful. In other words, junior residents should be encouraged to focus on broad disease process diagnosis and treatment, whereas senior residents would be prompted to use complications as a focus for their reflection.

Several limitations exist with this study. First, this is a study of a single site and single surgical program using a specific learning portfolio. The template used for the case-based discussion provides some guidance for content, but it does allow for significant variation in topic chosen. Therefore, we do believe this is a representative example of self-directed lessons learned. A second aspect that could alter the cases chosen, and thus the lessons learned, is that all residents regularly present complications at our morbidity and mortality conference. As the residents have studied these cases and performed background reading, they may choose to use these cases preferentially as SLIP entries to diminish the burden of the SLIP assignment. This may be an explanation for why the senior residents refer to complications more frequently in their SLIP entries. However, junior residents also present each of these complications at morbidity and mortality conference; therefore, preparation for this conference does not necessarily explain the different choices in lessons learned. Finally, although we reached theme saturation after analyzing 50 SLIPs, it is certainly possible that other significant themes would have been identified had we reviewed all SLIP entries.

Learning portfolios are becoming more widely used in medical education, but little is known about their efficacy for broad implementation. The term portfolio encompasses a wide spectrum of learning and organizational tools in the literature and in practice. More data are needed to demonstrate changes in knowledge, behaviors, and outcomes as affected by an individual's learning portfolio. The current study has given us a foundation to understand what residents are choosing to reflect on as lessons learned, but we would now like to understand why they make these choices. To understand why these choices are made, further qualitative studies focusing directly on this question would be beneficial.