Evaluating Professionalism and Interpersonal and Communication Skills: Implementing a 360-Degree Evaluation Instrument in an Anesthesiology Residency Program

The Accreditation Council for Graduate Medical Education (ACGME) in conjunction with the American Board of Medical Specialties developed a Toolbox of Assessment Methods to help assess the 6 general ACGME competencies.1 These competencies are patient care, medical knowledge, professionalism, interpersonal and communication skills, practice-based learning, and systems-based practice. Graduate medical education programs must incorporate the general competencies into their curriculum and have valid measurement tools to assess them. The ACGME has endorsed the use of 360-degree ratings to evaluate the competency of professionalism.

The use of 360-degree assessment has been advocated recently as a means of gaining additional feedback on resident physician performance from sources other than attending physicians.2 This type of assessment uses raters from a variety of groups to interact with trainees. Although the 360-degree feedback system has been used extensively by business organizations,3 it has not been used extensively in graduate medical education in anesthesiology.

Anesthesiologists provide the majority of postanesthetic care for patients4 after procedures under general or regional anesthesia and intravenous sedation. Anesthesiologists must possess a wide range of knowledge and skills to evaluate, treat, and provide recommendations to improve postanesthetic quality of life,5 factors that are aimed to reduce postoperative adverse events and provide optimal patient safety. The postanesthesia care unit (PACU) rotation is a requirement of the American Board of Anesthesiology and the ACGME Review Committee for Anesthesiology. The purpose of the PACU rotation is to provide the resident with the knowledge and technical skills needed to perform the routine PACU and perioperative care expected of a board-eligible anesthesiologist. The PACU residents work cooperatively with other health professionals including PACU nurses, secretarial staff, nurse aides, and medical technicians. For anesthesiology residents during the PACU rotation, such an approach is especially warranted given the traditional emphasis on teamwork and collaborative patient care.

In this study, we tested the 360-degree assessment for anesthesiology residents in a PACU rotation and determined whether using this evaluation instrument would be useful, feasible, and reliable to evaluate the professionalism and interpersonal and communication skills for a PACU rotation.

Our residency program has 15 residents per academic year and the PACU rotation of the residency program is a mandatory 2-week rotation occurring in the main hospital. A total of 15 residents were scheduled to rotate through the PACU during the 1-year study period. Twelve residents were postgraduate second-year residents and 3 were postgraduate third-year residents. The PACU used for this study has 52 beds, 40 full-time nurses, 10 part-time nurses, and 14 allied health professional staff including 6 secretarial staff, 4 nurse aides, and 4 medical technicians.

Professionalism and interpersonal and communication skills in anesthesiology have some unique aspects. It is difficult to design instruments for objective evaluation of these competencies. With institutional review board approval and after an extensive literature review concerning ACGME general competencies and resident physician professionalism, we developed a questionnaire that focused on professionalism and interpersonal and communication skills that should be addressed in the PACU rotation in anesthesiology.6–9 We thought the important components of the evaluation were clarity of communication, rapport with patients and nonphysician personnel, listening skills, management skills, and respect for others (table 1). We created a survey prior to implementing the evaluation. This survey allowed us to retrieve input and opinions from our residents and PACU nurses concerning this new format for general competency assessment of resident physician training. Before each resident rotated through the PACU, the resident was informed of the 360-degree projected evaluation and was asked to participate in this new evaluation process.

The rater categories included PACU nurses, allied health professional staff (secretarial staff, nurse aides, and medical technicians), and those who had direct observational knowledge of the residents' performance during their PACU rotation. The raters were oriented and instructed in the use of the evaluation form by one of the authors before implementation. Unfortunately, we were not able to include the PACU patients in this study because of residual anesthetic effects and ongoing pain control with opioids.

Each item was measured on a 9-point Likert scale (1, unsatisfactory performance, to 9, outstanding performance). Rating forms were distributed to raters after the completion of the PACU rotation. Completed forms were returned to the residency program within a 1-week period. Raters were encouraged to give comments. This was a single-blind study because the raters were identified by category only. Feasibility was evaluated based on survey feedback from residents and PACU nurses, compliance with the data collection, the time and training required to implement the instrument, and the potential for the behavioral change in residents.

Anesthesiology residents rotating through the PACU were traditionally evaluated only once by the faculty at the end of the rotation with a global rating form. The global rating form includes the 6 general competencies with a rating scale identical to the 360-degree form. We used the data of professionalism and interpersonal and communication skills items from the global rating form as the “gold standard” to compare the data from the 360-degree forms. We used the comparison to measure validity. Intraclass correlation coefficients were calculated to measure the reliability of ratings within each category of raters by the Pearson correlation coefficient. SPSS software (version 14.0, SPSS, Chicago, Illinois) was used for statistical analyses. Descriptive statistics were used to compare the differences between the rater categories. For all tests, P < .05 was considered significant.

Fifteen residents were evaluated during the 1-year study period. A total of 429 forms were returned. The average number of evaluations per resident was 22 (range, 16–28). The response rate was 88%. Average scores across the items were similar, ranging from 7.28 to 8.80. The residents were ranked highest for “Availability” (no. 7) and lowest for “Management skill (no. 5). The average rating score across all items combined was high (8.23).

table 2 shows the mean scores for each resident for each rater category with the intraclass coefficients of correlation. The average evaluations were similar between the rater groups, except for the data from the secretarial staff. Although there were no statistical differences, the average mean rating across all items from PACU nurses was higher (8.34) than from the secretarial staff (7.99, P > .08). We can see a trend across the different categories of raters: each resident was rated similarly, high or low. The residents ranked high by global ratings were also ranked high by the 4 categories of raters.

We compared the correlation coefficients of the global ratings versus the 360-degree ratings for each of the 3 groups of the raters. The intraclass coefficients of correlations were 0.8719, 0.7860, 0.8268, and 0.8575 for PACU nurses and allied health professional staff. This indicated the reliability of the score within each category of evaluation. This measure calculates the consistency of scores among the different raters: the higher the consistency, the higher the reliability of the scores. We noted that our data showed excellent correlation among the different rater groups (interclass correlation coefficients ranging from 0.87 to 0.77).

The PACU nurses were enthusiastic about this instrument. From the results of the survey, they all agreed that the PACU would be a good place to start using the 360-degree resident evaluation tool. They did not think that the process would increase their work. All categories of evaluators returned the completed form promptly, indicating their support for the process. From the survey, the residents showed a positive reaction to this instrument in general, but only a few residents had heard about the 360-degree evaluation before.

Using 360-degree instrument to evaluate resident competency is not very common practice in graduate medical education. To our knowledge, this study was the first report to assess anesthesiology residents' professionalism and interpersonal and communication skills during the PACU rotation. In an era of competency-based education, it is important for residency programs to appropriately assess resident knowledge and skills, as well as professional attitudes and behaviors.10 Since the ACGME and the American Board of Medical Specialties collaborated to develop a Toolbox of Assessment Methods, which includes the use of a 360-degree evaluation instrument,1 resident evaluation by peers, support staff, and faculty has been studied in other specialties including radiology, physical medicine and rehabilitation, and obstetrics-gynecology.11–13 

The 360-degree evaluation tool may not work for every rotation in anesthesiology. The PACU residents work in an environment where faculty may not always be present to evaluate the residents' interactions with the PACU nurses, allied health professional staff, and patients. Therefore, these individuals may provide additional information about residents' professionalism and interpersonal and communication skills. Furthermore, to successfully implement this instrument, both raters and those being rated must understand and accept the process. The raters must be willing to give fair and honest evaluations, and those being rated must respect the confidentiality of this process. We were able to introduce and implement the 360-degree evaluation instrument into our PACU rotation evaluation without any resistance from the participants; we also obtained high evaluation rates. All categories of evaluators returned completed forms promptly, indicating their support for the process. Our study demonstrated that the 360-degree assessment tool is feasible for evaluating residents' professionalism.11,12,14,15 

Our data showed that the average rating score across all items combined was high (8.23). One possible explanation is that the residents were informed of the evaluation before starting the rotation. This process may lead to a better performance and an enhanced productivity of our residents. Or the measuring instrument may not be sensitive enough to identify lower scoring behaviors, and perhaps there was rating bias by the raters (avoiding lower scores). Many valuable comments were documented on the evaluation forms. After the residents were provided the results of the evaluation, the majority of the residents felt positive about the experience, and they thought that this information helped them to improve their communication with patients and nonphysician personnel. Using this instrument had a positive impact on our residents' behavior in professionalism and interpersonal and communication skills during their PACU rotation. Other studies have shown that the 360-degree assessment of professionalism is useful for the evaluation of residents.14 In a pilot study of a 360-degree assessment instrument for a physical medicine and rehabilitation residency program, the tool was useful for providing formative feedback to residents regarding professionalism and performance.12 Our findings support that the 360-degree assessment tool can be useful in an anesthesiology residency program PACU rotation.

Previous studies comparing residents' evaluations by different professionals calculated intraclass correlations coefficients as a measure of ratings reliability within each group of raters.13,16 Our study showed that the reliability was high among all the groups of raters. The PACU nurses had the highest reliability. This suggests that the PACU nurse ratings of resident performance provided the most consistent information. We did not include the residents' self-evaluation in our study because the previous studies showed that self-evaluations do not meaningfully correlate with assessments by external evaluators.17,18 

This study has several limitations. First, even though our study covered a broad range of behaviors that were deemed important for the PACU rotation, some potentially important competencies might not have been included. Second, a small sample size, limited by our resident numbers, contributed to a general difficulty with further statistical analysis. In this study, our residents' performance was rated more frequently by female raters than by male raters because the majority of PACU professionals were women. The majority of the residents in the study were men. Could this have any potential gender bias? Or could this be the reason for the uniform high evaluation scores? Finally, there was significant time and effort involved in distributing, collecting, and ensuring confidentiality of the data. Perhaps the traditional faculty evaluation is much more time-efficient and provides the same amount of useful information. We hope that electronic data collection will solve this issue in the future.

In summary, a 360-degree assessment tool might be a feasible and reliable measurement of the residents' professionalism and interpersonal and communication skills during the PACU rotation in an anesthesiology residency program. In the anesthesiology literature, very little has been written regarding professionalism and how it should be taught. We hope that this study might help to stimulate the development of implementing assessment tools like this and to establish the appropriate evaluation methods for the competencies.