Development of and Preliminary Validity Evidence for the EFeCT Feedback Scoring Tool

Written feedback comments on assessment tools can inform both learning and assessment. Documentation of verbal feedback shared with a resident is valuable for subsequent reflection,^1,2  a key component of self-regulated learning.³  Documented feedback on assessment forms is often part of the constellation of evidence that is considered by competence committees or other groups who make high-stakes or summative progress decisions.⁴  In order for written feedback to be meaningful for both learning and decision-making, it must be of high quality.⁵  Given the importance of feedback to learning and assessment, training programs strive to engage in continuous quality improvement (CQI) with teachers to improve the feedback captured on assessment forms.

While some tools have been published to evaluate the quality of documented feedback, their practical value to training programs for CQI can be improved. For example, the Completed Clinical Evaluation Report Rating (CCERR) is intended as a tool for evaluating the quality of end-of-rotation high-stakes assessments.⁶  The CCERR serves well for its intended purpose but is cumbersome to use for assessments that include brief captures of formative feedback. Two recently published tools, the Quality of Assessment of Learning (QuAL) score⁷  and the Quality Improvement Instrument (QII),⁸  also have value, but include assumptions about how feedback will be structured, which limits their generalizability. The QuAL score emphasizes the description of the resident performance and requires that feedback include a suggestion for improvement. In the case of the QII, there is an expectation that feedback will include at least one strength and at least one weakness.

These tools all share 2 common elements that may affect how they are used for CQI directed at improving the content of documented feedback. First, all 3 tools incorporate specific assumptions of the structure of feedback, limiting their use to narratives that adhere to the assumed format. More significantly, while all 3 tools are well-suited for use by clinical educators, they may not lend themselves well for use by non-clinicians. This limits the value of these tools for CQI, as the task cannot be delegated to administrative or support staff.

The purpose of this project was to address the need for an evidence-based tool that can be used accurately by either clinical educators or support staff to evaluate the quality of feedback captured on any type of assessment form, regardless of structure. We describe the development of the Evaluation of Feedback Captured Tool (EFeCT) and present preliminary validity evidence.

Development of the EFeCT took place in a Canadian family medicine residency program and was completed in 2 iterative phases between 2013 and 2020. Participant details are included in the following phase descriptions. Validity evidence was collected and examined concurrently with both phases using the elements of Messick's unified concept of validity as a guide.^9,10 

Phase 1 used a narrative review methodology¹¹  with the identified purpose of mapping out common features of quality feedback as reported in original research. The findings from this literature review provided the foundation for our feedback quality evaluation tool. A literature search was carried out by a librarian using PubMed, MEDLINE, ERIC, CINAHL, Embase, Scopus, Web of Science, and PsycInfo databases from 1900 through 2013. Two members of the research team (S.S., S.R.) conducted primary screening of titles and abstracts. Inclusion criteria were descriptions of characteristics of good feedback and/or descriptions of best practices in sharing formative feedback as determined by the authors of the study. Exclusion criteria included the following: lack of descriptions outlined in inclusion criteria or referring to other studies for those descriptions rather than the study authors themselves stating their own descriptions; studies that focused on interpersonal aspects of the feedback relationship; and all studies where feedback was not the primary topic of the study.

In Phase 2, three members of the research team (S.S., S.R., S.A.S.) reviewed the full-text articles identified in Phase 1 and extracted a list of the characteristics of good formative feedback identified by the authors of the article. Two members of the research team (S.R., M.D.) discussed the lists to develop agreement about the essential elements of good formative feedback. This list was reduced using a consensus development panel methodology.^12,13  The panel included 2 master teachers¹⁴  (M.D., D.N.) who engage in continuous self-improvement, empower students to be independent learners, and have a mindset of personal accountability as professional educators¹⁵ ; a medical student (S.S.); one current (S.A.S.) and one former (P.H.) residency program director; and a researcher with extensive graduate training and experience in education and assessment (S.R.). The panelists independently reviewed the list of characteristics of good feedback from Phase 1, and then met in person to discuss their individual perspectives about which elements from the list were essential for the evaluation of documented feedback. This resulted in a final consensus list of key characteristics of good documented feedback. The consensus decision of the panel was to put the items into question form to make the tool more intuitive to use, resulting in EFeCT.

Validity evidence gathering was based on Messick's unified concept of validity⁹ ; essentially, is there evidence to support the proposed interpretation of the score generated by the instrument? Validity evidence was collected concurrently with development and piloting of the tool for the following facets of Messick's unified concept: content, substantive, generalizability, and consequential validity.

The tool was initially piloted by 4 members of the research team (D.H., S.R., S.A.S., D.N.), who used the tool to individually score the documented feedback on a sample set of 100 formative narrative assessment forms (FieldNotes)¹⁶  randomly selected from our residency program's online assessment portfolio. FieldNotes is a workplace-based assessment tool used as part of our residency program's programmatic assessment framework.¹⁷  Each FieldNote includes a brief description of what was observed and a free-text documentation of the feedback that was shared with the resident following the observation. All FieldNotes were completed about family medicine residents across the 2 years of residency, came from a variety of supervisors, and included highly variable formats for the documented feedback. This was an intentional sampling approach as it closely replicates the situations where the tool is intended to be used. Interrater reliability was calculated using Ebel's intraclass correlation (ICC) formula¹⁸  after the first 50 samples were scored. All score discrepancies were discussed. These discussions were used to refine the instructions before the team members scored the second 50 samples. Interrater reliability was calculated again. Following this piloting and refinement, 3 researchers (S.R., S.A.S., D.H.) scored 2 further sample sets of 100 FieldNotes, so that 3 full sample sets were available.

Evidence for content and substantive validity, as well as generalizability, was collected through multiple sessions with different groups of raters, using the 3 sample sets of 100 FieldNotes described above. Each session followed the same format: calibration, scoring, and debrief. Calibration entailed a research team member describing the tool and reviewing the instructions. Next, the participants each scored 5 samples of feedback (same samples for all participants). The facilitator then discussed the scores, with an emphasis on large discrepancies (2+ points) if any were found. The scores were reviewed in reference to the tool instructions, and the facilitator guided the group to consensus based on the instructions. The facilitator also answered any questions related to interpretation of the instructions. For scoring, each session participant independently scored the 100 samples of feedback in the set assigned to their group and gave the completed scores to the facilitator. In debrief, the facilitator gathered participants' reflections about the tool. The facilitator asked 3 questions: (1) What was your opinion of the ease of use and utility of the tool? (2) Did the tool reflect the way that you think about good feedback? (3) Do you have any other comments about the tool? Participants were recruited from 4 groups: clinical educators, health professions students, non–health professions students, and support staff. Five sessions were held over 3 years, and each session included a mix of participants from the 4 groups (3–6 raters/session). The same set of 100 samples was used for the first 2 scoring sessions, the second sample set was used for the next 2 scoring sessions, and the third sample set was used for the last session. All participants in a specific session scored the same 100 samples, to allow for analysis of interrater reliability.

Further validity evidence was collected by using the tool for research and CQI at 2 residency teaching sites between 2015 and 2019. Evidence for generalizability was collected using the EFeCT to score real-world documented feedback generated as part of the programmatic assessment framework. Findings from the research conducted using the EFeCT were shared back to the preceptors at the teaching sites either as a simple score report or verbally as part of CQI directed at improving teaching and assessment, and the tool itself was shared to preceptors on request. At these same 2 sites, evidence for consequential validity was collected by looking for “washback,” or whether introducing the scoring tool is associated with changes in quality of documented feedback,¹⁰  by comparing mean EFeCT scores in the first and last years using independent samples t tests.

Additional evidence for substantive validity was collected by repeating the literature search process from Phase 1 after the EFeCT had been in use in our program for 4 years, to ensure that the tool still reflected published literature about feedback theory.

Ethics approval was obtained from our institution's Human Research Ethics Board.

The initial literature review returned 5101 records; 1307 remained after duplicates were removed. After primary screening, 104 full-text articles were retrieved for secondary screening, which resulted in a final list of 89 articles included for data extraction (see online supplementary data).

The initial list of characteristics of good feedback extracted from those articles is shown in the left column of Table 1. These characteristics were reviewed by the consensus development panel in Phase 2. By consensus, the panel decided to remove all elements from the original consensus list (left column, Table 1) that pertained to the feedback process (the actual act of sharing of feedback), rather than the documentation. Where concepts were overlapping, statements were combined. The final list of 5 key components of good feedback is shown in the right column of Table 1.

The final version of the EFeCT is shown in Figure 1. Each element of good formative feedback is phrased as a simple question. Brief descriptions of each element allow raters to decide if the element is present or absent. For each element present in an example of documented feedback, a score of 1 is given, up to a maximum score of 5. The consensus panel suggested that there should be a way to indicate which elements were present when scoring a feedback sample, so that it was clear which elements made up a final score. This resulted in a “criterion” column, with criterion A being the requirement for written feedback to be present (so that there could be a score of 0 for those instances where a sample did not have written feedback). Simple instructions clarifying how to use the tool are provided in Figure 1. An exemplar set of sample feedback that has been scored using the EFeCT is presented in Figure 2.

Validity evidence is summarized in Table 2, where sources of evidence for each facet are described. For the piloting and refinement session, interrater reliability improved after the instructions were refined between the scoring rounds (first 50 samples: n=4, ICC=0.82; second 50 samples: n=4, ICC=0.94). Specific to the generalizability evidence, interrater reliability for the final version of the rating tool was found to be excellent for all sessions (Session 1: n=3, ICC=0.94; Session 2: n=6, ICC=0.90; Session 3: n=5, ICC=0.91; Session 4: n=6, ICC=0.89; Session 5: n=6, ICC=0.92),¹⁴  regardless of who participated in the session (see online supplementary data) or which of the 3 sample sets was scored.

The evidence for consequential validity is shown in Figure 3, where the mean of EFeCT scores demonstrate a general upward trend and significant increase between the first and last years in both teaching sites.

No new information was identified related to specific components of the quality of documented/written feedback when we searched literature from 2015 to 2020. All new information about feedback that was found discussed best practices in holding feedback conversations with learners.

The EFeCT is a rigorously developed feedback quality scoring tool that was intentionally constructed to be used by anyone—support staff or student research assistants as well as experienced clinical educators—with no compromise in the integrity or reliability of the scores produced. The collection of generalizability validity evidence for the EFeCT specifically included a diverse population of raters. While the instructions are simple (Figure 1), we consistently found a uniform application of the scoring tool, regardless of the experience or background of the raters. Additionally, the EFeCT was designed to capture the presence of specific evidence-based elements of high-quality feedback, and to do so without requiring that the feedback be in a specific structure or format. Both of these aspects of the EFeCT differentiate it from existing tools such as the CCERR,³  the QuAL score,⁴  and the QII,⁵  and make it a useful addition to these tools by offering a way to make feedback evaluation feasible and practical for programs.

Another advantage of the EFeCT is the clear and explicit language about the feedback element of “How was it done well or how can it be improved?” This element captures the importance of reinforcing a positive performance not just by saying “good job,” but by articulating what it was about a task or action that went well. This helps learners to know specifically what it is that they need to keep doing, which is just as important as providing specific information on what needs to be done differently for learners who need to improve or rectify a gap. This element is also important if the feedback is being considered by a competence committee making summative decisions, as this is the type of information that gives context to ratings or scores on assessment forms.^19,20 

Interestingly, the same information is valuable for both learning and assessment. The criteria on the EFeCT are derived from published research about characteristics of feedback that positively contribute to learning. This research often addresses assessment for learning²¹  and the role of formative feedback in self-regulated learning.^3,22–24  Much of this same literature has also informed assessment design in competency-based medical education (CBME) programs,^25–28  especially the incorporation of both formative and summative tools in programmatic assessment.^4,29  Summative assessment decisions in CBME are thus based on a large amount of data, including formative assessment forms containing feedback. There is clear value to including this information in summative decision-making,^5,19,20  but there is also inherent danger in doing so: the dual purposing of feedback for both formative and summative assessments may be detrimental to resident learning.³⁰ 

Regardless of this uneasy truce between the learning and assessment uses of feedback, the fact remains that it is the quality of the feedback captured on assessment forms that determines how meaningful the feedback will be for either purpose.⁵  It is therefore important for programs to have ways to monitor the quality of documented feedback beyond faculty or resident personal perceptions of good feedback, an approach that has been found to be both complex and problematic.^31,32 

The EFeCT is primarily a tool to be used to evaluate written, one-way, documented feedback captured on assessment forms. As such, it does not include measures of the process of sharing effective feedback through verbal conversations between educators and learners, and thus does not capture the environmental, contextual, and relationship components that are key features of sharing feedback in the workplace.²⁸  A potential future direction for research would be to examine whether use of the EFeCT in faculty development aimed at improving the quality of captured feedback will also affect the quality of feedback conversations between teachers and learners. An additional limitation is that our validity evidence was captured at one institution. The EFeCT has been adopted at other institutions, and we are planning for formal gathering of validity evidence in the future.

Additional future research with the EFeCT is planned to compare ratings between the EFeCT, the QUaL Score, and the QII on identical sample sets of feedback. Future research could also include further exploring the generalizability of the EFeCT by using it to score feedback captured on forms in other settings, such as objective structured clinical examinations.

There is preliminary validity evidence for the EFeCT as a tool for scoring the quality of narrative feedback captured on assessment forms. Interrater reliability evidence showed consistent scoring by all raters who used the EFeCT, regardless of their level of expertise in medical education or experience as clinical educators.

The authors wish to thank Amy Hegstrom, Mike Donoff, Paul Humphries, Adam Kulaga, Terra Manca, and Shirley Schipper for their contributions in helping to prepare this manuscript, and Rob Woods and Andrea Gingerich for their constructive reviews of earlier drafts.