Evidence-Based Selection, Development, and Testing of a Tailored Strategy to Improve Standardized Structured Reporting in Pathology: A Multicenter Study

Our study consisted of the following 3 steps: (1) selection of digital implementation strategy elements, (2) development, and (3) effectiveness and feasibility tests. In this section, the methodology used in each step is described.

In the Netherlands, SSR templates have been developed by the nationwide network and registry of histopathology and cytopathology, also known as the PALGA foundation (PALGA; Pathologisch-Anatomisch Geautomatiseerd Landelijk Archief) since 2009 (Table 1). In this article, we refer to the PALGA foundation as “the professional organization for developing standardized structured reporting templates.” Currently, there are 30 templates available.³⁷  The running software application (PALGA Protocol Modules) is linked within the laboratory information system. Our study focused on SSR templates used for urologic, gynecologic, and gastroenterologic oncologic diagnoses to limit the scope of the study. These templates cover most oncologic diagnoses, and major variation has been shown in the use of these templates among laboratories. The use of templates in 2019 varied between 41% and 70% for urologic tumor types, between 23% and 65% for gynecologic tumor types, and between 31% and 84% for gastroenterologic tumor types. The use of the SSR templates per laboratory are shown in Figure 1, A through C. Templates for colorectal biopsies and cervical cytology are mandatory for the national screening program and are therefore excluded. Detailed information about the included templates is provided in Supplemental Table 1 (see supplemental digital content containing supplemental data and 4 tables at https://meridian.allenpress.com/aplm in the December 2022 table of contents).

The selection of the implementation strategy elements included in the toolbox was based on an extensive analysis of influencing factors and determinants among multiple stakeholders of SSR implementation conducted in previous studies.^32,33  To further select implementation strategy elements to improve SSR implementation, we performed a literature search, organized a consensus meeting with a nationwide expert panel, and organized consensus meetings with the project team. The nationwide expert panel included representatives of the most important stakeholders regarding SSR implementation, particularly pathologists, oncology-treating clinicians, patient representatives, and database experts. Of each stakeholder type, 1 expert was already involved as an advisor during the preparation of the study application. The snowball method was applied to complement this group of advisors with additional experts. Ultimately, the nationwide expert panel consisted of 3 pathologists, 3 oncology-treating clinicians, 3 patient representatives (all related to the 3 domains of this study: oncologic urology, gynecology, and gastroenterology), 2 PALGA representatives, 2 Netherlands Comprehensive Cancer Organization representatives, 1 PALGA liaison, and 1 implementation expert. The project team consisted of a pathologist, PALGA representative, implementation expert, and researcher.

First, after analysis of the barriers and facilitators, the literature was consulted on useful theoretical constructs, associated implementation strategy elements, and evidence for these strategy elements. The latter specifically focused on the literature about implementation strategy elements regarding (1) evidence-based guideline implementation in general, (2) information technology solutions in health care, and (3) SSR implementation. The search was limited to articles in English.

Second, a consensus meeting with the nationwide expert panel was organized to brainstorm about potential implementation strategy elements related to the most important influencing factors regarding SSR implementation.³⁸  The meeting was chaired by a junior researcher with previous experience in chairing focus group discussions. Informed consent was given before the start of the meeting. At the start, the results of previous studies on barriers and facilitators of SSR and the literature search on potential implementation strategy elements were presented by the junior researcher. Next, in teams of 4, the experts discussed which influencing factors matched with which potential implementation strategy elements, both known from the literature and from the experiences of the experts. The experts then discussed the potential implementation strategy elements. Eventually, the junior researcher summarized the recommended implementation strategy elements, and additional remarks could be provided by the experts. The consensus meeting was audiotaped and transcribed verbatim. In addition, notes were taken by a senior researcher.

Third, multiple discussions with the multidisciplinary project team took place to discuss the final potential implementation strategy elements selected from the literature study and from the consensus meeting in order to achieve a toolbox that could be tested among 6 laboratories.

The different strategy elements were developed in close collaboration with the professional organization for developing SSR templates (PALGA) and with members of the multidisciplinary project team, and were directed to clinical pathologists. Before effectiveness and feasibility testing, the strategy elements were alpha tested for operability and functionality. In addition, for alpha testing of the strategy elements, we collaborated with a pathology resident and communications expert.

After the selection phase, specific content of the implementation strategy elements was developed, based on multiple resources from the professional organization for developing SSR templates, such as the Web site showing information about SSR templates, the PALGA liaison instructions, and knowledge about SSR usage from involved SSR experts. We used a Web-based platform for development of the e-learning.³⁹  At the end of the development phase, the strategy elements were alpha tested before the real-world testing. First, the content of the implementation strategy elements and their potential use in daily practice among a broad audience were assessed by presenting them during a yearly meeting with PALGA liaisons and a meeting of the SSR-template working group attended by pathologists. Second, 5 experts, 2 PALGA representatives, a pathologist, a pathology resident, and an implementation expert tested the functionality of the different implementation strategy elements. Third, these elements were tested for their usability and understandable content by 3 pathology experts and an additional expert in communications. Eventually, all digital strategy elements were combined in a single toolbox: a digital infographic including links to or additional explanatory information about the individual strategy elements.

We tested the effectiveness and feasibility of the toolbox to determine potential pitfalls before scaling up to a national level by conducting a small-scale study in 6 laboratories.^40,41  Laboratories were selected by region (north, middle, south), size (small, medium, large), type (internal general, internal university, external), and extent of use of SSR templates in 2018 (low, moderate, high). Two researchers of the project team visited the laboratories to provide additional information and to discuss study participation preconditions (having a PALGA liaison and being motivated to participate in this study). In addition, a contact person was designated and the restructured role of the PALGA liaison and the new role of the local opinion leader on SSR, called “SSR pathologist,” were allocated and explained. The medical manager of each laboratory signed a consent form to formalize their commitment and another form to authorize data collection from the PALGA database on SSR usage at the laboratory level. To assess the feasibility, we performed both an interview study with 2 pathologists per intervention laboratory and a survey among pathologists of the 6 intervention laboratories. Only pathologists using the included SSR templates in our study were included in the interviews and survey. To understand the effect, we determined and tested the changes in SSR usage among the 6 intervention laboratories compared to 34 control laboratories. The before measurement included all qualifying cases (all malignancies) between June 2019 and September 2019. The after measurement included all qualifying cases (all malignancies) between June 2020 and September 2020. Selection criteria for qualifying cases for effect evaluation are shown in Supplemental Table 2.

At the start, the toolbox was shared by email with pathologists of the participating laboratories by the study contact person. Pathologists were instructed to use the toolbox according to their own need. In 3 laboratories, pathologists were exposed to the toolbox for 8 months (February 2020 to September 2020), and at the other 3 laboratories, pathologists were exposed for 4 months (June 2020 to September 2020). Monthly telephone conversations were organized to inventory circumstances that may affect the test and to answer questions related to the test.

For assessing the effectiveness we used the indicator “Use of SSR.” Each pathology report that was created by using SSR was automatically flagged in the PALGA database. In this way, we were able to collect data on SSR usage. This was defined as the proportion of pathology reports composed by means of an SSR template.

Feasibility was tested in 2 phases, using qualitative and quantitative methods, including semistructured interviews and a survey at the end of the test period.^40–42  We conducted semistructured interviews with 2 pathologists per laboratory, who were recruited by the contact person of the laboratory, about their toolbox experiences. To preclassify these experiences, we developed an interview guide based on the first 4 categories of the feasibility framework of Bowen et al.⁴²  Informed consent was given before the interviews. Interview participants reported characteristics, for example, age, sex, work experience, and type of SSR template use. The interviews lasted between 17 and 60 minutes and were audiotaped.

After the interviews, the toolbox was adopted and emailed to all participants by the study contact person. At the end of the study, all participants received an eSurvey based on the results of the interviews on feasibility from the study contact person to determine the feasibility of the implementation strategy elements among all participants. The eSurvey consisted of multiple questions regarding the actual use, experiences, and self-reported effectiveness of the implementation strategy elements. Furthermore, questions about the experienced barriers of SSR implementation and the most promising strategy elements for future improvement of SSR implementation were included. Two people from the project team, 1 pathology resident, 2 experts on feasibility studies, and 1 communication expert tested the eSurvey. The survey did not accept unanswered questions. The survey was available from October 6, 2020, until November 6, 2020, and weekly reminders were sent.

We used IBM SPSS Statistics V25.0 (IBM Corp, Armonk, New York) to explore and test the effect of the intervention. First, laboratory characteristics (region, size, type, SSR use in 2018) were reported and the pathology report characteristic (type of SSR template) was extracted from the PALGA database. Second, descriptive statistics were used to gain insight into the proportion of the SSR that was used, as well as the laboratory and pathology report characteristics before and after distribution of the tailored strategy. Third, change in SSR use among the test laboratories and control laboratories was tested, taking into account the clustering of the cases within laboratories: We performed a multilevel logistic regression analysis by building a model, including covariates for strategy (none/tailored), measurement time (before/after), and its interaction term (strategy*measurement). The interaction term tests the difference in change between the test laboratories and control laboratories.

The interviews were transcribed verbatim for content analysis. Two researchers independently extracted experiences of the pathologists with the toolbox, using ATLAS.ti (version 8.3, ATLAS.ti Scientific Software Development, Berlin, Germany). These experiences were allocated to the first 4 categories of the feasibility framework of Bowen et al⁴² : demand, acceptability, practicality, and implementation barriers of the implementation strategy elements. Each encoder established an independent coding tree and discussed both coding trees to establish 1 refined coding tree. In case of discrepancies, codes were discussed to reach consensus.

We also used IBM SPSS Statistics V25.0 for the statistical analysis of the survey results. Missing data were coded as such and excluded from the data analysis. We performed descriptive statistics to determine the proportions of agreement of SSR users with the feasibility of the toolbox.

Ethical approval was obtained by the Human Research Committee (2018-4124).

By combining the barriers and facilitators of 2 previous studies, 18 different barriers and 14 facilitators were identified.^32,33  For all these factors, an implementation strategy element was determined. Six strategy elements were considered short-term, that is, defined as being able to be developed before the start of the test phase. The remainders were long-term, owing to long-term technical or organizational implications, and fall outside the scope of the current article. In Table 2, an overview of the previously determined barriers and facilitators that are connected to the implementation strategy elements selected by the expert panel and project group is given. The developed and tested short-term implementation strategy elements are distinguished from the long-term elements.

After selection, we developed a multifaceted strategy tailored to the barriers and facilitators found in previous research, which consisted of the following: (1) an improved Web site, (2) e-learning, (3) a communication manual, (4) an adapted feedback process, (5) a sheet within the SSR templates showing updates of SSR templates, and (6) monthly telephone conversations to discuss audit and feedback information regarding the use of different SSR templates within the laboratory. Key information about the implementation strategy elements is provided in Table 3. In Figure 2, the infographic of the toolbox is presented. Screenshots from all elements can be found in the Supplemental Data.

The aim of the adapted Web site containing information about the SSR templates was to inform SSR users on diverse aspects of the implementation of SSR templates. To develop a new version of the Web site showing information about SSR templates, we not only restructured the information already available, but also complemented it with information needed from multiple stakeholders regarding the SSR templates: the acreage of SSR templates, an explanation of the development and feedback processes of SSR templates, and the advantages of SSR usage. The latter was shown on the frontpage of the Web site section on SSR.

The e-learning aimed to improve the knowledge and skills of SSR users regarding the SSR templates. A storyboard in PowerPoint was constructed and the design principles of Mayer and Moreno⁴³  were applied to develop the e-learning modules. An overview of the different elements is provided in Supplemental Table 3. Each module focused on a learning goal based on the barriers and facilitators listed in Table 2. Next, we obtained advice from a senior researcher with experience developing e-learning for primary care. Eventually, the e-learning content was based on the documentation of the SSR templates, supplemented with instructional videos and challenging questions based on this documentation.³⁷  Pathologists were free to select modules according to their own needs, considering the influence of prior experience using SSR.³² 

The communication manual aimed to support pathologists in their communication with their PALGA liaison and PALGA regarding newly available SSR templates, updates of SSR templates, settings of SSR templates, the feedback process of SSR templates, and the advantages of using SSR. In addition, pathologists were encouraged to discuss the content of the pathology reports with other multidisciplinary team members. The communication manual was based on a previous version of the description of the role and tasks of PALGA liaisons and was supplemented with a newly created role and task description of a local opinion leader of SSR, called “SSR pathologist.” The laboratories were free to select a local opinion leader, but the head of the pathology department was most often selected.

The aim of the revised feedback process was to improve the transparency and efficiency of the current feedback process. To improve transparency, the feedback process of the SSR templates was published, for example, the experts participating in the PALGA SSR template working group, responsible for processing feedback. To improve efficiency, more emphasis was placed on the availability and additional value of the “Feedback Button,” instead of mailing or calling PALGA with feedback regarding the SSR templates. The feedback provider received an acknowledgment of receipt from PALGA, as well as information about the subsequent processing of the feedback. The Feedback Button streamlined the feedback mechanism and gave insights into perceived problems. This was supplemented with a “Frequently Asked Questions” page (part of the Web site) to deduct common questions and misconceptions of SSR users regarding SSR.

The aim of the additional sheet within SSR templates was to improve the awareness of SSR users regarding new elements within the SSR template. The sheet was developed by the professional organization for developing SSR templates in the same matter as the other tabs within SSR templates. The additional sheet included information on various updates, both technical and content related, and the most recent template version is shown. During the test, more emphasis was placed on the availability and additional value of this tab.

During the test period, contact persons were called monthly to discuss audit and feedback information on local SSR usage, specifically the absolute numbers of SSR used per SSR template per laboratory retrieved from the PALGA database. Furthermore, these meetings could be used to recall the aim of the test study.

In total, we included 31 052 pathology reports. The number of included pathology reports per period for both the test laboratories and the control laboratories is shown in Figure 3. Of the 31 052 included reports, 20 906 (67%) reported urologic oncology cases, 7306 (24%) reported gastroenterologic oncology cases, and 2840 (9%) reported gynecologic oncology cases. An overview of the characteristics of laboratories is provided in Supplemental Table 4.

The change in use of SSR between after- and before-measurement period (Δ10.1) significantly differed between the test laboratories and the control laboratories (P < .001): The average use of SSR among all test laboratories increased by 12.4% (33.7–46.1, Δ12.4), whereas among the control laboratories, the average use of SSR increased by 2.3% (64.4–62.1, Δ2.3).

For each laboratory, 2 pathologists were included in the interview study (n = 12), of which 4 were male and 8 were female, with an average age of 44 years (range, 35–60 years). The survey yielded 19 responses from 41 participants (46%): 11 female (58%) and 8 male (42%). The average age of the respondents was 44 years (range, 31–61 years), and their average work experience was 12 years (range, 1–31 years). Of the 19 respondents, 10 were specialized in gastroenterologic pathology, 6 were specialized in gynecologic pathology, and 4 were specialized in uropathology. All were familiar with SSR and used SSR templates in practice. Of the 19 respondents, 9 (47%) worked at an academic pathology laboratory, 8 (42%) worked in a nonacademic pathology laboratory, and 2 (11%) worked in an independent pathology laboratory.

We interviewed 12 pathologists to test the feasibility of the toolbox, including 6 tailored implementation strategy elements. The experiences of the pathologists with each strategy element are illustrated with some quotes (Figure 4). The quotes are translated literatim from Dutch to English.

Of the 19 respondents, 13 (68%) received the toolbox properly, and all of them (n = 12, 1 missing) looked at the content. The recommendation of toolbox use to colleagues varied, with a median of 7.0 (range, 2–9). According to 13 of the 19 respondents (68%), the provision of the toolbox could be improved by publishing the toolbox on the PALGA Web site, and, according to 10 respondents (53%), in the digital newsletter of the Dutch Pathology Association. In addition, 3 pathologists added the suggestion of a direct link to the toolbox included in the SSR templates.

Results on the actual use, self-reported effectiveness, and recommendation scores are provided in Table 4. The actual use of the 5 implementation strategy elements varied: the elements integrated in the SSR templates were used more often than elements only accessible via links in the toolbox. Respondents mentioned a lack of knowledge on the availability of specific implementation strategy elements, except for the sheet on SSR updates, as the main reason for not using them. The self-reported effectiveness of the respondents regarding the use of the implementation strategy elements to use SSR more often and in a better way varied between the elements: E-learning was considered most effective for using SSR more often, and both e-learning and use of the “Feedback Button,” for using SSR in a better way. The results on self-reported effectiveness were in line with the average recommendation score.

Furthermore, 2 of 6 SSR pathologists (33%) were encouraged by the rates on current SSR use at the laboratory level, and 3 of 17 respondents (18%, 2 missing) were encouraged to use SSR by the PALGA liaison of their laboratory.

As some strategy elements were not yet developed, we also evaluated the remaining barriers and promising strategy elements of SSR implementation. Of the respondents, 13 of 19 (68%) still perceived barriers related to SSR. Most experienced barriers related to the content of SSR templates, specifically the mandatory items and their rigidity. As a result, more than half of the 17 respondents (2 missing) mentioned content-related improvements as most promising. Of these 9 respondents, 8 (89%) mentioned the revision of the content of SSR templates by expert panels as a promising strategy element. Furthermore, 8 of 17 respondents (47%) mentioned improvements in readability of the SSR template, of which 6 (75%) wanted feedback from oncology-treating clinicians. Additionally, 6 of 17 respondents (35%) wanted the possibility to receive audit and feedback information regarding performance indicators based on SSR output, such as tumor grading, and 5 of 17 (29%) wanted technologic improvements, of which 5 wanted automatic transfer of clinical information (100%) and 4, integration of speech recognition (80%).

To improve nationwide implementation of SSR in oncologic pathology, we selected implementation strategy elements, developed a tailored strategy including these elements, and tested its effectiveness and feasibility in a real-world setting to optimize the toolbox before nationwide expansion. We showed both effectiveness and feasibility in our pilot experiment.

Our multifaceted strategy consisted of 6 implementation strategy elements: an updated Web site, e-learning, a communication manual, a feedback procedure, a sheet with information on updates incorporated into the SSR templates, and a monthly telephone conversation to discuss audit and feedback information regarding SSR usage. The tailored strategy showed a significant increasing trend in SSR usage among test laboratories compared to control laboratories. The actual usage of specific implementation strategy elements was highest when they were already part of the SSR template itself but could be improved for all strategy elements by emphasizing availability through national dissemination strategies familiar to pathologists. The e-learning and revised feedback procedure, mainly the use of the “Feedback Button,” were most feasible, effective, and had the highest recommendation score in improving SSR usage. The pathologists' experiences regarding the feasibility and effectiveness of the renewed Web site, communication manual, “Frequently Asked Questions,” and sheet on the updates of SSR were inconclusive, suggesting variability in (information) requirements among pathologists. Before nationwide expansion, feedback on element-specific content has to be processed and methods to increase toolbox exposure should be questioned to conform to the pathologists' needs.

This study is the first to develop and test a tailored strategy primarily focused on increasing the implementation of SSR among pathologists. Previous research has focused on the implementation of SSR, mostly deployed nationwide strategies, to inform pathologists about SSR. However, our study showed differences between laboratories and even pathologists in terms of information need regarding SSR templates. Nationwide use of the toolbox could provide us with insights into the influencing characteristics of laboratories in order to tailor the toolbox to local information needs. Moreover, Branston et al²⁵  organized both educational and training sessions on SSR. This is in line with the requests of the pathologists interviewed in our study. However, digital education was preferred instead of in-class education, and the facultative character of our e-learning was highly valued. Despite these positive aspects of its feasibility, the long-term effect of e-learning on professional behavior in health care is not yet clear.⁴⁴  Therefore, future research should evaluate the effectiveness of e-learning on the use of SSR among pathologists.

In our study, audit and feedback on local SSR usage was solely provided verbally to the contact person of the test laboratory. However, a Cochrane review on the effectiveness of audit and feedback strategy elements mentions the importance of having both verbal and written formats.⁴⁵  Another study discussing 15 suggestions for designers of practice feedback states the importance of constructing feedback through social interaction, which will become easier when it is provided in written formats.⁴⁶  Therefore, the provision of audit and feedback needs to be adapted to a more shareable format, challenging individual pathologists to increase their use of SSR in daily practice.

Studies of Renshaw et al^47–49  showed the impact of the method of report creation, content of SSR items, and format of these items on the accuracy of the pathology report and therefore patient care. In both studies on SSR-related strategy elements, discussions took place regarding SSR content.^25,27  Moreover, although stated as the ultimate goal by Nakleh et al⁵⁰  in 2017, a recent published article on the electronic cancer checklists of the College of American Pathologists shows no clear process on development and revisions of these checklists.⁵¹  In our study, the need for proper discussions on SSR content was clearly stated as the improvement with the most potential by the pathologists, and the availability of the “Feedback Button” was not deemed sufficient to support pathologists in their improvement of content of SSR templates. Because of the tremendous importance of these content features on SSR implementation, stated by pathologists in this study and in other studies, future research should be focused on the expansion of a widely supported SSR development and revision strategy and evaluate its effectiveness and feasibility on a nationwide scale. Part of this strategy could be aimed at evaluating current SSR output and testing the use of modifiers to quantitative values, holding a promise to improve SSR accuracy.^51,52 

The strengths of this study are the stepwise approach of selecting, developing, and testing a toolbox tailored to a previous analysis of barriers and facilitators and the additional step of testing this toolbox in multiple laboratories before introducing the toolbox on a nationwide scale. Moreover, by conducting the expert meeting to determine implementation strategy element selection and interviews to determine the feasibility of the strategy elements, the end users were actively involved in the selection, development, and testing phases of the tailored strategy. Others involved in the implementation of SSR may use the results of this study in their own process of incorporating SSR into daily pathology practice.

This study does have some limitations. First, the feasibility study was conducted during a nationwide lockdown period (coronavirus disease 2019). As a consequence, the number of pathology reports decreased in the after-measurement period, in line with the decrease in oncologic diagnosis in this period.⁵³  Furthermore, this crisis may have negatively impacted the communication between pathologists about the toolbox. However, by designing digitally accessible strategy elements and having monthly telephone conversations, we were able to continue the study during this period and to monitor changes in the working environment. Second, this study showed a significant stronger increasing trend in SSR usage among test laboratories compared to control laboratories. We were able to test the differences in change statistically, but we have to interpret this result with caution, because the used study design does not exclude other reasons for this change. In a subsequent next study in which the implementation strategy elements will be available for all Dutch pathology laboratories, effectiveness might be determined by an interrupted time-series analysis. Last, improving the implementation of SSR demands an investment, both monetary and in man-hours, from all involved stakeholders. However, implementation costs are negligible compared to losses due to suboptimal communication and treatment resulting from incomplete reporting.⁵⁴ 

Six implementation strategy elements were selected and developed, incorporated within an evidence-based multifaceted strategy, and combined in a toolbox, and their effectiveness and feasibility were tested to improve the implementation of SSR in oncologic pathology. Results on effectiveness showed a significant stronger increasing trend in SSR usage among test laboratories. Regarding feasibility, e-learning and the approachable feedback method were found to be most feasible and effective, owing to their additional value to current SSR practices. However, familiarity with all implementation strategy elements could be improved by using national dissemination strategies. Therefore, the next step is to optimize the toolbox and to share the toolbox nationwide and, thereafter, to evaluate both the effectiveness and feasibility of these strategy elements on a national level.

We thank the experts of the nationwide expert panel for their involvement in the consensus meeting. We thank the 6 laboratories and the contact persons for their contribution to this study. In particular, we thank the pathologists who participated in the interview and survey study. We thank Angelique Schlief, BSc, for her assistance regarding data collection and data analysis and Reinier Akkermans, MSc, for his assistance regarding data analysis.