Evaluating the Adoption of Laboratory Practice Guidelines

From July 2015 through January 2017 three modalities were used to track laboratory audience awareness: a written survey, telephone interviews, and focus group discussions. The survey development, analysis, and this publication were supported by Cooperative Agreement NU47OE000057-04, funded by the US CDC. The cooperative agreement with CDC required preapproval of the survey instrument by the US Office of Management and Budget (OMB No. 0920-1067). The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the CDC.

For the written survey, 3064 laboratories were selected for participation because they had enrolled in one of several CAP proficiency testing (PT) programs relevant to IHC testing. We also included an additional 448 laboratories that were not enrolled in any of the CAP PT programs. These non-CAP laboratories were identified by CDC using previous Centers for Medicare and Medicaid Services Part B reimbursement claims for IHC testing. The use of both CAP PT program participant laboratories and non-CAP participant laboratories was designed to assess any differences in survey results.

From July through November 2015, the 3064 laboratories enrolled in CAP PT programs relevant to IHC testing received the written survey with their regular PT mailings (see supplemental digital content at www.archivesofpathology.org in the January 2020 table of contents). The survey was also available electronically through the CAP's online portal. Nonparticipant laboratories were sent surveys via the US postal service. Only 1 response per laboratory was accepted. The questionnaire included the same questions from the original 2010 survey with additional questions that explored details regarding predictive and nonpredictive marker assays.⁵  The new questions pertained to decalcified specimens, overall concordance rates, the primary method for validation, and several revalidation specifics. The end of the survey queried laboratories as to their current awareness and adoption of the IHC VAL LPG.

Telephone interviews and focus group sessions (see supplemental digital content) were also conducted to allow for a fuller exploration of the factors that underlie adoption or nonadoption of specific recommendations within the LPG and to measure current use of the CAP's tools and resources. We planned for 40 telephone interviews (20 pathologists, 10 laboratory directors, and 10 laboratory managers) and 24 focus group participants in 2 groups—12 each arranged by peer group: (1) pathologists (eg, pathologists, pathology chairs, and medical directors) and (2) nonpathologist laboratory professionals (eg, laboratory managers, laboratory supervisors, and histotechnologists). An external consultant was hired to perform the telephone interviews and lead the focus group sessions.

The pool of laboratories for telephone interviews, conducted from November 2016 through January 2017, intentionally included CAP PT program participant and nonparticipant laboratories. Interviewees were randomly sampled from the CAP PT participant survey respondents and cross-checked for appropriate distribution of laboratory type and size. All 448 nonparticipant laboratories were included in the sampling owing to the lower number of respondents as described in the Results section.

Focus group participants were also randomly recruited from survey respondents as described above. The two 60-minute in-person focus group sessions were held in conjunction with a national laboratory medical society annual meeting in 2016. This session included some of the same probe questions from the telephone interview template and a specific subset concentrating on evaluating the CAP's current tools and resources.

Before developing the AL LPG, the CAP/ASH workgroup created an online baseline survey aimed at assessing the current state of initial leukemia diagnostic testing.⁶  The cooperative agreement with CDC also required preapproval of the survey instrument by the US Office of Management and Budget (OMB No. 0920-1067). For the AL baseline questionnaire, members of CAP, ASH, the Society for Hematopathology, and the European Association for Haematopathology were asked to complete a survey with the online link also posted externally. The CAP promoted the online survey with a news brief in CAP Today (a monthly trade periodical published by CAP), posts to social media outlets, the CAPconnect blog (a member peer network), and an announcement on the CAP Web site. The survey was open June 8, 2015 through July 24, 2015. Since the survey was disseminated broadly, the number of people made aware of the survey's existence cannot be determined. A total of 294 completed surveys were received, and 36 were excluded owing to survey abandonment. Survey results from 258 respondents were tabulated as the baseline data for AL testing practices and published by George et al.⁸  A follow-up survey is planned for 18 to 24 months after the AL LPG publication to compare to the baseline results and to evaluate the adoption of the 2017 AL LPG.

The CAP strived to maximize the response rates for both the IHC VAL and AL surveys by distributing flyers in PT products before the survey, writing an article in CAP Today, and promoting on listserv emails; however, the survey did not include questions about how participants learned about the questionnaire. Thus, the effects of the promotional efforts could not be captured.

Table 1 illustrates the estimated and actual response rates for the IHC VAL and AL projects. We initially estimated a 65% response rate for the IHC VAL surveys based on previous experience. Of the 3512 surveys sent, 1624 (46%) were completed and returned. These results included 1539 of 3064 surveys (50%) from laboratories participating in the CAP PT programs and 85 of 448 surveys (19%) from the nonparticipant laboratories. The quantitative IHC VAL survey results were tabulated, analyzed for comparison, and published separately by Fitzgibbons et al.⁹  In addition, Stuart et al¹⁰  were able to publish new immunohistochemical validation benchmark data for laboratories.

To have 40 completed telephone interviews, we estimated a need to contact 121 individuals based on 3 roles: IHC laboratory director, staff pathologist, and laboratory manager (estimated response rate, 33%). Interviewees were informed that the interview (1) was a joint CAP/CDC initiative, (2) would take approximately 15 minutes to complete, and (3) would address an evidence-based guideline, not a laboratory accreditation checklist. The call attempts exceeded 500 and we had to contact 298 to achieve 40 complete interviews (actual response rate of 13% based on 2 categories: pathologist and nonpathologist). Although all 85 nonparticipant laboratories were included in the sampling, none participated in the interview. To reach our goal of 24 focus group participants, we estimated that 200 laboratory professionals would need to be contacted to determine their availability, but after sending invitations to 747 unique individuals, only 19 agreed to participate. Of these 19 individuals, only 5 (0.7%; 4 pathologists and 1 laboratory manager) actually attended the in-person focus group sessions, and none were from the nonparticipant laboratories.

Regarding awareness of the IHC VAL LPG, two-thirds of survey respondents (691 of 1057) reported that they were aware of the recommendations before receiving the questionnaire, with most of the remaining laboratories planning to review them within the next 6 months.⁹  Telephone interviewees also reported awareness of its content. All interviewees suggested that the CAP should continue to make full efforts to keep awareness high, stating that current channels of notification and word of mouth are important. Further analysis demonstrated that 14 of the 20 pathologists (70%) and 7 of the 20 laboratory managers (35%) were aware of the LPG via direct CAP channels, while 11 of the 20 laboratory managers (55%) were informed of it by their pathologists. All focus group participants agreed that professional networking provided the widest LPG awareness and all participants encouraged electronic notifications and updates. The focus group members cited emails as the medium of preference, but notifications in proficiency testing packages and updates in CAP Today were also suggested. Several key qualitative results from the telephone interviews and focus group are summarized in Table 2.

As for the adoption rate, the survey revealed that some or all of the recommendations had been adopted by nearly 80% (550 of 688) of respondents with only 12 respondents (1.7%) stating they had no plans to adopt the recommendations unless required by their laboratory accreditor.⁹  More than half of laboratory respondents from the survey used or planned to use the guidelines prospectively for all new assays and for assay revalidations, with a minority of laboratories (110 of 687; 16%) stating they would use the recommendations to retrospectively validate existing assays.⁹  Comments from all but 2 of the interviews suggest that laboratories were both aware of the guideline and planned to comply.

Two pathologists out of the 40 completed interviews stated they initially tried to adopt the guideline, but ultimately decided it provided no useful value. Even though pathologists indicated they wanted a more comprehensive and prescriptive guideline, 55% (11 of 20) of all pathologists interviewed by telephone felt the guideline as it exists was useful. Eighty percent (16 of 20) of laboratory managers rated the current guideline as useful and indicated a positive effect was having more consistent written laboratory policies. Finding validation cases for rare antigens (367 of 682; 54%), the time and staff needed to run validations (319 of 682; 47%), and the additional expenses incurred (236 of 682; 35%) were the 3 difficulties cited most frequently by survey respondents.⁹  These barriers were also consistent across the telephone interviews and focus group conversations.

During the telephone interviews, the pathologists did not report substantive burdens in following the guideline, but instead emphasized that improved patient outcomes and better laboratory practices needed to be the main focus. In contrast, one-quarter (5 of 20) of the laboratory managers felt the guideline led to higher expenses, but declined to try to detail these impacts. When queried about staff impacts, almost half (9 of 20) of the laboratory managers saw the guideline as creating extra work for their staff. They felt this extra load was aggravated by the lack of guideline specificity and process detail. In the focus group session, one specific item that was cited was deciding when to use reference laboratories for a specific assay versus performing the validation to bring it in-house. Every interview found that the IHC VAL LPG needed more detail and specificity. Although the number of focus group participants was low, the IHC VAL LPG was accepted and adopted by the participants, several of whom commented on the motivation to comply with the recommendations because of the CAP's importance to the laboratory community and association with quality accreditation. They indicated that laboratory director awareness and education were deemed the most important factors to guideline adoption.

All CAP guidelines are freely available in both the online and print versions of Archives of Pathology & Laboratory Medicine. In addition, each guideline has a unique page with additional resources posted on the CAP Web site (IHC VAL: https://www.cap.org/protocols-and-guidelines/cap-guidelines/current-cap-guidelines/principles-of-analytic-validation-of-immunohistochemical-assays [accessed February 27, 2019]; AL: https://www.cap.org/protocols-and-guidelines/cap-guidelines/current-cap-guidelines/initial-diagnostic-workup-of-acute-leukemia [accessed February 27, 2019]). Scientometric data obtained from these sources are used to supplement direct survey data in order to gain a more comprehensive picture of the awareness of published guidelines and identify early indicators of potential guideline impact on laboratory practice. Guidelines are expected to be highly read and cited; our assessment confirms this expectation. To date, the IHC VAL LPG has been cited 83 times in publications based in 26 different countries,¹¹  the pdf version of the guideline has been accessed more than 21 881 times (Allen Press, Inc, email communication, October 12, 2018), and the guideline Web site has been accessed more than 11 713 times.¹²  The AL LPG, published more recently, has been cited 13 times in publications based in 8 different counties,¹¹  the pdf version has been accessed more than 13 583 times (Allen Press, Inc., email communication, October 12, 2018), and the guideline Web site page has been viewed more than 3998 times.¹²  Of note, the entire body of 17 CAP-developed guidelines are among the most highly accessed pdfs for the Archives, accessed more than 244 000 times as of this writing (Allen Press, Inc, email communication, October 12, 2018), and they have been cited more than 11 217 times in journals based in more than 115 different countries.¹¹  This confirms that our observations regarding the AL LPG and IHC VAL LPG are not unique phenomena. Long-term monitoring of these and other guideline impact framework indicators will help provide a broader view of guideline awareness and adoption, which is a useful supplement to essential direct survey data.

This investigation revealed the effectiveness and limitations of written surveys, telephone interviews, and focus group sessions on measuring the adoption of LPG dissemination; it also led to a better understanding of key factors related to the uptake of the IHC VAL guideline. The modalities were evaluated for sustainability in measuring the adoption beyond the cooperative agreement timeframe.

In our experience, written surveys, which were available in both paper and electronic format, showed the best ease of use for both developers and participants, a good response rate, and quantitative results that were readily analyzed and interpreted. By using the same questions in both the baseline and follow-up surveys, we were able to measure differences and draw more accurate comparisons about the differences found after IHC VAL LPG publication. Although precomparisons and postcomparisons of LPG adoption are not yet available for the AL guideline, we intend to use the abovementioned strategy for the follow-up survey.

The telephone interviews, conducted by the contracted consultant at a cost of $30,000, proved to be more time-consuming and difficult to conduct than the written surveys. The CAP database, from which the telephone interview participants were derived, contained several contacts for various responsibilities within an institution, and it was challenging to find the correct individual who also had time and willingness to discuss the nuances of implementing new guideline recommendations into the IHC laboratory. Interviewees were immediately asked if they had the required time and an inclination to provide their feedback. It turned out that many individuals declined the interview because they felt the time investment was too onerous for what they deemed to be simple “market research.” At times, there was a perception of intrusion by the independent interviewer and resistance in answering the questions; however, the feedback that was obtained was valuable as described in the Results section. None of the interviewees were willing to respond to open-ended questions that required them to give what they deemed “extra” feedback. They all, however, gave willing and concise answers to questions that asked for easily defined specific responses, and interviewees pointed out that they saw nothing that needed to be removed from the guideline. Based on this experience from the IHC VAL guideline it would be preferable to have preselected laboratories willing to participate in telephone interviews rather than random sampling for the AL guideline.

The focus group session was the most complicated modality to execute. To reach the intended audience of both pathologists and nonpathologists, we wanted to host this event at a national meeting attended by both; however, it was difficult to arrange the session at a time that did not interfere with educational activities. An artifact of this study was OMB prohibition against the use of small incentives (eg, gift cards or vouchers) to increase participation; this restriction will not apply to future CAP activities. We also did not have access to the attendee list, hence hundreds of invitations were sent out with a small return. Nonetheless, the focus group was held and the format was useful in that the dialogue and interaction allowed for suggestions for improvements in communication, resulting in an idea for an LPG-implementation tool that we had not previously considered. As with the telephone interviews, it might be better to have preselected laboratories and/or perform the focus group sessions under different circumstances to increase the effectiveness of the adoption analysis for the AL guideline.

Most participants from all 3 modalities indicated knowledge of the IHC VAL guideline's existence. To achieve this level of awareness, the CAP uses a variety of communication channels when an LPG is released including a targeted strategy toward CAP pathologist members, comprised of email blasts and educational courses, in addition to general audience channels such as press releases, CAP Today articles, and letters of announcement to targeted professional societies. In 2015, the CAP began distributing infographics (graphic visual representations of information, data, or knowledge intended to present information quickly and clearly) and increasing its social media tactics for subsequent LPG releases in addition to tracking communication channel uptakes such as press releases and social media. Participants confirmed that these approaches were effective and we believe this comprehensive communication strategy will continue to positively impact the perception of CAP-developed LPGs. Moreover, peer-to-peer promotion and a sense of professional benchmarking seem to be especially effective for CAP LPGs. Indeed, this knowledge of guideline update and awareness is confirmed by the scientometric data, which showed that both the IHC and AL LPGs were highly cited and accessed.

Most participants indicated that they already had adopted or planned to adopt all or most of the recommendations, but some interviewees seemed to conflate LPG recommendations with requirements in the CAP Laboratory Accreditation Program (LAP). About half of the interviewees expressed concern with being compliant with CAP “requirements,” suggesting that at least some interviewees did not understand that these LPGs are voluntary. The CAP LAP program accredits US laboratories performing IHC and could be perceived as a major facilitator to guideline adoption. As the CAP accredits more than 8000 laboratories in the United States and internationally, CAP-accredited laboratories may be more likely to adopt CAP-produced LPG recommendations. Failure to distinguish LAP accreditation requirements from guideline recommendations by the laboratory audience could also be considered a barrier among non–CAP-affiliated laboratories, since the LPG recommendations may not otherwise be mandated by these laboratories' accrediting agency.

The focus group participants claimed that other clinical organizations, such as the American College of Surgeons Commission on Cancer, are extremely effective in taking a “top down” approach to notification and updates. Participants reported that adoption was primarily influenced by laboratory directors and pathologists. They felt that credible related sources, including the CAP Cancer Protocol Templates, would enhance adoption and facilitate buy-in from administrators. Algorithms, templates, and teaching materials are needed to assist the laboratories, as participants indicated more prescriptive recommendations would be useful. The CAP began incorporating such items in subsequent published guidelines such as the “HPV Testing in Head and Neck Carcinomas.”¹³  In addition, all of the focus group participants were unaware of the toolkit materials on the CAP Web site, and lack of ease in navigating the Web site was also indicated. As a result of this feedback, an improved Web site design with a new header of “Protocols and Guidelines” was released in mid-2017.

The survey participants, telephone interviewees, and focus group attendees all provided specific feedback (eg, make recommendations more prescriptive and procedurally comprehensive), which will be reviewed when the IHC VAL guideline is formally updated (currently in progress). As indicated in CDC's paradigm model, the data collection and analysis (Step 4) directly informs the creation/revision process (Step 2) and will be used in future CAP LPGs.

There are many challenges when attempting to measure the translation of guideline recommendations into laboratory practices. Indeed, others in the academic and research world are faced with a similar dilemma when striving to gauge the community and societal value of their work, and numerous research impact frameworks have been developed to support these efforts.^14–17  The CAP has developed a similar framework to supplement information obtained from direct surveys and standardize its assessment of guideline project outputs. The 5 domains used in the Becker Model were selected as the foundation for the CAP Center Guideline Impact Framework,¹⁷  but indicators from all included framework models were reviewed.^14–17  Relevant indicators were selected, some indicators were modified, and additional indicators were added to account for our pathology and laboratory environment. Our 5 domains (advancement of knowledge; clinical implementation; community benefit; legislation and policy; and economic benefit) range from traditional bibliometric data to broader health and societal indicators that may take years to identify and measure.¹⁸  We can use the supplemental scientometric data to confirm the high-level guideline awareness in the wider laboratory community.

There were several limitations for both the quantitative and qualitative analyses. With both types of investigations, there could be nonresponse/selection bias due to lack of compliance or disagreement with the guidelines. For the IHC VAL 2015 survey, we acknowledge there was selection bias due to sampling that primarily included laboratories enrolled in one of CAP's PT surveys. To address this, the sample pool was supplemented with non–CAP-accredited laboratories, and statistical analysis was used to help adjust for these different groups. In addition, since CAP participating laboratories more commonly responded to the written survey, compared to nonparticipating laboratories (46% versus 19%), there is potential bias, since CAP accredited laboratories may be more focused on optimizing and harmonizing practices to adopt guideline recommendations.

For the AL baseline practices survey, we could not adjust or control for any laboratory demographic characteristics or duplicate responses owing to the electronic survey access on multiple listserves and Web sites. In addition, the follow-up data assessing the adoption of the AL LPG are not yet available. As such, we cannot yet assess the broader applicability of the findings we observed surrounding the IHC LPG on other guidelines.

There were also some considerations for the qualitative IHC VAL research. A high percentage of laboratory contacts declined to participate in the telephone interviews, thus we had to increase the sample size. For the IHC focus groups, the initial study design accounted for the different nonpathologist and pathologist roles, but the small sample size of 5 participants limited the interpretive and inferential value of the findings. Also, the low percentage of participants for our telephone interviews and focus groups, while providing an opportunity to obtain valuable qualitative information, introduced the substantial additional risk of limited sample bias. That is, feedback received during these small group sessions may not be generalizable to the larger laboratory community.

Of the various modalities of LPG evaluation outlined above, written practice surveys (paper and electronic) were the most effective method to evaluate LPG awareness, adoption, and effectiveness owing to their higher response rate. Although telephone interviews and focus groups provide opportunities to examine peripheral issues of guideline implementation not addressed in the written surveys, they are labor intensive, costly, and generally inefficient methods for acquiring useful feedback owing to the lack of laboratorian participation. Maintaining the uniformity in survey format and questions is essential for generating valid comparisons, acknowledging that no survey can anticipate novel and evolving practices. On the other hand, qualitative analyses can reveal unexpected findings that surveys can miss. For example, the follow-up telephone interviews and focus group discussions revealed some confusion between LPG recommendations and CAP LAP requirements. The larger number of written survey participants provided the opportunity to reach a more diverse population and gain a more accurate, data-driven picture of existing gaps in LPG awareness, adoption, and effectiveness.

The authors would like to acknowledge Ronald Brooks, MBA, contracted consultant, for his work with the telephone interviews and focus group sessions, and Carol Colasacco, MLIS, medical librarian specialist, for her work in preparing the manuscript. The authors would also like to thank Nicole Thomas, MPH, and Daniel Arber, MD, Tracy George, MD, Lauren Stuart, MD, MBA, and Joseph Tworek, MD, for their engaged participation on the “Guidelines Metric Expert Panel” (GMEP).