Context.—

To date, the College of American Pathologists (CAP) has developed 17 laboratory practice guidelines (LPGs) including updates. In 2013, the CAP was awarded a 5-year cooperative agreement grant from the United States Centers for Disease Control and Prevention to increase the effectiveness of LPGs.

Objective.—

To assess the awareness and adoption of 2 CAP LPGs: immunohistochemical (IHC) assay validation and initial workup of acute leukemia.

Design.—

Baseline surveys for each LPG were conducted in 2010 and 2015, respectively. To measure the adoption of guideline recommendations and inform future updates, a follow-up study consisting of surveys, telephone interviews, and focus group sessions was conducted in laboratories that indicated they perform IHC testing. A follow-up study for the acute leukemia LPG is planned.

Results.—

For the IHC Validation LPG, a total of 1624 survey responses, 40 telephone interviews, and discussions with 5 focus group participants were analyzed. The response rate for the aforementioned 3 modalities was 46%, 13%, and 3%, respectively. All modalities indicated most respondents were aware of the LPG and had adopted most or all of its recommendations. Respondents expressed needs for continued communication, increased specificity, and more prescriptive recommendations when the guideline is updated.

Conclusions.—

While data-driven development of evidence-based LPGs requires significant resources, active data collection to identify gaps and assess adoption contributes to improved laboratory testing practices in support of patient care. The CAP identified sustainable modalities to track metrics and developed multiple tools that should improve guideline development, adoption, and implementation. Of these modalities, written or electronic surveys were the most logistically feasible and had the highest response rate.

The United States Clinical Laboratory Improvement Amendments of 1988 (CLIA '88) established minimum requirements for assuring the quality of laboratory testing in US clinical laboratories.1  To ensure accurate diagnosis and patient management, many laboratories implement quality practices that extend beyond CLIA '88 by complying with relevant laboratory practice guidelines (LPGs). An LPG comprises specific recommendations for voluntary, standardized approaches to medical laboratory testing that may take into account processes for test selection, sample procurement and processing, analytical methods, and results reporting. Some LPGs are also disseminated to and used by clinicians to assist with test ordering and test result interpretation.

The College of American Pathologists (CAP), the world's largest organization of board-certified pathologists and leading provider of laboratory accreditation and proficiency testing programs, launched the Pathology and Laboratory Quality Center for Evidence-Based Guidelines (“The Center”) in 2009 to develop and promote LPGs, using the National Academy of Medicine's (NAM) standards for developing trustworthy guidelines.2  To date, The Center has published 17 evidence-based LPGs (including updated versions), using NAM's criteria. In addition, the CAP is an organizational member of the Guidelines International Network.

In 2013, the CAP established a workgroup under a cooperative agreement with the US Centers for Disease Control and Prevention (CDC) to define measures and collect information regarding dissemination, promotion, uptake, and adoption of LPGs on clinical testing and public health. The workgroup, designated the “Guideline Metrics Expert Panel” (GMEP), analyzed the collected data and explored how these processes differed among various intended users of LPGs. An important goal from the CDC's perspective is to help organizations that develop LPGs create a sustainable approach for continuous quality improvement through better collection of information, ultimately to improve uptake and implementation.

The analyses of qualitative and quantitative data generated by the GMEP were used to identify barriers to LPG implementation. In a published study that examined the adoption of clinical practice guidelines, Cabana et al3  identified several barriers such as lack of awareness and disagreement with the guideline recommendations. In addition to examining barriers to LPG implementation, we also wanted to explore the facilitators of adoption, such as guideline clarity, Web-based toolkits, and educational offerings, which can be used to assist laboratory professionals. With the information and identification of any potential external factors garnered from the GMEP data, the CAP aimed to increase adoption of CAP LPGs, thereby contributing to the improvement of patient care. The experience of gathering and analyzing the feedback on the dissemination and adoption of LPGs via the GMEP's activities has helped the CAP more effectively update existing LPGs and define recommendations for future guidelines.

The CDC cooperative agreement required that at least 2 LPGs be used as models to enhance uptake through the use of improved metrics. Funding applicants for the CDC agreement were asked to develop and apply metrics that addressed all 4 steps of the LPG lifecycle (Figure): (1) identification of gaps in laboratory practice, (2) creation/revision of LPG, (3) dissemination/promotion, and (4) uptake and use by laboratories and/or clinicians. The cooperative agreement between the CAP and CDC addressed 2 areas of laboratory testing: validation of immunohistochemical assays and initial workup of acute leukemia.

Four phases of the Centers for Disease Control and Prevention (CDC) on Improving the Impact of Laboratory Practice Guidelines (LPGs) with Metrics. Phase 1: Identify the gaps; Phase 2: Creation or revision; Phase 3: Dissemination and promotion; Phase 4: Data collection on awareness, adoption, adaptation, implementation, and impact. Data analysis in Phase 4 informs LPG revision and helps increase LPG uptake, utility, and impact. The life cycle then returns to Phase 1.

Four phases of the Centers for Disease Control and Prevention (CDC) on Improving the Impact of Laboratory Practice Guidelines (LPGs) with Metrics. Phase 1: Identify the gaps; Phase 2: Creation or revision; Phase 3: Dissemination and promotion; Phase 4: Data collection on awareness, adoption, adaptation, implementation, and impact. Data analysis in Phase 4 informs LPG revision and helps increase LPG uptake, utility, and impact. The life cycle then returns to Phase 1.

Before the creation of the immunohistochemistry validation (IHC VAL) guideline,4  the CAP had conducted a baseline survey in 2010 to collect information on IHC testing and validation practices.5  This study demonstrated a need for a formal evidence-based LPG. The “Principles of Analytic Validation of Immunohistochemical Assays,” published in 2014, details parameters for the initial validation of IHC assays and revalidation when testing procedures have changed.4  The intended users of the IHC VAL LPG include pathologists, clinical laboratory directors, and laboratory managers overseeing IHC testing.

The other LPG used in this demonstration project was CAP's “Initial Diagnostic Workup of Acute Leukemia” (AL) guideline, codeveloped with the American Society of Hematology (ASH),6  which was chosen owing to its direct relevance to the Healthy People 2020 public health impact initiative.7  The intended users of the AL LPG are pathologists and hematologists overseeing testing for acute leukemia patients.

This article outlines GMEP's analyses on the collection techniques for both qualitative and quantitative data before and after publication of the IHC VAL LPG and prepublication for the AL guideline. Under our cooperative agreement with CDC, we describe the overall findings including the efficacy of the various modalities of LPG evaluation with the intention of increasing awareness, further comprehending facilitators and barriers to guideline adoption, and informing LPG revisions. In addition, we began to explore scientometrics (the study of measuring and analyzing science, technology, and innovation) and included indicators such as guideline accesses via journal Web sites, and accesses of guideline pages on the CAP Web site, which may provide an early glimpse into laboratory community awareness and acceptance of published guidelines.

IHC VAL Guideline

From July 2015 through January 2017 three modalities were used to track laboratory audience awareness: a written survey, telephone interviews, and focus group discussions. The survey development, analysis, and this publication were supported by Cooperative Agreement NU47OE000057-04, funded by the US CDC. The cooperative agreement with CDC required preapproval of the survey instrument by the US Office of Management and Budget (OMB No. 0920-1067). The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the CDC.

For the written survey, 3064 laboratories were selected for participation because they had enrolled in one of several CAP proficiency testing (PT) programs relevant to IHC testing. We also included an additional 448 laboratories that were not enrolled in any of the CAP PT programs. These non-CAP laboratories were identified by CDC using previous Centers for Medicare and Medicaid Services Part B reimbursement claims for IHC testing. The use of both CAP PT program participant laboratories and non-CAP participant laboratories was designed to assess any differences in survey results.

From July through November 2015, the 3064 laboratories enrolled in CAP PT programs relevant to IHC testing received the written survey with their regular PT mailings (see supplemental digital content at www.archivesofpathology.org in the January 2020 table of contents). The survey was also available electronically through the CAP's online portal. Nonparticipant laboratories were sent surveys via the US postal service. Only 1 response per laboratory was accepted. The questionnaire included the same questions from the original 2010 survey with additional questions that explored details regarding predictive and nonpredictive marker assays.5  The new questions pertained to decalcified specimens, overall concordance rates, the primary method for validation, and several revalidation specifics. The end of the survey queried laboratories as to their current awareness and adoption of the IHC VAL LPG.

Telephone interviews and focus group sessions (see supplemental digital content) were also conducted to allow for a fuller exploration of the factors that underlie adoption or nonadoption of specific recommendations within the LPG and to measure current use of the CAP's tools and resources. We planned for 40 telephone interviews (20 pathologists, 10 laboratory directors, and 10 laboratory managers) and 24 focus group participants in 2 groups—12 each arranged by peer group: (1) pathologists (eg, pathologists, pathology chairs, and medical directors) and (2) nonpathologist laboratory professionals (eg, laboratory managers, laboratory supervisors, and histotechnologists). An external consultant was hired to perform the telephone interviews and lead the focus group sessions.

The pool of laboratories for telephone interviews, conducted from November 2016 through January 2017, intentionally included CAP PT program participant and nonparticipant laboratories. Interviewees were randomly sampled from the CAP PT participant survey respondents and cross-checked for appropriate distribution of laboratory type and size. All 448 nonparticipant laboratories were included in the sampling owing to the lower number of respondents as described in the Results section.

Focus group participants were also randomly recruited from survey respondents as described above. The two 60-minute in-person focus group sessions were held in conjunction with a national laboratory medical society annual meeting in 2016. This session included some of the same probe questions from the telephone interview template and a specific subset concentrating on evaluating the CAP's current tools and resources.

AL Guideline

Before developing the AL LPG, the CAP/ASH workgroup created an online baseline survey aimed at assessing the current state of initial leukemia diagnostic testing.6  The cooperative agreement with CDC also required preapproval of the survey instrument by the US Office of Management and Budget (OMB No. 0920-1067). For the AL baseline questionnaire, members of CAP, ASH, the Society for Hematopathology, and the European Association for Haematopathology were asked to complete a survey with the online link also posted externally. The CAP promoted the online survey with a news brief in CAP Today (a monthly trade periodical published by CAP), posts to social media outlets, the CAPconnect blog (a member peer network), and an announcement on the CAP Web site. The survey was open June 8, 2015 through July 24, 2015. Since the survey was disseminated broadly, the number of people made aware of the survey's existence cannot be determined. A total of 294 completed surveys were received, and 36 were excluded owing to survey abandonment. Survey results from 258 respondents were tabulated as the baseline data for AL testing practices and published by George et al.8  A follow-up survey is planned for 18 to 24 months after the AL LPG publication to compare to the baseline results and to evaluate the adoption of the 2017 AL LPG.

The CAP strived to maximize the response rates for both the IHC VAL and AL surveys by distributing flyers in PT products before the survey, writing an article in CAP Today, and promoting on listserv emails; however, the survey did not include questions about how participants learned about the questionnaire. Thus, the effects of the promotional efforts could not be captured.

Table 1 illustrates the estimated and actual response rates for the IHC VAL and AL projects. We initially estimated a 65% response rate for the IHC VAL surveys based on previous experience. Of the 3512 surveys sent, 1624 (46%) were completed and returned. These results included 1539 of 3064 surveys (50%) from laboratories participating in the CAP PT programs and 85 of 448 surveys (19%) from the nonparticipant laboratories. The quantitative IHC VAL survey results were tabulated, analyzed for comparison, and published separately by Fitzgibbons et al.9  In addition, Stuart et al10  were able to publish new immunohistochemical validation benchmark data for laboratories.

Table 1

Estimated and Actual Response Rates

Estimated and Actual Response Rates
Estimated and Actual Response Rates

To have 40 completed telephone interviews, we estimated a need to contact 121 individuals based on 3 roles: IHC laboratory director, staff pathologist, and laboratory manager (estimated response rate, 33%). Interviewees were informed that the interview (1) was a joint CAP/CDC initiative, (2) would take approximately 15 minutes to complete, and (3) would address an evidence-based guideline, not a laboratory accreditation checklist. The call attempts exceeded 500 and we had to contact 298 to achieve 40 complete interviews (actual response rate of 13% based on 2 categories: pathologist and nonpathologist). Although all 85 nonparticipant laboratories were included in the sampling, none participated in the interview. To reach our goal of 24 focus group participants, we estimated that 200 laboratory professionals would need to be contacted to determine their availability, but after sending invitations to 747 unique individuals, only 19 agreed to participate. Of these 19 individuals, only 5 (0.7%; 4 pathologists and 1 laboratory manager) actually attended the in-person focus group sessions, and none were from the nonparticipant laboratories.

Regarding awareness of the IHC VAL LPG, two-thirds of survey respondents (691 of 1057) reported that they were aware of the recommendations before receiving the questionnaire, with most of the remaining laboratories planning to review them within the next 6 months.9  Telephone interviewees also reported awareness of its content. All interviewees suggested that the CAP should continue to make full efforts to keep awareness high, stating that current channels of notification and word of mouth are important. Further analysis demonstrated that 14 of the 20 pathologists (70%) and 7 of the 20 laboratory managers (35%) were aware of the LPG via direct CAP channels, while 11 of the 20 laboratory managers (55%) were informed of it by their pathologists. All focus group participants agreed that professional networking provided the widest LPG awareness and all participants encouraged electronic notifications and updates. The focus group members cited emails as the medium of preference, but notifications in proficiency testing packages and updates in CAP Today were also suggested. Several key qualitative results from the telephone interviews and focus group are summarized in Table 2.

Table 2

Summary of Immunohistochemistry Validation Guideline Implementation

Summary of Immunohistochemistry Validation Guideline Implementation
Summary of Immunohistochemistry Validation Guideline Implementation

As for the adoption rate, the survey revealed that some or all of the recommendations had been adopted by nearly 80% (550 of 688) of respondents with only 12 respondents (1.7%) stating they had no plans to adopt the recommendations unless required by their laboratory accreditor.9  More than half of laboratory respondents from the survey used or planned to use the guidelines prospectively for all new assays and for assay revalidations, with a minority of laboratories (110 of 687; 16%) stating they would use the recommendations to retrospectively validate existing assays.9  Comments from all but 2 of the interviews suggest that laboratories were both aware of the guideline and planned to comply.

Two pathologists out of the 40 completed interviews stated they initially tried to adopt the guideline, but ultimately decided it provided no useful value. Even though pathologists indicated they wanted a more comprehensive and prescriptive guideline, 55% (11 of 20) of all pathologists interviewed by telephone felt the guideline as it exists was useful. Eighty percent (16 of 20) of laboratory managers rated the current guideline as useful and indicated a positive effect was having more consistent written laboratory policies. Finding validation cases for rare antigens (367 of 682; 54%), the time and staff needed to run validations (319 of 682; 47%), and the additional expenses incurred (236 of 682; 35%) were the 3 difficulties cited most frequently by survey respondents.9  These barriers were also consistent across the telephone interviews and focus group conversations.

During the telephone interviews, the pathologists did not report substantive burdens in following the guideline, but instead emphasized that improved patient outcomes and better laboratory practices needed to be the main focus. In contrast, one-quarter (5 of 20) of the laboratory managers felt the guideline led to higher expenses, but declined to try to detail these impacts. When queried about staff impacts, almost half (9 of 20) of the laboratory managers saw the guideline as creating extra work for their staff. They felt this extra load was aggravated by the lack of guideline specificity and process detail. In the focus group session, one specific item that was cited was deciding when to use reference laboratories for a specific assay versus performing the validation to bring it in-house. Every interview found that the IHC VAL LPG needed more detail and specificity. Although the number of focus group participants was low, the IHC VAL LPG was accepted and adopted by the participants, several of whom commented on the motivation to comply with the recommendations because of the CAP's importance to the laboratory community and association with quality accreditation. They indicated that laboratory director awareness and education were deemed the most important factors to guideline adoption.

All CAP guidelines are freely available in both the online and print versions of Archives of Pathology & Laboratory Medicine. In addition, each guideline has a unique page with additional resources posted on the CAP Web site (IHC VAL: https://www.cap.org/protocols-and-guidelines/cap-guidelines/current-cap-guidelines/principles-of-analytic-validation-of-immunohistochemical-assays [accessed February 27, 2019]; AL: https://www.cap.org/protocols-and-guidelines/cap-guidelines/current-cap-guidelines/initial-diagnostic-workup-of-acute-leukemia [accessed February 27, 2019]). Scientometric data obtained from these sources are used to supplement direct survey data in order to gain a more comprehensive picture of the awareness of published guidelines and identify early indicators of potential guideline impact on laboratory practice. Guidelines are expected to be highly read and cited; our assessment confirms this expectation. To date, the IHC VAL LPG has been cited 83 times in publications based in 26 different countries,11  the pdf version of the guideline has been accessed more than 21 881 times (Allen Press, Inc, email communication, October 12, 2018), and the guideline Web site has been accessed more than 11 713 times.12  The AL LPG, published more recently, has been cited 13 times in publications based in 8 different counties,11  the pdf version has been accessed more than 13 583 times (Allen Press, Inc., email communication, October 12, 2018), and the guideline Web site page has been viewed more than 3998 times.12  Of note, the entire body of 17 CAP-developed guidelines are among the most highly accessed pdfs for the Archives, accessed more than 244 000 times as of this writing (Allen Press, Inc, email communication, October 12, 2018), and they have been cited more than 11 217 times in journals based in more than 115 different countries.11  This confirms that our observations regarding the AL LPG and IHC VAL LPG are not unique phenomena. Long-term monitoring of these and other guideline impact framework indicators will help provide a broader view of guideline awareness and adoption, which is a useful supplement to essential direct survey data.

This investigation revealed the effectiveness and limitations of written surveys, telephone interviews, and focus group sessions on measuring the adoption of LPG dissemination; it also led to a better understanding of key factors related to the uptake of the IHC VAL guideline. The modalities were evaluated for sustainability in measuring the adoption beyond the cooperative agreement timeframe.

In our experience, written surveys, which were available in both paper and electronic format, showed the best ease of use for both developers and participants, a good response rate, and quantitative results that were readily analyzed and interpreted. By using the same questions in both the baseline and follow-up surveys, we were able to measure differences and draw more accurate comparisons about the differences found after IHC VAL LPG publication. Although precomparisons and postcomparisons of LPG adoption are not yet available for the AL guideline, we intend to use the abovementioned strategy for the follow-up survey.

The telephone interviews, conducted by the contracted consultant at a cost of $30,000, proved to be more time-consuming and difficult to conduct than the written surveys. The CAP database, from which the telephone interview participants were derived, contained several contacts for various responsibilities within an institution, and it was challenging to find the correct individual who also had time and willingness to discuss the nuances of implementing new guideline recommendations into the IHC laboratory. Interviewees were immediately asked if they had the required time and an inclination to provide their feedback. It turned out that many individuals declined the interview because they felt the time investment was too onerous for what they deemed to be simple “market research.” At times, there was a perception of intrusion by the independent interviewer and resistance in answering the questions; however, the feedback that was obtained was valuable as described in the Results section. None of the interviewees were willing to respond to open-ended questions that required them to give what they deemed “extra” feedback. They all, however, gave willing and concise answers to questions that asked for easily defined specific responses, and interviewees pointed out that they saw nothing that needed to be removed from the guideline. Based on this experience from the IHC VAL guideline it would be preferable to have preselected laboratories willing to participate in telephone interviews rather than random sampling for the AL guideline.

The focus group session was the most complicated modality to execute. To reach the intended audience of both pathologists and nonpathologists, we wanted to host this event at a national meeting attended by both; however, it was difficult to arrange the session at a time that did not interfere with educational activities. An artifact of this study was OMB prohibition against the use of small incentives (eg, gift cards or vouchers) to increase participation; this restriction will not apply to future CAP activities. We also did not have access to the attendee list, hence hundreds of invitations were sent out with a small return. Nonetheless, the focus group was held and the format was useful in that the dialogue and interaction allowed for suggestions for improvements in communication, resulting in an idea for an LPG-implementation tool that we had not previously considered. As with the telephone interviews, it might be better to have preselected laboratories and/or perform the focus group sessions under different circumstances to increase the effectiveness of the adoption analysis for the AL guideline.

Most participants from all 3 modalities indicated knowledge of the IHC VAL guideline's existence. To achieve this level of awareness, the CAP uses a variety of communication channels when an LPG is released including a targeted strategy toward CAP pathologist members, comprised of email blasts and educational courses, in addition to general audience channels such as press releases, CAP Today articles, and letters of announcement to targeted professional societies. In 2015, the CAP began distributing infographics (graphic visual representations of information, data, or knowledge intended to present information quickly and clearly) and increasing its social media tactics for subsequent LPG releases in addition to tracking communication channel uptakes such as press releases and social media. Participants confirmed that these approaches were effective and we believe this comprehensive communication strategy will continue to positively impact the perception of CAP-developed LPGs. Moreover, peer-to-peer promotion and a sense of professional benchmarking seem to be especially effective for CAP LPGs. Indeed, this knowledge of guideline update and awareness is confirmed by the scientometric data, which showed that both the IHC and AL LPGs were highly cited and accessed.

Most participants indicated that they already had adopted or planned to adopt all or most of the recommendations, but some interviewees seemed to conflate LPG recommendations with requirements in the CAP Laboratory Accreditation Program (LAP). About half of the interviewees expressed concern with being compliant with CAP “requirements,” suggesting that at least some interviewees did not understand that these LPGs are voluntary. The CAP LAP program accredits US laboratories performing IHC and could be perceived as a major facilitator to guideline adoption. As the CAP accredits more than 8000 laboratories in the United States and internationally, CAP-accredited laboratories may be more likely to adopt CAP-produced LPG recommendations. Failure to distinguish LAP accreditation requirements from guideline recommendations by the laboratory audience could also be considered a barrier among non–CAP-affiliated laboratories, since the LPG recommendations may not otherwise be mandated by these laboratories' accrediting agency.

The focus group participants claimed that other clinical organizations, such as the American College of Surgeons Commission on Cancer, are extremely effective in taking a “top down” approach to notification and updates. Participants reported that adoption was primarily influenced by laboratory directors and pathologists. They felt that credible related sources, including the CAP Cancer Protocol Templates, would enhance adoption and facilitate buy-in from administrators. Algorithms, templates, and teaching materials are needed to assist the laboratories, as participants indicated more prescriptive recommendations would be useful. The CAP began incorporating such items in subsequent published guidelines such as the “HPV Testing in Head and Neck Carcinomas.”13  In addition, all of the focus group participants were unaware of the toolkit materials on the CAP Web site, and lack of ease in navigating the Web site was also indicated. As a result of this feedback, an improved Web site design with a new header of “Protocols and Guidelines” was released in mid-2017.

The survey participants, telephone interviewees, and focus group attendees all provided specific feedback (eg, make recommendations more prescriptive and procedurally comprehensive), which will be reviewed when the IHC VAL guideline is formally updated (currently in progress). As indicated in CDC's paradigm model, the data collection and analysis (Step 4) directly informs the creation/revision process (Step 2) and will be used in future CAP LPGs.

There are many challenges when attempting to measure the translation of guideline recommendations into laboratory practices. Indeed, others in the academic and research world are faced with a similar dilemma when striving to gauge the community and societal value of their work, and numerous research impact frameworks have been developed to support these efforts.1417  The CAP has developed a similar framework to supplement information obtained from direct surveys and standardize its assessment of guideline project outputs. The 5 domains used in the Becker Model were selected as the foundation for the CAP Center Guideline Impact Framework,17  but indicators from all included framework models were reviewed.1417  Relevant indicators were selected, some indicators were modified, and additional indicators were added to account for our pathology and laboratory environment. Our 5 domains (advancement of knowledge; clinical implementation; community benefit; legislation and policy; and economic benefit) range from traditional bibliometric data to broader health and societal indicators that may take years to identify and measure.18  We can use the supplemental scientometric data to confirm the high-level guideline awareness in the wider laboratory community.

Study Limitations/Considerations

There were several limitations for both the quantitative and qualitative analyses. With both types of investigations, there could be nonresponse/selection bias due to lack of compliance or disagreement with the guidelines. For the IHC VAL 2015 survey, we acknowledge there was selection bias due to sampling that primarily included laboratories enrolled in one of CAP's PT surveys. To address this, the sample pool was supplemented with non–CAP-accredited laboratories, and statistical analysis was used to help adjust for these different groups. In addition, since CAP participating laboratories more commonly responded to the written survey, compared to nonparticipating laboratories (46% versus 19%), there is potential bias, since CAP accredited laboratories may be more focused on optimizing and harmonizing practices to adopt guideline recommendations.

For the AL baseline practices survey, we could not adjust or control for any laboratory demographic characteristics or duplicate responses owing to the electronic survey access on multiple listserves and Web sites. In addition, the follow-up data assessing the adoption of the AL LPG are not yet available. As such, we cannot yet assess the broader applicability of the findings we observed surrounding the IHC LPG on other guidelines.

There were also some considerations for the qualitative IHC VAL research. A high percentage of laboratory contacts declined to participate in the telephone interviews, thus we had to increase the sample size. For the IHC focus groups, the initial study design accounted for the different nonpathologist and pathologist roles, but the small sample size of 5 participants limited the interpretive and inferential value of the findings. Also, the low percentage of participants for our telephone interviews and focus groups, while providing an opportunity to obtain valuable qualitative information, introduced the substantial additional risk of limited sample bias. That is, feedback received during these small group sessions may not be generalizable to the larger laboratory community.

Of the various modalities of LPG evaluation outlined above, written practice surveys (paper and electronic) were the most effective method to evaluate LPG awareness, adoption, and effectiveness owing to their higher response rate. Although telephone interviews and focus groups provide opportunities to examine peripheral issues of guideline implementation not addressed in the written surveys, they are labor intensive, costly, and generally inefficient methods for acquiring useful feedback owing to the lack of laboratorian participation. Maintaining the uniformity in survey format and questions is essential for generating valid comparisons, acknowledging that no survey can anticipate novel and evolving practices. On the other hand, qualitative analyses can reveal unexpected findings that surveys can miss. For example, the follow-up telephone interviews and focus group discussions revealed some confusion between LPG recommendations and CAP LAP requirements. The larger number of written survey participants provided the opportunity to reach a more diverse population and gain a more accurate, data-driven picture of existing gaps in LPG awareness, adoption, and effectiveness.

The authors would like to acknowledge Ronald Brooks, MBA, contracted consultant, for his work with the telephone interviews and focus group sessions, and Carol Colasacco, MLIS, medical librarian specialist, for her work in preparing the manuscript. The authors would also like to thank Nicole Thomas, MPH, and Daniel Arber, MD, Tracy George, MD, Lauren Stuart, MD, MBA, and Joseph Tworek, MD, for their engaged participation on the “Guidelines Metric Expert Panel” (GMEP).

1
US Department of Health and Human Services
.
Clinical laboratory improvement amendments of 1988: final rule
.
Fed Regist
.
1992
;
57
(
40
):
7001
7186
.
Codified at 42 CFR §1405-494
.
2
IOM (Institute of Medicine)
.
Clinical Practice Guidelines We Can Trust
.
Washington, DC
:
National Academies Press;
2011
.
3
Cabana
MD
,
Rand
CS
,
Powe
NR
, et al.
Why don't physicians follow clinical practice guidelines: a framework for improvement
.
JAMA
.
1999
;
282
(
15
):
1458
1465
.
4
Fitzgibbons
PL
,
Bradley
LA
,
Fatheree
LA
, et al.
Principles of analytic validation of immunohistochemical assays: guideline from the College of American Pathologists Pathology and Laboratory Quality Center
.
Arch Pathol Lab Med
.
2014
;
138
(
11
):
1432
1443
.
5
Hardy
LB
,
Fitzgibbons
PL
,
Goldsmith
JD
, et al.
Immunohistochemistry validation procedures and practices: a College of American Pathologists survey of 727 laboratories
.
Arch Pathol Lab Med
.
2013
;
137
(
1
):
19
25
.
6
Arber
DA
,
Borowitz
MJ
,
Cessna
M
, et al.
Initial diagnostic workup of acute leukemia: guideline from the College of American Pathologists and the American Society of Hematology
.
Arch Pathol Lab Med
.
2017
;
141
(
10
):
1342
1393
.
7
Healthy People 2020
.
Centers for Disease Control and Prevention Web site
.
https://www.cdc.gov/nchs/healthy_people/hp2020.htm. Updated October 14, 2011. Accessed
December
19,
2018
.
8
George
TI
,
Tworek
JA
,
Thomas
NE
, et al.
Evaluation of testing of acute leukemia samples: survey result from the College of American Pathologists
.
Arch Pathol Lab Med
.
2017
;
141
(
8
):
1101
1106
.
9
Fitzgibbons
PL
,
Goldsmith
JD
,
Souers
RJ
, et al.
Analytic validation of immunohistochemical assays: a comparison of laboratory practices before and after introduction of an evidence-based guideline
.
Arch Pathol Lab Med
.
2017
;
141
(
9
):
1247
1254
.
10
Stuart
LN
,
Volmar
KE
,
Nowak
JA
, et al.
Analytic validation of immunohistochemistry assays: new benchmark data from a survey of 1085 laboratories
.
Arch Pathol Lab Med
.
2017
;
141
(
9
):
1255
1261
.
11
Scopus [database online]
.
Amsterdam, The Netherlands: Elsevier B.V.; 2017
.
December
12,
2018
.
12
Google
.
Google Analytics Web site
.
December
12,
2018
.
13
Lewis
JS
Jr
,
Beadle
B
,
Bishop
JA
, et al.
Human papillomavirus testing in head and neck carcinomas: guideline from the College of American Pathologists
.
Arch Pathol Lab Med
.
2018
;
142
(
5
):
559
597
.
14
Luke
DA
,
Sarli
CC
,
Suiter
AM
, et al.
The translational science benefits model: a new framework for assessing the health and societal benefits of clinical and translational sciences
.
Clin Transl Sci
.
2018
;
11
(
1
):
77
84
.
15
Panel on Return on Investment in Health Research
.
Making an Impact: A Preferred Framework and Indicators to Measure Returns on Investment in Health Research
.
Ottawa, Ontario, Canada
:
Canadian Academy of Health Sciences
;
2009
.
16
Description of the science impact framework
.
Centers for Disease Control and Prevention Web site
.
December
19,
2018
.
17
Sarli
CC
,
Dubinsky
EK
,
Holmes
KL
.
Beyond citation analysis: a model for assessment of research impact
.
J Med Libr Assoc
.
2010
;
98
(
1
):
17
23
.
18
Colasacco
C.
Adapting research impact assessment to laboratory practice guideline development projects
.
Poster presented at
:
Medical Library Association Annual Meeting; May 18–23
,
2018
;
Atlanta, GA
.

Author notes

The survey development, analysis, and this publication were supported by Cooperative Agreement NU47OE000057-04, funded by the US Centers for Disease Control and Prevention (CDC). The cooperative agreement with CDC required preapproval of the survey instrument by the US Office of Management and Budget (OMB No. 0920-1067). The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the CDC.

Supplemental digital content is available for this article at www.archivesofpathology.org in the January 2020 table of contents.

Mss Fatheree and Souers are employees of the College of American Pathologists. The authors have no other relevant financial interest in the products or companies described in this article.

Supplementary data