Laboratories must demonstrate analytic validity before any test can be used clinically, but studies have shown inconsistent practices in immunohistochemical assay validation.
To assess changes in immunohistochemistry analytic validation practices after publication of an evidence-based laboratory practice guideline.
A survey on current immunohistochemistry assay validation practices and on the awareness and adoption of a recently published guideline was sent to subscribers enrolled in one of 3 relevant College of American Pathologists proficiency testing programs and to additional nonsubscribing laboratories that perform immunohistochemical testing. The results were compared with an earlier survey of validation practices.
Analysis was based on responses from 1085 laboratories that perform immunohistochemical staining. Of 1057 responses, 65.4% (691) were aware of the guideline recommendations before this survey was sent and 79.9% (550 of 688) of those have already adopted some or all of the recommendations. Compared with the 2010 survey, a significant number of laboratories now have written validation procedures for both predictive and nonpredictive marker assays and specifications for the minimum numbers of cases needed for validation. There was also significant improvement in compliance with validation requirements, with 99% (100 of 102) having validated their most recently introduced predictive marker assay, compared with 74.9% (326 of 435) in 2010. The difficulty in finding validation cases for rare antigens and resource limitations were cited as the biggest challenges in implementing the guideline.
Dissemination of the 2014 evidence-based guideline validation practices had a positive impact on laboratory performance; some or all of the recommendations have been adopted by nearly 80% of respondents.
The College of American Pathologists (CAP) Pathology and Laboratory Quality Center (the CAP Center) develops and maintains evidence-based laboratory practice guidelines (LPGs) that follow the US Institute of Medicine's Clinical Practice Guidelines We Can Trust standards.1 In 2014, the CAP Center published a formal evidence-based guideline on analytic validation of immunohistochemical (IHC) assays.2 This LPG was based on a full systematic literature review by a professional methodologist rating the quality of evidence and strength of recommendations.
Also see p. 1255.
Intrinsic to the creation of practice guidelines is the need to establish appropriate metrics to measure their impact on clinical practice and identify guideline deficiencies and challenges to implementation. In fall 2013, the CAP Center was a recipient of a cooperative agreement from the US Centers for Disease Control and Prevention (CDC) to study and improve the impact of LPGs. The intent of this 5-year project—Improving the Impact of Laboratory Practice Guidelines: A New Paradigm for Metrics—is to identify gaps in awareness and uptake of LPGs and to develop metrics related to guideline implementation. One of the LPGs selected for this project was the IHC assay validation guideline.2 This particular LPG was selected because a survey of IHC assay validation practices had been done in 2010,3 before the guideline was developed, and provided a baseline for comparison. The 2010 survey documented significant interlaboratory variation in general validation practices and incomplete understanding of requirements for analytic validation. At the time this earlier survey was completed, evidence-based guidelines that addressed assay validation existed only for HER2 and hormone receptor assays.4–6 As a result, the CAP Center appointed a panel of IHC experts to systematically review published data and develop the evidence-based guideline for analytic assay validation.2
As part of the CDC agreement, the CAP/CDC Guideline Metrics Expert Panel created a new survey in 2015 to assess general awareness of the 2014 IHC validation LPG and determine what practice changes, if any, occurred following its publication. For meaningful comparison with the 2010 survey, the 2015 survey included many of the same questions. Analysis of these data allows an assessment of guideline implementation and effectiveness.
This manuscript represents a comparison of the 2010 and 2015 surveys, summarizes changes in laboratory practice for validation and revalidation of IHC assays, and represents the first study that directly assesses the impact of a CAP Center guideline on clinical practice. The CAP/CDC survey also included a number of new questions on IHC validation practices. These new benchmark data are the subject of a companion paper by Stuart et al.7
METHODS
In the latter half of 2015, the CAP distributed a survey of IHC validation practices and procedures to laboratories enrolled in the following CAP proficiency testing (PT) programs: the CAP/National Society for Histotechnology HistoQIP program, the Performance Improvement Program in Surgical Pathology, and the HER2 Immunohistochemistry Program. Not all subscribers of the HER2 program perform IHC staining. Laboratories that interpret HER2 slides that have been stained in another laboratory also participate in the HER2 program and were therefore included in the initial survey distribution, but their responses were excluded from the analysis. The same survey was also mailed to a selection of laboratories identified by Centers for Medicare & Medicaid Services Part B reimbursement claims for IHC testing; these Centers for Medicare & Medicaid Services–identified laboratories were not enrolled in any of the CAP PT programs. Laboratory accreditation status was not a factor in distribution of the survey or analysis of the results. The survey development, analysis, and this publication were supported by Cooperative Agreement NU47OE000057-04, funded by the US CDC. The cooperative agreement with CDC required preapproval of the survey instrument by the US Office of Management and Budget (OMB No. 0920-1067).
The survey contained 21 questions about validation policies and practices, guideline awareness and adoption status, and demographic factors. As noted above, many of the questions were essentially the same as those included in the 2010 survey. The survey questions did not apply to HER2 or hormone receptor assays, as separate guidelines for those markers had already been established. Survey answers included single-choice (yes, no, or unsure), multiple-choice, and numerical responses.
Because many of the laboratories participated in more than one of the CAP PT programs, duplicate responses were evaluated and only the single most complete response from each laboratory was included in the study. Data were also excluded from 74 laboratories that returned incomplete surveys. Results were analyzed to determine if and how validation practices had changed since the 2010 survey. For some responses, the results were stratified by laboratory size, as measured by surgical pathology accession volume.
Differences between the 2010 and 2015 surveys were analyzed by χ2 and Wilcoxon rank sum tests as appropriate. Statistical analysis of guideline awareness and adoption by laboratory volume and institution type used a multivariate regression model after adjusting volume to an ordinal format. The significance level was .05. Statistical analysis was done using SAS 9.3 (SAS Institute Inc, Cary, North Carolina).
RESULTS
Of the 3512 survey mailings, a total of 1624 completed surveys were available for analysis; this included 1539 of 3064 responses (50%) from laboratories participating in the CAP PT programs and 85 of 448 responses (19%) from the non-CAP PT laboratories. One hundred eighty-one of 1624 surveys (11%) were received from non-US laboratories. Analysis was conducted from 1085 respondents that indicated they performed IHC staining.
Tables 1 through 9 compare results of the 2010 and 2015 surveys. Table 1 demonstrates laboratory volumes with respect to number of antibodies in use at the time of the surveys, the number of new antibodies introduced in the year prior to the survey, and the number of surgical pathology accessions. There were no significant differences for any of these volumes between the 2010 and 2015 respondents.
Laboratory Has Written Procedure Outlining Steps Needed for Analytic Validation of New Assays
In 2015, laboratories were significantly more likely to have written validation procedures for both nonpredictive and predictive marker assays than in 2010 (Table 2). More than one-quarter of laboratories (28.4%; 206 of 726) reported not having a validation procedure for nonpredictive IHC assays in 2010, but this dropped to 14.3% (154 of 1077) in 2015. Similarly, the number of laboratories without a separate validation procedure for predictive markers dropped from 47.9% (312 of 651) in 2010 to 20.9% (225 of 1077) in 2015.
Tables 3 and 4 show that written procedures specifying the overall minimum number of cases needed for validation significantly increased for both nonpredictive (55.0% to 92.3%; 264 of 480 and 720 of 780, respectively) and predictive assays (66.3% to 93.4%; 195 of 294 and 584 of 625, respectively). Interestingly, having a specified number of positive and negative cases for validation did not significantly differ between 2010 and 2015 for either type of assay. Positive and negative minimum concordance rates were specified more often in 2015 for nonpredictive marker assays (Table 3), but for predictive marker assays, only negative concordance rates showed improvement from 2010 (Table 4).
Tables 5 and 6 show data on the minimum number of cases used for assay validation. Compared with 2010, the median numbers of cases needed for validation were the same or higher in all 2015 categories, but the ranges tended to narrow, with higher minimum and lower maximum numbers needed for all assay types.
Compliance with guideline recommendations to include specifications for validating IHC assays in cytologic specimens was improved in 2015 (Table 7), but only for nonpredictive markers. There was a slight decrease in the percentage of laboratories having written specifications for validating predictive IHC assays in cytology specimens in 2015, but this difference was not significant. Decalcification was not addressed in the 2010 survey, so there were no baseline data for comparison.
The 2015 survey asked laboratories if their procedures required assay revalidation for any of 12 specific changes in the conditions of testing. Six of these changes were also specifically addressed in 2010 and are included in Table 8. There was significant improvement in compliance with guideline recommendations for 4 of the 6 changes for nonpredictive assays, but for predictive assays, only introduction of a new antibody lot showed improvement in compliance.
Table 9 shows whether laboratories performed a validation study for their most recently introduced assay. There was dramatic improvement in compliance with validation recommendations from 2010, with 95% (735 of 776) of laboratories reporting validation of their latest nonpredictive assay and 99% (101 of 102) reporting validation of their latest predictive assay.
Tables 10 and 11 show the results of 4 survey questions that addressed participants' awareness of the recommendations in the IHC validation guideline and their plans for adoption. Two-thirds of respondents (691 of 1057) reported that they were aware of the guideline recommendations prior to this survey, and the majority of the remaining laboratories planned to review them within the next 6 months (Table 10). Some or all of the recommendations had been adopted by nearly 80% (550 of 688) of respondents; only 12 respondents (1.7%) stated they had no plans to adopt the recommendations unless required by their laboratory accreditor. More than half of laboratories used or planned to use the guidelines prospectively for all new assays and for assay revalidations. A minority of laboratories (16%; 110 of 687) stated they would use the recommendations to retrospectively validate existing assays. Finding validation cases for rare antigens, the time and staff needed to run validations, and the additional expenses incurred were the 3 most frequently cited difficulties in guideline adoption.
Multivariate analysis was used to test for demographic and practice characteristics associated with guideline awareness and adoption practices. Laboratory accession volume was significantly associated (P < .001) with guideline awareness, as high-volume laboratories reported greater awareness of the guideline than those with low test volumes (77.3% [102 of 132] for laboratories with more than 50 000 surgical specimens per year versus 58.5% [67 of 114] for laboratories with ≤5000); adoption rates did not vary by volume (Table 11). Compared with smaller laboratories, those with higher volumes were almost twice as likely to cite difficulties in validating rare antigen assays.
The awareness/adoption results were also stratified by location within or outside the United States. About two-thirds (68.0%; 597 of 878) of domestic laboratories were aware of the guidelines before the survey, compared with 52.5% (94 of 179) of international laboratories, but the percentage of respondents who reported having implemented all or some of the recommendations was almost identical (80.2% [477 of 595] versus 78.5% [73 of 93, respectively]). Interestingly, US laboratories were more likely than non-US laboratories to cite resource limitations as challenges to adoption: insufficient time/staff to run validations and additional cost/expense were cited by 49.5% (292) and 36.1% (213) of 590 US laboratories, respectively, as 2 of the 3 challenges cited most often, compared with 29.3% (27) and 25.0% (23) of 92 non-US laboratories, respectively.
Awareness and adoption were also stratified by CAP PT versus non-CAP PT laboratories (data not shown). The results were very similar, except non-CAP PT laboratories were much less concerned about validating assays in decalcified specimens (22.4% [147 of 656] of CAP laboratories versus 7.7% [2 of 26] of non-CAP laboratories) and cytology specimens (20.6% [135 of 656] versus 3.8% [1 of 26]) and were less likely to use the guidelines for validating predictive marker assays (75.6% [500 of 661] versus 50.0% [13 of 26]).
DISCUSSION
Evidence-based LPGs can advance the practice of laboratory medicine by promoting the most effective testing practices to achieve consistent, high-quality results. The authors of an earlier study of the extent of implementation of evidence-based guidelines concluded that guideline adoption is likely to be improved if the evidence is strong and the guideline is clear and is supported and disseminated by professional societies.8 The CAP Center has developed 10 guidelines, but until now has not systematically assessed their effectiveness or impact on pathology practice. By comparing the results of the 2010 and 2015 surveys, we have shown that the 2014 CAP Center guideline on analytic validation of IHC assays has had positive impact on clinical practice. A significantly higher number of laboratories now have written procedures for the analytic validation of both predictive and nonpredictive markers, and, compared with results obtained in 2010, a significantly higher number also have validated their most recently introduced predictive and nonpredictive assays. Additionally, details of validation procedures, including validation set composition and mandated concordance rates, are increasingly being included as standard operating procedures in immunohistochemistry laboratories.
In 2015, laboratories specified a higher minimum number of validation cases for both predictive and nonpredictive assays, but they also reported lower maximum numbers. This convergence may reflect growing familiarity with validation recommendations and the realization that appropriate assay validation does not require an excessive number of cases to ensure that the assay performs as expected.
Validation of IHC assays on alternatively fixed tissues, best exemplified by cytology specimens, has been a perpetually difficult area in the immunohistochemistry laboratory. A previous survey of 818 cytology laboratories found 12 different cytology specimen types used for IHC staining, and only 4 of 323 respondents reported validating a nonformalin fixative for HER2 testing on cytology samples.8 Despite this, increased numbers of laboratories reported having procedures for assay validation on cytologic specimens for nonpredictive assays. However, the proportion of laboratories with validation procedures for predictive markers on cytology specimens is unchanged compared with 2010. Performing a robust validation for predictive markers on cytology specimens is challenging for most laboratories because of the need to obtain sufficient numbers of appropriate validation specimens and variation in fixation methods used. Unfortunately, analogous data are not available for decalcified tissues because questions regarding validation practices on decalcified tissues were not included in the 2010 questionnaire; however, new benchmark data are now available.9
Changes in testing conditions that require assay revalidation were specifically addressed in the validation guideline. The follow-up data presented here show that an increasing number of laboratories now have procedures specifying what changes in assay conditions mandate assay revalidation. The increased recognition that shifts in assay conditions can alter results has led laboratories to improve their processes to assure accurate results when assay conditions change.
An additional component of the survey was intended to assess general awareness of the guideline, the state of laboratories' adoption of the recommendations, and challenges to guideline implementation. A significant majority of respondents were aware of the guideline's existence and are currently following its recommendations. A number of impediments were identified that inhibit guideline adoption. Not surprisingly, the 2 most commonly identified items were difficulties in obtaining cases for validation of rare antigens and the lack of resources available to perform validation procedures.
In our survey, larger laboratories were more likely than smaller laboratories to cite difficulties in validating rare antigen assays, despite the fact that higher-volume laboratories would be expected to have greater access to relevant specimens. This is probably because smaller laboratories are less likely than larger laboratories to include such assays in their test menu and therefore have no need to identify such cases. The procurement of tissues that are rarely positive for a particular assay is a well-known challenge and the bane of the IHC medical director's existence. This fact underscores the need for having alternative means for validating assays where it is difficult to obtain appropriate validation tissues. This could involve sharing rare specimens between laboratories. Other possibilities include xenograft tissues, cell cultures that contain a uniform amount of antigen, and, possibly, synthetic materials on which uniform amounts of extracellular antigen may be affixed.
This study contains flaws intrinsic to any survey-based data collection method. Possible biases include results skewed by a nonrandom sample and the lack of participation biased against potential respondents who are unaware of, noncompliant with, or opposed to the LPG. There were also differences in the 2 study populations: only subscribers to the HER2 PT program were surveyed in 2010, whereas the 2015 survey included participants in other PT programs besides HER2; however, the survey cohorts were similar in regard to institution type, antibodies in use, and test volume. The PT program difference would not be expected to represent a major bias as those laboratories have equal requirements for assay validation. Those who didn't respond to a question or to the survey may be less likely to have been aware of or compliant with guideline recommendations. Additionally, some questions were not asked in the 2010 survey that in retrospect would have been relevant. Examples include questions regarding decalcified tissues and revalidation processes upon change of antibody clone.
It is important to point out that the changes found in the survey cannot be attributed entirely to the LPG. Laboratories continually update their policies and procedures based on information from many sources, including laboratory accreditation. It is likely that some of the improvements noted here are not directly related to the guideline's publication. For example, Table 2 shows that, compared with the first survey, many laboratories had adopted written procedures, yet the numbers of respondents who reported positively for nonpredictive (866) and predictive (795) markers in the 2010 survey exceeded the number of respondents who were aware of the LPG (691). There may be several explanations, including that the respondents to the 2015 survey were not aware that someone else in the laboratory had been aware of the LPG and based the new requirement for having a procedure manual on it, or that it had influenced some other individual or group who secondarily influenced the laboratory director. Although attribution of motivations in a survey is difficult, these data show that the quality of laboratory practice has been significantly improved following the creation and dissemination of an LPG; specifically, some or all of the recommendations have been adopted by nearly 80% (550) of 688 respondents. To further address the study findings, we are conducting both in-person and telephone focus groups. Additional studies will add to our knowledge of the effectiveness of evidence-based guidelines and should continue to corroborate the hypothesis that evidence-based guidelines improve the quality of patient care.
This work was supported by Cooperative Agreement No. 1U47OE000057 from the Centers for Disease Control and Prevention.