A cooperative agreement between the College of American Pathologists (CAP) and the United States Centers for Disease Control and Prevention was undertaken to measure laboratories' awareness and implementation of an evidence-based laboratory practice guideline (LPG) on immunohistochemical (IHC) validation practices published in 2014.
To establish new benchmark data on IHC laboratory practices.
A 2015 survey on IHC assay validation practices was sent to laboratories subscribed to specific CAP proficiency testing programs and to additional nonsubscribing laboratories that perform IHC testing. Specific questions were designed to capture laboratory practices not addressed in a 2010 survey.
The analysis was based on responses from 1085 laboratories that perform IHC staining. Ninety-six percent (809 of 844) always documented validation of IHC assays. Sixty percent (648 of 1078) had separate procedures for predictive and nonpredictive markers, 42.7% (220 of 515) had procedures for laboratory-developed tests, 50% (349 of 697) had procedures for testing cytologic specimens, and 46.2% (363 of 785) had procedures for testing decalcified specimens. Minimum case numbers were specified by 85.9% (720 of 838) of laboratories for nonpredictive markers and 76% (584 of 768) for predictive markers. Median concordance requirements were 95% for both types. For initial validation, 75.4% (538 of 714) of laboratories adopted the 20-case minimum for nonpredictive markers and 45.9% (266 of 579) adopted the 40-case minimum for predictive markers as outlined in the 2014 LPG. The most common method for validation was correlation with morphology and expected results. Laboratories also reported which assay changes necessitated revalidation and their minimum case requirements.
Benchmark data on current IHC validation practices and procedures may help laboratories understand the issues and influence further refinement of LPG recommendations.
In 2015, the College of American Pathologists (CAP) conducted a survey of laboratory practices to examine the current state of immunohistochemical assay (IHC) validation procedures, assess general awareness of the CAP's 2014 evidence-based laboratory practice guideline (LPG) on IHC validation,1 and determine what changes, if any, occurred following its publication. The survey was developed as part of a cooperative agreement between the CAP and the US Centers for Disease Control and Prevention (CDC). As part of the collaboration between CAP and CDC, a Guideline Metrics Expert Panel was formed to oversee surveys regarding awareness and implementation of new and existing guidelines.
Also see p. 1247.
The 2015 survey included many of the same questions as those used in a 2010 survey of validation practices by Hardy et al,2 but also included new questions to collect data specifically related to the IHC validation guideline recommendations. As the IHC validation LPG was developed, gaps were identified in the literature when data were not available to describe current IHC validation practices; this included the issue of what circumstances required assay revalidation. Analysis of the survey results led to a before and after comparison of laboratory practices, which is addressed in a companion paper by Fitzgibbons et al,3 as well as new benchmark data from current laboratory practices that are described in this article. This article covers 3 broad aspects of IHC validation: laboratory policies on initial assay validation; revalidation practices; and the laboratory's actual practice during its most recent validation.
METHODS
In the latter half of 2015, the CAP distributed a survey of IHC validation practices and procedures to laboratories enrolled in 1 or more of the following CAP proficiency testing (PT) programs: the CAP/National Society for Histotechnology HistoQIP (HQIP-B), the Performance Improvement Program in Surgical Pathology (PIP-C), and the HER2 Immunohistochemistry Program (HER2-B). Not all subscribers of these programs perform IHC staining. Laboratories that interpret IHC slides that have been stained in another laboratory also participate in the programs and were therefore included in the survey distribution, but their responses were excluded from the analysis. The same survey was also mailed to a selection of laboratories identified by Centers for Medicare & Medicaid Services (CMS) Part B reimbursement claims that indicated they perform IHC testing; these CMS-identified laboratories were not enrolled in any of the abovementioned CAP PT programs. Laboratory accreditation status was not a factor in distribution of the survey or analysis of the results. The 2010 survey was sent to subscribers of CAP's HER2 program. Thus, some of the laboratories included in the current survey had also participated in the earlier survey.
The survey development, analysis, and this publication were supported by the Cooperative Agreement No. NU47OE000057-04, funded by the US CDC. The cooperative agreement with CDC required preapproval of the survey instrument by the US Office of Management and Budget (OMB No. 0920-1067).
The survey contained 21 questions that encompassed specific questions on validation policies and practices, guideline awareness and adoption status, and demographic factors. The survey questions specifically excluded HER2 and hormone receptor assays, as separate guidelines for those markers had already been established.4–7 Survey answers included yes/no/unsure format, multiple choice, and numerical responses. In addition to questions included in the earlier survey, the 2015 survey included a number of new questions prompted during the development of the IHC validation LPG.
Since many of the laboratories participated in more than 1 of the CAP PT programs, duplicate surveys were received and only the single most complete survey from each laboratory was included in the study. Results were also excluded from 74 laboratories that returned incomplete surveys. For some questions, the results were stratified by laboratory size (measured by surgical pathology accession volume) and marker type.
Differences between marker types for qualitative questions were analyzed by using χ2 and Fisher exact tests, and the Wilcoxon rank sum test was used for quantitative test differences. A significance level of .05 was used for the analysis. The survey results were summarized and analyzed with SAS 9.3 (SAS Institute Inc, Cary, North Carolina).
RESULTS
Of the 3512 survey mailings, a total of 1624 completed surveys were available for analysis; this included 1539 of 3064 responses (50%) from laboratories participating in the CAP PT programs and 85 of 448 responses (19%) from laboratories not enrolled in CAP PT. Most of the responses received were from US laboratories (domestic laboratories) but 181 non-US laboratories (international laboratories) also participated in the survey. Analysis was conducted on surveys from the 1085 respondents who indicated they perform IHC staining.
A range of personnel completed the survey with the most common being IHC supervisors (44.8%; 671), followed by MD or DO immunohistochemical laboratory directors (14.2%; 148), histotechnologists (11.4%; 119), staff pathologists (10.7%; 112), and department chair/laboratory medical directors (8.1%; 84); all others accounted for 10.8% (113 of the 1043) of respondents. Table 1 includes distribution volumes of the total number of antibodies, new antibodies introduced in 2014, and 2014 surgical pathology accessions from laboratories participating in the survey. The median number of IHC antibodies used by 1036 laboratories was 78 (5th–95th percentile range, 15–200); the median number of new antibodies introduced in 2014 by 1022 laboratories was 3 (5th-95th percentile range, 0–20); and the median surgical pathology accessions for 983 laboratories was 15 000 (5th–95th percentile range, 2800–79 568).
General Practices Reported for Initial Validation Procedures
Table 2 details the extent of written validation procedures for new IHC assays as reported by survey participants. More than 95% (809 of 844) reported always documenting the results of validation. Sixty percent (648 of 1078) reported having separate written procedures for predictive and nonpredictive assays, while 34% (366) did not. A lower proportion of laboratories had separate written procedures for verifying validation of unmodified US Food and Drug Administration (FDA)–approved assays (54.2%; 404 of 746) and laboratory-developed or laboratory-modified tests (42.7%; 220 of 515). A new question in this survey asked whether laboratories have separate validation procedures for decalcified and cytologic specimens, and approximately half of laboratories reported having validation procedures specific for these specimen types. For cytologic specimens, there was a significant shift in frequency from 37% (179 of 486) of laboratories responding affirmatively in 2010 to 47% (328 of 697) in 2015, but only for nonpredictive markers; there was no change for predictive markers over that same period (42% [126 of 298] in 2010 versus 40% [275 of 697] in 2015).
Laboratories also reported minimum requirements for initial validation of assays, as summarized in Tables 3 and 4. A minimum number of validation cases was specified by most laboratories (85.6% [720 of 838] for nonpredictive markers; 76.0% [584 of 768] for predictive markers) and most specified a minimum required number of positive and negative cases. However, only slightly more than half stipulated minimum concordance rates for either assay type. As reported by 714 laboratories, the median number of cases required for nonpredictive assays was 20 (5th–95th percentile range, 5–40), which is stated in the 2014 LPG for this assay type. The median number of cases required for predictive assays was 25 (5th–95th percentile range, 8–50; n = 579), which is below the LPG recommendation of 40. Table 4 also demonstrates the percentage of respondents that adopted the recommendation for initial IHC validations in terms of concordance rates (≥90% for positive concordance, negative concordance, and overall concordance) as specified in their procedures (93.2%, 93.7%, and 96.9%, respectively, for nonpredictive markers and 94.9%, 98.1%, and 97.0%, respectively, for predictive markers). In addition (data not shown in Table 4) roughly one-third of laboratories (32.2%, 35.4%, and 30.5% for positive, negative, and overall concordance, respectively) required 100% concordance for nonpredictive markers and a smaller proportion of laboratories (27.8%, 29.9%, and 23.8%, for positive, negative and overall concordance, respectively) required 100% concordance for predictive markers.
Testing Condition Changes Requiring Revalidation of Previously Validated Assays
For an existing, previously validated IHC assay, the LPG recommends having a written procedure that specifies when to reassess assay performance if there is a change in testing conditions.1 Having written procedures for assay revalidation was reported by 61.4% (607 of 988) of laboratories for both nonpredictive and predictive markers; 3.7% (37) reported having procedures for predictive markers only and 8.4% (83) for nonpredictive markers only. Sixteen percent (158) had no such procedure, and 10.4% (103) were unsure. Table 5 details which changes in testing conditions necessitate assay revalidation. The most common reasons included in the procedure for revalidating nonpredictive assays were change in antibody clone (92.0%; 613 of 666), followed by change in antigen detection system (86.4%; 572 of 662), change in antigen retrieval method (86.2%; 568 of 659), introduction of a new antibody lot (83.6%; 559 of 669), and change in antibody vendor (83.2%; 553 of 665). Roughly two-thirds of laboratories specified minimum numbers of cases needed for assay revalidation for specific changes (Table 6). Change in antibody clone required the most cases, with a median of 20 (5th–95th percentile range, 2–20; n = 388), while introduction of a new antibody lot required the fewest, with a median of 2 (5th–95th percentile range, 1–20; n = 342).
Revalidation of Previously Validated Assays: Changes Specified in Revalidation Procedures for Nonpredictive Markers
Revalidation of Previously Validated Assays: Minimum Number of Cases Specified for Nonpredictive Markers
Table 7 lists the changes in test conditions requiring revalidation of previously validated predictive markers. The most common conditions specified in the procedure for predictive markers were change in antibody clone (86.0%; 533 of 620), change in antigen detection system (82.6%; 511 of 619), change in antigen retrieval method (81.7%; 505 of 618), introduction of a new antibody lot (78.3%; 486 of 621), and change in antibody vendor (77.5%; 478 of 617). Table 8 summarizes the minimum numbers of cases needed for assay revalidation for specific changes, with most requirements listed as roughly double what was required for nonpredictive markers.
Initial Validations Reported From Most Recently Introduced Assays
Respondents also reported data from their most recently introduced IHC assay. The chronology of the most recent validations from 1019 laboratories was as follows: 4.1% (42) before 2013; 6.0% (61) in 2013; 21.6% (220) in 2014; 57.0% (581) in 2015; and 11.3% (115) were unsure. A validation study for the most recently introduced IHC assay was reported by 87.7% (902 of 1029) of laboratories; 3.5% (36) reported not having performed a validation study and 8.8% (91) were unsure. Table 9 demonstrates the methods laboratories used for their most recent validations. Correlation with morphology and expected results was the most common method for both 725 nonpredictive and 101 predictive markers (61.1% [443] and 46.5% [47], respectively). Direct comparison to previously validated assay results, performed either in-house or at an outside laboratory, was more common for predictive markers (50.4% [51] versus 34.7% [251]). The method validation differences between nonpredictive and predictive assay validation were statistically significant (P = .002, Fisher exact test). Table 10 presents the number of cases used in the most recent assay initial validation. The median case number for nonpredictive markers was 20 (5th–95th percentile range, 4–40; n = 685) and the median case number for predictive markers was 31 (5th–95th percentile range, 5–65; n = 97). The case number differences were statistically significant (P < .001, Wilcoxon rank sum test).
DISCUSSION
In conducting this survey, we gathered current practice data regarding laboratory policies and procedures on IHC assay validation. Information regarding changes in laboratory policies since 2010 is discussed in the companion article.3 This article describes issues covered in the 2015 survey that were not addressed in the earlier survey.2 Nearly 96% of laboratories documented validation procedures and 85% had written policies, 60% have separate written procedures for predictive and nonpredictive assays with 54.2% specifically addressing unmodified FDA-approved assays. Only 42.7% of laboratories also specifically address laboratory-developed or laboratory-modified tests. While this demonstrates increased attention to analytic validation of antibodies, some confusion may still remain regarding the need to have written procedures that distinguish between predictive and nonpredictive markers as well as FDA-approved/cleared tests and laboratory-developed tests. While it is not necessary to have separate procedures, differences in assay types must be addressed.
The responses to specific questions surrounding validation of IHC procedures for cytologic specimens were notable. In response to the question about having written procedures and specifications for validation of IHC tests for cytologic specimens, significantly more laboratories responded affirmatively as compared with 2010, but only for nonpredictive markers; there was no change for predictive markers over that same period. In view of the increasing utilization of predictive immunohistochemical markers for informing treatment options and the increasing reliance on cytologic specimens, one might anticipate growing motivation to validate these assays for cytologic specimens. The issue is somewhat obscured by what specimens are considered “cytologic.” Practically, in some departments, core biopsies are classified as cytology specimens despite being preserved and processed identically to tissue specimens. The protocols for “cell block” preparations are variable, encompassing cellular samples entrapped in clotted blood or centrifugally concentrated cellular specimens sometimes entrapped in a fibrin clot, all subsequently processed and embedded like tissue blocks. The matter is further complicated by the inclusion of various smear preparation methods (direct versus liquid based), alcohol fixation versus air-drying, and different preservatives and fixatives. It cannot be assumed that a procedure established and validated for one type of specimen would be equally valid for any other. Again, the issue becomes particularly important in the context of predictive markers.
The 2014 IHC validation LPG identified a number of changes in testing conditions that necessitate a reassessment of assay performance. When there is a change in antibody dilution, antibody vendor (same clone), or incubation or retrieval times, the guideline recommends confirming assay performance with at least 2 known positive and 2 known negative cases. If there is a change in fixative type, antigen retrieval method, antigen detection system, tissue processing or testing equipment, environmental conditions of testing, or laboratory water supply, the laboratory medical director is responsible for determining how many predictive and nonpredictive markers and how many positive and negative cases to test. Last, when an antibody clone is changed, the guideline recommends a full revalidation (equivalent to initial analytic validation). At the time of guideline development, there were insufficient data and evidence to support a defined recommendation and therefore the numbers offered to confirm performance were based on expert opinion and feedback from the open comment period. In this article, we present actual numbers of cases used to confirm performance for changes in testing condition including 6 situations not previously considered.
For predictive and nonpredictive antibodies, the median number of cases specified for revalidation met or exceeded the guideline statements in all scenarios, with the exception of a change in antibody clone for predictive markers. While the guideline recommends confirming predictive marker performance with 40 validation cases, the median number of reported cases was 25. Laboratories tended to test a higher number of cases when there was a change in antibody vendor for predictive markers (median = 10). Similar trends were identified for predictive markers when there was a change in antigen detection system, fixative type, or testing equipment. Although the guideline did not specify a minimum number of cases for these scenarios, laboratories reported using a substantially higher number of cases for predictive markers versus nonpredictive markers. These findings suggest that respondents generally recognize the increased clinical significance of predictive markers and design revalidation procedures commensurate with the increased importance of predictive assays.
The 2014 IHC LPG suggested methods used to validate tissues. In the current survey, 61.0% and 46.5% of laboratories correlated the new test's result with the morphology and expected results for nonpredictive and predictive assays, respectively. Seventeen percent and 26.7% of laboratories compared the new test's results with another laboratory for nonpredictive and predictive assays, respectively. Beyond HER2 assays, little was known regarding which methods were used to validate antibody assays. It is significant that a higher proportion of laboratories compare to another laboratory in the case of a predictive marker. This may be a more trusted method and implies greater rigor. This method is, however, dependent on the accuracy of the reference laboratory.
One-third (539) of laboratories responding to the 2015 survey indicated that they do not perform immunohistochemical staining. Pathologists in these laboratories interpret IHC slides that have been stained in another laboratory. This is a common practice, but its extent has not been well documented. It is important to recognize that the 2014 LPG focused exclusively on the technical aspects of immunohistochemical validation procedures and not on interpretation. While uniform, quality technical performance is an essential foundation for overall test performance, it does not, in itself, address interpretive elements. The need for appropriate and uniform interpretation of IHC assays, often in a quantitative manner, will only increase with the continued introduction of predictive biomarkers linked to specific therapies and appropriate education thereof.
In summary, this article offers benchmark data on current procedures and practices of IHC validation assays that were previously unknown. This information may help laboratories understand how other laboratories address issues of revalidation and also may help shape further refinement of LPG recommendations on IHC validation.
This work was supported by the Cooperative Agreement No. 1U47OE000057 from the Centers for Disease Control and Prevention (CDC).