Quantitative Imaging Analysis Fluorescence In Situ Hybridization Validation for Clinical HER2 Testing in Breast Cancer

HER2 FISH was performed on 4- to 5-μm–thick formalin-fixed, paraffin-embedded tissues using the US Food and Drug Administration–approved dual-color HER2 IQFISH probe (Dako, Carpinteria, California) as per manufacturer instructions. Each FISH slide was scanned manually in its entirety, and representative tumor nuclei were enumerated in each case by a technologist and reviewed by a pathologist with expertise in solid tumor FISH.

For QIA, slides were prescanned at ×4 for tissue matching with corresponding hematoxylin-eosin (H&E) or HER2 IHC slides. Regions of interest were selected and scanned at ×100 magnification using a PathFusion system (Applied Spectral Imaging, Carlsbad, California) available in our laboratory. Single filter images were taken at several different levels throughout the 4- to 5-μm thickness (z-stacking) and then combined into 1 image by the software. For digital counts, a minimum of 2 regions of tumor and 8 digital frames per slides were analyzed. PathFusion analysis performs nuclear segmentation, spot counting, and cell classification. Nuclear segmentation is the identification of nuclei and circling by the software. Spot counting involves counting of red and green signals that achieve a predetermined pixel-size threshold associated with the size of the probe being evaluated. Cell classification involves tabulation of the red and green signal spot counts and sorting them into predefined groups. The FISH images that are initially generated by the PathFusion system require review and manual editing by a trained technologist. Editing typically involves correcting automated classification errors due to overlapping nuclei, inaccurate nuclear segmentation, or erroneous inclusion of nontumor nuclei. Once the nuclei are edited from the FISH images, the classification scores are incorporated into a validation report.

HER2 FISH assay reference ranges recommended by the ASCO/CAP 2018 guideline²  for manual enumeration were used for digital enumeration. Because digital FISH enumeration uses z-stacking to mimic the manual focal adjustments made by the enumerator, validation of z-stacking was performed to: (1) help ensure that the z-stacking levels used by digital imaging capture hybridization signals through the entire depth of a nucleus and are similar to the number of signals seen on manual enumeration; (2) determine the number of cells that should be digitally enumerated to provide the most accurate result.

z-Stacking was set during development phase to 9 focal planes with a 0.7-μm interval between planes. This covers a 6.3-μm thickness, and given the 3- to 5-μm thickness specified by the HER2 IQFISH PharmaDx manufacturer’s instructions, should be sufficient to capture all signals in enumerated nuclei.

A minimum of 100 to 300 cells (range, 102–894) and 50 cells were scored on each slide digitally and manually, respectively, by 2 independent scorers on normal breast tissue. The scores were averaged and the 95th percentile calculated.

To standardize the results, cells/frames were deleted from right to left (to mimic the technologist stopping earlier), and results were recorded at 300, 200, 150, and 100 cell points for digital enumeration. Digital and manual scores were compared.

Current ASCO/CAP guideline and New York State regulations require enumeration of 20 to 40 cells and 50 cells, respectively, for manual HER2 FISH. A literature review for the optimal number of cells for enumeration for HER2 FISH showed that the average margins of error of HER2/CEP17 ratio and HER2 copy number were 0.40 and 0.53, respectively, when counting 20 cells. These decreased to 0.20 and 0.26 when counting 100 cells.⁷ 

We evaluated invasive tumor in 16 samples at different cell cutoffs similarly to the method used for the normal breast range samples to arrive at the decision of the number of cells to be enumerated digitally.

The software settings were optimized during the development phase, with adjustments for reoptimization done quarterly or upon changing of the light bulb or source.

The classifier was programmed to sort cells into 3 categories based on average copy numbers per cell: less than 4 HER2, 4 to 5.9 HER2, and 6.0 or more HER2 in the development workup in line with the classification groups of the ASCO/CAP 2018 guideline. For CEP17, the classifier binned nuclei as 0 (not present) or greater than 0 (present). Cells with no CEP17 or HER2 present were not enumerated. Halfway through the validation process, working with the instrument vendor, a new classifier was designed that sorted cells into individual ASCO/CAP group designations. All possible signal combinations up to 10 HER2 and/or 10 CEP17 were evaluated to verify that each cell was correctly classified. Then the new classification system was checked on 2 to 5 samples from each group to verify that the group designation did not change. This classifier was then used for the remainder of the validation and for clinical implementation.

The hardware and monitor quality necessary for digital review of samples was assessed to standardize across all users and for optimal image resolutions.

Accuracy of the PathFusion Imaging platform for use with HER2 FISH slides was established by comparison of results obtained using the digital QIA method to manual FISH scores. In the absence of the CAP guidelines for FISH QIA, the 2019 CAP guideline for QIA of HER2 IHC for breast cancer⁵  was extrapolated to FISH for the purpose of this validation.

A total of 168 cases were included in the study. A minimum of 40 clinical samples were selected from group 1 (n = 43), group 4 (n = 46), and group 5 (n = 42). Group 2 (n = 14) and group 3 (n = 23) tend to be uncommon, and all available cases were included for these groups. All cases were manually reviewed by a minimum of 2 reviewers and confirmed for group classification either before or after complementary HER2 IHC testing. Because the ASCO/CAP guidelines require equivocal HER2 FISH samples to have HER2 IHC performed and a second blinded FISH evaluation in the strongest circumferential membranous staining area of 2+ IHC staining, it was possible for samples to be assigned more than 1 group designation. For our validation, the ASCO/CAP group designation during the original manual H&E-guided enumeration was used. Digital FISH scans were performed in either tumor-enriched areas identified by an H&E slide, or, in the case of FISH groups 2 to 4, digital FISH scanning was performed in the strongest membranous protein-expressing areas marked on the HER2 IHC slides when HER2 IHC staining was 2+/equivocal. The technologist performing the digital scans was blinded to the manual results.

A sample was considered discordant between manual and QIA if on average: (1) there was a ±1.0 or more copy number difference without a change of group status, or (2) a sample changed group designation and had a ±1.0 or more copy number difference. However, if the group designation changed but the average difference was less than ±1.0 copy number the sample was considered “borderline but concordant.”

The only exception to the ±1.0 copy number rule were groups 1 and 3 cases, with an average HER2 copy number 6.0 or more, because higher HER2 copy numbers tend to have a higher level of variability in enumeration. Of note, such variability is also seen on manual scoring and has been reported in densely clustered FISH signals by others as well.⁸  These were considered discordant only if the group designation changed. There is a lack of literature consensus on the cutoff values for clinically significant changes between digital and manual results. Our lab uses the ±1 or more copy number cutoff because a normal cell has 2 copies of the gene of interest and if you lose 1 copy (2:1 control gene to target gene) you have a ratio of 0.5, which is a widely used criterion for gene deletion. On the reverse side, if you gain a copy of the gene, the cell is then defined as having trisomy of that gene and it becomes a clinically significant finding. The discordant samples were submitted for a blinded review by 2 pathologists to determine if the digital or manual count was more accurate. Concordance between digital and manual methods was calculated. Linear regression models and Bland-Altman plots were evaluated for correlation between the 2 methods and to identify outliers.

Instrument and operator precisions were evaluated as follows:

• Instrument within-run precision: One HER2 FISH slide from each of the 5 groups was scanned in triplicate on the same day by the same operator.

• Instrument between-run precision: One HER2 FISH slide from each of the 5 groups was scanned on 3 different days by the same operator.

• Operator precision: One HER2 FISH slide from each of the 5 groups was scanned and analyzed by a minimum of 3 different operators.

Differences between the 95th percentile cutoffs for each of the digital and the manual cell points ranged from −0.02 to 0.07 (Table 1). The average HER2 and CEP17 copy number varied from 2.00 to 2.07 and 1.85 to 1.90, respectively, which is close to the biologically expected normal of 2.0 copies per cell, establishing equivalence between digital and manual enumeration.

We also compared the digital and manual counts for each individual case by creating a Bland-Altman plot. The average results between manual and digital were plotted against the difference between the 2 results (Figure 1). There was no significant bias for either HER2 or CEP17, although there was a slight trend for HER2 to have a higher copy number on the digital method. This corresponds with the 2 outliers seen on the HER2 graph. A review of these cases showed that there were cells with artifact and overlapping nuclei that were not edited and may reflect expected variance in clinical practice.

In evaluating invasive tumor in 16 samples at different cell cutoffs (50, 100, and 150, 200) we found that there was 0.92 or less difference in HER2 copy number, with an average difference of 0.11, between all cell cutoffs. Four samples resulted in a change in group designation; in 3 of these cases, this change occurred between 100 and 50 cell counts. The fourth case was borderline (HER2 of 3.98 at 100 cells versus 4.02 at 200 cells). Stability in FISH group designation was seen in all other cases once the minimum number of cells counted was above 100.

The PathFusion HER2 classifier program correctly classified cells into different groups based on the 2018 ASCO/CAP guideline for breast carcinoma. Samples enumerated with CEP17 greater than 0 classifier maintain the same group designation and/or have less than ±1.0 copy number difference (excluding group 1 or 3 cases with HER2 copy number ≥6) when enumerated with the PathFusion HER2 Classifier.

The settings were optimized during the development phase as a cell diameter of 75 to 300 and cell circularity of 1.5, z-stack of 9 planes at a 0.7-m depth, with exposure times of 30 msec, 50 msec, and 75 msec for 4′,6-diamidino-2-phenylindole (DAPI), fluorescein isothiocyanate, and Texas Red, respectively. Per the laboratory’s standard operating procedure, the software exposure settings are adjusted quarterly to help correct for aging light sources or with a change of light bulb/source. There was 1 adjustment during the validation that changed the exposure times to 100 msec, 300 msec, and 300 msec for DAPI, fluorescein isothiocyanate, and Texas Red, respectively, per the manufacturer’s recommendation. The increase in DAPI exposure improved the prescan quality and autofocus portions of the digital scan with better segmentation of the cells by the software. Image quality improved significantly upon reoptimization.

A range of screen sizes and resolutions were evaluated (Table 2). Screens 24 inches or larger with a 1920 × 1080 resolution or higher were determined to be optimal. However, screens/laptops with smaller size and resolution were also deemed acceptable (Figure 2). The only major changes were the loss of the helpful close-up zoom box on the display, which was not critical for enumeration and required additional scrolling to fully view all regions of the FISH image.

For the final analysis, 21 of 189 cases (11.1%) were excluded because of inherent issues with the cases. Reasons for these included ink artifacts; quality issues, such as photobleaching; green mist; extensive stroma; and limited tumor/scattered tumor with fewer than 100 evaluable tumor cells (Figure 3). Six cases were excluded for circling errors or tissue selection errors on manual enumeration. No cases met the ASCO/CAP guideline for genetic heterogeneity.

ASCO/CAP requires a 95% concordance for HER2⁻ cases and a 90% concordance for HER2⁺ cases for HER2 IHC QIA. We observed an initial overall concordance between 78.3% and 100% for the individual groups and an overall concordance of 87.5% (147 of 168). The concordance rate for HER2⁺ cases (group 1) was 95.3% and that for HER2⁻ (group 5) was 98.2%. A total of 85.7% (18 of 21) of the initial discordant cases were resolved in favor of the digital count upon a second blinded pathologist review, with a final concordance of 95.3% to 100% across individual groups (Table 3). Details of discordant cases are provided in Table 4, with example cases depicted in Figure 4.

The linear regression R² values were 0.816 for HER2 and 0.7748 for CEP17, indicating good correlation between the manual and digital methods (Figure 5). Outlier scores for cases were identified on Bland-Altman plots (Figure 6). The HER2 and CEP17 Bland-Altman plots each had a few outliers. All the HER2 outliers occurred in 10 or more HER2 copy number samples with clumped signal, which intrinsically have more variability in scoring even on manual enumeration. The CEP17 outliers were more evenly distributed, but the most significant outliers occurred at higher copy numbers. There was a trend for decreased average CEP17 and HER2 copy numbers with the digital method, −0.13 and −0.97, respectively. The CEP17 trend was not significant for FISH enumeration. The HER2 trend, although concerning, was more significant in cases with higher HER2 copies. Upon eliminating the highly amplified samples (≥10 HER2 copies) the difference was reduced (−0.56) and was within the acceptable variance for FISH enumeration (Figure 6, C).

One HER2 FISH slide from each of the 5 groups was scanned in triplicate on the same day by the same operator. The overall group designation remained the same for each sample with a 100% within-run precision.

One HER2 FISH slide from each of the 5 groups was scanned on 3 different days by 1 operator. The overall group designation remained the same for each sample except for 1 group 2 sample, which showed borderline but concordant results per the discordant criteria established in the accuracy section. An additional sample from group 2 was subsequently tested, and the overall group designation remained the same between all 3 replicates for a 100% between-run precision.

One HER2 FISH slide from each of the 5 groups was scanned and analyzed by a minimum of 3 different operators. The overall group designation remained the same in 22 of 25 counts for an overall operator precision of 88%. These discordances were specific to certain technologists and were easily addressed with additional training on the removal of nonneoplastic cells and correct classifier editing.

To ensure similar results during clinical implementation and transition to the clinical laboratory, we used a side-by-side digital and manual review of all group 2, 3, and 4 cases requiring a second blinded IHC-guided scoring. In this phase, besides looking for concordance, we also monitored challenges for clinical implementation and specifically identified cases that would not be suitable for QIA and needed to be flagged for manual review. In our experience, we encounter genetic heterogeneity in approximately 0.5% of breast carcinoma cases,⁹  and these areas correspond to high protein expression that would in most instances be identified on IHC-directed FISH enumeration. Because IHC was not performed on all cases, the clinical implementation was limited to groups 2, 3, and 4 undergoing a second blinded IHC-directed FISH enumeration. Being a reference laboratory with a high volume of cases being reflexed to referral FISH testing, these groups comprise approximately 24% of our testing.¹⁰  Furthermore, because of their borderline nature, these cases also prove to be the most challenging cases for manual enumeration.

A total of 129 consecutive new samples prospectively collected (after completion of the validation phase) that were reflexed to IHC-guided scoring were evaluated by both manual and digital review as part of the clinical workflow. Four cases (3.1%) were unsuitable for digital review because of high background and poor image quality or fewer than 100 evaluable tumor cells. Of the remaining 125 cases evaluable by both digital and manual enumeration, 13 (10.4%) were borderline but concordant and 82 (65.6%) were concordant, for an overall concordance of 76%. The discordant cases comprised 16 group discordances (12.8%), 7 (range, 1.0–1.54 difference; mean, 1.20) HER2 copy number discordances (5.6%), and 7 (range, 1.02–2.76 difference; mean, 1.56) CEP17 copy number discordances (5.6%). However, only 10 (8%) of the discordant cases had a clinically significant discordance, that is, amplified versus not amplified (Table 5). The reasons for discordances were signal heterogeneity with the presence of scattered equivocal or amplified cells, inadequate matching of highest areas of protein expression to areas where the manual count was performed, improved unbiased digital counting, and cases with indistinct cell borders or artifact that should have been excluded from digital enumeration.

We compared the digital enumeration results to the original and IHC-guided manual enumeration results to create criteria that would allow us to capture any clinically discordant cases. We evaluated the 10 cases that changed clinical significance between the second manual count and the digital count in this cohort. Of the 10 cases, 7 also had clinically significant group changes between the original and the digital counts and could be captured by implementing a manual review of cases with group changes with clinical impact, that is, amplified to not amplified or vice versa. The remaining 3 cases had borderline HER2 copy numbers. To ensure these had the most accurate results we decided to require a minimum of 200 cells and to score additional tumor regions for large tumors (Figure 7). Based on the cohort results, we anticipate these extra steps will only be required in ∼10% of cases and the manual reflex for clinically significant changes would occur in ∼16% of cases.

Our validation study showed an excellent concordance of greater than 95% between manual and QIA across all groups of HER2 classification meeting the ASCO/CAP requirements of concordance. The QIA also showed equivalence for reference range, within-run, between-run, and observer precision to the manual methods. For our validation, at least 7 observers formally trained for both digital and manual FISH enumeration were involved in digital and manual enumeration of the slides, capturing interobserver variability that occurs in clinical practice. In this study, we used the scanner available in our laboratory that uses automated nuclear segmentation and spot counting with an option for manual review and editing before finalizing the results on a case. We also used clinical samples in our laboratory, thus providing a simulation of the actual clinical practice in a high-volume laboratory with similar results. Importantly, the exercise of clinical implementation allowed us to understand and create a clinical algorithm that would help triage and identify cases with potential clinical impact and flag them for additional manual review. This was very helpful in understanding the limitations of digital enumeration and excluding cases with high background, artifact, indistinct cell borders, and scattered tumor cells from digital review. Furthermore, cases with clinically significant group changes between the original manual enumeration and IHC-guided digital enumeration are best scored manually.

FISH QIA has thus far used different technologies, such as DAPI-based tile sampling using equal-sized tiles that include 1 to 2 nuclei or nuclear sampling using nuclear segmentation followed by spot counting of fluorescence signals by Metafer 4 (MetaSystems) and EIKONA3D Alpha Tec Ltd.^11–13  The Visia imaging D-Sight platform provides the option of screening at ×20 followed by scanning at ×100 for a focused scoring in frames in the region of interest captured using z-stacking.¹⁴  Other platforms include the Leica HER2 FISH system for Bond.¹⁵  More recently, convolutional neural network and difference of Gaussian-based signal enumeration methods have been explored.⁸  Top hat and bottom hat filters for image analysis is yet another promising technique.¹⁶ 

All these methods, like ours, have demonstrated a high degree of concordance between manual and digital enumeration, providing support for the adoption of QIA for FISH enumeration. Some of these also had the advantage of manual editing that improved the accuracy of HER2 analysis and was an important component of the analysis by eliminating truncated and nontumor nuclei, similar to the practice of manual enumeration. Although all these studies have shown a slightly lower concordance for equivocal cases, understanding the limitations of digital analysis and appropriate selection of cases can reduce these errors. These errors are mostly caused by issues related to the tissue, inappropriate selection of tumor regions for scoring or analysis methods, or, importantly, genetic heterogeneity. The option of scanning the whole slide at ×20 can overcome this limitation.^8,9,14,16  FISH enumeration can also be further simplified by matching areas of highest protein expression for digital enumeration in place of scanning the entire tumor on the FISH slide, which is very time intensive. This can be a useful tool to capture genetic heterogeneity.⁹  Furthermore, digital images provide for longer and more accessible image archival.

Lastly, the scope of our validation project did not include an analysis of cost and time savings. During the initial phase of transition, the time required for digital analysis may be more than for manual enumeration; we are hopeful that with familiarity and team effort, there will be time saving for each user with improved accuracy of FISH testing, specifically for equivocal cases.

In conclusion, this validation study demonstrates the utility of QIA in providing highly reliable HER2 FISH analysis by a more objective enumeration of a greater number of cells, removing manual scoring bias and, importantly, improving standardization across scoring, with similar results to manual methods.