Pathologist Concordance for Ovarian Carcinoma Subtype Classification and Identification of Relevant Histologic Features Using Microscope and Whole Slide Imaging: A Multisite Observer Study Open Access

This study included microscope and digital review of 212 individual sections from 109 ovarian carcinoma cases. As previously described in Seidman et al,¹⁰  the cohort was created by accruing 60 consecutive cases from the gynecology and gynecologic oncology service of a large community and tertiary care hospital, and then enriching for the less common subtypes (non–high-grade serous carcinomas), also as they appeared consecutively. The cases included 69 single-section and 40 multisection cases (144 sections, 2–6 sections per case), for a total of 213 sections derived from formalin-fixed, paraffin-embedded tumor tissue and stained with hematoxylin and eosin. The sections were selected by 1 of the experts as representative of the primary ovarian tumor. The average number of sections per case varied across subtypes (1.7 for HGSC, 2.0 for LGSC, 3.1 for MUC, 1.6 for EC, and 1.9 for CCC). One of the tissue slides could not be scanned and was excluded from this study.

The remaining 212 tissue glass slides (sections) in the cohort were digitized at ×40 using a Hamamatsu NanoZoomer 2.0 HT (Hamamatsu Photonics, Bridgewater, New Jersey) located at the Tissue Array Research Program, Laboratory of Pathology, Center for Cancer Research, National Cancer Institute, National Institutes of Health, Bethesda, Maryland. Basic scanner characteristics³¹  included a scanning resolution of 0.23 μm, a halogen 3250-K light source, an Olympus 20× UPlan Apo objective lens with 0.75 numerical aperture, and a 3-CCD camera referring to the use of 3 separate charge-coupled devices (CCDs), each one taking a separate measurement of red, green, or blue.

Pathologists from 5 different sites with varying experience in gynecologic pathology participated in this study. The study was designed for the review of individual sections (section-based review) instead of a case-based review. This strategy reflects a trade-off between maintaining the search aspect of a pathology review of an entire case versus narrowing the search area to single sections to reduce interobserver variability caused by differential weighing of the presence of histologic features in different tumor areas. The latter was particularly relevant in this study because of the intratumoral heterogeneity known to exist in ovarian cancer.⁹  Sections were reviewed in a randomized order with a control to avoid sections from the same cases being read consecutively. Even though immunohistochemistry has been shown to improve the interobserver reproducibility of ovarian carcinoma classification,¹³  it was not used in this study to refine classification because the focus was on differences between modalities for the histologic review of hematoxylin-eosin–stained tissue.

The study was conducted in 3 phases. In the first phase, 3 experienced gynecologic pathologists (EGyn1, EGyn2, and EGyn3), all of whom were co-authors of the 2014 WHO guidelines for the classification of gynecologic tumors,³²  reviewed each slide independently in a randomized order on a microscope and provided subtype diagnosis. The majority consensus diagnosis by this panel (a subtype diagnosis agreed by at least 2 of 3 pathologists) defined the reference diagnosis for the histologic subtype for each section.

In the second phase of the study, pathology review on the microscope was expanded to include the identification of 8 histologic features important for ovarian subtype classification. The features are defined below. This type of review was conducted by 1 of the 3 pathologists involved in the initial classification who was also available for this phase of the study (EGyn1), and 2 additional board-certified pathologists from a different academic hospital. One had 3 years of clinical practice in gynecologic pathology (GynP), and the other had specialized in hematopathology (HemP) and had completed general anatomic pathology training 20 years prior. All 3 pathologists reviewed all 212 tissue slides and provided a subtype classification and the presence or absence of any of the histologic features.

In the third phase of the study, 4 pathologists conducted a digital review of all 212 WSIs in randomized order and were instructed to report on the presence or absence of any of the 8 histologic features and to also provide a subtype diagnosis for each WSI. This group of pathologists consisted of 1 of the pathologists in the first group (EGyn1), an experienced gynecologic pathologist who was also a coauthor of the WHO guidelines (EGyn4), a fellow in gynecologic pathology (FeGynP), and a fellow in surgical pathology (FeSurgP).

The following 8 histologic features were selected after literature review and consultation with the senior author for their ability to distinguish between ovarian carcinoma subtypes. (1) High-grade atypia, based on 3-fold nuclear size variation: this feature is considered the primary feature for distinguishing between high-grade and low-grade serous carcinomas.³³  (2) High mitotic count (benchmarked at >12 per 10 high-power fields [×40 objective], although this was not specifically counted): this feature is considered the secondary feature for distinguishing between high-grade and low-grade serous carcinomas when there is doubt about the nuclear grade.³³  (3) Intracytoplasmic mucin: this feature supports the diagnosis of MUC.³⁴  (4) Hyalinized stroma: this feature can support the diagnosis of CCC rather than HGSC.³⁴  (5) Clear cell architectural patterns (hobnail, tubulocystic, or tubulopapillary patterns): these patterns define the diagnosis of CCC by architecture but do not include the cytoplasmic clearing.³⁵  (6) Sarcomatous component, characterized by malignant features typical of sarcomas, with or without more specific differentiation: this feature supports the diagnosis of carcinosarcoma rather than HGSC.³⁶  (7) Squamous differentiation in the form of morules or aggregates of spindle-shaped epithelial cells³² : this feature favors EC rather than MC or HGSC.³⁴  (8) Morphology suggestive of endometriosis: this feature favors EC rather than MC³⁴  and often, in low-stage ovarian carcinomas, displays endometriotic cysts showing an intact endometrial lining, stroma, and scattered foci of fresh or old hemorrhage. Features diagnostic of endometriosis were not required to be present.

The features above were not quantified, and therefore a small focus of a feature was sufficient to designate it as present. For this review pathologists searched for each of the 8 features and reported any that were present in a given whole slide image of a scanned slide.

For digital review, WSI images were displayed to the observers in each site on the same model and type of monitor to reduce variability associated with the use of different monitors. The monitor was an LED-backlit LCD HP Z24x 24-inch monitor (Hewlett-Packard, Palo Alto, California) with basic specifications, including native resolution of 1920 × 1200 at 60 Hz, 16:10 aspect ratio, and 1.07 billion color support. Each monitor was calibrated using an Eye-One calibration kit (X-Rite, Tewksbury, Massachusetts).

A customized interface based on the Philips Pathology Education Tutor (previously marketed as Path XL Tutor, Philips North America Corporation, Andover, Massachusetts) was used for WSI review and for collecting observer responses from a questionnaire that was shown alongside each WSI. Within the software, users were able to pan and view WSIs at multiple magnifications (4× up to 40×, native WSI resolution). The same computer and graphical interface were used by all observers. The questionnaire provided to observers instructed them to perform 2 tasks. For the first task, the instruction was to “Please select all histologic features present in this section.” Response choices included the 8 features described above along with a “None of these” option. For the second task, observers were instructed: “Assuming the current section is the only section available, please review and select the subtype of the tumor.” Response choices included HGSC, LGSC, MUC, EC, CCC, seromucinous carcinoma, carcinosarcoma, malignant Brenner tumor, undifferentiated, or mixed. The Figure shows a screenshot of the graphical interface used in the digital review phase of the study.

All pathologists were reminded of the instructions for handling problematic areas as outlined in Seidman et al.¹⁰  The 2 nonexpert pathologists in the second group (GynP and HemP) were provided with training in the form of a PowerPoint (Microsoft, Redmond, Washington) presentation showing multiple fields of view from each subtype along with textual descriptions and visual illustrations of relevant histologic features.

The training for the pathologists in the third group in the use of WSI consisted of a 2-step procedure conducted using the same interface on the same workstation. For the first step of training, observers were presented with digital textbook descriptions of ovarian subtypes along with images of subtype-specific histologic features and had access to 3 WSIs of each subtype for review practice. The second step of observer training consisted of an interactive practice session on 20 full-size WSIs within the digital viewing software, where observers were asked to repeat their review until the answer provided matched the reference standard. In addition to familiarity with the diagnostic task, this session provided familiarity with the controls and digital interface.

This analysis compared the concordance rates against the expert panel's subtype classifications of the observers using the microscope to the concordance rates of the observers using WSI. Two different approaches were used to calculate concordance with the reference standard by the expert panel. The first approach was to derive a reference subtype classification, determined as the consensus subtype classification for each section agreed upon by at least 2 of 3 expert panelists. For this analysis, concordance between a pathologist and the reference standard was defined as the percent of sections in the cohort for which the subtype classification by an observer was the same as the reference subtype classification. Differences in concordance were then calculated across observers using a microscope and observers using WSI. This approach required consensus by at least 2 of 3 pathologists on subtype classification; however, in some instances there was no consensus by the 3 experts (each one provided a different classification). Those cases, hereafter referred to as undetermined, had to be excluded from this analysis.

In order to account for uncertainty in deriving a reference diagnosis and avoid excluding any cases, a second approach to calculate concordance with the reference standard was used. The second approach calculated the average concordance of observers with each of the 3 expert panelists and compared the average concordance rates for pathologists using a microscope to those of pathologists using WSI. Concordance of observers with each individual panelist was also calculated. The 95% CIs on concordance and differences in concordance rates were derived using bootstrap resampling. Bootstrap resampling was performed on sections for paired concordance measures and on sections and observers for calculating 95% CIs on average concordance.

Expert panelist EGyn1, who also reviewed cases with WSI, was excluded from both analyses but was included in the interobserver analysis for feature identification described below. Intraobserver, intermodality analysis was not reported because the only one who conducted a review with both a microscope and WSI was the senior author of the study (EGyn1), whose familiarity with this particular cohort of carcinomas could bias findings and hinder their generalizability.

Interobserver agreement using a specific modality (microscope or WSI) for the identification of each histologic feature was quantified using percent positive agreement, defined as the percent of cases in a data set for which both pathologists identified a certain feature versus the number of cases where either of them identified that feature.

Three expert gynecologic pathologists (EGyns 1–3) conducted microscope reviews to provide a reference standard determination of subtype for all sections in the ovarian carcinoma cohort (Table 1). The table shows the breakdown of sections that had unanimous versus majority consensus diagnosis. Results show that of 212 sections, 56 (26.4%) had nonunanimous agreement, which, considering the expertise of the observers, indicates a challenging data set. Sections from seromucinous, EC, and MUC cases had the highest rates of nonunanimous agreement (4 of 8 [50.0%]; 7 of 16 [43.7%]; and 14 of 43 [32.6%], respectively). Subtype could not be determined (these sections are hereafter referred to as undetermined) for 12 of 212 sections. In these sections, all 3 experts reported different subtype classifications.

To determine a baseline of concordance with the expert panelists, 2 pathologists, GynP and HemP, reviewed glass slides of the ovarian carcinoma cases on a microscope. Concordance with the expert-based reference for GynP was 73.5% across all subtypes and increased to 75.8% when limiting the assessment to the 5 major subtypes (Table 2). Concordance with the reference for slides for which the expert panel was unanimous (84.0%) was substantially higher than for slides for which the panel determined subtype by majority (nonunanimous) consensus (46.4%). Concordance with the expert-based reference was significantly lower for HemP (55.5% across all subtypes, 55.1% 5 major subtypes only) than GynP (73.5% for all, 75.8% for major). The difference in concordance rate was 18.0% (95% CI, 10.0%–26.0%) across all subtypes and 20.7% (95% CI, 12.4%–29.2%) across the 5 major subtypes.

We then tested concordance with the expert-based reference for the pathologists reading WSI. Concordance rates with the reference subtype classification by EGyn4, FeGynP, and FeSurgP (also shown in Table 2) were 68.7%, 68.5%, and 58.5%, respectively, across all subtypes, and 67.8%, 66.3%, and 57.3%, respectively, across the 5 major subtypes. Similar to the pathologists reviewing on a microscope, concordance was substantially higher for sections that had unanimous consensus (76.8%, 75.0%, and 63.9% for the 3 pathologists) than for sections that had majority consensus (48.2%, 51.8%, and 44.6%, respectively). FeGynP's concordance with the expert-based reference was significantly higher than that of FeSurgP across all subtypes (difference: 10.0%, 95% CI [2.0–17.5]) as well as across the 5 major subtypes (difference: 9.0% [1.1–17.4]). In addition to having more experience with diagnosis of ovarian carcinomas, this finding could be partly attributed to the fact that FeGynP was trained under 2 of the expert panelists.

We wished to determine whether there were systematic differences in the propensity of microscope review or WSI review to give better agreement with the expert-based reference. We therefore compared the rates of concordance with the expert-based reference subtypes for GynP reading with a microscope to EGyn4 and FeGynP reading WSI (Table 3). Results are shown here for pathologists for whom, despite their difference in expertise, histotyping of ovarian tumors is a core skill. It can be observed that GynP (attending with subspecialty in gynecologic pathology) reading with a microscope had higher concordance with the expert panel than the pathologists with comparable experience reading on WSI (EGyn4 and FeGynP), but the differences only achieved statistical significance when limiting our analysis to the 5 major subtypes.

The specific subtype classifications provided by GynP, EGyn4, and FeGynP are shown in Table 4. It can be observed that the biggest contributor to differences in concordance rates between GynP (microscope) and EGyn4 or FeGynP (WSI) was MUC classification. GynP classified 29 of the 43 expert-based mucinous cases (67.4%) as MUC, whereas EGyn4 and FeGynP classified only 19 (44.2%) and 20 (46.5%), respectively, of these cases as MUC. Most of the misclassified MUCs were classified as ECs, with a rate of 11 of 43 (25.6%) for GynP, 17 of 43 (39.5%) for EGyn4, and 22 of 43 (51.2%) for FeGynP.

Average concordance rates with the 3 individual experts by observers using a microscope or WSI was used as a complementary metric to account for uncertainty in defining a consensus subtype classification. Findings shown in Table 5 were comparable to those shown above. Average concordance rates with the expert panel were higher for GynP reading on a microscope (69.5% across all sections and 74.0% across the 5 major subtypes) than EGyn4 (64.8% and 66.1%) or FeGynP (62.7% and 63.5%) reading on WSI, with the differences being statistically significant when the analysis was limited to the 5 major subtypes (Table 6). As a point of reference, interexpert concordance was 76.7% across all sections and 81.6% across the 5 major subtypes. Differences in concordance with individual experts between GynP and EGyn4 or GynP and FeGynP (Table 6) were all positive but varied in terms of whether the result was statistically significant.

The rates of agreement between pathologists for identifying the 8 selected histologic features using a microscope are shown in Table 7. Positive percent agreement (PPA) was on the order of 70% for identifying high-grade nuclear atypia and on the order of 60% for identifying high mitotic count across all pairs of pathologists. For the identification of intracytoplasmic mucin, PPA was 74.6% between the expert EGyn1 and GynP and was lower between EGyn1 and HemP (51.2%) and between GynP and HemP (60.3%). For the remaining histologic features, PPA between EGyn1 and GynP ranged from 50% for identifying clear cell architectural patterns to only 10% for identifying features suggestive of endometriosis, and varied widely across the other paired comparisons.

Positive percent agreement results in identifying the 8 selected histologic features between the pathologists using WSI are shown in Table 8. It is generally observed that PPA varied widely across different pairs of pathologists. For the identification of high-grade nuclear atypia, PPAs between EGyn1 and EGyn4 and between EGyn1 and FeGynP were 50.3% and 58.9%, respectively; in comparison, PPA for this feature between EGyn1 reading on a microscope and GynP was 71.4%. Similarly, PPAs for identifying high mitotic count between EGyn1 and EGyn4, and between EGyn1 and FeGynP were 39.4% and 41.1%; in comparison, the PPA for this feature between EGyn1 and GynP on the microscope was 62.0%. For identifying the presence of intracytoplasmic mucin, PPAs between EGyn1 and EGyn4, and between EGyn1 and FeGynP were 54.1% and 36.5%, respectively; in comparison, the PPA for this feature between EGyn1 and GynP on a microscope was 74.6%. For the remaining features, PPAs across pairs of pathologists varied widely, similar to the variation observed for pathologist pairs reading on microscope.

In this manuscript, we have presented the results of a reader study with pathologists from multiple sites focusing on examining pathologist performance for the histologic subtyping of ovarian carcinomas using both a microscope and digital review. We included comparisons of pathologist performance across subspecialties and for pathologists with varying levels of experience. In addition to subtype diagnosis, the study examined pathologist performance for identifying histologic features important for ovarian subtyping.

Results of the study showed that the concordance with the expert reference subtype classification of a pathologist specialized in gynecologic pathology reading on a microscope was significantly higher than the concordance of an expert gynecologic pathologist or a fellow in gynecologic pathologic reading on WSI. This result was primarily due to the substantially lower propensity of the 2 pathologists reading on WSI to diagnose mucinous carcinomas; they classified as mucinous 19 and 20 of the 43 mucinous carcinomas (44.2% and 46.5%, respectively) classified by the expert panel, compared with 29 of 43 (67.4%) by the pathologist reading on a microscope. Most of the missed MUCs were classified as EC, probably because of the difficulty in identifying the presence of intracytoplasmic mucin, as determined from our feature analysis. Another challenge found with the pathologists using WSI was distinguishing between high-grade and low-grade serous carcinomas. High-grade serous carcinomas as classified by the expert-based reference were diagnosed as low-grade serous carcinomas at a rate of 9 of 78 (11.5%) for the expert gynecologic pathologist and 11 of 78 (14.1%) for the fellow on WSI, as opposed to a rate of only 6 of 78 (7.7%) for the gynecologic pathologist reading on a microscope. These differences are probably due to difficulty in identifying high-grade nuclear atypia and high mitotic count, as evident by the results of our histologic feature analysis.

An important finding from this study regarding subtype classification was the substantial difference in pathologist concordance with the reference diagnosis, regardless of modality, between sections that were unanimously classified by the panel of experts and those sections that were classified only by majority consensus. For instance, GynP on microscope had 84% concordance on unanimous sections and only 46% on nonunanimously classified sections, whereas for EGyn4 and FeGynP on WSI those concordance rates were 76.8% versus 48.2% and 75.0% versus 51.8%, respectively. This result reflects the spectrum of diagnostic complexity that exists among cases and its impact on pathologist performance. The College of American Pathologists guideline for the validation of WSI recommended the inclusion of “easy and difficult cases”³⁷ ; however, only a few studies have reported on the distribution of challenging cases among study cases.^30,38  As such there is no information regarding the presence and proportion of challenging cases among data sets, making it difficult to make comparisons between findings in different research studies or to determine whether an observed diagnostic performance provides a realistic measure of expected performance in clinical practice, where challenging cases are not uncommon. Considering the probable difference in diagnostic complexity between such cases, it is important for studies examining observer performance for a variety of diagnostic tasks, including the evaluation of tools for clinical decision support³⁰  or computer-aided diagnosis,^39–41  to use data sets representing such differences.

The analysis of pathologist performance for identifying histologic features resulted in 2 main findings. One was that pathologists had relatively low agreement in identifying some important features for ovarian histologic subtyping with both viewing modes. That was apparent for identifying clear cell architectural patterns, sarcomatous component, squamous differentiation, and hyalinized stroma, for which positive agreement between experienced pathologists was ≤50%, and particularly apparent for identifying features suggesting endometriosis, for which agreement was as low as 10%. Even though the correct classification rate of endometroid carcinomas by experienced pathologists was relatively high, disagreement on identifying or evaluating squamous differentiation or features suggestive of endometriosis might have contributed to the misclassification of MUCs as ECs. As was determined during a consensus meeting between experts discussing discrepant ovarian carcinoma diagnoses,⁴⁴  reasons for such disagreements include the use of different thresholds for defining the presence of a feature (such as the minimum amount of sarcomatous component or squamous differentiation that must be present to be diagnostically important), whether there is any threshold at which a feature becomes a dichotomous classifier, and placing different amounts of importance on the presence of features in different tissue areas. This is an area for which specific guidelines for evaluating histologic features are lacking. The heuristics that lead to making a diagnosis of a mixed tumor are also very likely to differ between pathologists and contribute to diagnostic disagreement. On the other hand, agreement on the classification of tumors as CCC was high, despite low percent agreement on identification of clear cell architectural patterns or hyalinized stroma. This is unlikely to be surprising to pathologists, who know that tumor classification seldom relies directly on identification of discrete pathognomonic features and usually involves a synthesis of many features in a complex process influenced by experience. Still, better interobserver reproducibility in feature identification would likely contribute to improved concordance for diagnostic tasks, such as ovarian subtyping. More research is needed to determine thresholds and related criteria for linking histologic features to certain subtype diagnoses.

The second finding from our analysis of pathologist performance for feature identification was that there were challenges in identifying intracytoplasmic mucin, and to a lesser degree high-grade nuclear atypia and high mitotic count, with WSI. In the case of mucin (Table 8), the PPAs between experts EGyn1 and EGyn4 and between EGyn1 and the fellow in gynecologic pathology reading with WSI were 54.1% and 36.5%, respectively, whereas the PPA for this feature between EGyn1 reading on a microscope and GynP was 74.6%. Comparably lower were the interobserver agreement rates for identifying high-grade nuclear atypia and high mitotic count, 2 histologic features with significance across many tissue types. These findings are consistent with findings reported elsewhere on challenges identifying mucin or mitoses with WSI.⁴² 

Our study was not designed to tease out the possible reasons for the challenges in identifying histologic features or for classifying mucinous carcinomas observed in our study. One possible reason could relate to interobserver variability. In this study, we had a limited number of pathologists who may have used different criteria for subtype classification or for defining the presence of features. Moreover, the study was not fully crossed over, such that pathologists who reviewed using a microscope did not repeat their review using WSI. Likewise, the pathologists reading on WSI were not as experienced in the use of WSI compared with the use of a microscope, again potentially confounding our reported results. Experience with WSI in daily practice varied among the participants; none are using WSI for routine diagnosis of ovarian tumors. Focused training on reading with WSI for specific tasks, such as ovarian subtyping, could potentially improve pathologist performance or interobserver variability. Another possible reason for the observed challenges in identifying features with WSI could relate to limitations in the WSI scanning process, such as issues with color reproducibility, resolution, and depth of focus, or heuristic factors, such as propensity to review less than the whole slide or impatience due to lag time. Our study was not designed to differentiate limitations coming from the reader from those associated with the WSI process; however, identifying WSI limitations for specific diagnostic tasks would be particularly beneficial to pathologists using this technology as well as for manufacturers who could target further improvements.

One might consider that our study's use of commercial-off-the-shelf monitors contributed to the observed performance: it is possible that a medical-grade monitor could result in better feature identification. However, all pathologists in our study used monitors of the same make and model, calibrated in the same way, thus emulating the uniform specifications of a medical-grade monitor. A recent study⁴³  reported that there was no difference in the identification of mitotic figures or Helicobacter pylori burden between a medical-grade and a commercial monitor. That study did not evaluate the impact of display for identifying the histologic features, nor did it include the particular monitor used in our study. The findings of this study strongly suggest more research is needed to identify the impact of pathologist skill and experience, user training, WSI image quality, and display on feature identification and, more generally, on the ability to make a primary diagnosis. Moreover, given that WSI is likely to become part of workflow in the future, another issue to be addressed is how the lack of concordance in diagnosis affects patient management.

Regarding the impact of subspecialty, it was observed that the agreement of a nonspecialist in gynecologic pathology with the expert-based reference was significantly lower than the corresponding agreement of specialists with comparable years of clinical experience to that of the expert panel. This result is consistent with findings by Patel et al²⁵  on a smaller data set that consisted of only HGSC, CCC, and EC. We also observed that agreement between a fellow in general surgical pathology reading on WSI and the expert reference for overall histologic subtyping was lower by about 9% (without statistical significance) compared with the corresponding agreement between the fellow in gynecologic pathology with expert-based reference. A likely mitigating factor is that the gynecologic pathology fellow was a trainee of the expert panel pathologists. The study has a limited number of observers in each category, so the findings related to experience and subspecialization should not be generalized.

It should be noted that the diagnostic discordances observed in this study for either modality could be exaggerated due to the section-based, rather than case-based, review, and the use of a cohort that was enriched with the less common subtypes (non–high-grade serous carcinomas) in terms of both case numbers and numbers of selected sections per case.

In summary, our study found that the diagnosis of mucinous carcinomas and, to a lesser degree, the differential diagnosis between high-grade and low-grade serous carcinomas, were challenging for pathologists reading with WSI probably because of the difficulty in identifying intracytoplasmic mucin, and inconsistencies in identifying high-grade nuclear atypia or high mitotic count. Further research is needed to clarify the reasons for these challenges and to determine if these issues are fundamental to WSI or related to pathologist inexperience with the WSI review process. It was also found that pathologists had generally low agreement in identifying histologic features important to ovarian subtype classification, by either microscope or WSI. This result suggests the need for refined histologic criteria for identifying such features. Finally, the substantial differences in pathologist concordance between cases deemed as challenging and those that were more typical argue for the systematic inclusion and reporting of pathologist performance across cases spanning the range of diagnostic complexity.