Pathologists' Computer-Assisted Diagnosis: A Mock-up of a Prototype Information System to Facilitate Automation of Pathology Sign-out

Deidentified glass slides and corresponding pathology reports from matching breast core biopsy and breast lumpectomy specimens were used. Whole-slide images were created at 0.5 μm per pixel resolution with an Aperio ScanScope XT (Leica Microsystems, Wetzlar, Germany). Vendor-provided WSI viewing software was used for screen captures taken from relevant areas, termed regions of interest (ROIs). Desktop office software (PowerPoint, various versions) was used to create a pCAD prototype. Within PowerPoint, the captured images were cropped and annotated to simulate pCAD events such as proposed computer measurements, interactive question/answer scripts, and resulting pathology report data. The resulting PowerPoint pCAD prototype file was uploaded to the corresponding author's institutional Web site.

One automation approach for pathology involves conceptualizing pathology specimens as a series of concrete tasks and subtasks (Figure 2). In a breast cancer lumpectomy, these tasks might include (1) review of pertinent clinical data and prior specimens; (2) establishment of primary diagnoses (eg, invasive ductal carcinoma); (3) tumor characterization via tumor grading and staging; (4) correlation with clinical impression, radiology impression, and prior pathology; (5) documenting ancillary findings; and (6) creation of a complete pathology report data set. Most of these are high-level tasks that can be further degraded to make them more accessible for automation.

The review of prior clinical and pathology data serves several purposes, the main thrust being to help pathologists understand why the surgeon performed the operation and what they, the patient, and other related physicians are expecting to see in the specimen. This entails reviewing the provided clinical history, then following up with additional perusal of surgical clinic notes, radiology reports, and prior pathology reports. While an automated summary would be an advanced computational project, even a basic collection of hyperlinks would offer a productivity improvement for most pathologists. Integrated WSI systems can deliver previous pathology report data currently, but other vital data elements usually reside in alternative nonlaboratory information systems, and nonpathology data elements are typically displayed in suboptimal fashions. This is therefore an initial step in our pCAD workflow model.

Correlation tasks are related, also using radiology data, but are more specific with regard to documentation of lesional details. Radiographic data can include location, clip morphology, and tumor shape and size; it can also include intraoperative assessment of biopsy clips, tumor, and margins. This can be compared with the current specimen's gross description in the LIS and helps to establish that the specimen was grossed appropriately. Radiologic-gross pathologic correlations could also guide the pathologist in decisions about margin status, tumor size, and whether more histologic sampling is needed. Correlation of current and prior pathology includes a review of LIS and image data from a prior core biopsy specimen. This step varies widely in practice, but the purpose is for the pathologist to know what this patient's tumor looks like.

The third task is diagnosis; at this stage, the pathologist should already anticipate many of the tumor's features. Using the traditional workflow, a pathologist will look through glass slides one at a time, though often the pathologist can skip ahead to obtain a tumor dimension, either based on gross description, or based on the low-magnification appearance of the slide. For the primary (tumor) diagnosis, the pathologist must identify tumor and correlate its morphology with the prior core biopsy. The pathologist also records data about the tumor related to its stage, grade, and margin. In addition, the pathologist will also diagnose other important features, including biopsy site changes and in situ lesions.

The fourth and fifth task groups may occur during other steps and involve documenting ancillary findings and reviewing the preliminary pathology report. In traditional workflow, the pathologist has a variety of options such as taking notes, dictating, typing, or speech recognition. This is followed by a final report review, and sign-out.

The traditional workflow steps were reimagined in the context of whole-slide images and a hypothetical, “intelligent,” and interactive computer assistant (pCAD). The early data gathering and correlation tasks would be amenable to automation by gathering of the data sources into 1 area with links pointing to more detailed information. The diagnostic work could be divided into primary (tumor) tasks and secondary, nontumor tasks, because the pCAD system would be able to direct the end-user to relevant ROIs while simultaneously keeping track of what was reviewed. Further, during diagnostic tasks, the computer system would make preliminary assessments for the pathologist in a fashion similar to how trainees can provide their impression to a faculty pathologist. This might reduce the work labor, from manual microscope “driving” and manual note-taking, to an automated (or “chauffeured”) review of diagnostic areas with verbal confirmation of pCAD's assessments. Further, pCAD could automatically assemble the pathology data into a computer-generated report, which would then be available for a final review by the attending pathologist.

The pCAD prototype is a PowerPoint presentation that can easily be downloaded and viewed. It demonstrates a highly idealized, automated digital pathology workflow. It features a case dashboard that guides the pathologist through a series of sign-out tasks, for a breast cancer lumpectomy and axillary sentinel lymph node dissection. This PowerPoint mock-up presents a visual conceptualization of what was previously a simple description of pCAD, in which cutting-edge cognitive technologies are combined with large sets of heterogeneous data to create an intelligent computer agent intended to aid pathologists in routine clinical practice. The pCAD workflow, as presented in the PowerPoint scenario, is described below.

A pathologist starts his or her daily sign-out session by accessing pCAD, which presents the user with a list of cases pending review. This hypothetical information system understands normal human speech, and the pathologist begins his or her review by simply asking which case should be reviewed first. The pCAD system, which has enterprise-wide access to other information systems (eg, the surgeon's clinic schedule), alerts the pathologist to one case that should be reviewed urgently because the patient has a follow-up appointment soon. The pCAD system then proceeds to open the flagged, urgent case as it concurrently accesses a list of required work tasks, which is presented to the user on the left-hand side of the screen. These tasks, which are displayed to the pathologist in sequential fashion, also contain embedded hyperlinks so that the pathologist can jump directly to specific tasks immediately.

The initial task, Case/Patient Information, involves reviewing relevant clinical history/data, including radiographic and procedure-related information with convenient links to LIS and electronic health record data elements should more detailed information be required. This activity leads into the second set of tasks, Correlation, which directs the user to compare (and confirm) histopathologic findings to the (1) radiologic and (2) prior pathology findings. For radiologic Correlation, pCAD extrapolates the expected size of the lesion from radiologic studies (Figure 3, A) and queries the user to correlate this to the current histologic findings (Figure 3, B). The pCAD system is then asked to display additional information, including gross and more detailed radiologic findings—elements that are instantly accessible given the enterprise-wide integration. It is important to note that 3D scanners (Figure 4, A) and 3D cameras (Figure 4, B) can significantly enhance this step by enabling high-fidelity 3D digitization of gross specimens and tissue sections (Figure 4, C and D); the resulting 3D models, which provide basic coordinates that are vital to macroscopic-microscopic (Figure 4, G) and pathologic-radiologic (Figure 4, F) correlation, could then be pushed to pCAD (Figure 4, E) and made available for review during the sign-out period. While reviewing correlative tasks, the pathologist confirms the observations regarding concordance between macroscopic, microscopic, and radiographic findings. Tumor ROIs from current and prior pathology specimens are displayed to the pathologist, side by side.

The pathologist is then “chauffeured” to the third task, Primary Diagnosis, where triaged ROIs pertaining to the primary specimen diagnosis are displayed. These pCAD-triaged ROIs have been preliminarily tagged as suggestive of malignancy. The pathologist immediately recognizes fragments of probable tumor within an ROI and “takes control of the stage” by asking pCAD for control of the virtual microscope to scan and examine under more scrutiny. Since pCAD is equipped with advanced image analysis algorithms and machine-learning techniques, it can detect abnormal foci that are suggestive of malignancy. The pathologist confirms the tumor finding, thereby verifying the computer's preliminary diagnosis. Since tumor has been identified, the remainder of the review can be abbreviated; the pathologist could view all remaining ROIs at lower magnification, or could review another 20% of the suspicious ROIs at high magnification.

During the following task, Tumor Characterization, the pathologist must further classify the tumor lesions by navigating multiple subtasks, including histologic type/grade and biomarker evaluation. These subtasks are accomplished in a manner described for previous tasks—triaged ROIs and relevant, hyperlinked data elements are displayed to the user who confirms pCAD findings. As pCAD probes the user, persistently asking the pathologist to validate preliminary assessments, it simultaneously collects the user's input, displaying it on screen, as the information is seamlessly integrated into the working final report. This process of seamless integration of data into the final working report is a vital component of our hypothetical pCAD and is best demonstrated during the Tumor Characterization & Tumor Staging tasks. While piloting the user through the latter, pCAD displays a provisional measurement of the maximum size of the tumor for the pathologist's approval. If the measurement is not acceptable, it could be manually adjusted, but once approved by the pathologist then the final measurement would be automatically added to the report.

After completion of all the major work tasks, as described above, the pathologist eventually navigates to the penultimate task, Report Review. In completing this task, the pathologist reviews a computer-generated report that summarizes all pertinent findings related to the previous tasks. Since the system has successfully captured the interactive ROI review data, then this should be straightforward: computer-generated reports will almost certainly consist of structured data, which would facilitate additional utilization (eg, specialized oncologists' dashboard systems, educational packages for patients, deidentified research, clinical trials). The intelligent pCAD system has captured and compiled the data, presenting it as a preliminary synoptic report containing all findings previously confirmed by the reviewing pathologist. The pathologist then verifies the report, ideally without having to make many manual edits or corrections before sign-out.

It is difficult to understate the disruptive effect that proper automation would have on pathology practice in the 21st century. A dystopian viewpoint could see pCAD as a threat to pathologists (ie, fewer jobs or higher caseloads). If pathologists' only value to the health care organization is their ability to sign out cases, then this could be theoretically true. However, modern pathologists could find themselves with additional time to provide value-added contributions to the patient care team. These could include increased availability for consultation, directly consulting with patients, more actively engaging in analysis of laboratory data/metrics, or other tasks.

This mock-up of a breast pathology sign-out session driven by an “intelligent” computer agent may seem more grounded in science fiction than reality; however, complete automation need not occur all at once. Instead, the mock-up is intended to provide a framework to be used for facilitation of early efforts of automation—to begin the process of creating both specimen classes and required work tasks. Transforming pathology sign-out activities into concrete work tasks will enable computer scientists and engineers to help create pCAD.

The potential benefits of using pCAD are, at a minimum, 3-dimensional: (1) increased efficiency, (2) better patient care, and (3) true automation of pathology workflow. The first potential benefit is easy to understand given the amount of time that pathologists spend on tedious clerical tasks that require little diagnostic skills. A personal assistant, such as pCAD, with basic pathology knowledge, could automate the calculation of Nottingham grade, take notes, and write up the synoptic report based on the pathologist's input. A computerized pathologist's assistant could also alert users to problem cases early on (eg, cases needing immunostains or regrossing), proactively alert users to critical findings requiring urgent clinical communication (eg, positive “bug” stains, small-bowel tissue in an endometrial biopsy), automatically generate standardized reports, and transmit standardized reports to disparate information systems—likely as structured data rather than as free text. This latter ability would significantly facilitate clinicians' ability to access and rapidly understand pathology reports. Finally, such a system would inevitably improve both the value and the cost of the digital archive, since cases are automatically annotated during the interactive review. As a result, important ROIs such as tumor images will be easily available to future pathologists and researchers. Furthermore, entire digital slides could be purged from an archive to save space while important ROIs could be selectively retained at a fraction of the cost of retaining the entire case.

The second potential benefit is perhaps less comprehensible but equally plausible. One of the greatest fears of a histopathologic diagnostician is “missing” abnormal features, especially in specimens that contain only minuscule amounts of abnormal lesional tissue interspersed between large volumes of unremarkable, normal-appearing tissue (eg, a re-excised breast specimen). Double reading of slides by 2 pathologists would obviously reduce the risk of such errors but the additional cost renders this approach impractical.¹⁶  In our proposal, pCAD would partially fulfill the role of “the second opinion” and screen the histologic contents in parallel to a pathologist. If certain ROIs are to be missed, it is likely that pCAD and the pathologist would miss different ones, resulting in increased sensitivity of detection. Also, relevant to improved patient outcomes are the rich and intriguing modes of visualization offered by coupling pCAD with novel 3D scanning modalities. For example, macroscopic-microscopic and radiologic-pathologic correlation processes are currently carried out in a pathologist's mind; however, integrating 3D radiologic, gross macroscopic, and microscopic data sets into comprehensive 3D models will enable pathologists to visualize specimens in a more comprehensive fashion. Such a model could provide spatial details of the lesion in relation to the resected specimen and to the entire organ with a degree of accuracy and precision that cannot be matched by mental correlation, especially in complicated surgical cases. Thus, 3D reconstruction and 3D correlative imaging will enable better decision support, guide more accurate prosecuting, and enable more precise selection of microscopic sectioning.

The third benefit of pCAD, automation of pathology workflow, seems the most obvious of the 3, but is better appreciated in the context of a recent historical example. Consider the rise of departmental grossing manuals, which are the products of specimen classification. Grossing manuals divvy up specimens into distinct archetypes, include detailed grossing instructions, and are meant to facilitate the delegation of grossing such that a pathologist's personal attention is not required. Thus, grossing manuals automate the process of macroscopic specimen analysis and dissection, and yet the system's success is predicated on the ability of delegates (ie. pathology assistants) to recognize situations that require the attention of the pathologist. The pCAD system, as described above, would function in a similar capacity, aiming to automate the entire sign-out process, all while seeking input from pathologists in various situations. In fact, pCAD's ability to analyze gross pathology data and metadata would play a more crucial role in the context of automating sign-out, since many pathologists ascribe to the notion that diagnosis begins at the time of gross examination. As we envision it, pCAD's analytic functionalities would extend to the gross room, where it would image specimens, record gross observations, and push gross data elements (eg, descriptors, numerical measurements, weights) in a structured format, to the working draft of the pathology report. Additional efforts toward automation should, and likely will, improve integration of existing information systems to enhance pathologists' access to clinical and biological data housed outside of the LIS. The benefits of gradual automation should sharply increase as technologies evolve and as automated systems are validated and accepted by pathologists, clinicians, regulators, and patients.

There are likely to be many obstacles on the path to pCAD and full automation. Initial efforts must proceed slowly enough so that pathologists can start to feel comfortable entrusting computer assistants, much in the same way that they eventually relegated prosection duties to pathology assistants. In addition to physician buy-in, these automated computer assistants must undergo thorough validation studies, in much of the same way that current laboratory methodologies are validated. This will also require direct, ongoing supervision of the system as a part of existing quality assurance efforts; pCAD cases would likely require re-review at high rates initially, similar to how a junior or inexperienced pathologist's cases are often reviewed at a higher rate than those of established pathologists within a practice.

A more challenging issue will be the common misconception that the automation would eventually render pathologists obsolete. Fears about computers and advanced technology displacing jobs is not a new phenomenon—concerns about the effect of computers on work and on people has been around ever since they were first invented.^17,18  Historically, this fear of technologic unemployment has typically emerged in ages characterized by radical technologic changes. However, since the inception of these fears, economists and other researchers have keenly pointed out the existence of economic forces, which can spontaneously compensate for the reduction in employment due to technologic progress.¹⁹  More recently, there has been renewed debate and concern about technologic unemployment owing in large part to 2 assumptions: (1) the speed of progress in digital software and hardware is faster than in previous eras of technologic change, and (2) software and computer systems are increasingly able to automate cognitive tasks.²⁰  A recent study by Frey and Osborne,²¹  examining trends in AI, estimated that 47% of American jobs were at risk from automation. However, another research study²²  suggests a milder impact (over the next 3 to 5 years, at least), estimating that less than 5% of jobs can be entirely automated, using “currently demonstrated technologies.” More importantly, the researchers of the latter study²²  emphasized the potential for automation to enrich work, liberating people to focus on more creative tasks. Pathologists engage in many other clinical and nonclinical duties beyond sign-out responsibilities. Automation of sign-out or slide review might tempt highly industrious pathologists to simply do more work, but this would be a wasted opportunity. Instead, the fruits of automated slide review, namely, more free time, would create opportunities for pathologists to provide better value to patient care. This includes integration and reporting of ancillary specimen tests, more responsive consultation, and closer clinical cooperation with other physicians (eg, multidisciplinary breast cancer team).

Additional barriers can be anticipated by looking to other industries where complex automation has already taken hold. As automation systems, such as pCAD for example, are able to use more input sources, the automation inevitably becomes increasingly complex and the number of modes that a user must understand tends to increase. This increasing complexity of automated systems results in what is commonly referred to as “automation mode confusion,” wherein users are led to misinterpret the information being provided. Because users believe they are in a mode different than the one they are in, users may consequently make inappropriate requests or responses to the automation. Luckily, efforts are already underway to minimize the numbers of incidents caused by the human-automation issues, primarily by way of detecting the undesirable interactions between them in a timely fashion, giving more authority to the automation, enhancing operator training, and by changing user interfaces.

Another major hurdle to the development of pCAD involves incentive mechanisms for vendors, researchers, and engineers in the pursuit of building automated platforms. Most informaticians are already quite familiar with the paradoxical nature of information systems which, despite being designed to aid the needs of users, are deemed unusable owing to suboptimal human-computer interfaces. Many of the current generation of medical and LISs have been built primarily to capture billing information in an efficient manner, rather than optimizing clinical workflow. Current LIS vendors lack appropriate incentives for developing end-user–centric interfaces, and as with an ideal “agile” software development, the challenge is getting the iteration, feedback, and discussion between the end-users and the designers. Perhaps enhanced billing codes for pathology diagnoses, optimized by automated platforms and computer-assisted diagnoses, may be necessary to encourage vendors aiming to recoup the costs of their software development. Notwithstanding that, it is vital that pathologists, informaticians, and all other end-users exercise caution when implementing incentive models that are too heavily favored toward vendors—current LIS vendors are plagued by stagnation in image management due to dominance and complacency with vendor market share.

The theoretical intelligent computer assistant, pCAD, may seem far-fetched but there are existing examples of complex automata, including computer-screened Papanicolaou tests used in cervical cancer screening.²³  Outside of pathology, complex automata include Google's (Mountain View, California) self-driving car project, or Amazon's (Seattle, Washington) inexpensive autonomous flying drone aircraft.^24,25  Automation in surgical pathology is desperately needed, if it is not already too late. Cost control efforts, increasing clinical expectations, increasing amounts of clinically relevant data, larger workloads, and shifting metrics of pathologist productivity are legitimate threats that endanger the traditional practice of pathology.^26,27  Additionally, there may be an impending shortage of pathologists in the United States.²⁸  Future generations of pathologists must learn to harness cognitive technologies in order to neutralize these threats and boost productivity through legitimate, high-yield automation as we transition from digital to computational pathology.

The pace of technologic innovation continues to increase, with more sophisticated software, such as cognitive AI-derived technologies, disrupting labor markets. Strikingly, recent trends demonstrate that computerization and automation are no longer confined to routine manufacturing tasks. Autonomous driverless cars, currently being developed by Tesla (Palo Alto, California), Uber (San Francisco, California), Baidu (Beijing, China), and Google, provide one example of how manual tasks in transport and logistics will soon be automated.^29–31  In 2004, AI researchers went through great lengths to detail the difficulties of replicating human perception, asserting that driving in traffic would remain insusceptible to automation.³²  Six years later, in 2010, Google announced that it had modified several Toyota Priuses (Toyota, Aichi, Japan) to be fully autonomous.²⁰  Assuming a similar scenario takes place within the realm of pathology, an outline of a possible way forward, in the framework of an interactive computer-assisted image review with automated reporting, is presented above. Several of the advanced cognitive technologies, which are essential to pCAD, are currently being evaluated and/or developed for pathology, predominantly in the realm of computer vision and deep learning for image analysis.^33–37 

To facilitate progress toward pCAD, there must be a high-level vision for automation efforts, without which digital pathology may continue to stagnate. Explicitly stating required work goals and specimen archetypes is an exercise that empowers nonpathologist collaborators to understand the where, how, and what of the automation of surgical pathology work. While not an instantaneous solution, the mock-up outlines a path that can lead from contemporary manual practice via a route of increasing automation. Appropriate human supervision, in the form of quality assurance activities, will likely be a requisite to guiding the automation process to ensure that validated automation strategies proceed safely.