Efficient Visualization of Whole Slide Images in Web-based Viewers for Digital Pathology

The efficiency of 3 proprietary Web-based viewers has been evaluated in 2008⁹ comparing their pace to open an image (3–4 seconds) and display a field of view (FOV) (2–6 seconds). However, the included viewers and their underlying technologies do not exist anymore. Further, the authors compared the viewers based on 8 randomly selected slides, which provides limited statistical significance, and they did not include opening a case from an LIS, which is a crucial process for integrated DP. Finally, the authors did not discuss implementation details to explain or improve the performance.

Another aspect of viewer performance is image reproducibility and quality, for which the US Food and Drug Administration recently published official guidelines.^10,11  In contrast, our work focuses on streaming efficiency specifically for which no guidelines exist.

We describe our system setup and the performance metric definitions in the following Methods section. In the Results section, we report important relationships between specific viewer configurations and visualization changes using the metrics. We finalize the paper with a discussion of the results and their implications.

Our home-grown, Web-based slide viewer server runs in the hospital's data center. The server uses an internal solid-state-drive cache to store served image tiles, thumbnails, macros, label images, and other metadata. It is connected via 10‐Gigabit Ethernet to the digital slide storage (Dell, EMC Isilon H500-NAS-Platform).

Pathologists access the viewer via the LIS (Cerner, CoPath Plus) on their workstations using a consumer-grade 24” monitor with a resolution of 1920 × 1200 pixels. Workstations are connected via 1-Gigabit Ethernet with the hospital's backbone network. When opening digital images from within the LIS, the LIS launches a small local executable on the pathologist's machine that (1) receives the message file containing the selected image identifiers from the LIS, (2) signs the message via the hospital's active directory and a shared secret for automatic user authentication, (3) pushes the message to the viewer server, and (4) opens the local Web browser (such as Google Chrome, version 89) with a unique viewer URL to open the selected images. The viewer then displays the case including (1) the first WSI with its thumbnail, macro, label image, and image tiles needed to compose the first full-screen FOV, and (2) the thumbnail, macro, and slide labels of the remaining WSI in a virtual slide tray. To collect these data, each WSI has initially to be opened with its specific software development kit (SDK), which can be time consuming (on the order of seconds), such that the user has to wait for the case to be fully displayed. The time to launch slides of a given case and view the first image is part of our benchmark.

To visualize WSIs from the 4 different scanner models, Aperio AT2, Aperio GT450 (Leica Biosystems, Buffalo Grove, Illinois), IntelliSite Ultra Fast Scanner (Philips, Eindhoven, Netherlands), and Pannoramic 1000 (3DHistech Ltd., Budapest, Hungary), the vendor-agnostic viewer uses the open-source libraries OpenSlide (OS)¹²  and OpenSeadragon (OSD),¹³  as well as the vendors' proprietary SDKs from Philips¹⁴  and 3DHistech.¹⁵  OS and OSD are commonly used in Web-based slide viewers^16–21  as they are well suited to visualize and navigate a broad range of WSIs, being compatible with many slide formats. However, their impact on the WSI visualization speed has never been quantified.

To assess the viewer performance for this study, we defined 3 user-centric metrics and 1 technical metric to evaluate the turnaround time performance for Web-based slide viewers as shown in the following.

The time between opening the images of a case in the LIS and displaying the first digital slide in the viewer is defined as time to open a case. It is measured automatically in the viewer as the time between calling the viewer's API from the LIS and the full rendering of the first slide of the case.

The time between selecting a new WSI in the viewer's slide tray and displaying the corresponding slide in the viewer is defined as time to change a slide. It is measured automatically as the time between clicking on another slide of the case and the full rendering of that slide.

The time between zooming or panning to a different FOV and fully rendering that FOV is defined as time to show a field of view (TFOV). It is measured automatically in the viewer as the time between the start of a navigation and the completed rendering of the new FOV.

Note that this measure depends on the size of the FOV, as larger FOVs have to load more image tiles. To be comparable across experiments, we included only navigation events with an FOV size of 1920 × 790–1040 pixels, which corresponds to a full-screen view on the hospital workstations. Further, to exclude continuous zooming and panning events that we did not want to consider in this study, we included only FOV events with an effective number of loaded tiles between 0.5 and 2 times the number of expected tiles for the FOV size.

The time between querying and displaying a single tile is defined as time per tile. This is measured as the time between querying a tile for an FOV and receiving this tile. This metric is technically interesting for specific experiments in our work, but not directly relevant for pathologists, as they are not reviewing individual tiles, but rather a whole FOV.

All metrics were automatically measured using the viewer's JavaScript code and the specified events, and empirically collected in the running system over 3 months. We illustrate the significant influence of specific caching strategies, different compression techniques, and block and tile size configurations on these metrics and show how we improved our setup accordingly.

We use the word “block” to refer to the static image patches stored in the WSI file as they are generated during the scanning process. After scanning, they cannot be changed. We use the word “tiles” to refer to the image rectangles that compose the viewer's FOV. Tiles are generated by stitching and cropping blocks to the configured viewer's tile size.

During our study period of 3 months, 239 distinct pathologists reviewed a total of 60 080 WSIs (37 570 from Leica AT2, 20 899 from Leica GT450, 781 from Philips UFS, 830 from 3DHistech P1000). All slides were scanned at ×40 except for Leica AT2 slides scanned at ×20. In the following sections, we describe specific scanner and viewer settings that had measurable influence on the performance to visualize WSIs from a user-centric point of view.

Our scanners store the WSIs on a dedicated storage connected via 10-Gigabit Ethernet. Viewers can either stream WSIs directly from their original location on the storage cluster or alternatively cache them first temporarily on the viewer server at the time of accessing the corresponding case. With prior caching, the viewer server will first download the complete WSI from the storage to provide a faster streaming to the client afterward. Of note, download of the first WSI of a case must happen before visualization of the case to avoid concurrent access of the file for visualization, thumbnail generation, and label file extraction. Download of additional WSIs in the same case can happen in parallel in background while the first WSI is being displayed and navigated.

The viewer server will maintain cached versions of the WSIs and their thumbnails, macros, labels, and all previously visited tiles. When visiting a cached slide, the WSI does not need to be opened with the SDK, as the metadata are already prepared. Hence, the slide can be opened faster. This cache is cleared regularly every night on the server to save disk space.

Note that if a user opens a case in the same browser again, the browser might even reuse cached versions of the corresponding images, thereby increasing the speed of displaying the case further. However, Web-based viewers must rely on (limited) browser caching controls, optimized for websites and not necessarily for WSI visualization. This is a major difference from local desktop application viewers with advanced client-based caching strategies.

Figure 1, A through C, illustrates the tradeoff between opening of a case and tile streaming with and without WSI caching. While cases are opened quicker when disabling WSI caching (Figure 1, A and B), tiles are streamed slower (Figure 1, C and D) as each tile has to be generated on a remote file share attached via network. This data set includes 5691 cases with 12 165 WSIs accessed by our pathologists from the LIS between September 11 and September 24, 2020. On September 18, we enabled caching of the entire WSI in the viewer settings, and thereafter, we disabled caching. For the accessed WSIs, we measured 29 399 FOV and 35 049 tiles loaded (every third tile measured). We find that disabling WSI caching accelerates opening of a case significantly from 6.3 to 3.9 seconds and changing to another slide from 1.1 to 0.1 seconds, both improvements noticeable by the user. At the same time, directly streaming the WSI from the storage slightly decelerates the viewer performance at the FOV level (from 270–314 ms) and on a single tile level (77–163 ms). Because the performance gain to open cases and slides greatly outweighs the loss in streaming, we decided to keep WSI caching disabled.

WSIs from Leica AT2 Scanners can be compressed with either the JPEG or JPEG2000 standard. We evaluated how the alternative compression techniques affect the streaming of a Web-based viewer. The data set used in this experiment is a total of 2255 Leica files for which the viewer uses OS to access the WSI. The slides have an internal default block size of 512 pixels and a compression quality factor of Q = 70. Figure 2, A, indicates that JPEG2000-compressed slides need 515 ms to change the FOV of such a WSI, while JPEG‐compressed slides need 446 ms to change each FOV. This time difference might be hardly noticeable by the user. On the other hand, JPEG2000-compressed slides use on average a remarkable 40% less space on disk with 36 kB per block instead of 60 kB per block for JPEG-compressed WSIs (Figure 2, B). The superior compression rate of JPEG2000 comes at the cost of a longer decompression time due to a higher complexity.

We included WSIs from the scanners Philips UFS, Leica AT2, Leica GT450, and 3DHistech P1000 to evaluate the influence of the file format on the visualization performance in our viewer. Note that this comparison does not include the native vendors' viewers, but our home-grown, Web-based slide viewer. Therefore, this experiment does not make any claim about the particular vendors' systems. Rather, it gives an impression of how well one can use the file formats outside of their native systems.

To access JPEG-compressed Leica SVS files, we use both OS and alternatively our own implementation of an SVS reader. This implementation uses a byte reader that points to the SVS file and streams the individual JPEG-compressed blocks in the WSI. As a result, the viewer server can send the blocks directly to the client as they are saved in the WSI, without any additional decompression and compression as is done in OS to stitch and crop the blocks to the queried tiles. While this is fast and efficient, it limits the viewer's tile width to be identical with the WSI's block width, as horizontal block stitching would require decompression and recompression of the blocks. For the tile height, we can stitch multiple blocks together without decompression leading to nonquadratic tiles.

Figure 3, A and B, shows the viewer performance of all 3 scanner types on all 60 080 distinct WSIs reviewed by our pathologists. All WSIs were directly accessed from the storage cluster (ie, without prior caching on the viewer server). In total, 102 302 change-to-slide events were recorded (Figure 3, A). For changing to a slide in the viewer, it is important to differentiate between first-time access and consecutive access of the same slide. For the first-time access, the slide object has to be instantiated on the server (first call of OS, SVS Reader, or SDK). This takes a median of 333 ms for iSyntax files using Philips' SDK, 439 ms for SVS files using OS, 562 ms for SVS files using our byte reader, and 568 ms for MRXS files using 3DHistech's SDK. If a user reopens a case that has been opened earlier, the slides of all 4 vendors are visible almost immediately (below 100 ms). For the impact of the slide type on slide navigation, we measured a total of 163 017 relevant FOV events in the viewer (Figure 3, B). Loading FOVs showed a median of 318 ms for iSyntax files using Philips' SDK, 256 ms for SVS files using OS, 156 ms for SVS files using our byte reader, and 204 ms for MRXS files using 3DHistech's SDK.

When serving tiles in an FOV in a Web-based viewer, the tile size directly influences the performance of the viewer; with smaller tiles, an FOV can be served in parallel, but multiple network requests and file accesses are needed to provide the tiles. If there are too many tiles, browser limitations for concurrent requests per client come into play. On the other hand, with larger tiles, fewer tiles are requested for the FOV, but tiles are larger in size and need longer to be generated and transferred. We therefore investigated if there is an optimal tile size for efficient FOV loading. We separated all FOVs served in the viewer by WSI vendor (Philips, Leica, 3DHistech), WSI reader (OS, own, or vendor's SDK), and tile size (240, 256, 512, 720, 1024, and 1902 pixels) and illustrate this relationship in Figure 4, A through H. In the left column, the time per tile is plotted against the different tile sizes for the Philips (Figure 4, A and B), Leica (Figure 4, C through F), and 3DHistech (Figure 4, G and H) slides. As expected, small tiles are the fastest to be delivered in the viewer for all vendors. However, the right column plotting the TFOV demonstrates that small tiles still do not necessarily lead to faster FOVs. Throughout all vendors, there is an empirically optimal TFOV at a tile size of 1024 pixels. Further, for all vendors' slides, the time it takes to generate the FOV is approximately equal to the time per tile multiplied by the number of tiles loaded (yellow bars), indicating that OS, our own implementation, and the SDKs serve the tile queries in sequence rather than in parallel.

When experimenting with different tile sizes in the viewer, we received the following valuable feedback from our users: when small tiles were loaded in an FOV, there was a visible mosaic effect in the FOV that was perceived as disturbing. Larger tiles of 1024 pixels and above, instead, reduced this mosaic effect greatly, which was preferred by our users.

Finally, we investigated the relationship between block size and tile size when generating the FOV. The Leica scanners allow for the control of the block size, and we changed their configuration to use a block size of 240 (original), 256, 512, or 1024 pixels. This setting was not configurable for Philips or 3DHistech scanners and was preset to 128 and 256 pixels, respectively. After 3 months, our pathologists reviewed 22 345 WSIs included for this experiment. Figure 5 shows the TFOV stratified by block size and tile size. Similar as before, we observe an optimal block size for Leica WSI at 1024 pixels for fastest FOV streaming (median 167 ms per FOV) when using OS and a tile size of 1024 pixels. When using our own streaming technique without decompression and compression, this can be further reduced to 108 ms per FOV. The fastest configurations for Philips and 3DHistech WSIs in our viewer were tile sizes of 1024 pixels (median 238 ms per FOV) and 2048 pixels (median 125 ms per FOV), respectively.

For wide adoption of DP, digital slide viewers must be able to efficiently open cases and display and navigate slides. While microscopes visualize glass slides instantaneously, digital viewers are affected by slide loading times, image rendering times, and mosaic effects. Therefore, improved slide streaming techniques to accelerate loading and visualization times are needed. Owing to the large size of digital slides of several gigabytes compressed image data and due to their random access during slide review, these streaming techniques need specific considerations for pathology data, such as intelligent caching, efficient tiling and close connection of image data, and viewer servers. Still, little effort is done to systematically quantify relevant key measures for reviewing cases and slides, and therewith to discover common bottlenecks.

A most optimal viewer would have an ideal response time of zero seconds for all user interactions. This is of course not reachable, but it has been shown that response times below 1 second increase user productivity significantly (commonly known as Doherty's 400 ms threshold).²² 

To our knowledge, we present the first study to define and systematically quantify visualization efficiency via 3 user-centric key metrics for opening a case from the LIS, opening an individual image, and rendering an FOV. These metrics were measured in a Web-based digital viewer in a production system for routine pathology diagnostics at a large academic medical center with 60 080 different slides reviewed by 239 pathologists. Our results indicate an overall acceptable benchmark performance of less than 4 seconds (3.9 seconds) to open a case from the LIS, less than 1 second (333–568 ms) to change to another slide within the viewer, and less than one-fourth of a second (108–238 ms) to change the FOV. This is not only faster than what has been reported before,⁹  but also reaches productive reaction times.²²  Further improvements should be made for the time to open the case, which still takes most of the loading time.

This setup further allows for the comparing of different WSI file formats and critical system configurations, such as caching strategies, tile sizes, and even block sizes. We find that a relatively large block and tile size of 1024 pixels is favorable to both the speed to load an FOV and the user's perception (fewer mosaics). Conventional scanners and viewers use a block and tile size of 128 to 512 pixels, which might be optimal for smaller FOVs. Considering that computer screens get larger with higher resolution in future, we propose to adjust scanners and viewers accordingly.

Of note, we did not compare different viewers with each other, including the native viewers of the vendors' file formats. The reported numbers therefore do not reflect the overall performance of the specific file formats in their native systems, but only of these formats in our own Web-based viewer. While we can measure the proposed metrics in our home-grown system easily, no other viewer reports similar statistics of its own performance over time. With this study, we define key metrics that are meaningful to pathologists, provide a first baseline on what Web viewers can achieve on these metrics with nonstandard configurations, and encourage viewer developers to consider these metrics in their development to make viewers comparable to each other. We give insight in technical details on how to improve DP workflows with specific scanner and viewer configurations.

Not all improvements presented in this study have the same impact. While disabling slide caching accelerated case opening most notably in the order of seconds, changing the block and tile sizes increased the rendering of an FOV in the order of milliseconds, hardly noticeable by the user. Still, it provided a positive association due to reduced mosaic effects. However, the provided metrics will enable us to objectively compare further system configurations and additional viewers and scanners, to drive the development toward a seamless DP workflow. For example, further improvements of digital viewers that were not discussed in this study include pretiling of WSIs, new image compression techniques,^23,24  and advanced Web streaming.^25,26  With improved digital systems, we hypothesize, digital sign out will be on par or even more efficient than with a microscope.