Augmented Reality Technology Using Microsoft HoloLens in Anatomic Pathology

A Microsoft HoloLens (Development Edition) was connected to the Internet via our institution's (Pittsburgh, Pennsylvania) secure wireless network. The HoloLens contains an 802.11ac, 2×2 wireless fidelity (WiFi) radio, which means it can only connect to a network supporting the 5-GHz band. Connecting the device to a WiFi network was similar to connecting a laptop or mobile device to a WiFi network. The institution's secure wireless network required specifications such as Encrypted and authenticated Transport Layer Security (ETLS) or Protected Extensible Authentication Protocol (PEAP). The secure network provided a 5-Mbps minimum bandwidth connection. To gain access to this network, we required a service account in the institution's Identity Management System; once approved, the mobile device could access the secure network through a designated username and password.

The HoloLens device is a lightweight (579 g), wearable holographic computer, enabling human-computer interaction in a mixed-reality environment. Holographic computer images can be viewed within a user's point of view of native surroundings. It consists of a widescreen stereoscopic head-mounted display (resolution 1268 × 720 pixels per eye, 16:9 aspect ratio, and 60-Hz refresh rate) containing tinted holographic lenses, a depth camera (120° × 120°), and 4 additional cameras with built-in sensors for environment sensing and detecting ambient light. The sensors receive user input through inertial measurement units that include an accelerometer, gyroscope, and magnetometer. It also has integrated speakers and 4 microphones for 2-way communication. The device's Holographic Processing Unit can handle 1 trillion calculations per second, and has 2 gigabyte (GB) of random access memory (RAM). Further specifications include 64 GB of internal flash storage and support for wireless connectivity with WiFi (IEEE 802.11ac) and Bluetooth (4.1 Low Energy). Battery life carries up to 3 hours of active use or a standby time of 2 weeks. The HoloLens has a clicker accessory, connected via Bluetooth, allowing users to make physical commands instead of using gaze, voice commands, and/or hand gestures.

AR sessions during testing were captured with a Microsoft Surface Pro 3 computer (weighing 1.76 lb). This is a 12” high-definition (2160 × 1440) touchscreen tablet computer (64-bit) with 64 GB of storage and 4-GB RAM. The tablet was powered by a fourth-generation Intel Core i5 processor (Santa Clara, California) and used the Windows 10 operating system.

The HoloLens used a Windows Holographic platform under the native Windows 10 operating system. The platform included a Microsoft application repository. Applications specific to the HoloLens were directly downloaded and installed on the device via the Windows store. The device's interface used gaze input (head tracking), gesture (bloom, air tap, press and hold), and voice commands.⁸  Three gestures were used to interact with the AR environment:

1. Bloom: upward facing palm, starting with fingertips together, then spreading fingers outward; for application start-up and closure.

2. Air tap (tap and release): with the dorsal aspect of the user's hand facing them, raising and flexing the index finger (ie, up, down, and up again) in a pinchlike fashion (press and release); for selecting an operation.

3. Tap and hold: raising and flexing the index finger to the thumb and motioning the pressed fingers together (press and hold); in the user's 3D space (ie, up then down)—for manipulation of selected objects.

A Bluetooth Clicker was also provided with the device that can mimic hand gestures. Voice commands were interchangeably used (eg, “select” instead of the air-tap gesture, or “take a picture” to capture a live point-of-view mixed-reality screen-capture). Internet connectivity was established via the secure hospital network, which caused no problems. No patient health information data were recorded. Photos and videos were captured with the HoloLens and backed up on our institution's Microsoft OneDrive cloud storage. Voice command instructions using Microsoft's software intelligent personal assistant, Cortana, were used to capture photos and video. With a remote desktop agent the HoloLens was used to create a dynamic virtual workstation. By means of this virtual “cockpit” users were able to access and work with various applications (eg, Microsoft Outlook, laboratory information system, electronic medical record, whole slide image [WSI] viewer).

After establishing secure Internet connectivity in the autopsy suite, Skype (version 7.26.0.101, Microsoft) was installed on the Surface Pro with a Skype for HoloLens plug-in, and Skype Beta was installed on the HoloLens device. These applications allowed for simultaneous live streaming and bidirectional virtual annotation between both devices. By annotating in real-time, users could virtually point an arrow or freehand draw on their surrounding environment. By using these applications, remote communication was initiated for users to contact and interact with one another. Our director of autopsy was selected as an expert pathologist user to virtually guide pathology trainees through their autopsy procedure.

A Shining 3D Einscan-S 3D scanner (Hangzhou, China) with Einscan 3D software (version 1.7.3) was used to capture 3D images of gross specimens. Detailed methods of specimen scanning and software enhancement to produce a 360° 3D virtual image have been previously reported.⁹  Digital files of scanned 3D specimens were saved in object file format (.OBJ) and imported into open-source Blender software (Amsterdam, the Netherlands) to be scaled, edited, and oriented for viewing. The final 3D specimen image was then exported from Blender in Filmbox (.FBX) file format. The Filmbox format was the only readable 3D holographic file format that could be natively imported into the Microsoft HoloLens 3D Viewer Beta application. 3D viewer Beta application was used to view 3D model holograms of our scanned specimens. The 3D viewer allowed users to interact with holographic images (eg, resize, rotate, flip plane axes, mirror).

Similar to the autopsy setup, the HoloLens and Surface Pro devices with Skype Beta for HoloLens were used for live streaming and bidirectional virtual annotation to support gross specimen telepathology. Pathologists' assistants and trainees wearing the HoloLens received instructions (eg, selection of representative areas from a specimen to obtain tissue sections) from a remote pathologist using audio, visual communication, and virtual annotations.

Conventional and AR workflows were compared. Pathologists' assistants wearing the HoloLens device at the grossing workbench were asked to manipulate (eg, move, scale) virtual radiographs (eg, mammogram) and overlay them atop corresponding gross specimens. Findings in the radiograph (eg, calcification, biopsy marker clip) were used to locate corresponding tissue areas within the gross specimen (eg, mastectomy, breast lumpectomy excision). Patient specimens included 12 total mastectomies and 12 excisional breast resections (ie, lumpectomies, excisional breast biopsies); 9 patients underwent neoadjuvant chemotherapy with or without radiation divided among both workflow methods. Pathology diagnoses included invasive ductal carcinoma (n = 20), invasive lobular carcinoma (n = 1), atypical ductal hyperplasia (n = 1), and papilloma (n = 2). Specimen radiographs from the picture archiving and communication system (PACS) were de-identified and imported to the Microsoft HoloLens as JPEG image files through the device's OneDrive cloud storage account. Conventional workflow included manual detection during specimen dissection and pathologists' assistants having the specimen radiograph display on the monitor if retrieved from the PACS.

Microsoft Edge Web browser was used on the HoloLens to access the Internet. The remote desktop application Remote Desktop Preview App was downloaded and investigated for remote navigation of digital slides stored on an institutional server. Several Web-based WSI viewers were tested with the HoloLens (Table). All viewers were freely accessible. Supported gestures for image navigation included scroll, pan, and zoom. Compatibility and usability of each WSI viewer was documented. Features of interest include loading of WSI Viewer Web page, rendering of WSI, and gesture functionality (ie, scroll, pan, and zoom navigation of WSI).

Formalin-fixed paraffin-embedded blocks of human and animal tissue samples were processed by 3Scan (San Francisco, California) with their knife-edge scanning microscopy technology. This process generates images from thousands of regular, serial tissue sections with fiber optic microscopy. The images are subsequently reconstructed in 3D (OBJ image files) to study cellular and subcellular structures.¹⁰  The HoloLens was used to view these images with the 3D Viewer Beta application.

Overall, the Microsoft HoloLens device was light and comfortable to wear, easy to use, and provided sufficient computing power. Gesture input, although limited to only 3 gestures, did provide adequate interaction for all pathology use cases to be performed successfully. For all use cases in this study, the bloom, air tap, and tap and hold were used as well as associated interchangeable voice commands or calling on Cortana for expedited command operation. The device facilitated remote, real-time, bidirectional audio, and mixed-reality annotation. Image resolution was sufficient to support digital pathology including telecommunication during autopsy, telepathology during grossing, viewing and manipulating 3D pathology specimens, and WSI. Captured photos, using the HoloLens, averaged 300 KB in size, with a resolution of 1408 × 792, and 72 dots per inch. The creation of a virtual workstation successfully allowed multiple virtual windows to be displayed by the HoloLens in the user's visual field. Interacting with these windows allowed users to resize and reposition them. Setting up a virtual desktop permitted a variety of secure clinical applications (eg, laboratory information system, electronic health record, PACS) to be remotely accessed from any location (eg, at the gross workbench) (Figure 1).

A pathologist using Skype was able to successfully establish remote communication with pathology house staff wearing the HoloLens device through the Skype for HoloLens application during an autopsy conducted in the autopsy suite located in the basement of the hospital. The Microsoft Surface Pro tablet was used by the attending pathologist, whereas the HoloLens was donned by house staff. Face shields were worn in such a fashion as to not obstruct the HoloLens sensors. Audio quality and visual resolution were sufficient for both users. For example, the attending pathologist was able to easily determine that there was no pulmonary embolism in one case while remotely directing the trainee during dissection of the pulmonary vasculature. The pathologist and trainees both used the air tap and the tap and hold to create virtual arrows and freehand drawings during AR sessions. Using the HoloLens by means of interactive gestures, trainees performing an autopsy successfully recorded organ weights, pathology notes, and voice-commanded image acquisition of gross findings including screen-captures with annotations. For attending pathologists, advantageous uses of Skype included remote portability, remote supervision, quick access to other pathologists if needed, a more timely response to trainee inquiries that avoided going to the morgue, and the ability to record or document the interaction. Educational autopsy live video streaming or recordings could be used for future teaching and education.¹¹ 

Multiple gross pathology specimens (eg, whole brain including subsequent coronal slices, prostate, amputated digit, kidney with tumor) were scanned with the Einscan 3D scanner. A 360° 3D rendition of these scanned gross specimens was generated. Specimens with a coarse surface texture, as opposed to those with a shiny reflective surface, were most successfully visualized. Gray and white matter tracks of coronal brain slices were also less clear. Imported 3D gross specimens were successfully viewed with the HoloLens (Figure 2). Using hand gestures, these virtual specimens could be scaled and manipulated (eg, rolled around the x-, y-, and z-axes). Live streaming allowed 3D gross specimens in the AR field to be shared for educational purposes.

Virtual images remotely transmitted for review by a pathologist demonstrated adequate resolution for a variety of gross specimens (eg, nephrectomy, prostatectomy). Telepathology was performed, without difficulty, by pathology residents and pathologists' assistants wearing the HoloLens. Bidirectional annotation via hand gestures was used to successfully guide gross specimen examination. For example, a pathologist was able to remotely specify tissue areas to be sectioned during intraoperative consultations and remote gross dissections.

Both methodologies for coregistration of breast resection specimen radiographs and gross specimens for biopsy clip detection and correlation were performed by using conventional workflow and the head-mounted AR device. The HoloLens, worn by pathologists' assistants during gross examination of breast excision specimens, was effectively used to retrieve corresponding specimen radiographs from the PACS. There was sufficient resolution in these virtually displayed radiology images to clearly identify breast lesions and metallic biopsy clips. The imported radiographs were easily manipulated with gestures in order for them to be accurately coregistered over gross specimens in real-time (Figure 3, A through D). This process was used to rapidly identify the precise location of a biopsy clip or lesion such as calcifications in the breast resection specimen. Pathologists' assistants were successfully able to manually coregister the radiograph and gross specimen by using hand gestures and voice commands to scale and position the radiograph atop the gross specimen. Identification of the metallic biopsy clip averaged 10.6 minutes when using conventional workflow applications and 1.5 minutes when using the Microsoft HoloLens. One hundred percent of users rated the Microsoft HoloLens 4 or greater on a 5-point scale in being a useful and usable technology for radiograph-specimen coregistration.

Whole slide image viewing in 2 dimensions within the HoloLens AR environment was possible. A range of gestures permitted users to navigate digital slides and zoom in on selected fields of view. Drag mode allowed users to pan around the WSI by directing their fingers toward the intended direction of the digital slide. Zooming in was performed by directing the end-users' fingers upward and zooming out was performed by directing their fingers downward. Of the 9 WSI viewers tested (Table), 3 were not compatible owing to Flash plug-in encoding built into the viewer (DBViewer [Hanover, Germany], Webscope [Aperio, Buffalo Grove, Illinois], Zoomify [Zoomify, Aptos, California]). These viewer Web pages did not load in the HoloLens device Edge browser, or render the WSI for navigation. Three WSI viewers (AJAX [MSTechnology, Charlotte, North Carolina], Google Maps API [Google, Mountainview, California], and OpenLayers) permitted viewing, but the gestures in all navigation modes did not function as intended. For instance, while attempting to use the tap and hold gesture in the zoom navigation mode, the WSI would continuously pan in one direction and not zoom in. The remaining compatible 3 WSI viewers (DigitalScope [Aptia Systems, Houston, Texas], ImageZoomer [ThietkewebDrupal, Hai Ba Trung, Hanoi, Vietnam], and OpenSeadragon) permitted viewing, and all gestures worked as intended in all navigation modes. A Bluetooth-connected keyboard and mouse were also successfully used for navigation input. Whole slide images can successfully be viewed on the HoloLens device through Web-enabled viewers.

Tissue from mouse gastrointestinal tract, lungs, and blood vessels was successfully serially sectioned, imaged, and reconstructed by using the knife-edge scanning microscopy technique.¹⁰  3D image file data sets were converted from .OBJ files to .FBX files to be imported into the Microsoft HoloLens (Figure 4, A and B). Image file sizes ranged from 17 MB to 1 GB. Large files (>50 MB) were not compatible with HoloLens 3D viewer applications. Performance load capabilities were also limited to 150 000 vertices or more than 400 meshes. While optimization of image files allowed them to be imported into the HoloLens, they could not be manipulated.

While the capability to deliver AR experiences began decades ago, only recently has this technology become easily accessible, affordable, and transportable. Compared to AR, VR solutions are more widely developed and used, even for use in pathology.^1,12–17  To the best of our knowledge, we are not aware of any other studies demonstrating AR-specific applications using the HoloLens in pathology.

AR is “active” technology, whereby users can interact with virtual objects relative to real life. One of the drawbacks of using Google Glass for telepathology was poor image resolution.¹⁸  Similarly, low image resolution of the Oculus Rift device was a major limitation for viewing WSIs.¹  This study showed that image quality with the HoloLens was sufficient for viewing gross and microscopic (eg, WSI) pathology images. The resolution of the HoloLens of 2536 × 1440 (1268 × 720 per eye) surpasses that of earlier VR/AR devices. For example, the Oculus Rift (Development Kit 1) by comparison only has an image resolution of 1280 × 800 (640 × 800 per eye). AR also has other advantages over VR devices. For instance, it was reported that users were able to complete tasks related to 3D object manipulation up to 22.5% faster in an AR environment than with VR.¹⁹ 

The Microsoft HoloLens is a novel AR tool with potential for clinical and educational applications in health care. Medical education is transforming by using safer simulation technologies.²⁰  A recent study²¹  reported higher achievements and lower cognitive burden from medical students learning neuroanatomy with AR than with traditional textbook methods. Educational content specific to the Microsoft HoloLens today is mostly devoted to anatomy (eg, HoloAnatomy, HoloHeart [Made in Point GmbH, Erlangen, Germany]). There is still a gap regarding educational content for pathology. AR also has novel clinical applications such as 3D volumetric reconstruction to coregister radiology, gross anatomy, and histopathology to support radiation treatment planning in patients with cancer.²²  AR using 3D-integrated intraoperative imaging has proved to be reliable in performing minimally invasive procedures in a variety of neurosurgical diseases, showing superior accuracy of hardware placement as compared to free-hand techniques.^23,24 

Security for health care–related AR or VR head-mounted devices is critical. Indeed, the main vulnerability when connecting these wearable devices to the Internet is security. Usually, for a mobile device to have access on our institution's network, a passcode for the device is necessary. However, the HoloLens does not have login entry password-protected capability. This makes it hard to lock down. Also, for mobile devices (eg, smartphones) our institution uses MobileIron (Mountain View, California) software to secure content on these devices. The MobileIron app, unfortunately, was not yet available or compatible with the HoloLens. Hence, with repeated use of this “open device” on our network, hospital information technology security officers may not recognize the HoloLens and hence remove it from the network. Fortunately, we did not have any of these problems. The other problem with relying on WiFi connectivity was the geographic limitation of using the device only within our network.

Our experience shows that wearing the HoloLens is feasible and permits pathologists, pathologists' assistants, and trainees to easily manipulate their surrounding mixed-reality environment for a variety of clinical and nonclinical use cases. This includes supervising prosectors and pathology residents during an autopsy or while grossing surgical pathology specimens, evaluating 3D gross specimens as holograms, remotely interacting with WSIs, reconstructed microscopic images (volumetric pathology), facilitating telepathology, and aiding in real-time pathology-radiology correlation. Not all WSI viewers are yet configured for use in such a mixed-reality environment. Future directions should include developing WSI viewers for easy navigation with AR and/or VR devices. Remote telepathology applications for the HoloLens has tremendous potential. Volumetric pathology represents the study of pathology in 3 dimensions, where 3D visualization of hundreds or thousands of coregistered histopathology WSIs is possible. Coregistration of radiology images (eg, computed tomography, magnetic resonance imaging) and digitized slides from postmortem human brain specimens has been successfully mapped with an accuracy of 0.56 ± 0.39 to 0.87 ± 0.42 mm.²⁵  It remains to be seen if this can be improved with the HoloLens. The HoloLens also supports a wide variety of hands-free tasks (eg, voice commands, documentation, annotation, video recording) and digital imaging applications (eg, whole slide imaging). A limitation of using both AR and VR head-mounted devices is a type of visually induced motion sickness, better known as simulator sickness. Simulator sickness has been described in VR more than AR devices, possibly due to native surrounding visualization in AR. This discomfort is characterized by nausea, disorientation, eye strain, or other oculomotor symptoms, and can negatively impact the user's experience, acceptance, performance, and safety. We did not find this in our testing, perhaps because the HoloLens was not used for prolonged periods of time. Individual consecutive user times ranged from a few minutes (eg, during WSI navigation, rad-path correlation, 3D gross specimen manipulation) up to 1 hour (eg, during an autopsy). Users did not report episodes or symptoms of simulator sickness. Indeed, some evidence shows that shorter footage duration viewed on a head-mounted display may be better tolerated in VR.²⁶ 

In summary, we found that using AR in pathology for clinical and nonclinical reasons is promising. The HoloLens introduces many novel opportunities that could enrich the practice of pathology, such as creating virtual workstations and exploiting dynamic ways to manipulate spatial digital data. However, further testing is required to validate the HoloLens for use in routine clinical practice.

We thank 3Scan for generating and sharing their 3D images.