Immersive technologies, including virtual reality (VR), augmented reality (AR), and mixed reality (MR), can connect people using enhanced data visualizations to better involve stakeholders as integral members of the process. Immersive technologies have started to change the research on multidimensional genomic data analysis for disease diagnostics and treatments. Immersive technologies are highlighted in some research for health and clinical needs, especially for precision medicine innovation. The use of immersive technology for genomic data analysis has recently received attention from the research community. Genomic data analytics research seeks to integrate immersive technologies to build more natural human-computer interactions that allow better perception engagements. Immersive technologies, especially VR, help humans perceive the digital world as real and give learning output with lower performance errors and higher accuracy. However, there are limited reviews about immersive technologies used in healthcare and genomic data analysis with specific digital health applications. This paper contributes a comprehensive review of using immersive technologies for digital health applications, including patient-centric applications, medical domain education, and data analysis, especially genomic data visual analytics. We highlight the evolution of a visual analysis using VR as a case study for how immersive technologies step, can by step, move into the genomic data analysis domain. The discussion and conclusion summarize the current immersive technology applications' usability, innovation, and future work in the healthcare domain, and digital health data visual analytics.

As the years pass by, the technologies in healthcare continue to improve patient treatment outcomes, help enhance the quality of care, and lower cost. Innovative technologies including immersive technologies are used for augmented reality (AR) training, healthcare data leveraging, patient-customer experience personalization, big data analytics, and health outcome retrieval from data.[1] Immersive technologies allow the users to have the perception of being physically present in a nonphysical world using different stimuli such as images and sound. These technologies are becoming more affordable, user-friendly, and pervasive. They have also been adopted and embraced by several industries, including healthcare.[2] This is particularly true with the advanced analytical methods and massive quantities of healthcare data being collected from patients and the general population. Healthcare experts will address the extensive unmet information, make a prediction, and select a treatment method based on a patient's genetic profile instead of a one-size-fits-all approach.[3] Artificial intelligence (AI), precision medicine, and immersive technology are among the most promising health technologies.[4] The convergence of AI and immersive technology trends offer opportunities currently available to health domain users that will change many of the experiences that they already have. As health data continue to become more diverse, the need for tools that facilitate interactive, versatile, and integrative visualization of complex big data is also growing. Immersive technology is developed to offer natural interactions between humans and machines, and incorporate other perception dimensions, such as sound,[5] over conventional data visualization approaches. AI, when added to the novel data visualization tools, makes them even more effective when processing complex data.[6]

Machine learning, as a branch of AI,[7] is used as one of the leading health technologies for enhancing accuracy in clinical insights, improving decision making, avoiding errors such as misdiagnosis and unnecessary procedures, helping in the ordering and interpretating of appropriate tests, and recommending treatment methods.[8] Machine learning has been used in the field of “precision medicine” for years. According to the U.S. National Library of Medicine,[9] precision medicine is “an emerging approach for disease treatment and prevention that takes into account individual variability in genes, environment, and lifestyle for each person.” Precision medicine enables physicians to determine personalized treatments for patients by interrogating the data of a patient's genetic history, location, environmental factors, lifestyle, and habits instead of a blanketed approach for all patients. Machine learning can make data visualizations dynamic with real-time analytics, find more granular and actionable insights, create better searches for visualization dashboards, and create better predictive models.[10]

Virtual reality (VR) provides users with a virtual or simulated experience that excludes their actual physical surroundings. It helps humans perceive the digital world as real and gives learning output with lower performance errors and higher accuracy. VR is an immersive technology that allows the user to explore and manipulate computer-generated three-dimensional (3D) environments in real time.[11] A VR environment, in comparison with other media, has fewer distractions, more space, and more natural interactions. VR has become more portable, immersive, and vivid, which has enabled the technology to be used in a broad range of medical applications,[12] such as neurological disease, and other domains, including education[13] and construction safety.[14] AR superimposes digital elements on top of the user's physical environment through a device. AR is another immersive environment that provides a real-time overlay of reality with digital information.[15] Mixed reality (MR) blends the digital world with the physical environment integrated with the user's physical surroundings. Immersive technology platforms can precisely translate movements from the physical world to the virtual world.[16] VR, AR, and MR together are also called extended reality (XR).[17] With the mature immersive game engine architecture, we can design controls that mimic novel physical actions, such as grabbing a data point to extend workflows and presenting data into the virtual world.[18]

The field of healthcare discovery demands such visualization tools for the clear and comprehensive representation of data exploration to lead to new insights and discovery. Immersive technologies anchor virtual objects to the real world and allow users to interact with the virtual objects.[19] Machine learning and immersive technology are both technologies that present opportunities independently; however, combining them will make various experiences even more interactive and engaging.[20] XR and immersive environments open up new opportunities for diagnostics, medical education, preoperative planning, and intraoperative support because their 3D interactive and immersive nature allows users to absorb and retain more information.[21] VR has been used in pain management, physical therapy, fears and phobias, and cognitive rehabilitation for its unrivaled engagement ability.[22] Visual data analytics has proven essential in genomics research to gain insight into biological processes, find correlations and trends in large data sets, and communicate outcomes to others.[23] Big complex genomic data analysis needs methods to support flexible and dynamic queries to search for informative insights over very large collections in very high dimensions.[24] Immersive technologies could improve scientific genomic data visualization and interpretation by combining the natural 3D environment with natural human pattern recognition to uncover multidimensional relationships in data, especially when VR visualization tools are integrated with machine learning models. The VR display environment could amplify human perception and assist users in recognizing patterns in genomic data, as has been shown in multiple studies[2527] in the digital health domain.

The AR and VR industries were valued at $14.1 billion in 2017 according to Statista. The compound annual growth rate of the AR/VR industry is expected to be in the range of 40% to 80%. Statista predicts that the AR/VR industry will be valued at $209 billion by 2022.[28] There is no doubt that healthcare will also jump in on the action. More commercial VR headsets are launched, which is expected to accelerate the growth of the market. Making the VR experience more realistic is a key driver for market adoption and penetration.[29] As immersive technology is only now starting to find its way into the healthcare domain, its innovative potential has yet to be fully realized. For example, there are thousands of exciting and innovative immersive technology projects coming along in clinical studies listed at ClinicalTrials.gov[30] and NIH (National Institutes of Health) RePORTER.[31] There was a dramatic fund increase from 2018 to 2021. At this stage, more than 85% of the projects are for the clinical management of children or older adults. It is encouraging to see almost 5% of the projects that are using immersive and XR technologies for health data analysis. However, there are not many reviews and research work that highlight the gaps and discuss the possible solutions, and there are limited works on genomic data visualization in immersive environments, especially on the approaches to integrate different technologies such as machine learning and game engine theory. This paper contributes a comprehensive review on the existing immersive projects for healthcare with a focus on patient-centric projects, medical education, and data visual analytics. We use a case study to find how immersive technologies step, can by step, move into the genomic data analysis domain. We review and discuss the combination of immersive technology, human-machine interaction, data visualization, and machine learning to solve real clinical cases. Current issues and future trends also are discussed.

The following section, “Immersive Technology Projects for HealthCare,” reviews the current immersive projects and applications directly used by the patient, physician, or practitioner in healthcare, including dementia, remote consultation, aged care, children's anxiety, psychiatric care, distraction therapy, and healthcare education. Immersive technology used for healthcare education inspires the use of digital health data visualizations because users potentially engage more in an immersive environment to understand the patterns and insights from the complex big health data. The section, “Immersive Technologies for Visual Analytics of Health Data,” focuses on reviewing immersive technology used for data analysis, especially for health data analysis. A research use case is also reviewed for the evolution of immersive technology used for healthcare data analysis. The final two sections discuss the current issues and the future work for VR technologies in healthcare data analytics.

Interactive visualization seeks to go beyond the traditional environment into emerging platforms such as VR, AR, MR, large and high-resolution displays, and mobile devices. Complex data visualization and interaction tools in the XR environments could enhance humans' perception of where the interactions could be more natural or more effortless.[32] There has been increasing use of immersive technologies in the healthcare sector, such as medical training, patient treatment, medical marketing, and disease awareness.[33]

We frame this onto healthcare data resolution with a specific focus on the use of both immersive technology and machine learning for genomic data visualizations for the period between 2000 and 2020. It commences with a general search on a search engine, such as Google Scholar and Google search engine, and then in several databases, such as Springer, Nature, Genome Research, IEEE, and ACM. We also collect information through relevant market reports, such as the Mordor Intelligence Analysis Report. The search terms included “Genomic VR project,” “Healthcare VR applications,” “Machine learning combines immersive technology,” and “VR data visualization tools.” These words were used for all the other database searches. Only studies published in the English language were included for review. The main reviewer extracted and analyzed data from all articles in consultation with the other authors.

Healthcare Immersive Technology Projects

In clinical practice, immersive technologies offer the potential to facilitate the manipulation of patient-specific needs.[34] Immersive technology is being used in many ways in the healthcare domain, as shown in Figure 1. Each branch represents one type of use in the healthcare domain. Immersive technologies are gaining insight into life with dementia, enabling remote general practitioner (GP) consultation, activating aged care, easing child anxiety, automating psychiatric care delivery, supporting healthcare education, providing distraction therapy for patients with cancer, and delving into health data analytics. Some finished and ongoing immersive healthcare projects related to Figure 1 are listed in Tables 1 and 2. Detailed project description and discussion are listed in the following paragraphs “Patient-Centric Immersive Solutions in the Healthcare Domain” and “Physician- or Practitioner-Centered Immersive Technology for Healthcare Education.” We can find that immersive technology can enhance learning and improve productivity in healthcare domain training from these projects. Data analytics and visualization then use this better perception advantage to help users find patterns and insights from complex big data. Data analytics projects and health data analytics with immersive technology are listed in Table 2. The discussion about these projects is also listed and discussed in section “Immersive Technologies for Visual Analytics of Health Data.”

Figure 1

Immersive technology has been used in the healthcare domain in many ways including health data analytics. Each branch stands for one type of healthcare domain that immersive technology is used for. See Tables 1 and 2 for additional details on each technology.

Figure 1

Immersive technology has been used in the healthcare domain in many ways including health data analytics. Each branch stands for one type of healthcare domain that immersive technology is used for. See Tables 1 and 2 for additional details on each technology.

Close modal
Table 1

Immersive technologies and innovations used in the healthcare domain

Immersive technologies and innovations used in the healthcare domain
Immersive technologies and innovations used in the healthcare domain
Table 2

Immersive technologies for data analysis

Immersive technologies for data analysis
Immersive technologies for data analysis

Patient-Centric Immersive Solutions in the Healthcare Domain

User-centric design (UCD) is an iterative design process to bring the users into every stage of the design process to create highly usable and accessible products for users. Many immersive technology applications have been prioritized for patient-centric immersive solutions.

Immersive technologies can help with distraction therapy. Patients are often anxious, which is known as white coat syndrome.[35] Immersive technologies can help provide an immersive experience for patients to tour a health facility visually, to help patients cope with pain, and to ease children's anxiety.[36] For example, there is a project Distraction Therapy for Cancer Patients,[37] which Samsung ran under a collaboration of Start VR and Chris O'Brien Lifehouse. When patients become worried about unknown outcomes during sitting and waiting for periods of time, the VR application allows them to escape the experience of chemotherapy and give them a perception of 3D blended space to temporarily forget what is going on. Also, such an immersive experience before surgery distracts patients and allows them to keep their spirits up.

Immersive technology is being used for Alzheimer's disease, for example, the project “Gaining Insight into Life with Dementias” created by Alzheimer's Research UK and VISYON.[38] An innovative smartphone app named “A Walk-Through Dementia” is provided to users with an immersive space to feel what it is like to live with dementia. The main goal of this application is to provide the users with a clearer view of the challenges that patients with Alzheimer's disease face in everyday life and give a feeling into the emotional impact of symptoms. The Android phones application uses a combination of 360-degree video and VR environments to illustrate how even the most common everyday tasks for someone living with dementia, such as making a cup of tea, can become a challenge. Immersive technology helps people to engage with the impact of dementia on a new level.[39]

In addition, VR can be used for remote GP consultation. One example is the Remote GP Consultation project, which is run by the Silver Chain Group.[40] This project made it possible for a nurse to check the patients' health data and consult with a remote holographic doctor enabled with AR technology. This project is seeking methods to help a nurse who needs to speak to a doctor but is busy helping patients, so he or she can look through Microsoft HoloLens[41] to see patients' data and speak to a holographic doctor. The HoloLens VR/AR/MR technology makes it possible for the doctors to stay in their current location while still helping out and finishing their job. The project can potentially save the healthcare system considerable time and money. Patients then could receive similar care in the comfort of their own homes.

Immersive technologies can be used for aged care, such as aged care project[42] run by a Melbourne aged care, called BlueCross Care. They use VR technology in aged care to enable elderly individuals to experience travel, adventure, aquatics, even classical concerts and theatrical performances that they may have not been able to reach due to their age, from the comfort of their homes. Staff and family can also watch simultaneously on another device or screen to enjoy the VR experience.

Furthermore, immersive technologies can be easily accepted by children for easing anxiety. At St John of God Health Care hospitals, a promising future is seen in VR goggles that are used to help children when they have blood tests and other painful procedures to feel less anxious.[43] The immersive technology distracts children by taking them on an underwater adventure, and the children who are in the virtual world with the VR goggles are visibly more relaxed. Moreover, when clinicians need to complete simple but sometimes painful procedures for children, using VR goggles can also reduce the time. Another similar VR research study is being conducted at Monash Children's Hospital and Monash University; the clinicians use immersive technologies to help distract pediatric patients from painful procedures in the Pathology and Emergency Departments.[44] With immersive technologies, the users are transported to an engaging and interactive 3D “virtual world” that provides an escape from the real world. When the painful procedure is performed, the fear and pain associated with needle-based procedures are reduced.

There are more possible applications of VR/AR/MR in the healthcare domain. Adult automated psychiatric care delivery can use immersive technologies as well. For example, the patients can be empowered by VR to confront the sources of their anxieties when maintaining their dignity and skirting painful consequences. The research study, led by the University of Oxford's Department of Psychiatry,[45] shows that immersive technologies not only potentially help patients with schizophrenia silence the voices in their heads, but is also proven to help patients overcome their fear of heights. New evidence shows that the results are even better than what is expected from face-to-face therapy.

Physician- or Practitioner-Centered Immersive Technology for Healthcare Education

Immersive technology implementations for a physician not only have a distinct focus on the patient, but also concentrate on aspects of healthcare education for its better engagement feature. XR creates a computer-generated real or artificial 3D multimedia sensory environment and allows users to explore and manipulate it in real time.[46] Immersive technologies allow the human to percept the digital world as real and interact with objects and/or perform a series of actions in this digital world.[11] Through different levels of immersion based on engagement, empowerment, and self-actualization, immersive technologies also allow for a first-person active learning experience.[47]

Immersive technologies have the ability to enable the individualized repetitive practice of motor function while engaging and stimulating cognitive processes,[48] so the technologies are suitable to be used in training such skills as laparoscopic surgery, education of orthopedic residents, gynecology residents, suturing, ultrasound, nursing procedures, and paramedical interventions.[49] Immersive technologies provide a great way to teach and train surgeons, assist nurses and other medical professionals, and offer the next level of education in the healthcare domain.

Immersive technologies in surgery provide an opportunity to be fully immersed in a situation that is nearly identical to an actual operation.[50] Immersive technology such as VR can potentially improve and standardize both cognitive and technical skills and make it free from the demands of traditional clinical environments.[51] VR used for teaching purposes has already been successfully implemented in many clinics around the world. For example, Stanford University has a VR system that is used to help train residents, assist surgeons in planning upcoming operations, and educate patients.[52] The system also helps surgeons in the operating room by guiding them in a 3D space. VR and MR surgical simulators provide a risk-free training environment and can work as an indispensable part of physicians' training. For example, MR is used for rasping procedures in the artificial cervical disc replacement surgery with a VICON motion tracking system.[53] VICON is an active optical motion tracking system that allows users to in real-time track the motion of the surgical tools and plastic spine models. Evidence showed that people trained by VR/MR had lower performance errors and higher accuracy than those trained by conventional approaches.[54,55] Immersive technologies also start to be used in digital health data visualizations because users potentially engage more when in the immersive environment to understand the patterns and insights from the complex big health data.[56,57]

VR Benefits Visual Data Analytics

As immersive analytics combine the use of immersive technologies, natural user interface, and visualization, best-known techniques to achieve transparent cognitive experience and to enable visual analytics in the immersive environment,[5860] it provides novel opportunities to enhance digital data analysis. As a platform for interactive, collaborative, visual exploration, VR is one of the immersive technologies and is also used as big data visual analytics. VR for data visual analytics has advantages such as more data visualization possibilities, intuitive approach, multiple users, and elimination of distractions.[61] VR technology can lead to a demonstrably better perception of a data scape geometry, more intuitive data understanding, and better retention of the perceived relationship in the data.[62] Interactive 3D VR platforms can reduce distractions and apply more interactions to maintain attention and focus on learning.[63] VR allows users to interact with the data surrounding them just like the data are in front of, behind, above, and to either side of them. VR uses a human's natural instinct[64] to think about and process data in multiple dimensions and makes it possible to communicate data attributes in numerous positions.[65] VR can make data analysis more fun, as it allows “stepping into the data” and avoids the exercise of pouring over complex data. More humans then can be involved in monitoring machine learning models to ensure the machine's decisions continue to be ethical, fair, and reasonable.[66]

Some VR platforms for other domain data analysis combine all the modern technologies together to get better data analysis outcomes, as shown in the top part of Table 2. For example, iViz[67] is VR for a visual data analytics tool that uses Oculus Rift VR goggles as a display device to show large digital sky surveys data. The user interface for iViz enables the user to easily select and shuffle which data parameters are mapped to determine the optimal mapping choice for a given scientific. This flexibility allows for a more powerful visual data exploration and discovery. Bader[68] immersive data analytics platform allows users to see the whole picture and real-time collaboration. Users can adjust the data and visualize the results and then collaborate on how to proceed. Virtualitics[69] is another VR platform that is an innovator in the data visualization space with an AI-driven feature. This platform can help practitioners get actionable insights that are quicker than with traditional data analytics tools, and it also supports multiple users. 3Data[70] is another VR 3D platform that is developed for enterprise information technology and cybersecurity operations. With such VR platform, users can get illusory experiences that they are in reality, “stepping into” the command and control of complex networks.[71]

VR for Health Data Analytics

Genomic data is getting bigger and more complex for advanced computing technologies. For example, a single-cell data set can have tens of thousands to millions of cells and hundreds to tens of thousands features created by a total-seq platform.[72] For complex health data, especially genomic data, VR is a remarkable technology to control the subconscious mind and understand patterns in big data sets efficiently.[73] Immersive technology is used for health data analysis in many ways, such as data simulation, data visualization, and the combination of data visualization and machine learning. The bottom part of Table 2 shows some immersive using healthcare data analysis.

Immersive technology can be used for genomic data simulation. For example, VR is used to combine data on the genome sequence with data on gene interactions to create a 3D model that shows where regulatory elements and the genes they control sit relative to each other. It also makes it easier to understand the processes going on within a living cell.[74] With this technology, researchers can efficiently combine their data to gain a much broader understanding of how the organization of the genome affects gene expression, and how mutations and variants affect such interactions. Immersive technology is also used for showing the molecular structure of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2).[75] Immersive environment tools such as Molecular Rift[76] and VMD[77] help to improve the understanding of the virus action mechanism and to accelerate the drug discovery process. Using VR to visualize such genomic data is important because the human brain is very good at pattern recognition, and people think visually. Immersive technology can assist microscopy data visualization and colocalization analysis.[78]

In such systems, fully immersive manipulation of the microscopy data and analysis tools are offered by using hand gestures interface and a conventional gamepad as an alternative input method. The user's perception in data handling is maximized in the immersive environment.

Immersive technology is used for genomic visualization in many projects such as Genome3DExplorer[79] and iCAVE.[80] The immersive tools help biologists highlight some global topological characteristics of data and make better decisions based on the visualizations. Some data visualization tools combine VR and AI as a better solution for health data analysis, such as Deep Learning Development Environment in Virtual Reality.[81] It is also used to explain deep learning algorithms. This software runs as a standalone application on the Oculus Rift VR headset.[82] Another health data visualization tool, the Children Cancer Data Visualization tool,[83] already combines AI and VR to enable an important possibility of AI-based pattern recognition reporting results in a VR display.[32] This tool can be used to show the whole group of patients' data processed with AI algorithms in a 3D scatter plot, and then individually check each patient's detail. It can also zoom and rotate the visualization plot, compare genes among several patients, as well as interact with users and show the comparison visualization between selected patients. The tool can run on different VR environments such as HoloLens and Oculus Rift. It allows the users to see and manipulate the data in different and immersive ways in comparison with ordinary displays and portable devices. Then the users can look and feel the patients' avatars moving and positioned in a cohort and interact with the 3D visualization.

An Evolution of Integrating Immersive Technologies to a Visual Analytics of Genomic Data

Visual analytics or machine learning platforms for genomic data analysis have been in the research domain for more than a decade. Genomic data research trends to integrate immersive technologies. For example, as shown in Figure 2, a genomic data visual analytic research project starts from focusing on statistics and data mining, and then on blending 3D visualizations to 2D screen and mobile devices, and recently focuses on building it to an immersive 3D VR environment. Machine learning models are also integrated into the genomic data visualization in the immersive environment.

Figure 2

The genomic data analytics timeline that uses machine learning, mobile, and immersive technologies in the development evolution.

Figure 2

The genomic data analytics timeline that uses machine learning, mobile, and immersive technologies in the development evolution.

Close modal

Genomic data analysis focused on statistics and data mining before 2010, such as Kadupitige et al [84] created a “MINER” to apply exploratory analysis of gene interaction networks by machine learning from expression data. And then 3D visualization tools were developed, such as Nguyen et al[85] for visual analytics of clinical and genetic data sets of acute lymphoblastic leukemia. This paper highlighted methods to support the decision-making process by processing data mining, interactive visualization, analytical views and gene comparison. Then the research team created an improved work[86] for a novel 3D interactive visualization for medical data analysis. This work used better selection strategies to choose a group of data in a blended 3D environment. Machine learning models were then added to the 3D visualization models in the research, for example the study by Nguyen et al,[87] unlocked the complexity of genomic data of patients with rhabdomyosarcoma by using visual analytics. Then another interactive visualization work for patient-to-patient comparison was created by Nguyen et al.[88] This work improved the work by adding multiple interactions and provided a comprehensive framework solution for analyzing large and complex integrated genomic and biomedical data.

Following this, Khalifa et al[89] developed a visualization system for analyzing biomedical and genomic data sets using the Unity3D platform. This system applied the use of a 3D platform to analyze biomedical and genomic data. Furthermore, Nguyen et al[90] used 3D gaming environments and interfaces to improve the user-centric interaction and exploration experience. Khalifa et al[91] started to integrate the game engine to create an interactive visualization of complex genomic data. It presented a visual analytics model that enables the analysis of large and complex genomic data using Unity3D game technology. Genomics data visualization was also extended to the other platforms, including mobile[92] and large- and high-resolution display.[93]

Some platforms combined machine learning models with 3D space visualization, such as that of Qu et al,[94] which used visualization to illustrate machine learning models for genomic data. New research should apply popular visualization techniques in biology fields, such as scatter plot and heatmap visualization to train genomic data and explain a decision tree machine learning algorithm. Some visual analytical tools have been built in the VR platform; one of them was by Lau et al[83] for immersive intelligence genomic data visualization. This tool visualizes a 3D space scatter plot with data processed by a machine learning model. A review [95] of intelligent and immersive visual analytics of health data discusses the intelligent visualization, AI, and immersive technologies in the health domain with various VR case studies in genomic data visual analytics.

Immersive technology can potentially improve humans' perception. Humans are born into and are best adapted to a multidimensional world. Immersive VR technology allows the users to see, hear, smell, or touch things as in the real world.[96] VR makes it possible for sensory information processing in more natural conditions.[97] More and more VR clinical uses are created because of the benefits of its warm, inviting environment for engagement, and distraction-free and therapeutic care environment.[96]

Immersive technology, especially VR, is a potential technology for collaboration among different teams in different locations. Most VR is single user now, but some have or can be changed to multiple users. This enables the opportunities on the technology side for developers to create and share multi-user immersive MR experiences.[98] VR has the ability to collaborate in 3D environments and link data with natural human pattern recognition to uncover multidimensional relationships in data and to extract actionable knowledge that may not be discoverable by any other means.

Immersive technology as a promising technology combines different data analysis frameworks such as machine learning, and data visual analytics. VR can bridge knowledge gaps between experts and newcomers in the digital health domain.[99] With better perception and natural interaction ability, VR can be used to visualize and explain complex machine learning models to clinical users to improve trust. Researchers use immersive technology to assist in explaining AI in the medical domain, such as for simulation-based training in surgery and medicine[100] and reinforcement learning as a tool to make people move to a specific location in Immersive VR.[101] More and more research projects seek to combine XR with machine learning approaches to improve medicine or precision therapy through digital data genetic analysis.

Immersive technology has been used in the health industry for both the patient and the professional, such as aged care, remote GP, dementia, child anxiety, psychiatric care, distraction therapy, education, and health data analytics. These existing and ongoing projects are already helping to enhance patient outcomes, improve medical education, and interact with data visualizations in an intuitive way. All the projects are very new, most started from 2014, and most are ongoing projects. VR used for the digital health domain still needs time to develop its full potential. Data visualization starts to extend to the VR environment for its more interactive and more natural way to understand big genomic data.

Although there has already been some research on immersive technology for genomic data visual analytics, it is still in the early stages. Big complex genomic data analysis needs innovative immersive technology to improve human engagement and find more patterns and insights. Knowing how to make the immersive technology applications useful for the domain users also has a long way to go. Future research needs to know “why” before “how” by collecting end users' domain knowledge before and after developing the genomic data visual analytics tools in the immersive technology environment. The users' real requirements should be merged into the tool design and development. The immersive technology projects for genomic data visualization have the ability to inherit and integrate the previous research outcomes such as derived through AI and add new features to make the outcome really useful for domain users. Usability studies with domain users also need to address more ergonomic assessment to improve data-driven and UCD with immersive technology.

This paper provides a comprehensive review on the current immersive and XR technology projects for healthcare, XR for medical domain education, and health data analysis, especially genomic data visualizations. We also use an evolution of a genomic data analysis research case study to demonstrate how VR integrates with other technologies such as AI and mobile devices to solve clinical questions. Immersive technology applications for healthcare are still in their infancy.

Within the authors' knowledge, there are few works on genomic data visualization in VR environments, especially on how to integrate different technologies such as machine learning and game theories. The potential remains largely unexplored for VR used for genomic data analysis. A new framework may be useful for analyzing and interacting visualization genomic data in the XR environment. The new framework should aim to solve real clinical cases such as combining data processing, machine learning models, visual design and interactions, game optimization, and visual analytics with domain expert knowledge to deliver an effective analysis process to the domain users. New usability studies should evaluate the integration of all the preceding technologies to analyze complex genomic data in a user-friendly way. Ergonomic assessment in the virtual environment is also needed to get the immersive space design more data-driven and user-centric. Moreover, the genomic data visualization user interface also needs to improve to better use the XR and 3D environment as a more natural way of interacting. Last, the genomic data analysis ways need to combine different analysis strategies such as machine learning and add suitable explanations to make the results useful, trustable, and interpretable for domain users. Immersive technology as an innovative technology will be used more and more to improve the accuracy and effectiveness of current procedures and enhance the capabilities of humans in the digital health domain.

The individual nature of XR implementation has not allowed for widespread adoption within larger healthcare networks yet, so this paper could not review leading examples of widespread immersive technologies, nor study the quantifiable impacts. Immersive technologies combined with cameras and cloud servers have the potential to lead to widespread adoption. This article might exclude some potential works in other languages because we only searched to find the immersive projects written in English.

The authors thank The Sony Foundation and Tour de Cure Foundation for funding support under the Virtual Reality Cancer Research Scheme.

1.
10 Health care quality improvement trends you can't ignore. Prometheus Research. May
2019
.
Accessed Jun 4, 2020.
2.
Prabha Susy M, Anitha SP. Role of immersive (XR) technologies in improving healthcare competencies: a review.
In:
Giuliana
G,
Anitha
SP,
Eds.
Virtual and Augmented Reality in Education, Art, and Museums. IGI Global;
2020
:
23–46.
3.
Krumholz
HM.
Big data and new knowledge in medicine: the thinking, training, and tools needed for a learning health system
.
Health Aff (Millwood)
.
2014
;
33
:
1163
1170
.
4.
Reddy
M.
9 health technologies every executive should be excited about in
2020
.
Venture-Med. 2019. Accessed Nov 10, 2019.
5.
Filimowicz
M.
Foundations in Sound Design for Interactive Media: A Multidisciplinary Approach.
Routledge,
Taylor
&
Group;
Francis
2020
.
6.
Bakshi
SK,
Lin
SR,
Ting
DSW,
Chiang
MF,
Chodosh
J.
The era of artificial intelligence and virtual reality: transforming surgical education in ophthalmology
.
Br J Ophthalmol
.
2021
;
105
:
1325
1328
.
7.
Campesato
O.
Artificial Intelligence, Machine Learning, and Deep Learning. Mercury Learning and Information;
2020
.
8.
Topol
E.
Deep Medicine : How Artificial Intelligence Can Make Healthcare Human Again. Basic Books;
2019
.
9.
What is precision medicine? MedlinePlus. U.S. National Library of Medicine. Accessed Jan 24,
2022
.
10.
Blitz
S.
How machine learning improves data visualization. Accessed Nov 20,
2021
.
11.
Kyaw
BM,
Saxena
N,
Posadzki
P,
et al
Virtual reality for health professions education: systematic review and meta-analysis by the digital health education collaboration
.
J Med Internet Res
.
2019
;
21
:
e12959
12.
Birckhead
B,
Khalil
C,
Liu
X,
et al
Recommendations for methodology of virtual reality clinical trials in health care by an international working group: iterative study
.
JMIR Ment Health
.
2019
;
6
:
e11973
.
13.
Radianti
J,
Majchrzak
TA,
Fromm
J,
Wohlgenannt
I.
A systematic review of immersive virtual reality applications for higher education: design elements, lessons learned, and research agenda
.
Computers & Education
.
2020
;
147
:
103778
.
14.
Li
X,
Yi
W,
Chi
H-L,
Wang
X,
Chan
APC.
A critical review of virtual and augmented reality (VR/AR) applications in construction safety
.
Automation in Construction
.
2018
;
86
:
150
162
.
15.
Müller-Wittig
W.
Virtual reality in medicine
.
In:
Kramme
R,
Hoffmann
KP,
Pozos
RS,
Eds.
Springer Handbook of Medical Technology
.
Springer;
2011
:
1167
1186
.
16.
Cipresso
P,
Giglioli
IAC,
Raya
MA,
Riva
G.
The past, present, and future of virtual and augmented reality research: a network and cluster analysis of the literature
.
Front Psychol
.
2018
;
9
:
2086
.
17.
Doolani
S,
Wessels
C,
Kanal
V,
et al
A review of Extended Reality (XR) technologies for manufacturing training
.
Technologies
.
2020
;
8
:
77
.
18.
Atsali
G,
Panagiotakis
S,
Markakis
E,
et al
A mixed reality 3D system for the integration of X3DoM graphics with real-time IoT data
.
Multimed Tools Appl
.
2018
;
77
:
4731
4752
.
19.
Izard
SG,
Juanes Méndez JA, Palomera PR. Virtual reality educational tool for human anatomy
.
J Med Syst
.
2017
;
41
:
76
76
.
20.
Chen
M,
Saad
W,
Yin
C.
Resource management for wireless virtual reality: machine learning meets multi-attribute utility. Presented at GLOBECOM 2017–2017 IEEE Global Communications Conference; Singapore; Dec 4–8,
2017
.
21.
Verhey
JT,
Haglin
JM,
Verhey
EM,
Hartigan
DE.
Virtual, augmented, and mixed reality applications in orthopedic surgery
.
Int J Med Robot
.
2020
;
16
:
e2067
.
22.
Powell
W.
Five ways virtual reality is improving healthcare.
2019
.
Accessed Dec 11, 2019.
23.
Sheikh
SA,
Neda
R,
Kamran
S,
Amit
XG,
Eric
M.
Visual analytics for dimension reduction and cluster analysis of high dimensional electronic health records
.
Informatics
.
2020
;
7
:
17
.
24.
Stephens
ZD,
Lee
SY,
Faghri
F,
et al
Big data: astronomical or genomical?
PLoS Biol
.
2015
;
13
:
e1002195
.
25.
Wright
WG.
Using virtual reality to augment perception, enhance sensorimotor adaptation, and change our minds
.
Front Syst Neurosci
.
2014
;
8
:
56
.
26.
Nielsen
SL,
Sheets
P.
Virtual hype meets reality: users' perception of immersive journalism
.
Journalism
.
2019
;
22
:
2637
2653
.
27.
Ventura
S,
Baños
RM,
Botella
C.
Virtual and Augmented Reality: New Frontiers For Clinical Psychology. IntechOpen;
2018
.
28.
Gallagher
C.
End of year summary of Augmented Reality and Virtual Reality market size predictions. Accessed Nov 20,
2021
.
29.
Metzarchive
R.
The step needed to make virtual reality more real. Accessed Nov 20,
2021
.
30.
US National Library of Medicine website. Accessed Jan 24,
2022
.
ClinicalTrials.gov
31.
National Institutes of Health RePORTER website. Accessed Jan 25,
2022
.
32.
Qu
Z,
Lau
CW,
Nguyen
QV,
Zhou
Y,
Catchpoole
DR.
Visual analytics of genomic and cancer data: a systematic review
.
Cancer Inform
.
2019
;
18
:
117693511983554
.
33.
Virtual reality in healthcare. Visualise website. Accessed Dec 11,
2019
.
34.
Velazquez-Pimentel
D,
Hurkxkens
T,
Nehme
J.
A virtual reality for the digital surgeon
.
In:
Atallah
S,
Ed.
Digital Surgery
.
Springer International Publishing;
2021
:
183–201.
35.
Pioli
MR,
Ritter
AM,
de Faria
AP,
Modolo
R.
White coat syndrome and its variations: differences and clinical impact
.
Integr Blood Press Control
.
2018
;
11
:
73
79
.
36.
Mosadeghi
S,
Reid
MW,
Martinez
B,
Rosen
BT,
Spiegel
BMR.
Feasibility of an immersive virtual reality intervention for hospitalized patients: an observational cohort study
.
JMIR Mental Health
.
2016
;
3
:
e28
.
37.
Johnston
R.
Using virtual reality as distraction therapy for cancer patients. Gizmodo website. Accessed Jun 3,
2021
.
38.
Visyon Looking Beyond. A walk through dementia. Accessed Jun 3,
2020
.
39.
Spanswick
E.
Virtual reality app provides unique insight into life with dementia. Accessed Jun 4,
2020
.
40.
Lammens
M.
Silver Chain Group reveals the next big thing in the healthcare world. HealthiAR+VR+MR+XR. Accessed Jun 4,
2021
.
41.
Microsoft HoloLens 2. Accessed Jun 4,
2020
.
42.
Virtual reality technology used in dementia care. The Senior. Dec 5,
2016
.
Accessed Jun 4, 2021.
43.
O'Leary
C.
WA first sees virtual reality goggles used at St John of God Health Care Hospitals to ease anxiety. Sep 27,
2017
.
Accessed Jun 4, 2021.
44.
Virtual reality to distract children during medical procedures. Monash Children's Hospital. Apr 21,
2017
.
Accessed Jun 4, 2021. monashchildrenshospital.org/virtual_reality_to_distract_children_during_medical_procedures
45.
Castles
T.
VR could automate psychiatric care delivery, extending help to millions. Aug 9,
2018
.
Accessed Jun 4, 2021. www.
https://www.chiefhealthcareexecutive.com/ view/vr-could-automate-psychiatric-care-delivery-extending-help-to-millions
46.
Gazit
E,
Yair
Y,
Chen
D.
The gain and pain in taking the pilot seat: learning dynamics in a non immersive virtual solar system
.
Virtual Reality
.
2006
;
10
:
271
282
.
47.
Coleman
DS.
Technological Immersion Learning: A Grounded Theory. ProQuest Dissertations Publishing;
2017
.
48.
Mirelman
A,
Maidan
I,
Shiratzky
SS,
Hausdorff
JM.
Virtual reality training as an intervention to reduce falls
.
In:
Montero-Odasso
M,
Camicioli
R,
Eds.
Falls and Cognition in Older Persons: Fundamentals, Assessment and Therapeutic Options
.
Springer International Publishing;
2020
:
309–321.
49.
Samadbeik
M,
Yaaghobi
D,
Bastani
P,
et al
The applications of virtual reality technology in medical groups teaching
.
J Adv Med Educ Prof
.
2018
;
6
:
123
129
.
50.
Khor
WS,
Baker
B,
Amin
K,
et al
Augmented and virtual reality in surgery—the digital surgical environment: applications, limitations and legal pitfalls
.
Ann Transl Med
.
2016
;
4
:
454
454
.
51.
Chavez
B,
Bayona
S.
Virtual reality in the learning process
.
Presented at
:
Trends and Advances in Information Systems and Technologies; Italy; Mar 27–29
,
2018
.
52.
Erickson
M.
Virtual reality system helps surgeons, reassures patients. July 11,
2017
.
Accessed Jun 4, 2021.
53.
Halic
T,
Kockara
S,
Bayrak
C,
Rowe
R.
Mixed reality simulation of rasping procedure in artificial cervical disc replacement (ACDR) surgery
.
BMC Bioinformatics
.
2010
;
11
:
S11
.
54.
Eversbusch
A,
Grantcharov
TP.
Learning curves and impact of psychomotor training on performance in simulated colonoscopy: a randomized trial using a virtual reality endoscopy trainer
.
Surg Endosc
.
2004
;
18
:
1514
1518
.
55.
Grantcharov
TP,
Carstensen
L,
Schulze
S.
Objective assessment of gastrointestinal endoscopy skills using a virtual reality simulator
.
JSLS
.
2005
;
9
:
130
133
.
56.
Parekh
P,
Patel
S,
Patel
N,
Shah
M.
Systematic review and meta-analysis of augmented reality in medicine, retail, and games
.
Vis Comput Ind Biomed Art
.
2020
;
3
:
21
.
57.
Galati
A,
Schoppa
R,
Lu
A.
Exploring the SenseMaking process through interactions and fNIRS in immersive visualization
.
TVCG
.
2021
;
27
:
2714
2724
.
58.
Luboschik
M,
Berger
P,
Staadt
OG.
On spatial perception issues in augmented reality based immersive analytics. Presented at 2016 ACM International Conference on Interactive Surfaces and Spaces; Niagara Falls, Ontario, Canada, Nov 6–9,
2016
.
59.
Müller
C,
Krone
M,
Huber
M,
et al
Interactive molecular graphics for augmented reality using HoloLens
.
J Integr Bioinform
.
2018
;
15
:
20180005
.
60.
Polys
N,
Mohammed
A,
Iyer
J,
et al
Immersive analytics: crossing the gulfs with high-performance visualization. Presented at 2016 Workshop on Immersive Analytics; Greenville, SC; Mar 19–23,
2016
.
61.
Millais
P,
Jones
S,
Kelly
R.
Exploring data in virtual reality: comparisons with 2D data visualizations. Presented at Conference on Human Factors in Computing Systems; Montreal, Canada; Apr 21–27,
2018
.
62.
Okada
K,
Yoshida
M,
Itoh
T,
Czauderna
T,
Stephens
K.
VR system for spatio-temporal visualization of tweet data and support of map exploration
.
Multimed Tools Appl
.
2019
;
78
:
32849
32868
.
63.
Zhang
H,
Yu
L,
Ji
M,
et al
Investigating high school students' perceptions and presences under VR learning environment
.
Interactive Learning Environments
.
2020
;
28
:
635
655
.
64.
Calogiuri
G,
Litleskare
S,
Fagerheim
KA,
et al
Experiencing nature through immersive virtual environments: environmental perceptions, physical engagement, and affective responses during a simulated nature walk
.
Front Psychol
.
2017
;
8
:
2321
65.
Nguyen
NVT,
Virgen
L,
Dang
T.
Advances in visual computing
.
In:
Bebis
G,
Parvin
B,
Boyle
R,
et al
, Eds, VRParaSet: A Virtual Reality Model for Visualizing Multidimensional Data
.
Springer International Publishing;
2019
:
129–140.
66.
Marr
B.
Using VR to step inside your data: VR or AR-enabled analytics. Feb 12,
2021
.
Accessed Jan 25, 2022.
67.
Donalek
C,
Djorgovski
SG,
Cioc
A,
et al
Immersive and collaborative data visualization using virtual reality platforms. Presented at the 2014 IEEE International Conference on Big Data (Big Data); Washington DC, Oct 27–30,
2014
.
68.
BadVR website. Accessed Nov 20,
2021
.
badvr.com
69.
Virtualitics website. Accessed Jun 4,
2021
.
virtualitics.com
70.
3Data website. Accessed Jun 4,
2021
.
3data.io
71.
Gonzalez-Franco
M,
Lanier
J.
Model of illusions and virtual reality
.
Front Psychol
.
2017
;
8
:
1125
.
73.
Hérisson
J,
Gros
P,
Férey
N,
Magneau
O,
Gherbi
R.
DNA in Virtuo visualization and exploration of 3D genomic structures
.
In:
Proceedings of the 3rd International Conference on Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa
.
ACM
.
Nov
2004
,
35
40
.
74.
Virtual reality tool developed to untangle genes. Sep 21,
2017
.
Accessed Jun 5, 2020.
75.
Calvelo
M,
Piñeiro
Á,
Garcia-Fandino
R.
An immersive journey to the molecular structure of SARS-CoV-2: virtual reality in COVID-19
.
Comput Struct Biotechnol J
.
2020
;
18
:
2621
2628
.
76.
Norrby
M,
Grebner
C,
Eriksson
J,
Boström
J.
Molecular rift: virtual reality for drug designers
.
J Chem Inf Model
.
2015
;
55
:
2475
2484
.
77.
Stone
JE,
Kohlmeyer
A,
Vandivort
KL,
Schulten
K.
Immersive molecular visualization and interactive modeling with commodity hardware. Presented at Proceedings of the 6th International Conference on Advances in Visual Computing; Berlin, Heidelberg, Germany; Nov 29,
2010
.
78.
Theart
RP,
Loos
B,
Niesler
TR.
Virtual reality assisted microscopy data visualization and colocalization analysis
.
BMC Bioinformatics
.
2017
;
18
:
64
.
79.
Férey
N,
Gros
PE,
Hérisson
J,
Gherbi
R.
Immersive graph-based visualization and exploration of biological data relationships
.
Data Sci J
.
2005
;
4
:
189
194
.
80.
Liluashvili
V,
Kalayci
S,
Fluder
E,
Wilson
M,
Gabow
A,
Gümüs
ZH.
iCAVE: an open source tool for visualizing biomolecular networks in 3D, stereoscopic 3D and immersive 3D
.
Gigascience
.
2017
;
6
:
1
13
.
81.
Vanhorn
K,
Zinn
M.
Deep learning development environment in virtual reality. arXiv: 1906.05925. Jun 13,
2019
.
82.
Oculus Rift. Accessed Jun 5,
2020
.
83.
Lau
CW,
Qu
Z,
Nguyen
QV,
Simoff
S,
Catchpoole
D.
Immersive intelligence genomic data visualisation. Presented at Proceedings of the Australasian Computer Science Week Multiconference; Sydney, NSW, Australia; Jan 29,
2019
.
84.
Kadupitige
SR,
Leung
KC,
Sellmeier
J,
et al
MINER: exploratory analysis of gene interaction networks by machine learning from expression data
.
BMC Genomics
.
2009
;
10
:
S17
.
85.
Nguyen
QV,
Gleeson
A,
Ho
N,
et al
Visual analytics of clinical and genetic datasets of acute lymphoblastic leukaemia. Presented at the 2011 International Conference on Neural Information Processing (ICONIP 2011); Shanghai, China; Nov 14–17,
2011
.
86.
Alzamora
P,
Nguyen
Q,
Simoff
S,
Catchpoole
D.
A novel 3D interactive visualization for medical data analysis
.
In:
Proceedings of the 24th Australian Computer-Human Interaction Conference
.
ACM, Inc.;
2012
:
19–25.
87.
Nguyen
QV,
Alzamora
P,
Ho
N,
Huang
ML,
Simoff
S,
Catchpoole
D.
Unlocking the complexity of genomic data of RMS patients through visual analytics. In: 2012 International Conference on Computerized Healthcare (ICCH)
.
IEEE
.
2012
:
134
139
.
88.
Nguyen
QV,
Nelmes
G,
Huang
ML,
Simoff
S,
Catchpoole
D.
Interactive visualization for patient-to-patient comparison
.
Genomics Inform
.
2014
;
12
:
21
34
.
89.
Khalifa
NH,
Nguyen
QV,
Simoff
S,
Catchpoole
D.
A visualization system for analyzing biomedical and genomic data sets using Unity3D platform. In Proceedings of the 8th Australasian Workshop on Health Informatics and Knowledge Management. Australian Computer Society.
2015
:
47–53
90.
Nguyen
QV,
Khalifa
NH,
Alzamora
P,
et al
Visual analytics of complex genomics data to guide effective treatment decisions
.
J Imaging
.
2016
;
2
:
1
17
.
91.
Khalifa
NH,
Nguyen
QV,
Simoff
S,
Catchpoole
D.
Interaction visualisation of complex genomic data with game engines. In: 2017 21st International Conference Information Visualisation (IV)
.
IEEE
.
2017
:
133
139
.
92.
Nguyen
QV,
Qu
Z,
Huang
ML,
et al
A mobile tool for interactive visualisation of genomics data. In: 2018 9th International Conference on Information Technology in Medicine and Education (ITME)
.
IEEE
.
2018
:
688
697
.
93.
Brunker
A,
Catchpoole
D,
Kennedy
P,
Simoff
S,
Nguyen
QV.
Two-dimensional immersive cohort analysis supporting personalised medical treatment. In: 2019 23rd International Conference in Information Visualization–Part II
.
IEEE
.
2019
:
34
41
.
94.
Qu
Z,
Nguyen
QV,
Zhou
Y,
Catchpoole
DR.
Using visualization to illustrate machine learning models for genomic data. Presented at ACSW 2019 Proceedings of the Australasian Computer Science Week Multiconference; Sydney, NSW, Australia; Jan 29,
2019
.
95.
Qu
Z,
Lau
CW,
Catchpoole
DR,
Simoff
S,
Nguyen
QV.
Intelligent and immersive visual analytics of health data
.
In:
Maglogiannis
I,
Brahnam
S,
Jain
LC,
Eds.
Advanced Computational Intelligence in Healthcare-7: Biomedical Informatics
.
Springer;
2020
:
29–44.
96.
Cipresso
P,
Giglioli
IAC,
Raya
MA,
Riva
G.
The past, present, and future of virtual and augmented reality research: a network and cluster analysis of the literature
.
Front Psychol
.
2018
;
9
:
2086
.
97.
Scarfe
P,
Glennerster
A.
Using high-fidelity virtual reality to study perception in freely moving observers
.
J Vis
.
2015
;
15
:
3
.
98.
Nichols
G.
Data visualization via VR and AR: how we'll interact with tomorrow's data. ZDNet. Mar 26,
2019
.
Accessed Nov 20, 2021.
99.
El Beheiry
M,
Doutreligne
S,
Caporal
C,
Ostertag
C,
Dahan
M,
Masson
JB.
Virtual reality: beyond visualization
.
J Mol Biol
.
2019
;
431
:
1315
1321
.
100.
Mirchi
N,
Bissonnette
V,
Yilmaz
R,
et al
The virtual operative assistant: an explainable artificial intelligence tool for simulation-based training in surgery and medicine
.
PLoS One
.
2020
;
15
:
e0229596
.
101.
Rovira
A,
Slater
M.
Reinforcement learning as a tool to make people move to a specific location in immersive virtual reality
.
Int J Human-Computer Studies
.
2017
;
98
:
89
94
.
102.
Khadra
C,
Ballard
A,
Dery
J,
et al
Projector-based virtual reality dome environment for procedural pain and anxiety in young children with burn injuries: a pilot study
.
J Pain Res
.
2018
;
11
:
343
353
.

Competing Interests

Source of Support: The Sony Foundation and Tour de Cure Foundation provided funding support under the Virtual Reality Cancer Research Scheme. Zhonglin Qu is funded by a Western Sydney University Research Scholarship.

Conflict of Interest: None.

This work is published under a CC-BY-NC-ND 4.0 International License.