Avatar-Mediated Home Safety Assessments: Piloting a Virtual Objective Structured Clinical Examination Station

Home-safety assessments are an important way for health care professionals to identify and modify risk factors for falls in elderly individuals.1,2 Home-safety assessments have been shown to reduce the incidence of falls.3–5 Geriatric fellowship directors consider real or “simulated” home-safety assessments to be an important skill for fellows to learn.6,7 Because traditional home visits are not practical to incorporate into an objective structured clinical examination (OSCE) exercise, educators have used various illustrative materials to identify safety hazards. Emerging Internet-based technologies offer various innovative and practical solutions to assess fellows' skills during an OSCE.

A virtual world is an Internet-based, computer-generated, 3-dimensional (3-D) simulation that allows the user to “live in” and interact via his or her graphic representation, or “avatar.”8 Virtual worlds are engaging, media-rich simulation environments that promote “experiential learning”9 and that have been used for training and skills assessment in different medical specialties.10–12 Home-safety assessment in a virtual world offers medical educators the advantage of being able to manipulate the environment to include the desired common hazards to prepare learners for the real-world experience.

With this pilot study, we sought to determine the feasibility and acceptability of a 3-D interactive home as a virtual OSCE (VOSCE) station, the first step toward incorporating such simulations in the competency assessment of internal medicine learners.

We built a 3-D virtual house in Second Life (Linden Lab, San Francisco, CA), and populated it with digital objects representing hazards. The virtual house and digital objects cost about $150; we spent 120 hours developing the virtual house (approximately $3000). The simulated house was a 2-story, New England–style home surrounded by a lawn and garden. It included a family area, dining room, kitchen, first-floor master bedroom with bathroom, and 3 second-floor bedrooms (figure). The virtual real estate was based in a private educational island with a monthly subscription fee ($295/month for 3 months). The proctor needed approximately 1 day of training to become familiar with the navigation ($17.40/h for 8 hours  =  $139.20).

Eight geriatric medicine fellows enrolled in a 1-year, Accreditation Council for Graduate Medical Education–accredited fellowship participated in this study. Their mean age was 37 ± SD 10 and participants had a diverse ethnic background. The computer proficiency instrument rated the participants as “proficient.”15 The University of Miami's Institutional Review Board granted exemption of our study as an educational activity.

Fellows participated in a required, 16-station (8 standardized patients and 8 computer-based stations), 1-day OSCE examination in January 2010. The authors have previously described the components of this examination.16,17 In this session, we pilot-tested the 3-D virtual home safety simulation we developed to assess learner competency in home-safety assessment.8,18 

The fellows rotated through each of the 15-minute OSCE clinical stations. An exam proctor oversaw the 3-D virtual house simulation in one clinical room equipped with a 15-inch screen Apple Mac Book Pro (Intel Core Duo 2.53Ghz, NVIDIA GeForce 9400M graphics, 4GB RAM) laptop (Apple, Cupertino, CA) and provided navigation and technical assistance as needed. Participants were seated at the laptop running the virtual-world simulation. The proctor instructed them as follows: “You are about to enter the home of an elderly woman named Mrs Harrison. Conduct a home safety assessment of the ground floor. Write down on the paper provided all the safety hazards that you identify in Mrs Harrison's house and then make specific recommendations to fix the hazards.” We limited the testing area to the garden and first floor of the virtual house to conserve time.

We used an inventory-based instrument to score fellows' home-safety assessments. This instrument revealed a high internal consistency (Cronbach 0.9) when used to score a similar group of 30 fellows identifying hazards of the virtual house in another study.18 The investigators gathered information on home hazards for the elderly from a peer-reviewed checklist developed at our center19 and from peer-reviewed checklists of home-safety hazards in the literature.20,21 Two geriatricians with expertise in home care and the director of the geriatric medicine fellowship at the University of Miami created an inventory of 50 common safety hazards after a virtual walk-through of the simulation. We displayed a selection of these hazards in and around our 3-D virtual house.

Two reviewers scored each fellow with the inventory-based instrument, rating each specific recommendation on a scale of 0 to 3 (0, no recommendation; 1, weak recommendation; 2, moderate recommendation; 3, appropriate recommendation). We debriefed the exam proctor and surveyed participant perceptions on what they liked and disliked and their suggestions to improve the simulation. The investigators entered the quantitative data of the participants' performance into Statistical Package for the Social Sciences v. 16 (formerly SPSS Inc., Chicago, IL; now IBM Corporation, Armonk, NY). We analyzed task performance and compared the performance of men to women using the 2-tailed paired Student t test.

Participants (n  =  8) identified a mean ± SD of 43% ± 9 of all the home safety hazards in the time allocated. The 2 reviewers had an interrater reliability of 88%. Self-reported participant confidence in hazard identification was a median score of 6, with a range of 4 to 7 on a 7-point Likert scale (anchors: 1, not confident; 7, extremely confident). Four of 5 female fellows (80%) requested navigation assistance from the proctor; 2 requested navigation assistance and guidance from one area of the house to the next, and 2 requested navigation assistance but no guidance. None of the 3 male fellows (0%) requested navigation assistance (table). Six fellows (75%) reported the experience was an excellent method of instruction.

We found that the avatar-mediated, 3-D home OSCE was technically feasible to set up and, in general, considered usable and acceptable to the geriatric medicine fellows. The fellows identified less than half of the 50 hazards located in the 7 living areas of the simulated home. The women reported some navigation difficulties.

The generally encouraging results of our pilot study show that the avatar-mediated VOSCE station may represent an acceptable, practical, and efficient approach to assess competency in home-safety assessment. Although the encounter required a proctor familiar with the virtual world environment, this person would not have to be physically present.

VOSCE participants' low scores in identifying home-safety hazards may be due to the complexity of the task. Fellows had to navigate within an unfamiliar virtual environment while identifying and writing down hazards and giving specific recommendations. In addition, although all participants had previously conducted home-safety assessments in real environments, their low performance in the VOSCE could be because this task was timed.

Women requested more navigation assistance than men, which is consistent with the literature on sex-related differences in spatial ability favoring men.30,31 In one study set in a 3-D virtual grocery store, women recalled more items and were more efficient at locating items than men, which suggests that sex performance is task-dependent.32 Medical educators intending to use virtual worlds for assessment of instruction must consider the role of sex differences.

Our pilot-study limitations include a small convenience sample and an unvalidated 3-D virtual home, as well as a lack of a baseline skill assessment in performing real-home safety evaluations. We did not identify the participants' prior experience in 3-D virtual worlds. No subgroup analysis that could account for confounders, such as sex or age, was possible because of the small numbers in this introductory/pilot study.

Yet, our study has important implications for skills assessment in graduate medical education. Avatar-mediated, 3-D virtual worlds may not only serve as practical methods for home-safety skills assessment but also for other skills that are currently difficult or impractical to assess, such as team-based skills, disaster management, and cultural competencies. Heinrichs et al33 assessed the team skills of emergency room residents and medical students using a 3-D, virtual emergency room, and Yellowlees et al34 used a 3-D virtual world to develop and test a bioterrorism defense-training environment for health care professionals. In assessing communications skills, avatars can serve as alternatives to standardized patients,36 who are expensive to train and maintain and may not be consistent in portrayal.37 We are confident that this pilot study was an important first step in showing that an avatar-mediated, 3-D virtual home is a practical and acceptable assessment of learners' ability to perform home-safety skills within an OSCE.