Individuals with intellectual and developmental disabilities (IDD) have complex healthcare needs, which are often unmet. Nominal group technique (NGT) uses a mixed-methods approach, which may engage the IDD population in the research process in a person-centered manner and address the shortcomings of traditional research methods with this population. NGT was used with a group of 10 self-advocates to evaluate a series of healthcare tools created by and for individuals with IDD. Participants provided helpful input about the strengths of these tools and suggestions to improve them. NGT was found to be an effective way to engage all participants in the research process.
In recent years, healthcare access for people with intellectual/developmental disabilities (IDD) has evolved from institutional-based care to community-based services internationally. This shift in healthcare services has created debate about the quality of community-based health services people with IDD are currently receiving (Kozma, Mansell, & Beadle-Brown, 2009; Lewis, Lewis, Leake, King, & Lindemann, 2002; Martínez-Leal, et al., 2011), especially because it has been found that this group has higher rates of unmet health needs (Iacono, Bigby, Unsworth, Douglas, & Fitzpatrick, 2014; Ouellette-Kuntz, 2005) and more difficulty finding and receiving healthcare than other individuals (Freedman, Nichols, & Ward, 2010; Krahn, Reyes, & Fox, 2013). A need for systematic re-evaluation of the delivery of primary and community-based care for this vulnerable population is warranted (Balogh, Ouellette-Kuntz, Bourne, Lunsky, & Colantonio, 2008).
Canada's Strategy for Patient-Oriented Research (SPOR) encourages patients, researchers, healthcare providers and decision-makers to actively collaborate to build a sustainable, accessible and equitable healthcare system (CIHR, 2014). (For the purposes of this article we use patient as an overarching term inclusive of individuals with personal experience of a health issue and informal caregivers, including family and friends [CIHR, 2014].) As explained in a recent Patient Centred Outcomes Research Institute (PCORI) report and posted on the Canadian Institutes for Health Research SPOR website, “engaging patients in healthcare research makes [investments in] research more accountable and transparent, provides new insights that could lead to innovative discoveries, and ensures that research is relevant to patients' concerns” (Nass, Levine, & Yancy, 2012). Unfortunately, research evaluating healthcare tends to ignore the voice of people with IDD; even through it can be it done, it has yet to become a standardized research practice (see Fujiura, 2012 for review).
There are different ways that researchers can successfully seek input from users with IDD about their healthcare experiences. Several researchers have used individual interviews for this purpose (e.g., Bell, 2012; Webber, Bowers, & Bigby, 2010). For example, Bell (2012) conducted interviews with eight adults with IDD about their experience using a healthcare communication tool (a “hospital passport”) in the hospital. However, individuals with IDD may perceive the interview process as intimidating, which can result in an undue power imbalance (Irvine, 2010). They may also not be accustomed to being asked their opinion about their healthcare experience and as a result, they may feel the need to acquiesce for social desirability (Fujiura, 2012).
Focus groups about individual healthcare experiences have also been carried out with success (Bollard, 2003; Brown & Gill, 2009; Lunsky & Gracey, 2009; Freedman et al., 2010; Parish, Moss, & Richman, 2008,); however, dominant personalities can limit the range and comfort of other group members to participate in group discussion (Gallagher, Hares, Spencer, Bradshaw, & Webb, 1993). It is critical that people with IDD are not excluded from patient-oriented research because they, too, have a right to meaningfully engage in developing healthcare policies and practices that will affect their quality of care (Harkness, 2005). One well-studied research method, which has the strengths of focus groups and flexibility of individual input, is nominal group technique (NGT).
Nominal Group Technique
Developed as an organizational tool by Delbecq and Van de Ven, NGT is commonly used in healthcare settings as a mixed-methods data collection tool (Delbecq & Van de Ven, 1971; Delbecq, Van de Ven, & Gustaffson, 1975). Facilitated meetings are held with small groups of stakeholders who generate and discuss responses to a specific question, followed by individual voting to rank the generated responses (Gallagher et al., 1993). NGT allows for quantitative and qualitative analyses and it is recommended that analysis considers themes across the entire group rather than just top ranked responses (McMillan et al., 2014). NGT has been used to bridge the gap between researchers, healthcare providers, patients, caregivers, and community stakeholders (Carney, McIntosh, & Worth, 1996; Elliot & Shewchuk, 2002; Dewar, White, Posade, & Dillon, 2003; Fawkes, Leach, Mathias, & Moore, 2014). The majority of NGT studies have been conducted in general medicine and nursing, although less frequently with other health professionals and consumers (Potter, Gordon, & Hamer, 2004). Elliot and Shewchuk (2002) proposed that NGT is particularly valuable in the early development phase of patient-oriented interventions as a means for problem identification and problem solving, as well as in the evaluation of practices.
NGT and People With IDD
With increasing importance being placed on involving individuals with IDD in patient-oriented research and evaluation of healthcare systems/services, the structured approach of NGT may be particularly suitable for this purpose. In the context of healthcare, NGT affords people with IDD the opportunity to provide input on their needs resulting in the development of healthcare that is for them, designed by them (Harvey & Holmes, 2012). NGT is an accessible research method that can be adapted to meet the needs of participants and may be a more adaptive approach than focus groups as it allows for open group discussion of ideas but also provides support to individuals who are less vocal. Furthermore, NGT can eradicate polarization of ideas as it facilitates equitable collaboration in idea generation, by preventing dominant personalities from monopolizing the discussion (Gallagher et al., 1993; Carney et al., 1996; Fawkes et al., 2014; Harvey & Holmes, 2012). NGT may be empowering in that the act of generating and subsequently voting on ideas gives a voice to a marginalized population (Hares, Spencer, Gallagher, Bradshaw, & Webb, 1992), which is particularly relevant for people with IDD.
Despite the promise of NGT, few studies have utilized it with people with IDD. The first NGT study focused broadly on major problems faced by young people with IDD (Bostwick & Foss, 1981). Tuffrey-Wijne, Bernal, Butler, Hollins, and Curfs (2007) used NGT to explore views held by people with IDD about end-of-life care. Most recently, Friedman, Arnold, Owen, and Sandman (2014) found that NGT was an effective method to explore definitions and experiences of sexuality. Thus, NGT may be an appropriate and innovative technique to gain insight about sensitive and complex subject areas directly from those with IDD (Tuffrey-Wijne et al, 2007; Friedman et al, 2014). This preliminary evidence suggests that NGT can meaningfully engage people with IDD in the research process; however, to our knowledge, NGT has yet to be adopted as a way to obtain feedback on and further inform healthcare innovations. The purpose of the current study is to investigate whether NGT is a feasible participatory action research methodology for evaluating patient-oriented healthcare tools developed by, and created for people with IDD.
Members of three self-advocacy groups located in the city of Toronto (the largest city in Canada with a population of 2.6 million) were invited to take part in a consumer consultation meeting with researchers to evaluate healthcare tools that were developed by, and created for, patients with IDD. Ten community-dwelling, adult, self-advocates, (5 men and 5 women) who expressed an interest in improving healthcare access for people with IDD participated. Two support staff attended the consumer consultation meeting acting strictly in a support capacity for specific participants, but did not influence their opinions or thoughts. The study was reviewed by the hospital Research Ethics Board and upon approval, participants provided consent to take part in the study.
NGT was the primary method used in the current study. The consumer consultation meeting began with a brief brainstorming session where participants explored positive and negative aspects of going to see their doctor. Two lists were developed; one list addressed reasons why participants like going to see their doctor while the other list addressed reasons why they do not. Upon completion of the lists to the satisfaction of the group members, each participant voted on their top three reasons why they either like or do not like going to see their doctor. This first task served three purposes: 1) it allowed for open discussion about general perceptions toward healthcare, 2) it allowed for participants to become comfortable sharing different opinions while emphasizing that there are no right or wrong answers, and 3) it familiarized participants with the NGT process. Following this orientation exercise, the three healthcare tools evaluated were 1) a patient healthcare communication booklet titled “Today's Health Care Visit,” a tool used by patients with their healthcare providers. It contains information such as the reason for the healthcare visit, fears or concerns about receiving healthcare, ways to aid and ensure the individual's understanding of the healthcare visit, along with feedback about what occurred during the visit including changes to medication and information about follow-up appointments. 2) A public awareness poster campaign about healthcare and disability depicting an individual with IDD and a direct quote of the individual speaking to their personal hobbies or personality, along with their thoughts about healthcare and strategies to improve. The final tool that was evaluated was 3) an instructional video which portrays a woman with IDD at the hospital, having blood drawn by a nurse. The video includes a voiceover of this woman describing how she feels when she needs to have blood work completed as well as strategies she uses to manage this stressful experience (these tools can be found at www.hcardd.ca under healthcare resources).
The researcher team presented the first tool to the group followed by a group brainstorming session. Two lists were developed: 1) what we like about the tool and 2) what we do not like about the tool. A member of the research team recorded summarized points of discussion by the group under the appropriate heading to develop a comprehensive list. Once the group felt satisfied with both lists, each participant came to the front and used color coded stickers to vote on the top three items they liked and did not like about the tool and further, what they would like to see changed in the tool. It was noticed by the research team that some participants were unable to select three responses, thus participants were informed that they did not have to use all three votes if they chose not to. This procedure was repeated for the other two tools.
Analysis drew from 3 data sources: 1) the raw data generated by participants during brainstorming sessions, 2) participant voting, and 3) field notes taken by two members of the research team during group discussions. Descriptive statistics were used to summarize the votes generated by participants for each of the four topic areas. It is important to note that due to not all participants voting three times for each topic; votes shown are based on the number of people who voted for that particular item. Axial coding was used to qualitatively analyze the data in which codes were identified and themes naturally emerged. Two researchers coded the raw data, and the research team met three times to discuss how themes were identified, described, and organized (Corbin & Strauss, 1990). The team discussed at length any discrepancies about codes and themes until a consensus was reached.
The NGT responses were coded for each of the following four discussion topics: 1) what we like/don't like about going to the doctor, 2) what we like/don't like about the “Today's Health Care Visit” booklet, 3) what we like/don't like about the poster, and 4) what we like/don't like about the instructional video. These were organized into themes developed by the researchers during the analysis process and are discussed in detail below.
Discussion Topic 1: “What we like/don't like about going to the doctor”
The open group discussion about “what we like/don't like about going to the doctor” was organized into the following themes: doctor-patient relationship, health promotion, costs, access difficulties, and other. Doctor-patient relationship was an important factor contributing to why participants liked going to see their physician. Participants stated, “I like doctors who have experience,” “I like doctors who know me,” and, “I like keeping the same doctor.” Participants also stated that they appreciate, “professionalism in my doctor,” however, do not like it when their “doctor cancels appointments” when “they use restraints” or when, “appointments feel rushed,” leading them to feel as though the doctor is not interested in them. Participants also mentioned they sometimes feel as though their doctor is only there for monetary purposes. The theme health promotion emerged from the data as participants stressed the importance of going to the doctor, which included the following responses: “getting checked to make sure everything is okay,” “finding out [about] what problems you don't [know you] have,” and making sure medications are correct. The theme of costs emerged from the data as participants discussed how they do not like when they are charged a fee for canceling an appointment, ambulance use, or when they need a doctor's note. The theme of access difficulties to care includes participant's dissatisfaction with long wait times, difficulty finding a family doctor, and not getting required medication. The other theme included topic areas such as enjoying riding in ambulances, having a fear of needles, and how going to the doctor is seen as, “a chance to go out in the community.” Table 1 represents the most commonly endorsed responses by participants.
Discussion Topic 2: “What we like/don't like about the ‘Today's Health Care Visit' booklet”
The open group discussion about “what we like/don't like about the ‘Today's Health Care Visit' tool” was categorized into the following themes: communication tool and tool usability. Participants discussed why they thought the “Today's Health Care Visit” booklet was an effective communication tool because it “gives a voice to someone who cannot talk,” provides, “good additional information for [the] doctor,” and, “helps me tell the doctor where I have pain when I can point to a picture.” Participants also discussed how the booklet can be a memory aid, reminding them about appointments, medication, and suggestions to improve visits. The themes of tool usability included discussions by participants about the usability of the tool. For example, participants stated that they liked the graphics and that “the size is good,” it is “user-friendly,” and “the wording is good.” However, participants also spoke about the inaccessible aspects of the tool, such as there not being enough pictures, “not enough room for medication [if prescribed more than one],” “not enough space to write,” “the print being too small,” and they “would need help to fill it out.” Table 2 contains the list of prioritized votes generated by participants specific to what they like and do not like about the “Today's Health Care Visit” tool.
Discussion Topic 3: “What we like/don't like about the poster”
The themes that emerged from the open group discussions about “what we like/don't like about the poster” were tool design, empowerment/advocacy, and representation of disabilities. In terms of tool design participants spoke about the tool's appearance. For example, they liked the logo on the poster, how the person in the poster was being portrayed as attractive and happy, and that the posters were found to be “straight to the point.” Participants also spoke about aspects of the aesthetic/design of the poster that they would like to see changed, such as the size of the front and size of the poster itself. Participants critically discussed the poster content, suggesting it to be empowering/advocating for themselves and others with disabilities. The group talked about how they found the posters “can make a difference in people's lives” because it, “teaches people that I'm more than a disability.” They liked how the poster content included a direct quote from a person with IDD about the person's hobbies, interests, and experience with healthcare because they felt it represented people with IDD with their own voice. Participants expressed how they would like to appear on such a poster because it makes people feel special. The theme of representation of disabilities emerged from the data as participants discussed how the posters only showed people with IDD and do not represent other disabilities. The group discussed how they thought representation of other disability groups is important and how such groups should not be excluded from the posters. Table 3 represents the prioritized voting results for the discussion, “what we like/don't like about the posters.”
Discussion Topic 4: “What we like/don't like about the instructional video”
The following themes emerged from the open discussion about “what we like/don't like about the video”: video content, video purpose, and emotional reactions to video. When participants discussed the video content of the instructional video, they described how they liked the fact that the video was about the experience of having a blood-work procedure performed. They liked that it was clear and showed the blood-work procedure step-by-step and expressed that the nurse explaining the blood test was helpful. Additionally, the group liked the audio effect of hearing the voice of the person with IDD explain what she was feeling during the procedure. However, participants also identified missing content that they thought would contribute to its effectiveness. For example, “there was no list at the end [of the video] of how to make getting blood work done easier,” it was, “hard to tell what she [the patient] was doing to relax,” and it “didn't explain different spots where you can get a needle.” Additionally, participants spoke about how they would have liked the video to show what happens when the blood-work appointment does not go well, for example, if the nurse is not being helpful, or if the patient does not want to get the blood work done. The video purpose was described by participants as, “it helps people prepare to go get blood work done,” which would make their appointment a more positive experience. The theme of emotional reactions to video emerged from the data as some individuals expressed that watching the video made them feel uncomfortable because they do not like getting blood work done or are afraid of needles. Others spoke about how the video was unrealistic in that they felt that they could not “get the feeling of what taking blood feels like” and would have liked to see a real blood get drawn, as opposed to a dramatization. Table 4 represents the prioritized votes for “what we like/don't like about the instructional video.”
Despite growing recognition of the active role patients should play in building accessible and equitable health care (Crawford et al., 2002; Tuffrey-Wijne et al., 2007), there is a lack of healthcare improvement research that includes people with IDD. This study is the first to explore how NGT can be used to evaluate patient-oriented healthcare tools for individuals with IDD and addresses the importance of patient engagement in health-systems research. In the current study, NGT proved to be an effective method to meaningfully engage people with IDD in the evaluation of clinical tools that were created for, and by people with IDD to improve their access to health care.
Consistent with past research findings, the NGT method provided each participant with a platform to express the participant's views regardless of level of disability (Friedman et al., 2014; Tuffrey-Wijne et al., 2007). Although some participants were vocal about their opinions, others who were less vocal took a more passive role during the brainstorming session. However, the voting exercise allowed less vocal participants the opportunity to be equally involved in voting on which items were most important to them. Interestingly, one rather shy participant was empowered by the voting exercise; and she stood up in front of the group on her own accord to assist one of the researchers in announcing the voting results. The NGT process allowed for participants' ideas and opinions to be validated by their peers and the research team; observing that their thoughts were transcribed on large chart paper by the research team validated that the individual's contributions were not only important, but would also contribute to the research knowledge being created during the evaluation session.
Through adapting NGT to meet the needs of people with IDD, it provided a structured format for the group to organize their thoughts, think critically about the discussion items, and vote on their main concerns from the list of items generated by the group (Friedman et al., 2014; Tuffrey-Wijne et al., 2007). NGT allowed participants to feel comfortable in expressing their views about health care in a group setting, allowing diverse issues to be brought forth by participants. With both positive and negative views being voiced, it was evident that participants were not acquiescing. This is an important finding because it has been a well-documented concern of researchers when using self-reported measures with this population (Finlay & Lions, 2002; Fujiura, 2012; Irvine, 2010). NGT provided participants with a platform to think critically about each healthcare tool. This was evident from the range of participants' responses, in that they reflected different aspects of the tools and offered insightful critiques about how they can be improved. Further, votes were typically spread out, which suggests that individuals did not feel pressured to vote on the most popular item, but instead voted on what was actually important to them. The variation in votes suggests that further research is needed to understand what people with IDD want included in healthcare tools targeted towards them and their healthcare providers. From both the opening discussion and comments related to each tool, it was clear that individuals had concerns about how their health care is managed and about which tools would improve their experiences.
Results of this study are consistent with past research that has found that people with IDD feel as though they are not receiving adequate care from healthcare providers (Freedman et al., 2010; Mitchell, 2012; Walmsley, 2011). During the discussion about why they do not like going to the doctor, some participants expressed feeling that they were not getting the care they require because appointments felt rushed and the doctor was disinterested in them; each of which could be improved with the use of appropriate healthcare tools. In their evaluation of the posters, for example, participants described the posters as empowering and suggested they could be used as a tool for advocacy in various settings. This was an interesting finding because the posters were originally designed for healthcare providers, and not patients, to create awareness about better serving patients with IDD in hospital settings. Research has shown that people with IDD do not have equal access to health care nor do they feel in control of their health (Bell, 2012; Iacono et al., 2014; Lunsky & Gracey, 2009; Mitchell, 2012; Walmsley, 2011). Therefore, development of healthcare tools that represent people with IDD, such as advocacy posters, can be an empowering experience for people with IDD, reminding them that they have the right to equal access and standards of care, as well as the ability to be in control of their own health.
The current study had several limitations that should be addressed in future investigations. The first was the number of healthcare tools being evaluated during one session. The evaluation session was an intense 3-hr work period of critical thinking; and it may have been beneficial to have separate sessions to discuss each tool individually. Because of the length of the session, there may have been fewer ideas generated for the tools presented later in the session. Second, during the brainstorming session researchers recorded ideas generated by the group on large chart paper that participants voted on directly. The advantage of this approach was that each idea was given equal weight and allowed for everyone's opinions to be showcased to their peers. However, for some individuals, particularly nonreaders, there may have been too many ideas to select from; and it would have been beneficial for the research team to help participants consolidate similar items before voting. Technology could help in this regard. Future evaluation studies should designate time to analyze the list as a group and collapse similar points so that participants are not overwhelmed by voting on a long list of items with conceptual overlap. A third limitation was that the voting stickers were color coded to represent first, second, and third votes. This was a source of confusion for participants, because the stickers required constant clarification and reminders of which color was associated with each rank. Having stickers with the numbers printed on them or stickers of different sizes might be different ways to address this barrier and would have been beneficial to alleviate any confusion. This limitation is not unique to this study: Tuffrey-Wijne et al. (2007) reported a similar challenge around the procedure of ranking votes; further, participants in a study by McMillan et al. (2014) also reported difficulty with attributing greater importance to one idea over another. Last, the sample size may limit generalizability. Continued refinement of healthcare tools would benefit from engagement of different groups using NGT at various stages of the development process.
Overall, NGT proved to be an effective participatory action research method to gain valuable information and feedback from people with IDD about their healthcare experiences and healthcare tools. Future health-systems research should use NGT as method to ensure meaningful engagement of people with IDD in the development and evaluation of such healthcare tools. By doing so, health systems can become more transparent and accountable in the development of healthcare policies and practices, ensuring that they are relevant to the needs of people with IDD. Additionally, accessible participatory action research methods such as NGT can teach people with IDD how to become stronger health advocates, not only for their own care but for the wellbeing of others.
This study was supported by funding from the Ministry of Health and Long-Term Care Health Services Research Fund program award (Ministry Grant #06671) and is part of the Health Care Access Research and Developmental Disabilities (H-CARDD) Program. The opinions, results, and conclusions reported in this paper are those of the authors and not necessarily those of the funding sources. No endorsement by the Ontario MOHLTC is intended or should be inferred. The authors thank participants for taking part in this study.
This study was reviewed and approved by the Centre for Addiction and Mental Health Research Ethics Board.