This study used data from the National Core Indicators (NCI) Survey to derive an empirically validated measurement model for social outcomes and associated constructs for both individuals with Autism Spectrum Disorder (ASD) and individuals with other disabilities. Items consistent with the survey structure of the NCI were selected as initial indicators of the latent constructs Social Relationships, Community Inclusion, and Opportunity for Choice in factor analyses. Results yielded a novel factor structure that is different from the original NCI survey structure. Three factors emerged as a result of these analyses: Personal Control, Social Determination, and Social Participation and Relationships. The factor structure of each of these constructs was consistent although not identical across individuals with ASD and individuals with developmental disabilities other than ASD.
Social deficits are the cardinal feature of Autism Spectrum Disorder (ASD) and include impairments in the use of nonverbal behavior to regulate social interaction, difficulty establishing and maintaining peer relationships, a lack of shared enjoyment of interests and accomplishments with others, and a general lack of social or emotional reciprocity (American Psychiatric Association, 2013). These social difficulties continue through adolescence and into adulthood for individuals with ASD (Howlin, 2003; Howlin, Mawhood, & Rutter, 2000; Seltzer et al., 2003; Seltzer, Schattuck, Abbeduto, & Greenberg, 2004). Persisting social deficits contribute to significantly poorer social outcomes for adults with ASD than both typically developing individuals and individuals with disabilities other than ASD, including lower quantity and quality of friendships, less inclusion and integration in the community, and negligible opportunities for choice making and personal initiative with regards to socialization (Eaves & Ho, 2008; Howlin, 2005; Howlin et al., 2000; Howlin, Goode, Hutton, & Rutter, 2004; Levy & Perry, 2011; Lord & Venter, 1992; Mawhood, Howlin, & Rutter, 2000; Smith & Matson, 2010; Whitehouse, Watt, Line, & Bishop, 2009).
Research on adults with ASD has primarily focused on objective measures of outcome including functional independence and employment and has identified ASD severity, verbal abilities, and IQ as reliable predictors for adult outcomes (Billstedt, Gillberg, & Gillberg, 2005, 2011; Nordin & Gillberg, 1998). Less is known about the predictive value of environmental factors that improve social outcomes, including social relationships and community involvement for adults with ASD (Heller, Miller, & Hsieh, 2002). Better understanding of the impact of environmental factors such as opportunity for choice on the social functioning of adults with ASD would provide useful information to guide future policy and practice (Ruble & Dalrymple, 1996).
Previous research using Washington state's 2002 National Core Indicators Survey examined relationships between Choice and Quality of Life indicators (Community Inclusion, Rights, and Social Relationships) in adults with intellectual disability by creating manifest variables from sum scores of items from various subsections of the NCI Consumer Survey (Neely-Barnes, Marcenko, & Weber, 2008). The present study aims to expand on this paradigm in that survey items will be treated as indicators of latent constructs as opposed to components of measured variables. The rationale for this approach is that it is likely that different survey items vary in their association with each latent construct as a function of group membership (ASD versus non-ASD; Dimitrov, 2006). In the present study, survey items initially selected as indicators for each latent construct subjected to exploratory and confirmatory factor analysis are consistent with previous research (Neely-Barnes et al., 2008) and are also consistent with the structure of the NCI survey instrument.
Results of this study will offer an empirically derived and valid measurement model for social outcomes in both individuals with ASD and with disabilities other than ASD. This study will lay the foundation for the making of meaningful comparisons across groups with regards to Social Relationships, Community Inclusion, and Opportunity for Choice, as well as for further exploration of the relationship between social outcomes and environmental predictors. Specific aims of this study include, first, the development of an empirically derived measurement model of the constructs Social Relationships, Community Inclusion, and Opportunity for Choice using exploratory factor analyses and, secondly, the validation of that measurement model in an independent validation sample using confirmatory factor analyses. Finally, this study will conclude by recommending the appropriate subset of questions that should serve as indicators of each construct of interest in individuals with ASD.
Data were obtained from the 2009–2010 and 2010–2011 versions of the National Core Indicator (NCI) Consumer Survey (Human Services Research Institute & National Association of State Directors of Developmental Disabilities Services, 2011, 2012). The NCI survey is a component of a national project on the assessment of outcomes for the purpose of quality assurance and enhancement coordinated by the Human Services Research Institute (HSRI) and the National Association of State Directors of Developmental Disabilities Services (NASDDDS). The NCI survey is administered only in states that choose to participate in the NCI program. The purpose of the NCI survey is to identify and measure core indicators of performance of state developmental disabilities services.
The 2009–2010 and 2010–2011 NCI surveys were administered to a random sample of individuals 18 years of age and older who were receiving, at the time of the survey, at least one state service besides case management. States were asked to complete 400 interviews with members of the random sample; thus, most states draw an oversample to account for refusals. Some states did not complete 400 interviews, and others exceeded this goal. A sample size of 400 allows valid comparisons across states with a 95% confidence level. Those states with samples below 400 participants are also included in the data.
Sixteen states, the District of Columbia, and one county in California administered the consumer survey in 2009–2010 and together collected background and demographic information and survey data on a total of 11,599 individuals. The participating states represented were: AL, AR, GA, IL, KY, LA, ME, MO, NC, NJ, NY, OH, OK, PA, TX, and WY, as well as DC and Orange County, CA. Of the 11,599 individuals, 10.4%, or 1,206 individuals, have an ASD. Twenty-four states and the District of Columbia administered the NCI consumer survey in 2010–2011 and together collected background, demographic, and survey data on a total of 8,796 individuals. The participating states represented were AL, AR, AZ, CA, FL, GA, HI, IL, KY, LA, MA, ME, MO, NC, NH, NJ, NM, OH, OK, PA, SD, TX, VT, and WA, as well as DC. Of the 8,796 individuals, 9.3%, or 816 individuals, have an ASD.
For this study, data analyzed include individuals from both the 2009–2010 and 2010–2011 datasets. Thus, the total number of possible participants for this study is 20,395. From this survey population, there is a total of 2,022 (9.9%) individuals with ASD. Of the 2,022 individuals with ASD, 886 individuals have a valid1 response to both sections 1 and 2 of the NCI Consumer Survey2. For this study, four groups were drawn from the population of individuals who have valid responses to both section 1 and 2 of the NCI consumer survey. Two groups consisted of individuals with ASD and two of individuals with developmental disabilities (DD) other than ASD. Individuals with ASD who had valid responses (N = 886) were split into two groups, an exploratory group (N = 443) and a confirmatory group (N = 443) to ensure that exploratory analyses were validated on an independent confirmatory sample. To create the comparison groups, two groups of 443 individuals with DD other than ASD were drawn from the pooled 2009–2010 and 2010–2011 dataset of individuals with valid responses to section 1 and 2. Individuals for the exploratory non-ASD (N = 443) and confirmatory non-ASD (N = 443) comparison groups were matched for gender with the ASD groups.
The NCI adult consumer survey
The NCI survey is made up of questions intended to collect information regarding various indicators giving a snapshot of system performance and outcomes for individuals with developmental disabilities. Performance indicators explored in this study include survey questions under the categories of: “Friends and Family,” “Community Inclusion,” “Choices,” “Access to Needed Services,” and “Satisfaction With Services/Supports.”
The NCI Adult Consumer Survey is conducted as a face-to-face interview by trained interviewers. To increase standardization and reduce interviewer bias, all interviewers receive consistent training. The NCI Adult Consumer Survey protocol is supported by a national training program for interviewers, including training manuals, presentation slides, training videos, scripts for scheduling interviews, lists of frequently asked questions, picture response formats, and a review of the survey tool.
Threats to the NCI Adult Consumer Survey's validity include those related to selection bias. States differ in their eligibility requirements for the diagnosis of ASD and qualification requirements for services. In states that have related clauses in their eligibility criteria, a higher percentage of individuals with ASD are served; thus, the sample of individuals used in this study may not be equally representative of all participating states. In addition, NCI Adult Consumer Survey data is only collected in states that agree to participate in the program, which may contribute to bias. The NCI Adult Consumer Survey has good reliability as demonstrated by 92% inter-rater agreement for past versions of the survey and 80% agreement in test stability over time (Smith & Ashbaugh, 2001).
Survey items initially selected as indicators for the latent constructs Social Relationships, Community Inclusion, and Opportunity for Choice subjected to exploratory and confirmatory factor analysis are consistent with previous research (Neely-Barnes et al., 2008) and with the designated subscales of the NCI survey instrument.
Five questions from the “Friends and Family” section of the NCI Adult Consumer Survey were chosen as initial indicators of overall quality of social relationships for this study. Respondents were asked whether they: 1) have friends, 2) have a best friend, 3) see friends when they want, 4) can go on a date if they want, and 5) ever feel lonely. Responses were coded as 0 (no), 1 (sometimes), or 2 (yes). Responses to “never feel lonely” were reverse-coded so that, for all indicators, high scores indicate higher levels of social support. These five variables serve as initial indicators of the latent construct of social relationships in exploratory factor analyses.
Seven questions from the “Community Inclusion” section of the NCI Adult Consumer Survey were chosen as indicators of overall inclusion in the community for this study. Respondents were asked whether in the past month they had gone: 1) shopping; 2) out on errands; 3) out for entertainment; 4) out to eat; 5) out to religious services; 6) out for exercise; and, in the past year, 7) if they had gone on vacation. Responses were coded as either 0 (no inclusion in the community) or 2 (access to the community). Each of these seven questions serves as an initial indicator of the overall latent construct of community inclusion in exploratory factor analyses.
Opportunity for choice
Eleven questions from the “Choices” section of the NCI Adult Consumer Survey were chosen as indicators of the individual's overall perception of his or her opportunity for choice (as the individual's actual opportunity for choice, i.e., number of options presented in each perceived choice, was not taken into account). Respondents were asked whether they chose: 1) where they live, 2) who they live with, 3) who helps them at home, 4) their daily schedule, 5) how to spend their free time, 6) their place of work, 7) their last job, 8) where they go during the day, 9) who helps them during the day, 10) what they buy, and 11) their case manager. Responses were coded as 0 (someone else chose), 1 (person had some role in the choice), or 2 (person made the choice). Each of these 11 questions serves as a variable used as an initial indicator of overall latent perceived choice for exploratory factor analyses.
The presence of an ASD diagnosis for the purposes of the NCI Adult Consumer survey was determined by a review of the individual's records on state computer databases during the presurvey process. Information related to diagnoses was provided by the service coordinator/case manager and verified by the individual or family member during completion of the survey. Therefore, in the context of this survey, the clinician completing the diagnostic assessment and the instruments used by these clinicians are unknown. However, to be eligible for developmental disability services in all states, a medical diagnosis is required. Despite this common requirement, states vary in their diagnosis of ASD and eligibility requirements to qualify for DD services within that state, which may lead to systematic differences in individuals identified as having an ASD across states.
The NCI survey data were used as a population from which two samples of interest were drawn for data analyses. One sample included individuals with ASD and the second sample consisted of a comparable group of individuals with DD other than ASD. The two samples were matched on gender and subsequently randomly split into exploratory and confirmatory groups. The de-identified data from the survey materials were analyzed to identify relationships between the variables of interest. Following approval from The Ohio State University Institutional Review Board (IRB), the NCI data was obtained from HSRI/NASDDDS. All NCI data was de-identified. The date of birth had been replaced with chronological age and county of residence was removed for all survey respondents included in the data provided. After completion of the study, the NCI dataset will be returned to HSRI/NASDDDS in compliance with their policy on the access and use of research data.
The SPSS Missing Values Analysis (MVA) module was used to examine missing data patterns. Items with high levels of “missingness” (>40%) in either group were dropped from further analyses in both groups. These items included: “did you choose where you work” and “did you choose your job staff.” For analysis of descriptive statistics, exploratory factor analyses, and t-tests carried out in SPSS, pairwise deletion of missing data was used. In confirmatory factor analyses, carried out in Amos, full information maximum likelihood (FIML) procedures were used to estimate missing data. Simulation studies have shown that imputing missing data using FIML estimation procedures more accurately represents associations among variables than does listwise or pairwise deletion of missing data, even when data are not missing entirely at random (Little & Rubin, 1989; Muthén & Kaplan, 1985; Muthén, Kaplan, & Hollis, 1987). It is likely that consumers with missing data from this survey failed to respond for nonrandom reasons related to their ability to communicate or comprehend the content of the questions. Therefore, FIML was used in the Amos software for estimating relationships among latent variables and indicators in the present analyses.
Exploratory and confirmatory factor analyses
Exploratory factor analyses (EFA) were conducted using the exploratory ASD (N = 443) and exploratory non-ASD (N = 443) groups. Initially, EFA was conducted separately in each group using pairwise deletion of missing values, maximum likelihood extraction, and oblimin factor rotation. An oblimin factor rotation was chosen over an orthogonal solution due to the presence of correlation among the items of interest. Use of the oblimin rotation can result in reduced interpretability of the factors and cross-loading of items on multiple factors, but was chosen given the correlation among items. The initial indicators of the three NCI latent constructs (Social Relationships, Community Inclusion, and Opportunity for Choice) were then entered into one analysis in each group respectively to allow indicators to freely load on any number of possible factors. These analyses serve the purpose of exploring the factor structure of each of the latent variables, Social Relationships, Community Inclusion, and Opportunity for Choice, and informing the selection of indicators with high factor loadings (>.20) in both groups to be used as indices of each latent construct in further analyses. Due to the initial exploratory nature of these analyses, some indicators may be excluded from further analyses if they are found to be minimally related (factor loadings <.20) to the latent variables of interest in one or both samples (Mulaik, 2010). Following this initial EFA, items loading most strongly on each derived factor were then tested in individual EFAs for that given factor separately in each sample (ASD and non-ASD).
After EFA procedures were completed, the confirmatory ASD and confirmatory non-ASD groups were used in separate confirmatory factor analyses (CFA) to validate the structure of the factors derived during the EFA procedures and to test the adequacy of the hypothesized measurement model. The CFA was carried out in Amos and used FIML estimation of missing values. All factors and their measured indicators were entered together in an initial analysis, restricting indicators to only load on the factor found to be most associated with that indicator during EFA procedures. Following this initial CFA, each factor and its indicators were tested in separate CFAs to further verify the adequacy of the measurement model for each latent construct.
Descriptive Statistics and Correlations
The ASD and non-ASD samples were matched for gender, as the significant majority (N = 656, 74%) of individuals with ASD in this sample are male, reflecting the gender distribution of the population (Center for Disease Control and Prevention, 2012). The ASD and non-ASD samples were found to be comparable on other key demographic variables including gender, age, race, ID diagnosis, level of ID, psychiatric diagnosis, seizure disorder diagnosis, and residential placement (Table 1 presents descriptive statistics by group for gender, age, diagnosis of ID, psychiatric diagnosis, seizure disorder, race, and residential placement). Individuals with ASD were younger on average (average age = 33.2 years; SD = 12.0) than individuals with DD other than ASD (average age = 43.8 years; SD = 14.6), t(1767) = 16.75, p = 0.00; Glass's Δ = 0.73 (Glass, McGaw, & Smith, 1981). Of individuals diagnosed with ASD, 86% were found to also have a diagnosed intellectual disability compared with 94% of individuals with DD other than ASD, although this difference was nonsignificant, t(1756) = 0.93, Glass's Δ = 0.04. This increased percentage of ID diagnosis in individuals with DD other than ASD was primarily accounted for by an increased prevalence of mild ID in the non-ASD sample; as a result, mean differences were found in the level of ID across groups although the effect size of this difference is small, t(1742) = −2.62, p = 0.009, Glass's Δ = 0.15. Despite this difference, the distribution of level of ID across groups is similar. There were no significant group differences in race between groups, t(1742) = −0.74, p = 0.46; Glass's Δ = 0.03. Individuals with DD other than ASD were more likely to be diagnosed with a seizure disorder, t(1770) = 3.67, p = 0.00; Glass's Δ = 0.17, and with a psychiatric diagnosis, t(1770) = 4.113, p = 0.00; Glass's Δ = 0.19, than individuals with ASD, although the effect sizes for these differences were small. Individuals with ASD and with DD other than ASD were similar in their residential placement, t(1763) = −0.54, p = 0.519, Glass's Δ = 0.02, with most individuals in both samples living either with a parent or relative or in a group home. Correlations between measured variables included in the CFA models are moderate in both individuals with ASD (see Table 2) and individuals with DD other than ASD (see Table 3).
Measurement Model of Latent Constructs for Individuals With and Without ASD
Initial EFA using pairwise exclusion of missing values, maximum likelihood factor extraction, and oblimin rotation was conducted separately in the ASD and non-ASD exploratory groups. In both exploratory samples, a three-factor structure emerged. In both groups, items that did not load adequately on any of the three factors were dropped. Items dropped in the ASD exploratory group included: 1) did you go out for religious services, 2) are you ever lonely, 3) can you see your friends, and 4) did you help make your service plan. Items dropped in the non-ASD exploratory group included: 1) are you ever lonely, 2) can you see your friends, and 3) did you help make your service plan. Following this initial EFA, items loading on each specific factor were entered in separate EFAs in each group to confirm adequate fit. The EFA results indicated a factor structure differing from that of the NCI structure. Whereas NCI indicators are organized into three domains and, as such, were hypothesized to load on three factors (Social Relationships, Community Inclusion, and Opportunity for Choice), the indicators from these NCI domains cross-loaded across factors, yielding three scales more appropriately named Social Determination, Social Participation and Relationships, and Personal Control. This new emerging measurement model was largely consistent across individuals with and without ASD. Table 4 lists each factor and its measured indicators and factor loadings for the ASD group and the non-ASD group, as well as indicators that loaded on each derived factor consistently across both groups.
Following EFA, CFA was conducted using only the indicators that loaded on each derived factor across both groups to verify that the measurement model derived in EFA fit the data in an independent validation sample. The CFA was conducted separately in each group using the independent validation samples (confirmatory ASD, N = 443; confirmatory non-ASD, N = 443). Several CFAs were conducted, including a CFA of the measurement model derived using EFA, a CFA testing the hypothesized factor structure based on the structure of the NCI survey, and a CFA testing the hypothesized structure of the NCI survey only including indicators that EFA analyses indicated as having adequate factor loadings (>.20 on any factor). Fit indices for the three CFAs in both the ASD and non-ASD group are presented in Table 5. Results confirmed the factor structure derived in the EFA as the best-fitting measurement model in both the ASD and non-ASD groups (RMSEA = .060, CFI = 0.835; RMSEA = .051, CFI = 0.873, respectively) when compared with the NCI survey measurement model, which had mediocre fit in both the ASD and non-ASD group (RMSEA = .083, CFI = 0.588; RMSEA = .088, CFI = 0.561, respectively; Curran, West, & Finch, 1996). Factor loadings for the CFA of the derived measurement model are presented in Table 6. After evaluating the fit of the full CFA model, model specification and fit was calculated for each latent variable subcluster of the hypothesized model in each group to confirm the measurement model for all three latent variables individually (see Table 7 for goodness of fit statistics for the factor structure of each latent variable by group). Overall, CFA results confirm the measurement model derived using EFA procedures of the three latent variables, Social Participation and Relationships, Social Determination, and Personal Control, and confirms the superior fit of the derived measurement model over the NCI survey measurement model in both the ASD and non-ASD confirmatory groups.
Every year more than 12,000 individuals are assessed across the country and form a randomly selected sample of individuals used to track outcomes of DD services across more than 30 states. These data are a valuable source of information on the status of individuals with ASD, intellectual disabilities, and other related developmental disabilities. Most relevant published research on the NCI Adult Consumer Survey to date has involved creating sum scores of survey subdomains and then comparing constructs across various groups based on those sum scores. This study offers a novel data-driven approach to the analysis of the NCI Adult Consumer Survey data. The NCI survey was designed to be administered to all persons who receive state DD services rather than specific subpopulations; thus, it is important to consider how survey items function in a given population of interest before drawing conclusions regarding any given construct in a specific subpopulation. The approach taken in this study first explored the contribution of each survey item to the latent construct it is intended to measure in each sample (adults with ASD and adults with DD other than ASD) before testing the relationships between these latent constructs. Results of EFAs yielded an empirically derived factor structure comparable in both groups of individuals (with and without ASD) that differs from a factor structure consistent with the theoretical organization of the NCI survey items. This emerging empirically driven factor structure captures an alternative and theoretically compelling conceptualization of social outcomes and associated environmental factors for adults with and without ASD.
Three novel factors emerged as a result of these analyses: Personal Control, Social Determination, and Social Participation and Relationships. The Personal Control factor is a latent variable capturing the extent to which an individual has control of basic choices that dictate important aspects of that individual's daily life—primarily the extent to which individuals choose their provider staff. The Social Determination factor is a latent variable capturing the extent to which a given individual is making socially relevant choices in their life with regards to how they spend their time, what they buy, and whether or not they date. Items making up the Personal Control and Social Determination factors primarily came from the “Choice” section of the NCI survey. Dividing choice items in this way makes theoretical sense in that there is an important distinction between making long-term-oriented choices regarding one's staff and making day-to-day social choices regarding how one spends one's time.
The Social Participation and Relationships factor is a latent variable capturing the extent to which individuals are participating in their community and forming friendships. Indicators of the Social Participation and Relationships factor came from both the “Friends and Family” and “Community Inclusion” sections of the NCI Survey. These items logically fit together to the extent that they capture the essence of the level of meaningful social involvement an individual has both in their community and in the lives of others. Taken as a whole, these latent constructs offer a novel conceptualization of social outcomes for individuals with and without ASD grounded in empirical validation and may be useful in future research on social outcomes for individuals with and without ASD.
Although the latent variables Social Determination, Social Participation and Relationships, and Personal Control emerging in both populations shared several indicators across groups, it is important to note that the factor structure of each latent construct was not entirely equivalent across groups. This finding has both practical and theoretic implications. From a practical standpoint, it is critical to note that analyses in this study were conducted using exclusively shared indicators of each latent construct to facilitate comparison across groups and, thus, do not capture the slightly unique flavors of each latent construct in the ASD versus non-ASD groups. Further research is warranted both on the shared and unique factor structure of each latent variable in both populations to shed light on the nature of these constructs between and within given populations of interest.
Overall, this study offers a data-driven approach for the use of NCI Adult Consumer Survey data as part of research on outcomes for adults with developmental disabilities by highlighting the critical first step of validating a measurement model before using that model to make comparisons across groups. Furthermore, this study offers a novel conceptualization of social outcomes for adult with and without ASD and offers constructs including Social Determination, Social Participation and Relationships, and Personal Control that may serve as useful outcome measures in future research. Furthermore, this study offers a validated measurement model of those constructs for adults with ASD that may be useful in future studies. These results open the door for future research including the validation of these latent constructs in novel samples of adults with ASD, adults with DD other than ASD, as well as adults with other specific disabilities. Our findings also lay the foundation for research associated with the exploration of the relationships between Personal Control, Social Determination, and Social Participation and Relationships in both adults with and without ASD, as well as comparisons of levels of these constructs across groups.
A portion of this manuscript was presented at the AAIDD Annual Meeting, Pittsburgh, PA, June, 2013, and at the APA Annual Convention, Honolulu, HI, August, 2013. This study was supported in part with funding from the U.S. Administration on Intellectual and Developmental Disabilities grant #90-DD-0691. This project received approval from The Ohio State University's Social and Behavioral Sciences IRB. The authors would like to thank the National Core Indicators for making available their data and Robert Cudeck, Ph.D., and Rebecca Andridge, Ph.D. for their consultation on this project.
If the interviewer feels that either Section 1 or 2 contains invalid responses they may indicate this on the survey form. All data marked as invalid were excluded from these analyses. Responses to Section 1 may be marked invalid if the individual receiving the services is unable to nswer the questions him/herself, as this section contains questions that are subjective in nature and must be answered by the individual (e.g. “do you ever feel lonely?”, “do you have a best friend?”, etc.).
The NCI consists of three sections: Background Information, Section 1, and Section 2. Section 1 contains questions that can only be answered by a face-to-face interview with the individual receiving services as they contain subjective questions. Section 2 contains questions that may be answered by an individual who knows the consumer well or the consumer themselves.
Margaret H. Mehling, Department of Psychology, The Ohio State University; and Marc J. Tassé, The Ohio State University Nisonger Center.
Correspondence concerning this article should be addressed to Margaret Mehling, The Ohio State University, Department of Psychology, Nisonger Center, McCampbell Hall, 1581 Dodd Dr., Columbus, OH, 43210 (email: firstname.lastname@example.org).