In the past 10 years, we have witnessed significant changes in the nature and extent of the publicly financed systems of services and supports for individuals with developmental disabilities and their families. Growing service needs, declining state revenues, mounting pressures on federal and state Medicaid budgets, and heightened federal quality management expectations are placing increasing demands on state developmental disabilities agencies to improve accountability, more effectively track outcomes, and strengthen their capacity to assess and improve service quality and responsiveness. The initiation of the National Core Indicators collaboration 10 years ago was in direct response to these and other forces described below (National Core Indicators, 2004, 2005a, 2005b).
Public systems for people with intellectual and developmental disabilities have become highly diverse and widely dispersed, making on-site monitoring by public managers increasingly difficult. As the places where people live and work and the supports that they receive become more individualized and idiosyncratic, the necessity for launching new performance appraisal systems has continued to grow.
Recent changes in the administration of the Home and Community Based Medicaid Waiver program by the U.S. Department of Health and Human Services Centers for Medicare and Medicaid Services (CMS) have created an impetus for states to “modernize” their quality and performance management systems. The enhanced federal expectations included in the revised formal Waiver application require that each state agency explain the operational features of its planned HCBS Medicaid Waiver program in detail. States also describe the components of the comprehensive quality management strategy that they intend to employ to monitor and improve the accessibility and quality of services offered to Waiver participants and to demonstrate that required Waiver assurances are being met.
Through this new approach, CMS will draw conclusions regarding the effectiveness of a state's quality management strategy based on the presence or absence of objective, measurable data demonstrating that the state is actively managing the quality of Medicaid Waiver services and supports through a continuous process of discovery, remediation, and improvement. To respond to the new emphasis on performance and outcome data, CMS has recommended that states consider the use of standardized tools, such as the National Core Indicators, to gather and analyze information on Waiver participants' outcomes and satisfaction with the services they receive. The changing expectations on the part of CMS regarding quality monitoring and performance measurement parallel gubernatorial and legislative initiatives in several states aimed at increasing government accountability, improving service outcomes, and strengthening program responsiveness.
History and Structure of National Core Indicators
In the late 1990s, mounting financial pressures caused many states to re-evaluate their existing programs and fiscal policies. Rising Medicaid costs, flat or declining state revenues, and the potential of fiscal savings through the implementation of managed health care under the Medicaid program spurred state officials to seek a better value for their service dollar. In many areas the resulting cost–benefit analysis led to a new emphasis on outcome tracking and performance measurement in human service delivery systems. In 1997, representatives of 13 state intellectual and developmental disabilities agencies convened to discuss the possibility of entering into an unprecedented collaboration, a collaboration involving a cross-state agreement regarding key performance indicators and a resolve to collect data pegged to those indicators. The initial group has since expanded to 24 states and four regional developmental disabilities centers in California, with new states being added each year. The effort, termed National Core Indicators, represents an impressive multistate collaboration that has been supported almost entirely by state resources. Only a modicum of federal funding has been directed to the project.
The National Core Indicators program has made it possible for the participating states to compare their performance on key outcome indicators with that of other states as well as with national norms, a level of exposure that many public managers would have shied away from when developmental disabilities systems were more embryonic. The presence of National Core Indicators is a testament to the maturation of community services and supports and an increased confidence in policy choices that have continued to enhance and expand community resources.
The performance indicators developed by the collaborators provide a macro-level view of system performance in order to gauge trends and identify potential problems. They are intended to be used in tandem with other quality assurance processes that states operate under broader quality management systems (e.g., critical incident and reporting systems, risk assessments, provider monitoring). As such, the core indicators do not encompass all elements of system performance but, rather, are designed to target critical areas of concern.
The selection of core indicators was based on key criteria identified by state developmental disabilities policymakers as essential to the accomplishment of vital program outcomes. Core indicators are designed to (a) be directly relevant to major organizational or systemic goals, (b) reflect activities that can be influenced by the organization or system, (c) be relevant to the major constituencies served by the organization or system, (d) have directional qualities so as to reflect changes over time, (e) be expressed as rates or proportions, and (f) include a standard or goal for the desired level of attainment of each outcome (Bradley, 2003).
The domains within which the 61 indicators are organized reflect that major components of the mission of public developmental disabilities systems. They include work, community inclusion, self-determination, choice and decision-making, relationships, family and individual participation, access, safety and health, and family support—all areas that, regardless of the state, describe the larger aspirations for intellectual and developmental disabilities systems. Over the past 10 years, the measures and data-collection protocols have changed somewhat but the overarching desired outcomes have stayed the same.
Another key criterion for the selection of National Core Indicators measures was, and continues to be, the extent to which the indicator makes benchmarking between and among states possible. Therefore, indicators must yield comparable data. Comparability is accomplished through the use of common data-collection protocols (e.g., consumer and family surveys) as well as through common definitions of the particular phenomenon and data source addressed by the indicator. During the past decade, we have gotten closer to comparability on such previously idiosyncratic measures such as mortality and serious injuries.
The heart of National Core Indicators is the survey of individual outcomes. The survey is based on a tool that was developed in collaboration with the National Core Indicators Technical Advisory Committee, which was made up of state officials with survey and research experience. The committee recommended that for certain more subjective questions that require a personal response (e.g., Are you happy with your job? Are you happy with where you live?) proxies would not be allowed. As a consequence, in addition to demographic and service information, the survey is divided into two major parts: one that only the individual can answer and the second that can be answered by proxies if the individual is unable to respond. The latter questions relate to more concrete, objective facts, such as whether the individual needs a communication device. Over the last 10 years, we have found that between 65% and 70% of respondents are capable of responding.
The survey has undergone a number of validity and reliability tests and is accompanied by a comprehensive training package. Each state (either every year or every other year) collects information on at least 400 individuals served in the range of settings supported by the public developmental disabilities system. In order to compare results from state to state, the data are “risk adjusted,” based on the functional characteristics of the individuals served. This process reduces significant differences in the characteristics of the individuals served and the outcomes they experience.
The consumer survey database reflects information on over 9,000 individuals, an enormously rich source of information about the life experiences of people with intellectual and developmental disabilities around the country. Data from this source were recently analyzed in connection with an evaluation of Home and Community Based Services completed for CMS that paired individual- and service-related information with Medicaid payment files (University of Minnesota, 2006). Results of the study provided important insights into the relationships between outcome, satisfaction, and expenditure variables and service-related factors, such as personal characteristics, residential setting size and type, and self-direction.
Information on the perspectives of family members is gathered through three distinct National Core Indicators family surveys: (a) Child Family Survey (for families of children with disabilities living at home), (b) Adult Family Survey (for families of adults with disabilities living at home), and (c) Family Guardian Survey (for families of adults with disabilities living outside the family's home).
Other data-collection protocols
In addition to the consumer and family surveys, several other data-collection protocols align with particular domains and key performance indicators. Instruments have been developed to collect information on staff turnover, consumer and family representation on boards of directors, employment of people with intellectual and developmental disabilities, serious incidents, and mortality.
Implications for Policy and Practice
The multistate database that has been generated by National Core Indicators has been the source of some interesting, provocative, and sometimes surprising findings. In one of the Data Briefs periodically circulated to participating states, factors affecting access to health services were reviewed (National Core Indicators, 2005a). Some of the issues highlighted included the finding that non-Whites were less likely to get regularly scheduled health check-ups, that availability of transportation was linked to the ability of people to secure health care, and that people who lived with their families were less likely to get routine check-ups than were people who lived in structured living arrangements.
Another Data Brief (National Core Indicators, 2004) focused on self-determination reported data suggesting (albeit with some caution regarding variation in data collection) that people who had individual budgets and who used a fiscal intermediary were more likely to exercise greater control over their environments, to make more choices, live somewhat more independently, and have improved relationships with their case managers.
More recently, in a study funded by the CMS, National Core Indicators data on 3,000 individuals from 6 states (Alabama, Kentucky, Indiana, Oklahoma, Massachusetts and Wyoming) were used to determine, among other things, the differences in the characteristics and outcomes of individuals served in Intermediate Care Facilities for People with Mental Retardation (ICFs/MR), and in HCBS Waiver funded services. The findings, although not really surprising, were none the less sobering. They indicated that people with more severe disabilities were more likely to live in ICFs/MR than people living in HCBS supported settings; people living in ICFs/MR made fewer choices about where and with whom they lived; and satisfaction with services and supports was lowest among people with dual diagnoses (mental illness and intellectual/developmental disability, and highest among people living at home with families) (University of Minnesota, 2006).
With respect to the results of the family surveys, a Data Brief circulated in August of 2005 revealed statistically significant differences in families' experiences and satisfaction with information and planning, levels of choice and control, service and support delivery, and connections to the community. These differences existed between families with a family member at home versus out-of-home and between families with children versus families with adults living at home.
Most interesting, perhaps, were comparisons with the Family Guardian Survey (out-of-home adults with developmental disabilities). In contrast to the other survey results, families of people who live outside the home are clearly more satisfied with the planning and delivery of services, although having less involvement and control in the process. In nearly all instances, comparisons between out-of-home respondents and families with adults and children living at home demonstrate significant differences at the .001 level. Furthermore, families with a family member with a disability living at home felt more isolated and more cut off from their communities. Out-of-home families were also more likely to know how much money was being spent on their family members.
These selected findings reflect only a small sampling of the policy and practice implications that can be mined from these rich data on outcomes for people receiving services and supports around the country.
Current State Activities
In reviewing state experiences over the past 10 years, we found many examples of the application of National Core Indicators to bolster strategic planning, transparency, and quality improvement. In Massachusetts, for example, the results are linked to strategic-planning objectives and are published as an annual report each year. In the state of Washington, results of the family surveys are reviewed by families on the Developmental Disabilities Council and policy recommendations are generated. In Pennsylvania, the consumer survey is administered to people with disabilities and family members, and the results are reviewed by local county intellectual disabilities agencies. In Rhode Island, the results have been used over the past 3 years to provide information for a statewide quality council that identifies priority issues and reviews progress toward improvement. Several states have also built their CMS-mandated Quality Management Plans around National Core Indicators, including the states of Massachusetts and Washington.
These are only some of the steps that states have taken since 1997 to take advantage of the opportunities that National Core Indicators have provided them to both look within their own systems and compare themselves with accomplishments nationally. It has also provided participants with a growing network nationally of similarly situated public managers who can provide advice, experience, and examples of what has worked and what has not worked in their attempts to integrate National Core Indicators in quality management systems.
National Core Indicators Going Forward
The National Core Indicators program has continued to grow in the 10 years since its implementation in 1997, expanding both the scope of its performance measures and the number of states involved. Future activities will continue to include analyses of the information included in the exhaustive National Core Indicators database. Additional objectives include the following:
Support states to further embed performance measurement into their quality management systems
Develop the capacity among public managers to use, display, and disseminate the information gleaned from National Core Indicators
Expand the involvement of self-advocates and families at various levels of the National Core Indicators effort
Increase the focus on developing peer-interviewer training resources and report formats that are accessible to a wider audience
Develop better measures to describe how effective state systems are in supporting individuals and families to direct and manage their own supports
Play an increasingly important role as a forum for states to exchange information and explore new methods of measuring performance and outcomes
The National Core Indicators represent an unprecedented interstate collaboration. It has increased and enriched the conversation about performance measurement and assisted more than half of the states in the country to respond to the increasing pressure to manage systems in a more data-based fashion. As this collective enterprise continues to expand, the hope is that it will also increase the quality and responsiveness of services and supports and, therefore, will improve the lives and well-being of people with intellectual and developmental disabilities and their families.
Authors: Valerie J. Bradley, MA, President (email@example.com), Human Services Research Institute, 2336 Massachusetts Ave., Cambridge, MA 02140. Chas Moseley, PhD, Director of Special Projects, National Association of State Directors of Developmental Disabilities Services, 113 Oronoco St., Alexandria, VA 22314