Abstract

Proxies typically serve as information providers in studies of persons with intellectual disabilities. However, little is known about the concordance between different proxy categories and how proxy characteristics influence the information provided. We compared 89 pairs of relative and staff reports on the living conditions of persons with intellectual disabilities, using percentage agreement and Cohen's kappa statistics. Results demonstrate differences between relative and staff reports for most of the domains investigated, with moderate agreement for objective items and fair agreement for subjective items. Relative and staff proxies contributed different information related to diverse viewpoints and varying types of information. Thus, we suggest that information provided by proxies should not be treated as being interchangeable but, rather, as complementary.

Efforts have often been made in studies involving people with intellectual disabilities to obtain information directly from the persons with the disability. The validity of self-reports, however, has been questioned because of (a) the limited verbal abilities of some persons with intellectual disabilities and (b) response bias (Heal & Sigelman, 1995;Matikka & Vesala, 1997; Perry & Felce, 2002). Thus, proxies, such as informed relatives and staff members, are typical sources of information in studies focused on persons with intellectual disabilities. Individuals with severe intellectual disabilities and limited communicative ability will normally be excluded from study samples unless proxy responses are used. However, informed proxies are considered to provide a satisfactory approximation of the responses the individual with intellectual disabilities would give. Examinations of the agreement between proxy and self-reports of people with intellectual disabilities suggest that the validity of proxy responses is acceptable for objective circumstances (Perry & Felce, 2002; Rapley, Ridgway, & Beyer, 1997; Schwartz & Rabinovitz, 2003; Stancliffe, 1999; Tøssebro, 1998).

Proxy information can even be more reliable than self-reports concerning actual living conditions because proxies control the situations of everyday life and survey the comprehensive situation (Tøssebro, 1990). However, proxy reports and self-reports should not be treated as being interchangeable with regard to subjective evaluations and views (Cummins, 2002; Perry & Felce, 2002; Rapley et al., 1997; Schwartz & Rabinovitz, 2003; Stancliffe, 1995, 1999). Responses by proxies and people with intellectual disabilities may differ in perspective and frame of reference (Schwartz & Rabinovitz, 2003; Stancliffe, 1995, 1999; Wehmeyer & Metzler, 1995).

Only a few researchers have compared reports provided by different proxies; and in those that do exist, the results are contradictory. Stancliffe (1999) reported significant correlations between staff ratings on objective items in the Quality of Life Questionnaire Empowerment scale. In a Norwegian study, Tøssebro (1990) found agreement between staff reports on objective features of living conditions of people with intellectual disabilities, but he suggested that subjective items may be biased by proxy valuation and proxy characteristics. Schwartz and Rabinovitz (2003) examined whether parents are better than staff in predicting self-reports on the Life Satisfaction Scale by comparing total life satisfaction scores. Although scores were positively correlated, the authors found significant differences between reports by parents and reports by staff. Further, parents´ reports were correlated with self-reports, whereas differences were found between staff reports and self-reports. Thus, if persons with intellectual disabilities cannot provide information themselves, Schwartz and Rabinovitz suggested that parents should be the preferred responders. Gaudet, Pulos, Crethar, and Burger (2002) reported lack of concordance between service providers and family members on the Consequence of Traumatic Brain Injury scale.

There are further problems in this area of investigation. Persons who give information are typically treated as a homogeneous group, and little attention has been given to whether proxy responding is influenced by proxy characteristics (gender, age, and educational level), the nature of the relationship (family ties or caregiver–care receiver), and the frame of reference (e.g., people in general and people with and intellectual disabilities).

The living conditions of people with intellectual disabilities in Sweden have radically changed during the last century. These individuals live in the community like other citizens. Service and support are linked to the equality–inequality discourse, and persons with intellectual disabilities share generic social and health services. In addition, they are eligible for complementary special social services and supports typically provided by local government. The living conditions approach is a well-established measure for monitoring welfare and equalization policy in Sweden and some other countries as well (Tøssebro & Kittelsaa, 2004). Since 1974, Statistics Sweden has conducted annual surveys of living conditions of the general population (Statistics Sweden, 1999). One objective has been to compare various sociodemographic groups by focusing on objective conditions rather than people's needs and subjective assessments of these conditions (Erikson, 1993). People with intellectual disabilities are included in these general studies by random; however, they have not been reported separately.

During the last decade, few special surveys of the living conditions of people with intellectual disabilities have been published in Sweden (e.g., Kebbon et al., 1992; Umb-Carlsson & Sonnander, 2005). Though information has been provided by various sources (self-reports, relative responders, staff responders), to our knowledge no Swedish researchers have compared relative and staff reports of living conditions.

Our aim in the present study was to compare the reports of relatives and staff on a questionnaire on living conditions of a selected population of adults with intellectual disabilities and to examine whether proxy characteristics (gender, age, and level of education) influenced the participants' responses.

Method

Participants

In 1974, 213 children born in Uppsala County (228,680 inhabitants) between 1959 and 1974 were registered at the Board for Provision and Services for the Mentally Retarded. At the time of the present study, 57 persons were deceased and 2 had moved abroad. The remaining 154 persons were invited to participate in a study on living conditions; of these, 112 persons gave informed consent. An immediate family member was asked to recommend one relative and one staff member to provide information for each of the 112 persons. The criteria for their choice were that (a) proxies should have extensive direct contact with the person with intellectual disabilities and (b) they should be well-acquainted with the living conditions of the person concerned. No relative was known for 10 persons with intellectual disabilities, and 6 did not have any contact with staff. Thus, 96 relative–staff pairs were identified and constituted the study sample. Of those, 96% (91 relatives and 94 staff) agreed to participate; our sample consisted of 89 relative–staff pairs. Our analyses were based on these 89 pairs of relative–staff reports. The Ethical Committee at the Faculty of Medicine at Uppsala University approved the study.

Measures

A questionnaire was compiled to obtain information from proxies on living conditions of adults with intellectual disabilities. It was based on questionnaires employed in earlier studies of living conditions in Sweden (Kebbon et al., 1992) and Norway (Tøssebro, 1990).

We conducted a pilot study with 14 staff members in housing and daily activities to test face validity of the questionnaire. These participants were experienced professionals, trustees, relatives, and persons with intellectual disabilities. Following the outcome of the pilot study, we made minor adjustments of response alternatives in the questionnaire.

The final version of the questionnaire contained 233 items, divided into 10 domains: Education (14 items regarding type of education), Housing (53 items regarding type of housing, privacy markers (e.g., extent to which the individual had possessions typical of a private home, such as a private mailbox, house key, television, computer, and private space as well as participation in everyday matters, being with neighbors and guests), Employment and Daily Activities (12 items regarding type of occupation and working conditions), Finances (18 items regarding income and financial situation), Recreation and Culture (39 items regarding holiday trips and recreational activities), Transportation (7 items regarding public transportation and mobility service), Family and Social Relations (66 items regarding family circumstances, contact with family and friends, and professional support), Personal Safety (8 items regarding bullying, violence, threats, and fear), Community Participation (6 items regarding association activities, voting, and being informed of the news); and General Items (10 items regarding influence, well-being, and happiness).

The response alternatives on the questionnaire items were in fixed format on a yes/no (25%) and a 3- (18%), 4- (20%), 5- (19%), 6- (13%), or 10- (5%) point scale. In addition, 171 items included a don't know response alternative.

Further, relatives and staff members were asked to estimate the communicative ability of the person with intellectual disabilities (understanding language and using language) and to summarize the level of disability on a 5-point scale from no disability to very severe disability. Finally, the questionnaire included information on proxy characteristics (age, gender, level of education) and type, length, and frequency of personal contact with the person who had intellectual disabilities.

Procedure and Analyses

Relatives and staff (n = 192) were asked via a phone call to participate in the study. They were sent the questionnaire by mail and asked to return it to the first author in a self-addressed and prepaid envelope.

Items were categorized according to the type of information asked for in the questionnaire. Sixty-nine percent of the items (160 of 233) described objective circumstances (e.g., type of housing, occupation, income, and leisure time activity). The remaining 31% of the items (73 of 233) were categorized as subjective; the respondent was asked for proxy opinion (12) or proxy estimation of the opinion of the person with intellectual disabilities (61). Responses on objective items and the two types of subjective items were analyzed separately.

The Statistical Package for the Social Sciences (version 11.5 for windows, SPSS, Inc. Chicago) was used for all analyses. Nonparametric tests (Pearson chi-square test with Yates's correction and Fisher's exact test) were used for statistical analyses with the significance level set at .05. All comparisons, which were two-tailed, were made across characteristics of proxies (gender, age, and level of education), characteristics of the persons with intellectual disabilities (age, level of disability, and communicative ability) and frequency of personal contact and type of relation (relative ties or caregiver/care receiver) between the proxy and the person with intellectual disabilities.

Percentage agreement and Cohen's (1960) kappa statistics were used to analyze the relationships between the reports of the relatives and those of the staff. The kappa coefficient corrects for the agreement that can be expected due to chance. The labeling of agreement of Cohen's kappa was set at less than .20, poor agreement; .21 to .40, fair agreement; .41 to .60, moderate agreement; .61 to .80, good agreement, and .81 to 1.00, very good agreement (Taube, 1995). A value of zero indicates that agreement is no better than chance.

In order to correct for differences in response alternatives, we treated relative reports and staff reports as agreeing if the observed difference, on items with a 10-point response scale, was only one step.

Results

Proxy Characteristics

The analyses were based on reports of 178 proxies (89 relatives, 89 staff members). The majority of the proxies were women (see Table 1). Relatives were significantly older than staff, t(82) = −9.20, p = .00. Although most of the relatives were parents (79%). some were siblings (9%) or trustees (12%). Staff responders provided support in housing (80%), daily activities (19%), and personal counseling (1%). The most frequent proxy combination was a parent and a staff person in housing (62% of 89). The chi-square test did not reveal any relation between proxy responses and proxy characteristics (gender, age, or level of education) and frequency and type of relationship with the person who had intellectual disabilities.

Table 1

Distribution of Responders Across Gender, Age, and Level of Education (n = 178)

Distribution of Responders Across Gender, Age, and Level of Education (n = 178)
Distribution of Responders Across Gender, Age, and Level of Education (n = 178)

Nonresponses and Don't Know Responses

Percentage distribution of nonresponses and don't know responses are presented in Table 2. The frequency of nonresponses on objective as well as on the two types of subjective items in relative reports (5.1%) was similar to the frequency in staff reports (4.0%). On objective items, relatives chose the response alternative don't know less frequently than did staff responders (5.3% vs. 10.7%, respectively), whereas on subjective items corresponding frequencies were found (15.4% vs. 13.8%). We note that the don't know alternative was more frequently chosen by both relative and staff responders on items asking the proxy to estimate the opinion of the person with intellectual disabilities than on items asking for proxy opinion (15.2 vs. 9.3%, respectively).

Table 2

Percentage of Frequency of Nonresponses and Don't Know Responses Across Questionnaire Domains and Item Category by Responder

Percentage of Frequency of Nonresponses and Don't Know Responses Across Questionnaire Domains and Item Category by Responder
Percentage of Frequency of Nonresponses and Don't Know Responses Across Questionnaire Domains and Item Category by Responder

Percentage Agreement and Cohen's Kappa

Percentage agreement was calculated for all items. The value of Cohen's kappa was reported for 75% of the items (129 objective and 45 subjective items). For the remaining 25% (31 objective and 28 subjective items), the responses were concentrated to one response alternative, and, thus, the value of kappa was not considered meaningful to report for these items. Percentage agreement and calculation of Cohen's kappa for most domains were consequently based on different numbers of items (Table 3).

Table 3

Case-by-Case Agreement for Objective and Subjective Items Across Questionnaire Domains

Case-by-Case Agreement for Objective and Subjective Items Across Questionnaire Domains
Case-by-Case Agreement for Objective and Subjective Items Across Questionnaire Domains

Characteristics of the Persons With Intellectual Disabilities

All 89 persons with intellectual disabilities (34 women, 55 men) were born between 1959 and 1974 (M age = 34.4 years, SD = 4.1) and registered in 1974 at the Board for Provision and Service to the Mentally Retarded in Uppsala County. Relatives and staff agreed that 34 persons with intellectual disabilities had a severe disability; 11, a moderate disability; and 5, a mild disability, yielding a total percentage of concordance of 61.7 (50 of 81) and a kappa value of .24. Thus, proxies disagreed on level of disability for 31 persons. Relatives rated 19 persons with more severe disability than did staff respondents, and staff members rated 12 persons with more severe disability than did relatives. For the remaining 8 persons, no comparisons were carried out because of missing responses from relatives or staff responders.

Relatives and staff agreed on 63 persons with intellectual disabilities having good or limited communicative ability and 11 having no communicative ability. For 13 persons, relatives and staff disagreed on whether the individual was able to communicate or not. For 2 persons with intellectual disabilities, either relative or staff information was missing.

Living Conditions

The agreement between relative and staff reports was high on items describing type of housing (97% of 89) and type of employment and daily activities (93% of 89). Relatives and staff agreed that 60 persons lived in a group home; 10, independently in a private apartment; and 16, with their parents. For the remaining 3 persons, relatives and staff disagreed whether the housing was an apartment in a group home or an independent apartment. A group home consists of a small number of apartments, either in a villa or scattered in a few residential blocks with staff service in the vicinity. Staff members also visit the private apartments when their service is needed.

A majority of the proxy pairs (83) agreed that the persons with intellectual disabilities were not part of the labor force but, rather, were provided supported daily activities. Daily activity is a community-based special service for people with intellectual disabilities, providing care and enriching experiences or productive activities.

The mean percentage of agreement across all 233 items was 72%, though the level of agreement varied across item domain and item category (see Table 3). High agreement on both objective and subjective items (over 78%) was evident in three domains (Housing, Transportation, and Personal Safety). Low percentage agreement (less than 66%) was noted on objective and subjective items regarding recreation and culture and family and social relations, except for an item asking whether the proxy considered the person with an intellectual disabilities to have a close friend or not. In general, the agreement was higher on objective than on subjective items. The two types of subjective items, asking for proxy opinion and asking the proxy to estimate the opinion of the person with intellectual disabilities, demonstrated corresponding low mean percentage agreement (67% vs. 69%) and fair mean values of kappa, .36 vs. .27.

Item comparison revealed no systematic response pattern between the ratings of relatives and staff when their reports differed. Furthermore, analyses (Pearson chi-square test and Fisher's exact test) indicated that proxy reports were not significantly related to the communicative ability of the person with intellectual disabilities.

Discussion

Results of this study demonstrated fair to moderate relations between the reports of relatives and staff members on most items describing the living conditions of people with intellectual disabilities. A difference was observed between relatives and staff on age and level of education, though the analyses revealed no association between proxy responding and proxy characteristics.

A high percentage of agreement (over 78%) in conjunction with a moderate to very good value of kappa (over .41) indicate sufficient agreement; and, correspondingly, a low percentage of agreement and a low value of kappa indicate disagreement. However, interpretation becomes more difficult when there is a high percentage agreement and a low value of kappa. Such paradoxes may be due to imbalance in the marginal totals of a concordance table, with responses concentrated to one or a few response categories (Feinstein & Cicchetti, 1990). In the present study, we found contradictory mean values of percentage agreement and kappa, particularly for objective items. Furthermore, imbalanced marginal totals with responses concentrated to one response alternative, entailed that the value of kappa was not considered meaningful to calculate for 19% of objective and 38% of the subjective questionnaire items. These findings support the claim that the value of kappa depends not only on the two reports to be compared, but also on the structure of the data. Nevertheless, an advantage of the kappa coefficient is its applicability in nominal scales and its correction for agreement that can be expected by chance. Thus, the value of kappa calculations should not be neglected. In the present study, kappa coefficients as well as percentage agreement demonstrate that there is higher agreement between relative responders and staff responders on objective items than on subjective items on living conditions. However, percentage agreement and kappa statistics merely demonstrate whether there is agreement between two reports: They do not provide any information about whether one proxy is more reliable than the other.

Analyses of nonresponses and don't know responses indicate that relatives and staff were familiar with different aspects of living conditions of the person they rated. For example, fewer relatives than staff members gave nonresponses or responded don't know on objective items about education and finance. A reasonable explanation to this observation is that staff providing support to adults have limited knowledge about their background and, thus, lack information about previous education. Relatives, on the other hand, have a lifelong relationship with the individual and, therefore, have a great deal more information about his or her schooling and education. Correspondingly, response patterns on items of finance may be related to relatives carrying the main financial responsibility, whereas staff were only familiar with economic, everyday matters. Surprisingly, relatives gave more nonresponses than did staff responders on items of employment and daily activities. This may reflect relatives being less involved in, and thus less knowledgeable about, work conditions.

In the present study, case-by-case comparisons between reports of relatives and staff indicated differences for most of the domains investigated, with moderate kappa values for objective items and fair kappa values for subjective items. In addition, objective items demonstrated higher percentage agreement than did subjective items. Furthermore, both responder groups chose the don't know response alternative more frequently on subjective than on objective items, particularly regarding items asking the proxy to estimate the opinion of the person with intellectual disabilities. These findings are consistent with studies in which researchers have cast doubt on the reliability of proxy judgments about subjective preferences and experiences (Cummins, 2002; Perry & Felce, 2002; Rapley et al., 1997; Schwartz & Rabinovitz, 2003; Stancliffe, 1995, 1999). A possible explanation is that objective items are observable, whereas subjective items usually have to be communicated (verbally or nonverbally). The limited agreement on items about recreation and culture and social relations may be because the person with intellectual disabilities considers some issues private and, consequently, chooses not to share them with all proxies. Further, individual proxies may also differ in how interested they are in the private life of the person with intellectual disabilities.

Cognitive difficulties, limited verbal ability, and response bias are typically referred to as impediments to valid responding on interviews and questionnaires. Even though measures are modified and sign language, symbols, and communication technology are used to facilitate the communication, experience has shown that many people with intellectual disabilities are not able to provide information themselves (Perry & Felce, 2002, Tøssebro, 1990). People who cannot communicate with spoken language usually develop their own expressions to communicate, and there is reason to believe that all people communicate basic emotions (i.e., joy, loathe, interest, surprise) in a similar way. This can provide relatives and staff with a good enough base for interpretation of how people with severe intellectual disabilities experience their everyday life (Goode, 1994; Tøssebro, 1990). However, studies have shown that proxy responding is not always correlated with self-reports, particularly regarding subjective issues (Cummins, 2002; Perry & Felce, 2002; Stancliffe, 1995, 1999; Wehmeyer & Metzler, 1995), and there is an increasing advocacy for the use of self-reports as data sources whenever possible. Accordingly, a necessary task for researchers is to develop methods and strategies to obtain information from as many people with intellectual disabilities as possible.

Nevertheless, it seems likely that proxies will continue to be required for some individuals who are unable to respond satisfactorily to interview questions and questionnaires. Unless proxy responses are used, people with severe intellectual disabilities are at risk of being excluded. However, researchers must be aware of the biases introduced with proxy responders and gather data from multiple sources.

The fair to moderate relation between proxy reports may be explained by relatives and staff using different reference groups in making their ratings (Stancliffe, 1995). In the present study relatives may have compared the living conditions of the person with intellectual disabilities with that of people without intellectual disabilities, whereas staff may have used peers with intellectual disabilities as a yardstick, or vice versa. Accordingly, relative and staff differences may be explained in terms of differing perspectives.

Conclusion

Given the fair to moderate relation between the responses of relatives and staff at the case-by-case level, we suggest that such response information should not be treated as being interchangeable. Relative proxies and staff proxies seem to contribute complementary information related to different viewpoints and varying types of information. It seems appropriate, therefore, to consider the type of information sought when choosing the most suitable type of proxy as the data source.

References

References
Cohen
,
J.
1960
.
A coefficient of agreement for nominal scales.
Educational and Psychological Measurement
20
:
37
46
.
Cummins
,
R.
2002
.
Proxy responding for subjective well-being.
International Review of Research in Mental Retardation
25
:
183
207
.
Erikson
,
R.
1993
.
Descriptions of inequality: The Swedish approach to welfare research.
In M. Nussbaum & A. Sen (Eds.), The quality of life (pp. 67–87). Oxford: Clarendon Press
.
Feinstein
,
A. R.
and
D. V.
Cicchetti
.
1990
.
High agreement but low kappa: I. The problems of two paradoxes.
Journal of Clinical Epidemiology
43
:
543
549
.
Gaudet
,
L.
,
S.
Pulos
,
H.
Crethar
, and
S.
Burger
.
2002
.
Psychosocial concerns of adults with developmental disabilities: Perspectives of the self, family member, and provider.
Education and Training in Mental Retardation and Developmental Disabilities
37
:
23
26
.
Goode
,
D.
1994
.
A world without words. The social construction of children born deaf and blind.
Philadelphia: Temple University Press
.
Heal
,
L. W.
and
C. K.
Sigelman
.
1995
.
Response biases in interviews of individuals with limited mental ability.
Journal of Intellectual Disability Research
39
:
331
340
.
Kebbon
,
L.
,
K.
Sonnander
,
S-I.
Windahl
,
K.
Ericsson
,
M.
Tideman
, and
B.
Åkerström
.
1992
.
KOM–UT–projektet. Utvärdering av kommunalisering av omsorger om utvecklingsstörda. Bakgrund och metod [The KOM UT–project. Evaluation of the decentralization of services for people with intellectual disabilities. Background and method].
Uppsala, Sweden: Uppsala University, Centre for Disability Research
.
Matikka
,
L. M.
and
H. T.
Vesala
.
1997
.
Acquiescence in quality-of-life interviews with adults who have mental retardation.
Mental Retardation
35
:
75
82
.
Perry
,
J.
and
D.
Felce
.
2002
.
Subjective and objective quality of life assessment: Responsiveness, response bias, and resident:proxy concordance.
Mental Retardation
40
:
445
456
.
Rapley
,
M.
,
J.
Ridgway
, and
S.
Beyer
.
1997
.
Staff: staff and staff:client reliability of the Schalock & Keith (1993) Quality of Life Questionnaire.
Journal of Intellectual Disability Research
42
:
37
42
.
Statistics Sweden.
1999
.
Undersökningen av levnadsförhållanden (ULF) 99:3–4.
[Survey of living conditions, SLC 99:3–4]. Retrieved May 19, 2000, from http://www.scb.se
.
Schwartz
,
C.
and
S.
Rabinovitz
.
2003
.
Life satisfaction and community living.
Journal of Intellectual Disability Research
47
:
75
84
.
Stancliffe
,
R. J.
1995
.
Assessing opportunities for choice-making: A comparison of self- and staff reports.
American Journal on Mental Retardation
99
:
418
429
.
Stancliffe
,
R. J.
1999
.
Proxy respondents and the reliability of the Quality of Life Questionnaire Empowerment factor.
Journal of Intellectual Disability Research
43
:
185
193
.
Tøssebro
,
J.
1990
.
Fortellinger om velferd for psykiskt utviklingshemmete, metodiske betraktinger over intervju i 3.person.
[Welfare of people with intellectual disabilities, methodological concerns of proxy interviews]. Trondheim, Norway: Helsefaghøgskole
.
Tøssebro
,
J.
1998
.
Researching the living conditions of people with intellectual disabilities.
In E. Hjelmquist & L. Kebbon (Eds.), Methods for studying the living conditions of persons with disabilities (pp. 24–35). Stockholm: Swedish Council for Social Research
.
Tøssebro
,
J.
and
A.
Kittelsaa
.
2004
.
Studying the living conditions of disabled people: Approaches and problems.
In J. Tøssebro & A. Kittelsaa (Eds.), Exploring the living conditions of disabled people (pp. 17–43). Lund: Studentlitteratur
.
Taube
,
A.
1995
.
To judge the judges–kappa, ROC or what?
Anatomie et de Cytologie Pathologiques
43
:
227
233
.
Umb-Carlsson
,
Õ.
and
K.
Sonnander
.
2005
.
Comparison of the living conditions of adults with intellectual disabilities in a Swedish county and in the general population.
Journal of Policy and Practice in Intellectual Disabilities
2
:
240
248
.
Wehmeyer
,
M. L.
and
C. A.
Metzler
.
1995
.
How self-determined are people with mental retardation? The national consumer survey.
Mental Retardation
33
:
111
119
.

This work was supported by the Sävstaholm Foundation, the Swedish Council for Social Research (Project 2001-1039) and the Erik and Carola Tengströms Foundation. We express our thanks to all the persons involved in the study. We also thank Hans Bertil Andersson for valuable statistical advice.

Author notes

Authors: Õie Umb-Carlsson, Doctoral student (oie.umb-c@swipnet.se), Department of Neuroscience, Psychiatry, Ulleråker, SE-750 17 Uppsala, Sweden. Karin Sonnander, PhD, Professor, Department of Public Health and Caring Sciences, Uppsala Science Park, SE-751 85, Uppsala, Sweden