Early intensive behavioral intervention is generally effective for children with autism spectrum disorder but is associated with variability in treatment outcome and quality of treatment delivery may contribute to this. This study examined the relationship between therapist personality, attitude toward individuals with a disability, and perceived relationship between therapist and child on procedural fidelity. Discrete Trial Teaching (DTT) was provided at a preschool for children with intellectual disabilities. Seventy DTT sessions between 22 therapists and 35 children were videotaped and analyzed. Data on therapist's attitude toward individuals with a disability, therapist's personality traits, and perceived relationship between therapist and child were also collected. Procedural fidelity was high and significantly related to therapist's attitude toward individuals with a disability, therapist's openness to experience, and perceived relationship between therapist and child. Therapists with high procedural fidelity tended to have a more positive attitude toward individuals with disabilities on the cognitive dimension, a more negative attitude toward individuals with disabilities on the affect dimension, lower levels of openness to experience, and perceived the relationship between themselves and the child as less positive.
Early intervention based on applied behavior analysis (EIBI) is currently recognized the treatment of choice for children with autism spectrum disorder (ASD; Eldevik et al., 2009; Peters-Scheffer, Didden, Korzilius, & Sturmey, 2011; Rogers & Vismara, 2008). A frequent approach used in EIBI is Discrete Trial Teaching (DTT), which is particularly important in the early stages of treatment and mostly combined with more naturalistic approaches (e.g., incidental teaching) to foster generalization (Lovaas, 2003; Smith, 2001).
Although EIBI is generally effective for children with ASD, there is great variability in outcomes (Eikeseth, 2009; Peters-Scheffer et al., 2011; Reichow & Wolery, 2009). Results of studies investigating predictors of outcome are ambivalent, but a subset of studies shows that outcome is associated with child and family factors, including age of treatment onset (e.g., under the age of 4; Granpeesheh, Dixon, Tarbox, Kaplan, & Wilke, 2009), pretreatment IQ, autism symptom severity and speech acquisition early in intervention (Ben-Itzack & Zachor, 2007; Smith, Groen, & Wynn, 2000), comorbid conditions (Smith, Eikeseth, Klevstrand, & Lovaas, 1997; Smith, Klevstrand, & Lovaas, 1995) and parental stress (Osborne, McHugh, Saunders, & Reed, 2008), and treatment features, such as treatment intensity and duration (Eldevik, Eikeseth, Jahr, & Smith, 2006; Granpeesheh et al., 2009; Lovaas, 1987; Peters-Scheffer, Didden, Mulders, & Korzilius, 2010) and intensity and quality of supervision (Eikeseth, Hayward, Gale, Gitlesen, & Eldevik, 2009).
Even small changes in treatment procedures can influence child outcome (Kelly, Green, & Sigman, 1998), suggesting that the quality of treatment delivery can predict treatment outcome (Allen & Warzak, 2000; Bibby, Eikeseth, Martin, Mudford, & Reeves, 2001). Treatment quality is dependent on therapists reproducing intervention during daily sessions (procedural fidelity) and the extent to which treatment is implemented as designed (treatment integrity). While treatment integrity centers on the role of consultant to generate a treatment program that meets the child's needs, procedural fidelity focuses on the accurate use of particular treatment procedures and the skills and judgements of the therapist shown during treatment (Symes, Remington, Brown, & Hastings, 2005).
During EIBI research, treatment in general is implemented with good treatment integrity and procedural fidelity (Sallows & Graupner, 2005; Smith, Buch, & Gamby, 2000) but in community-based programs, procedural fidelity and treatment integrity is likely lower (Love, Carr, Almason, & Petursdottir, 2009; Mudford, Martin, Eikeseth, & Bibby, 2001). Since children learn better when therapists are appropriately trained (Dib & Sturmey, 2007; Sarakoff & Sturmey, 2008) many researchers have improved treatment quality by training paraprofessionals, professionals, and parents to implement EIBI correctly (Crockett, Flemming, Doepke, & Stevens, 2007, Dib & Sturmey, 2007; Leblanc, Ricciardi, & Luisselli, 2005; Sarakoff & Sturmey, 2004; 2008; Thomson, Martin, Arnal, Fazzio, & Yu; 2009); however, there is little research to determine the relationship between therapist individual differences and their performance during EIBI. An exception is Symes et al. (2005) in which 19 therapists who implemented DTT were interviewed about factors they considered to enhance or reduce correct treatment delivery. According to them, both training in behavioral analysis theory and techniques, and therapist's patience and child factors such as likeability, motivation, compliance, intellectual competence, and behavior problems all related to child outcomes. Symes et al. (2005), however, did not observe procedural fidelity. Therefore, this study investigated the relationship between therapist characteristics and the therapist–child relationship and treatment integrity using objective measures of treatment integrity. Given the characteristics of EIBI and the significant learning deficits and challenging behavior displayed by children with ASD and ID, therapists might need specific skills or personality traits to provide adequate treatment. “Personality” may seem an unusual variable for behaviorists to study, yet, Skinner (1953) did indeed discuss personality as a “functionally unified set of responses” (p. 285). The observations made in this study can be interpreted usefully in this light.
Among researchers, there seems to be consensus that there are five replicable, broad dimensions that represent personality at the broadest levels of abstraction: (1) extroversion, (2) agreeableness, (3) conscientiousness, (4) neuroticism (emotional stability), and (5) openness to experiences. In short, extroversion refers to an energetic approach to the social and material world and involves traits like assertiveness, positive emotionality, activity, and sociability. Conscientiousness involves prescribed impulse control that facilitates task- and goal-directed behavior like following norms and rules, planning, organizing and prioritizing tasks, thinking before acting, and postponing gratification. Agreeableness describes a prosocial and communal orientation toward others and comprises traits such as trust, tender-mindedeness, modesty, and altruism. Openness to experiences refers to the complexity, originality, width, and depth of the person's mental and practical life, while neuroticism describes negative emotionality like feeling nervous, sad, anxious, and tense. This classification facilitates the possibility to study a specified personality domain instead of observing all behaviors that comprise an individual's personality (John & Srivastava, 1999).
Over different jobs and settings conscientiousness predicted overall job performance, while the other dimensions predicted job performance for job categories or specific tasks (Barrick & Mount, 1991). As EIBI needs to be delivered accurately and precisely, we hypothesized that therapists with higher scores in conscientiousness and lower scores in openness to experiences, will show higher levels of procedural fidelity. We also expected that therapists with higher scores in agreeableness show higher procedural fidelity and that increased scores of extroversion will be related to lower levels of procedural fidelity. The dimension neuroticism was examined on an exploratory basis. Therapists in Symes et al. (2005) suggested that competent, likeable, motivated, and compliant children are associated with more effective service delivery and that effective treatment delivery was hindered by the therapists' negative reaction to child behavior (Symes et al., 2005). Therefore, we expect higher procedural fidelity in therapist–child dyads with a better therapist–child relationship than in dyads in which the relationship is of lesser quality. Furthermore, we expect that therapists with a negative attitude toward individuals with a disability have lower procedural fidelity scores.
Participants and Setting
At the time of the study, 34 paraprofessional line therapists worked at the ASD treatment program, of which 22 therapists participated. All were female and provided DTT to one or more children with ASD and ID in an educational setting for young children and ID (M = 3.82; SD = 2.67; range: 1–11). All had worked for at least 2 months in this setting. On average, therapists had 22.77 months experience in providing DTT (SD = 25.36; range: 2–103) for 6.32 hours per week (SD = 4.10; range: 2–20). Therapists were aged 20–58 years (M = 28.82; SD = 9.98). Thirty-five children participated and all had autism or Pervasive Developmental Disorder-Not Otherwise Specified (PDD-NOS) and mild to severe ID with a mean ratio IQ of 40. A clinician independent of the study determined diagnoses based mainly on the Autism Diagnostic Observation Schedule (ADOS; Lord, Rutter, DiLavore, & Risi, 2006), the Autism Diagnostic Interview–Revised (ADI-R; Lord, Rutter, & Couteur, 1994), the Mullen Scales of Early Learning (MSEL; Mullen, 1995), the Bayley Scales of Infant Development Second Edition (BSID-II; Van der Meulen, Ruiter, Spelberg, & Smrkovsky, 2002) and/or the Vineland Adaptive Behavior Scales (VABS; Sparrow, Balla, & Cicchetti, 1984). Children's ages ranged from 3.5-10.1 years with a developmental age between 1.5-3.4 years. They were chosen for the program based on availability.
The study took place at an ASD treatment program, which provided EIBI services in the eastern part of the Netherlands to children between 2 to 10 years of age who are classified with ASD and ID. The treatment format was Applied Behavior Analysis (ABA) with an emphasis on DTT, combined with incidental teaching methods. Each child's team included the program consultant, a senior therapist, and several therapists delivering the DTT program. Senior therapists had master's degrees and at least one year of experience in delivering ABA at the project. The consultant held a master's degree in special education and had eight years of experience in treatment of children with ASD with or without ID using ABA treatment procedures.
Procedure and Measurements
Three master special education students collected data on procedural integrity. They selected video fragments at random from a container that had all the video clips of each dyad (n = 37) and, when unavailable, they randomly filmed a treatment session (n = 33) and scored them using an instrument (see below) developed for the purpose of this study.
For each trainer–child dyad one session was scored (average duration 7.6 minutes, range 5 to 10 minutes; minimal of 10 trials) and the number of sessions scored per therapist ranged from 1 to 11 (M = 3.05; SD = 2.52). Although DTT is one method of teaching used during EIBI, only DTT sessions conducted at the table were scored, as the structured character of these trials facilitates the measurement of procedural fidelity.
Personal and job related questionnaire
Therapists were asked to provide personal and job-related information including their age, level of education, experience in delivering DTT, and prior acquaintance with an individual with a disability (ASD and/or ID).
Attitudes toward individuals with a disability
The Multidimensional Attitudes Scale toward individuals with disabilities (MAS; Findler, Vilchinsky, & Werner, 2007) was used to assess the therapist's attitude based on three dimensions: affect, cognition, and behavior. A social scenario vignette was used so that respondents project their own emotions, thoughts and behavior onto the given situation. After reading the vignette describing an interaction between a women called Michelle and an individual in a wheelchair, participants marked 47 items on a 5-point Likert-type scale, indicating to what degree they believed the item accurately reflected how Michelle would feel, think, or act in that situation. The MAS yields a total score (range: 47–235) and scores on the three dimensions. A high score on the MAS represents a negative attitude toward individuals with a disability.
For each therapist–child dyad, therapists completed the Dutch version of the Student-Teacher Relationship Scale (STRS; Cornelissen & Verschueren, 2002) to measure the perceived relationship between therapist and child. The STRS is a questionnaire comprised of 28 items with a 5-point Likert-type scale (range: 28–140). It consists of three subscales: closeness (positive aspects of the relationship between the therapist and the child), conflict (negative aspects of the relationship between the therapist and the child such as child challenging behavior), and dependency (negative aspects of the relationship related to possessive behavior). The STRS has good psychometric properties for individuals with ID (Roeden, Maaskant, Koomen, Candel, & Curfs, 2012). Twenty-two teachers and 35 children were included, forming 70 therapist–child dyads. Therapists completed the STRS for 1 to 11 children (M = 3.05; SD = 2.52), while one to seven therapists completed the STRS for each child (M = 2.29; SD = 1.55). In the analysis, each dyad was used once.
The Dutch version of the NEO-Five Factor Inventory (NEO-FFI; Hoekstra, Ormel, & De Fruyt, 2007) was used to measure neuroticism, extroversion, openness to experiences, agreeableness, and conscientiousness. For each scale, therapists rated twelve items on a 5-point Likert-type response format. Raw factor scale scores were transformed into stanine scale scores using Dutch norms (M = 5; SD = 2).
The dependent variable was the therapist's correct use of the components of DTT during 5 to 10 minutes of one DTT session (minimal of 10 trials). An observation instrument was developed to measure correct use of DTT based on treatment manuals (Leaf, McEachin, & Harsh, 1999; Lovaas, 2003), interviews with senior therapists, the training package of the ABA provider and individual education plans of the children, and included the following components: discriminative stimulus, prompt, response time, feedback, error correction, positive practice trial, and intertrial interval (for a description, see Appendix A). Using event recording, the observer scored the correct or incorrect implementation of the component, omission, or commission of the component and, when applicable, procedural errors that occurred (e.g., incorrect discriminative stimulus, multiple stimuli, incorrect prompt [location, level], and absence of contingent feedback).
The percentage of correct implementation was calculated for each trial using the following formula: number of correct components per trial divided by the total number of components of the trial (i.e., correct and incorrect implemented components, with exclusion of irrelevant components, such as a positive practice trial after a correct response of the child), and multiplied by 100. Total procedural fidelity for each session was computed as the average percentage of correct implementation across trials. In addition, for all fragments, the correct implementation of each component of DTT was calculated by dividing the frequency of the correct implementations of the component over all trials by the total number of trials and multiplied by 100.
Prior to scoring, the three observers and first author practiced scoring using videotaped sessions until interobserver agreement (IOA) of at least 90% was achieved, which was obtained in two sessions (total time: 1.5 hours). An agreement was defined as observers scoring an item on the checklist identically. A disagreement was defined as the observers scoring an item on the checklist differently (including omission and commissions). Over five treatment sessions the three observers obtained a mean IOA of 95% (range 91–100%). The remaining videotapes were scored by one of the three observers and 20% of the sessions was scored by a second observer. The mean IOA was 97% (range 96%–100%).
The average procedural fidelity for each fragment ranged from 74.29% to 100% (M = 90.38, SD = 5.86) and average scores for individual therapists ranged from 82.93% to 98.61% (M = 91.50, SD = 4.57). Procedural fidelity and procedural fidelity for the components were >90% for all components except for the component feedback (67.82%; see Table 1).
Predictors of Procedural Fidelity
Therapist's characteristics are presented in Table 2. Since stanine scores of one to three are below average, four to six average, and seven to nine above average, therapists had average personality scores on the NEO-FFI as compared to the Dutch norms.
First, a hierarchal multiple regression analysis was conducted to assess the contribution of personality traits and attitude toward individuals with a disability to procedural fidelity. Attitudes and traits are both seen as relatively stable, but in general, personality traits are more resistant to transformation than attitudes, as the latter might change as new information or experiences become available (Ajzen, 2005). Therefore, in a stepwise manner, the personality traits neuroticism, extroversion, openness to experiences, agreeableness, and conscientiousness were entered in the first step and scores on the cognition, behavior, and affect scales of the MAS in the second step (see Table 3). R2 was significantly different from zero at the end of each step and the three models significantly improved the ability to predict procedural fidelity (Model 1: F(1,19) = 4.90, p = .04; Model 2: F(2,18) = 6.65, p < .01; Model 3: F(3,17) = 7.58, p < .01). After step 3, in which openness to experiences and the affect and cognition scale of the MAS were included, 57% of the variance of the procedural fidelity was accounted for. However, as only openness to experience and the affect and cognition scales of the MAS contributed significantly to the model, in step 3 neuroticism (t = −0.50; p = .62), extroversion (t = 0.90; p = .38), agreeableness (t = −0.79; p = .44), conscientiousness (t = 1.46; p = .16), and the behavior scale of the MAS (t = −0.17; p = .87) were excluded from the analysis.
To assess the relationship between procedural fidelity and the therapist–child relationship, Pearson correlations were computed using data from 70 therapist–child dyads.
Based on Cohen's (1992) criteria, associations with a medium effect size were found between procedural fidelity and the total scale (r = −.28; p = .02), and between procedural fidelity and the closeness scale of the STRS (r = −.39, p < .01). Hence, therapists who experienced a positive relationship with the child displayed lower procedural fidelity than therapists who experienced the relationship as less positive. No significant relationships were found between procedural fidelity and the dependency scale (r = .06, p = .63) and the conflict scale (r = .09, p = 48) of the STRS.
Until recently, relatively little attention has been given to the impact of individual therapist characteristics on the procedural fidelity and the outcome of treatment for children with ASD and/or ID. This study found that the therapist's personality trait openness to experiences was associated with procedural fidelity. Therapists who displayed more openness to experiences showed lower procedural fidelity, which is in line with our expectations as these individuals experience more difficulties in conforming to rules and schedules (Hoekstra et al., 2007). Also, therapists who experienced a positive relationship with the child displayed lower procedural fidelity than therapists who experienced the relationship as less positive. Finally, therapists with negative emotions toward persons with a disability displayed lower procedural fidelity as opposed to individuals with positive emotions toward individuals with a disability. However, higher procedural fidelity was seen in therapists with less negative cognitions toward individuals with ID. These observations suggest that therapist characteristics do indeed predict their performance during DTT.
Skinner's analysis of personality as an explanatory fiction may yield some interesting implications. For example, for those therapists who experience a positive relationship with the child engaging in interactions with children is often highly reinforcing. Therefore, it might be beneficial for them to engage in other activities where sociable interaction is desirable at times other than DTT, thereby reducing the reinforcing value of engaging in sociable interactions during DTT that may inhibit learning. Alternately, or in addition to the previously mentioned strategy, therapists whose characteristics interfere with certain essential components of DTT may require more training and supervision than other therapists to ensure that they teach the children correctly and effectively.
While it may be possible to speculate that personality tests might be useful in selecting therapists, the magnitude of this relationship is very modest and hence not likely to be helpful in reliably predicting the performance of individual therapists. Future studies might investigate whether the ability in ABA principles on questionnaires with a vignette and/or role play might predict future therapist performance. Also, a work sample before and during training and therapist's responsiveness to training and supervision might predict future therapist behavior.
Overall procedural integrity was relatively high; however, these data may be deceptive. Integrity data on the feedback component of DTT were low and this component includes reinforcement of correct responses and error correction, which are essential components of DTT that may lead to child progress. Therefore, next to overall procedural fidelity, fidelity scores on subscales should be reviewed to obtain a full picture of therapist performance. Further, even where treatment integrity appears relatively high, the ranges were very wide, which might hide problems with individual therapists. Only when these data are validated with data on child response acquisition and reduction in maladaptive behavior or other evidence of learning (Dib & Sturmey, 2007; Sarakoff & Sturmey 2008) can we be sure that these measures of staff performance is valid. Although we do expect that there is a clinical difference between therapists with low and high scores on procedural fidelity, future studies should further explore this by including measures of child behavior in addition to staff behavior, and explore data on overall procedural fidelity and the components of DTT.
The present study has several limitations. First, limited data were collected on child variables. Future studies might investigate to what extent child characteristics influence therapist procedural fidelity. Children's challenging behavior, compliance, motivation, and learning rate might influence how demanding and reinforcing treatment delivery is to a therapist. Also, child factors such as likeability, motivation, compliance, intelligence, and behavior problems might be related to the relationship between the therapist and child and may indirectly influence procedural fidelity. Therefore, future research should adress the gap between child characteristics and procedural fidelity. Second, the present sample of therapists was drawn from one service setting where there may have been relatively good staff training and support, which might have attenuated the relationship between therapist individual differences and treatment integrity. This may limit the generality of these results. Likewise, this study only studies staff implementation of one intervention method. Thus, future research should also determine to what extent these findings can be generalized to other less structured teaching methods such as Pivotal Response Training (Koegel, Koegel, Harrower, & Carter, 1999). Finally, the small sample size may have enlarged the chance of making a Type II error. In addition, the number of predictors in the regression analyses was relatively large in relation to sample size, and therapists and children were included in multiple therapist–child dyads. Therefore, these are tentative results that need to be corroborated in larger samples in future research.
This study was funded by Driestroom, Elst (The Netherlands). We are greatful to the children, parents, therapists and senior therapists of Driestroom (Elst, The Netherlands) for their participication.
Editor-in-Charge: Glenn Fujiura
This study was funded by Driestroom (Elst, The Netherlands). We are greatful to the children, parents, therapists and senior therapists of name facility for their participication.
Nienke Peters-Scheffer (e-mail: email@example.com), Radboud University Nijmegen/Driestroom, Behavioural Science Institute, Montessorilaan 3/Postbus 139, Nijmegen/Elst, Gelderland, Netherlands; Robert Didden, Radboud University Nijmegen–Behavioural Science Institute; Hubert Korzilius, Radboud University Nijmegen–School of Management; Peter Sturmey, Queens College, Department of Psychology, New York.