Reliability and Validity of the Revised Transfer Assessment Instrument

A panel of four individuals with experience using the TAI 3.0 completed initial revisions. Items with lower than acceptable reliability in the TAI 3.013 were first targeted for revision. These items and a summary of subsequent revisions can be found in Table 1. Due to the low reliability of some items in Part 2 and redundancy with items in Part 1, Part 2 was removed from the tool. To improve ease of use for clinicians and remove the need for training, clinical terms were removed and replaced by greater detailed explanations and/or pictures. Separate scoring was added for manual and power wheelchair users with a “not applicable” (N/A) option as appropriate. For potential future use as a self-report measure, the tool was revised to use first-person text and multiple choice answers, rather than a response of “yes” always indicating correct technique as is the case with the TAI 3.0.

Following initial revisions, two external reviewers with more than 20 years clinical experience provided feedback on the content, wording, and scoring of items. Their additional changes included adding introductory text describing the scope of the assessment, creating separate versions of the tool for assisted and dependent transfers, and removing head-hips terminology with a focus just on the direction of trunk lean. The final version of the tool, TAI 4.0, consists of 16 core questions (eAppendix; provided as supplemental digital material). There are two additional questions that relate to assistive technology (transfer board or lift) that rely on reports of when assistive technology is used and how it used.

The TAI 4.0 is scored on a 0 to 10 scale. Instructions for scoring are provided as part of the tool (eAppendix). On the clinician version, scoring prompts are provided next to each question. A score of 1 on each item would indicate perfect technique, with 0 as very poor technique. Partial credit (0.5) is allowed for some items. All item scores are added together, multiplied by 10, and averaged, resulting in a score from 0 to 10 points:

Items scored as “not applicable” or “not possible” are removed from the total score calculation. For some items, points are subtracted from the total item score for certain answer options (items 14–15), but the lowest possible score for each item is still 0 (ie, a negative score is tallied as 0 when calculating the final score). For items 10–11, scoring was revised to reflect the potential use of multiple hand placements during the transfer. For these items, the scores for the different hand placements are tallied and then averaged, but the total item score still ranges from 0 to 1.

In a similar manner, we also calculated three subscores that correspond to the three phases of a transfer: wheelchair setup (6 items), body setup (7 items), and flight/landing (3 items). The items that correspond to each of the subscores can be found in the eTable (provided as supplemental digital material).

Subjects were recruited from the 2017 National Veterans Wheelchair Games (NVWG) in Cincinnati, Ohio, and signed consent forms were approved by the Veterans Affairs Pittsburgh HealthCare System Institutional Review Board. Each participant met the following criteria: >18 years old, used a wheelchair for their primary means of mobility (≥40 hours/week), could independently transfer to/from a wheelchair surface within 30 seconds, and spoke English. Participants were excluded if they had a recent (within the past 3 months) or current history of pressure sores, stood to transfer, had a neurologic condition that could impair learning, or had arm pain that limited their ability to transfer or bear weight through their arms.

During administration, the participant was directed to set up his or her wheelchair and body as he or she normally would for a transfer. The height of the mat table was set to 22 in. prior to each participant's first transfer, and participants were instructed that the height of the mat table was adjustable. After participant setup but before the actual transfer, a ruler was used to measure the distance between the front corner of the chair and the mat table as well as the difference in height between the cushion and the mat table. Additionally, a goniometer was used to measure the angle between the chair and the mat table. All measurements were read aloud. Participants were then instructed to transfer between the wheelchair and mat table (session 1). The participants were then asked to transfer back to their wheelchair. After at least a 10-minute delay, participants completed a second transfer (session 2) to assess intrarater reliability. To assess test-retest reliability, participants returned 1 to 2 days later to complete another transfer (session 3).

General demographic information was collected at baseline and included age, gender, and diagnosis. Four raters used the TAI 4.0 to assess each transfer. As with the original version of the TAI, a visual analog scale (VAS) was completed as a global assessment of each transfer to assess concurrent validity.12 The VAS was anchored by 0% (poor transfer) and 100% (excellent transfer). For ease of comparison between the VAS and TAI 4.0, VAS scores were divided by 10.

Data analysis was performed using IBM SPSS 24.0 software (IBM Corp., Armonk, NY). Descriptive statistics were calculated for subject demographics and TAI 4.0 total, subscore, and item scores. Pairwise comparisons were used to determine if differences existed between session 1 versus 2 and 1 versus 3 for all TAI 4.0 raters as well as VAS scores. Differences in TAI 4.0 scores based on subject demographics were investigated using Mann-Whitney U or Kruskal-Wallis tests. Disability groups were recoded as SCI, amputee, or other for this purpose. Spearman rank correlation was used to investigate a relationship between age, years since injury, and average TAI 4.0 scores for each participant.

Intraclass correlation coefficients (ICCs) were used to assess interrater reliability (session 1), intrarater reliability (session 1 vs 2), and test-retest reliability (session 1 vs 3) for the TAI 4.0 total score.15,16 Based on previous studies, we set cutoffs a priori of ≥0.8 for strong, 0.60–0.79 for acceptable, 0.40–0.59 for moderate, and ≤0.39 for poor reliability.12,13,17,18 

The standard error of measurement (SEM) was calculated for each session as SEM = SD × [1-r]^1/2 where SD is the SD of the dataset and r is the interrater reliability coefficient.13 The minimal detectable change (MDC) was calculated as MDC = 1.96 × 2^1/2 x SEM. Both SEM and MDC were compared to the TAI 3.0 from a previous study.13 

Items on the TAI 4.0 are targeted to specific components of the transfer to identify deficit areas for training and intervention. To evaluate consistency at the item level, we also calculated the percentage of participants who were deficient in each item (score <1) according to each rater across all three sessions.

The VAS was used as the gold standard to evaluate construct validity of the TAI 4.0.12 Pearson correlations were used to estimate the relationship between the TAI 4.0 and VAS for session 1. Due to the global nature of the VAS score, it is unclear what drives the global assessment and which aspects of a transfer clinicians find most important (or most concerning). The TAI 4.0 improves upon the VAS by including possible total scores, subscores, and item-level scores, allowing for a more gradient analysis of the aspects of transfer. To further investigate the individual factors associated with VAS scores, we evaluated the relationships between the VAS and TAI 4.0 subscores (Pearson correlations) and item scores (point biserial correlations). We completed this analysis only for session 1 (which had the largest sample size) and one rater (experienced rater and TAI 4.0 total score had the strongest correlation to VAS). Additionally, we evaluated if clinician scoring of the VAS differed for “good” and “bad” transfer technique. To explore these differences, the file was dichotomized based on the mean TAI 4.0 total score for the same rater used in the above described analysis (good technique: score ≥7.76; bad technique: score <7.76) and the correlations repeated.

Forty-four participants completed at least the first transfer as part of this study. One individual did not complete the second transfer, and 16 individuals did not return to complete the third transfer due to either scheduling conflicts or their first visit being completed on the final day of the NVWG with no days remaining for follow-up. The majority of the sample was male, had a spinal cord injury, and used a manual wheelchair (Table 2). The average age since injury was 56.5 ±12.7 years, with the average time since injury 17.4 ±11.4 years.

The average TAI 4.0 score across all transfers was 7.58 (SD = 1.12). There were no differences in TAI 4.0 score based on diagnosis (p = .078), wheelchair type (p = .621), or gender (p = .684). TAI 4.0 score was also not significantly correlated with age (R = −0.337, p = .080) or years since injury (R = −0.269, p = .193). The average subscores across all transfers were as follows: 7.55 ± 1.95 (wheelchair setup), 7.10 ± 1.49 (body setup), and 8.69 ± 2.51 (flight/landing).

The raters included two physical therapists, a doctor of physical therapy student, and a kinesologist/certified athletic trainer. Because novice clinicians as well as researchers may be utilizing the tool, we included a student and kinesiologist/athletic trainer as raters in order to provide a range of level of experience and diversity in training backgrounds, respectively. They ranged in age from 24 to 36 years and from 2 to 13 years of experience working with wheelchair users. All raters had experience providing transfer training to wheelchair users in a clinical or research setting. The VAS was completed by a physical therapist with 29 years of experience and knowledge of the clinical practice guidelines on transfers.19 

No adverse events occurred as part of the testing. Raters completed the TAI 4.0 for the first transfer in an average of 3.3 ± 0.9 minutes. No additional training was provided. Raters indicated that a consolidated single scoring sheet might have been useful once they were familiar with the items of the tool.

For the TAI 4.0 total score, we found strong interrater reliability at both session 1 and 2 (Table 3), indicating that, overall, the four raters agreed on total scores assigned to users. We also found acceptable to strong intrarater reliability for total TAI 4.0 scores for each rater between sessions 1 and 2 (Table 3). These lower values were generally expected, as the participants performed each transfer twice and their underlying technique was expected to vary slightly. The ICCs for test-retest reliability were mostly acceptable to strong (≥.6), although rater 3 only exhibited moderate agreement between sessions (Table 3).

For subscores, ICCs were mostly moderate to strong. Interrater reliability was highest for wheelchair setup (>.9 for both sessions) and lowest for body setup (Table 3). Intrarater reliability for subscores was mixed over the subscores, with certain raters having much higher ICCs than others (Table 3). For example, rater 1 had very high intrarater reliability over sessions for body setup (1.0), while rater 2 had highest intrarater reliability for flight/landing (.86). All raters had the lower agreement between sessions for wheelchair setup (all <.65), which might mean there is more variability within a user when performing repeated transfers. For test-retest reliability, wheelchair setup had the lowest agreement for all raters, with body setup and flight/landing having mostly acceptable values (most >.7).

In a post hoc analysis, the first five cases were omitted and the analysis was repeated to detect potential learning effects. Analysis remained stable; thus learning effects were not an issue (data not shown).

The calculated SEM for TAI 4.0 total scores were 0.24 and 0.23 for session 1 and 2, respectively, while the associated MDCs were 0.68 (session 1) and 0.63 (session 2). There were no significant differences in total scores between session 1 and 2 or 1 and 3 for any rater (p values = 0.16–0.93).

The percentage of participants who were deficient in each item (score <1) according to each rater across all three sessions were examined (eTable). Only two users utilized a sliding board so consistency is not reported in item-level statistics, but items (item 17–18) are included in the total score. Items where raters disagree on scores for >2 participants include foot position, scooting, push hand grip, leading hand grip, leading hand distance, lean, and movement between surfaces.

On the TAI 4.0, there are five items that have a “not applicable” or “not possible” option. All participant wheelchairs were equipped with brakes and no participants reported being unable to scoot to the front of their seat secondary to feeling unstable. All participants had the potential to complete a level transfer. Fifty-two percent of wheelchairs were not equipped with armrests, whereas 71% did not have sideguards or they could not be removed if present.

Relationships between the TAI 4.0 scores and the 10-point VAS were examined. Figure 1 shows the average values for all three sessions. While the mean VAS (7.44 ± 1.78) was similar to the mean TAI 4.0 total scores, the variability over the sample was much greater for the VAS. There was a moderate relationship between the two scores, with correlations averaging 0.59 ± 0.10.

When examining the relationship between the TAI subscores and the VAS, we found that body setup (R = 0.487, p = .001) and flight/landing (R = 0.701, p < .001) were significantly correlated to VAS. Specific items in the body setup that correlated significantly to the VAS were scooting (R = 0.505, p < .001) and leading hand grip (R = 0.326, p = .031), while significant items in the flight/landing category were leaning (R = 0.538, p < .001), moving between surfaces (R = 0.454, p = .002), and landing (R = 0.454, p = .002). We found that for those with more deficiencies in their technique (ie, “poor” transfer technique), VAS was correlated to only flight/landing (R = 0.768, p < .001); while for those with “good” technique, VAS was correlated to only body setup (R = 0.485, p = .004). Two outliers were noted in the data based on VAS score for session 1 (Figure 1). When examining their TAI 4.0 subscores, these individuals had many deficits in flight/landing but average values for body and wheelchair setup, indicating that the VAS might be driven more by skills in flight/landing.

The knowledge translation of research findings into clinical practice is often slow, with an average lag time often cited of 17 years.20 During revisions of the TAI 3.0, specific efforts were made to address implementation barriers of time and resources required to utilize the tool. Refinements to the TAI resulted in an accessible outcome measure that does not require any additional training. The tool was administered in a reasonable time (<5 minutes; average of 3 minutes) across raters with varying levels of experience. The only additional items required to complete the examination were a ruler and goniometer, which are low cost and common to clinical practice settings.

Wheelchair users in this study represented a range of ages, years of experience, and diagnoses. Our sample was predominantly male, however this is reflective of the general population of mobility device users.13,21,22 We found no bias in TAI 4.0 scores secondary to demographic characteristics. The average TAI 4.0 score was comparable to the average score reported for the TAI 3.0 (7.30 ± 1.42) when taken from a similar population.

The interrater reliability was strong for total score indicating comparable results to the TAI 3.0. Based on the diversity of raters, these results support the reliability of the tool across different levels of experience as well as varying practice settings (research vs clinic). Intrarater reliability ICCs were lower but still met acceptable levels. Test-retest reliability was moderate to acceptable. Based on item level analysis and within subject changes in the VAS, we believe differences between sessions are likely attributed to a change in transfer technique rather than true rater disagreement. Future evaluation with an added gold standard of video recording may shed light on differences in measurement error versus changes in technique between sessions.

At the item level, we found discrepancies in items associated with body setup and flight/landing. A better definition of what is considered a sufficient scoot forward to prepare for the transfer may be warranted. For hand placement, a revised definition of what is considered the transfer versus what is considered setup may be beneficial, as sometimes placement changes between body positioning and actual loading while moving between surfaces. Further, viewing angle may affect a rater's perception of hand placement. Similar to scooting, scoring the leaning item requires a subjective judgment on behalf of the rater regarding what amount of lean differentiates an upright posture from a forward posture. The lean item is closest to the head-hips item on the TAI 3.0, which had moderate to acceptable reliability. For between surface movement, scoring varied regarding whether or not abrupt movement was present. As the flight phase happens relatively quickly compared to the setup phase, this feature may be more difficult for raters to capture. For all of these items, having the ability to review a recording of the transfer may assist with improving reliability. Low variability was reported for trailing hand distance, landing, and alternating arm placement, indicating consistent technique across participants in our sample. Before using the score in clinical settings or as a research outcome, further investigation into the reliability for these items is needed.

While a VAS may provide overall insight into whether or not someone would benefit from transfer training, it is highly dependent on clinical experience and does not highlight specific areas for improvement. In contrast, the TAI 4.0 breaks the transfer down into phases and specific items in each phase that can provide a road map for targeting interventions and also a quantitative measure of improvements. Additionally, when compared to the VAS, TAI 4.0 scores had a smaller range, which we believe indicates that the tool is less sensitive to deficits in only one of the three areas of transfers skills. This was further supported by the examination of the two individuals with the lowest VAS scores; they had average ability in wheelchair and body setup but significant deficiencies in flight/landing. Further, the moderate correlation between the total score and VAS may indicate that clinical judgment in scoring the VAS is variable and depends more strongly on easily seen errors in technique.

The MDC indicated that a change of at least 1.30 in the TAI 4.0 total score is needed to detect significant difference in transfer skills. However, minimal clinically important differences still need to be identified using longitudinal data and an external anchor in future studies. Additional research into potential weighting of items or subscales (perhaps body setup or flight/landing) when calculating the total score should also be explored.

Participants in this population were recruited from the NVWG, a group who is likely more active than the general population. As such, conclusions about transfer skill deficits in the general population should be interpreted with caution. Overall, the majority of scores for both the VAS and TAI 4.0 were high, indicating good transfer technique. Future studies would benefit from participants with a greater range of transfer abilities. While all transfers in this study were from a wheelchair to a firm mat table, scoring options are also present in the revised tool for soft surfaces, armchairs, and the commode/grab bars. Additional testing may be warranted to evaluate the reliability and scoring implications of these situations. Results from this study are limited to individuals who can transfer independently without standing. Future studies should evaluate the newly created TAI 4.0 assisted and dependent assessments. Future studies should also evaluate the responsiveness of the tool to training and criterion validity and change in technique over time.

The TAI 4.0 provides a generally reliable, quantitative assessment of an individual's transfer without the need for comprehensive training of raters. The tool can be completed in less than 5 minutes in a clinical setting with only a ruler and goniometer. The correlation of the TAI 4.0 to a global assessment may be limited by specific components of the transfer having a stronger influence on clinical judgment. Compared to global assessment, the TAI 4.0 identifies specific targets for interventions.