Establishing the Test-Retest Reliability and Minimal Detectable Change of the Multiple Sclerosis Resiliency Scale

Persons with multiple sclerosis (MS) can be confronted with several challenges related to their disease, to which they can have any number of responses. Among persons with MS, resilience has been described as “bouncing back,”¹  “living well,”¹  and “want[ing] to thrive.”¹  Resilience can have several positive effects on the lives of persons with MS. For example, high resilience has been associated with greater levels of physical activity and participation, better diet and psychological well-being, and lower levels of fatigue.²  On the opposite end of the spectrum, persons with lower levels of resilience may experience more psychological distress.^2–5 

Measuring resilience to MS-related challenges provides the opportunity for clinicians to identify relative strengths as well as areas of needed support, both of which can inform targets for intervention. The Multiple Sclerosis Resiliency Scale (MSRS) is a multidimensional measure based on the conceptual framework that MS-related resilience is an interaction between protective factors and risk factors, which include psychological, physical, and social components.^1,4,6–9  The 25-item MSRS produces a total score as well as 5 subscale scores: Emotional and Cognitive Strategies (13 items), Physical Activity and Diet (3 items), MS Peer Support (2 items), Support From Family and Friends (5 items), and Spirituality (2 items).⁶  For each item, respondents use a 4-point Likert scale ranging from “strongly agree” to “strongly disagree.”⁶ 

Several of the psychometric properties of the MSRS have been evaluated, including its internal consistency, convergent validity, and divergent validity.^6,10  Meaningful scores have also been derived with the Connor-Davidson Resilience Scale as a reference,¹⁰  including high (81) and low (75) resilience cutoff values and values for detecting psychological distress (ie, anxiety and depressive symptom severity; 68–72).³  However, the test-retest reliability and minimal detectable change (MDC) of the MSRS have yet to be determined. Besides being an important component of a measure’s psychometric properties, test-retest reliability and MDC are necessary for clinical trials to examine whether there has been a reliable change after an intervention.

Given the physical and psychological benefits of resilience,²  there have been several interventional studies focusing on promoting resilience in MS,^11,12  but these studies have used the 10-item Connor-Davidson Resilience Scale¹¹  or the 15-item Resilience Scale,¹²  which are not specific to MS. As such, this study aimed to establish the test-retest reliability and MDC of the MSRS to increase its utility as an outcome measure for resilience-focused interventions for persons with MS.

The study was conducted at the Joyce D. and Andrew J. Mandell Center for Comprehensive Multiple Sclerosis Care and Neuroscience Research at Mount Sinai Rehabilitation Hospital, Trinity Health Of New England. Surveys were collected between November 2020 and March 2021. Individuals were eligible for the study if they were diagnosed as having MS by a neurologist, were aged 18 to 89 years, were able to read and respond to English-language questionnaires, and had access to the internet and a valid email address. Participants were individuals who had previously expressed interest in research and were contacted about the present study via email or phone call. A sample size of 62 was determined to be adequate to power the test-retest reliability analyses and account for a 20% dropout rate.¹³ 

The study was approved by the Trinity Health Of New England institutional review board. All data collection was performed using REDCap electronic data capture tools hosted at the University of Connecticut Health Center.^14,15  At time 1, participants completed the MSRS, a demographics questionnaire, and the Patient-Determined Disease Steps (PDDS) scale to measure MS-related disability.^16–19  Two weeks later, participants received an email from REDCap to complete the MSRS a second time. This time frame was selected because resilience was not thought to change during this small window. Another REDCap email was automatically sent a week later if participants had not yet taken the second survey. Follow-up phone calls were made as needed to ensure that participants were receiving the survey links. Participants who completed both parts of the study were compensated with a $25 gift card.

A statistical software program (IBM SPSS Statistics for Windows, version 26.0; IBM Corp) was used to conduct the analyses. First, descriptive statistics were run for the demographics and the MSRS scores at time 1 and time 2. If there were missing items on the MSRS (n = 2 at time 1; n = 1 at time 2), the value was imputed using the participant’s mean score on the particular subscale (eg, Emotional and Cognitive Strategies). No participants had more than 1 missing response on the MSRS, and none of the missing responses occurred in the smaller subscales (eg, MS Peer Support). The internal consistency of the total and subscale scores was evaluated using Cronbach α or, if there were only 2 items in the subscale, Spearman-Brown statistic. Test-retest reliability was assessed using a 2-way, random-effects, single-measurement intraclass correlation coefficient (ICC),²⁰  with agreement between time 1 and time 2 visualized with a Bland-Altman plot.²¹  Proportional bias in the Bland-Altman plot was tested using a simple linear regression, with the mean of the time 1 and time 2 measurements as the independent variable and the difference between the 2 measurements as the dependent variable. Based on the ICC, the test-retest reliability could be described as poor (<0.50), moderate (0.50–0.75), good (>0.75–0.90), or excellent (>0.90).²²  The standard error of measurement (SEM) was determined using the mean square error term’s square root,²⁰  which was then used to calculate the MDC with a 95% CI using the following equation: MDC = SEM * 1.96 *√2.

Of the 64 participants who completed time 1, 62 (96.9%) completed time 2 (TABLE S1, which is available at IJMSC.org). Most participants were White (82.3%), were female (77.4%), and had relapsing-remitting MS (85.5%). They had MS for a mean ± SD of 14.42 ± 8.32 years, with a median PDDS scale score of 2 (moderate disability). The mean ± SD time between the 2 assessments was 16.60 ± 3.97 days (range, 14–30 days).

Total scores could range from 25 to 100. Participants had mean ± SD scores of 77.19 ± 11.97 at time 1 and 76.38 ± 12.75 at time 2 (TABLE 1). The total score had good test-retest reliability (ICC = 0.88), and the Bland-Altman plot (FIGURE 1A) was not indicative of proportional bias (mean = 0.81; 95% CI, −11.13 to 12.75; t = −1.04; P = .302). With an SEM of 4.31, the total score’s MDC was 11.95.

Emotional and Cognitive Strategies subscale scores could range from 13 to 52, with mean ± SD scores of 40.48 ± 7.97 at time 1 and 40.55 ± 8.18 at time 2 (Table 1). Good test-retest reliability was noted (ICC = 0.85), and the scale had an MDC of 8.79. The Bland-Altman plot (FIGURE 1B) was not indicative of proportional bias (mean = −0.07; 95% CI, −8.85 to 8.71; t = −0.37, P = .716).

With a score range of 3 to 12, the mean ± SD Physical Activity and Diet subscale scores were 8.98 ± 2.00 at time 1 and 8.82 ± 1.95 at time 2 (Table 1). The Physical Activity and Diet subscale had good test-retest reliability (ICC = 0.86), with an MDC of 2.05. The Bland-Altman plot (FIGURE 1C) was not indicative of proportional bias (mean = 0.16, 95% CI, −1.88 to 2.20; t = 0.38; P = .705).

The MS Peer Support subscale scores could range from 2 to 8, with mean ± SD scores of 5.76 ± 1.98 at time 1 and 5.44 ± 2.03 at time 2 (Table 1). Good test-retest reliability was noted (ICC = 0.77), and the scale had an MDC of 2.66. The Bland-Altman plot (FIGURE 1D) was not indicative of proportional bias (mean = 0.32; 95% CI, −2.33 to 2.97; t = −0.30; P = .769).

With a score range of 5 to 20, mean ± SD Support From Family and Friends subscale scores were 15.92 ± 3.17 at time 1 and 15.56 ± 3.48 at time 2 (Table 1). The Support From Family and Friends subscale had good test-retest reliability (ICC = 0.87), with an MDC of 3.30. The Bland-Altman plot (FIGURE 1E) was not indicative of proportional bias (mean = 0.36; 95% CI, −2.93 to 3.65; t = −1.48; P = .145).

Spirituality subscale scores could range from 2 to 8, with mean ± SD scores of 6.05 ± 2.21 at time 1 and 6.02 ± 2.08 at time 2 (Table 1). Excellent test-retest reliability was noted (ICC = 0.93), with the scale having an MDC of 1.55. The Bland-Altman plot (FIGURE 1F) was not indicative of proportional bias (mean = 0.03; 95% CI, −1.52 to 1.58; t = 1.23; P = .223).

The MSRS demonstrated good test-retest reliability, both in terms of the overall measure and its individual subscales. The present results suggest that a change of 12 points or more on the MSRS total score is indicative of reliable change. This value can be used to identify individuals who have significant improvements (or declines) after a resilience training program. The MDCs for the subscales ranged from 2 to 9. Although clinicians or researchers may focus on change on the overall MSRS, focusing on an individual subscale’s change may be beneficial for targeted interventions. For example, if the goal is to increase the use of active coping strategies (vs emotion-focused coping), an increase of 9 points on the Emotional and Cognitive Strategies subscale may be more meaningful than a 12-point increase across the entire measure.

Although these findings strengthen the known psychometric properties and clinical utility of the MSRS, there are limitations to the present study. First, although the obtained sample size was adequate for the intended test-retest reliability analyses,¹³  the lower number of participants may have contributed to the reduced Cronbach α for the Physical Activity and Diet (α = 0.68) and Support From Family and Friends (α = 0.69) subscales at time 1. For reference, in larger cohorts, the internal consistency for these subscales has ranged from 0.77 to 0.84 for Physical Activity and Diet^6,10,23,24  and from 0.74 to 0.80 for Support From Family and Friends.^6,10,23,24  In addition, the alphas were greater than 0.70 at time 2, and the averaged alphas over time points always exceeded 0.70. Second, although nearly half of the participants completed time 2 exactly 14 days later, there were several outliers, with almost 18% (n = 11) responding to the second survey 3 weeks or more after the first. That said, these outliers do not seem to have a significant effect on the test-retest reliability scores of the MSRS. For example, the ICC for the total score decreased from 0.88 to 0.86 after removing these 11 individuals. For the subscales, the change in ICCs ranged from 0.007 (Spirituality) to −0.013 (MS Peer Support). Finally, data were not collected about any stressors, such as relapses, COVID-19, or other life events, that occurred between administrations of the MSRS and may have skewed responses and/or contributed to differences in responses between time 1 and time 2. Because all but the Emotional and Cognitive Strategies subscale had lower average scores at time 2, there is the possibility that participants experienced additional stressors within the 2-week period that could have lowered their scores at the second administration. Future research may consider validating the present findings in a second sample of persons with MS in a more controlled time frame and accounting for potential stressors that may affect scores.

Overall, the MSRS has good test-retest reliability, with a change of 12 points or more indicating reliable change. Each of the MSRS subscales had good to excellent test-retest reliability, with their MDCs ranging from 2 to 9. Combined with previous work on its psychometric properties,^6,10  these results support use of the MSRS as a potential outcome measure for MS-related resilience.

The authors thank Dotty Wakefield, MS, for coding and management of the REDCap data collection and Aprille Gangi, Carly Gillen, and Amelia Kroschwitz for their assistance with data verification. The PDDS scale is provided for use by the North American Research Committee on Multiple Sclerosis (NARCOMS) Registry (http://www.narcoms.org/pdds). NARCOMS is supported in part by the Consortium of Multiple Sclerosis Centers (CMSC) and the CMSC Foundation.