BACKGROUND

Assessing motor functioning is important to monitor the disease course of multiple sclerosis (MS). Video-assisted rating of classic neurologic tests and activities of daily living may improve the detection of changes in motor functioning. We investigated the value of using video-assisted composite measures for the detection of changes in mobility and upper extremity function (UEF).

METHODS

Forty-three patients with MS were recorded performing motor function tests before and during treatment with fampridine. Patients were classified as improved or not improved on mobility composite (MOB-COM) and UEF composite (UEF-COM) measures based on neurologists’ ratings of the tests. The proportional agreements between the composite measures and the conventional measures—the Timed 25-Foot Walk test (T25FW) and the Nine-Hole Peg Test (NHPT)—were determined and compared with patient-perceived improvement, which was determined using patient-reported ratings of changes in mobility and UEF.

RESULTS

Agreement between MOB-COM and T25FW was 79.5%, and agreement between UEF-COM and NHPT was 82.1%. Twenty-six of 39 patients (66.7%) reported mobility improvement; 6 of these reports were confirmed by both T25FW and MOB-COM, 4 were confirmed by T25FW only, and 2 were confirmed by MOB-COM only. For UEF, 13 of 39 patients (33.3%) reported improvement; 3 of these were confirmed by the NHPT and 3 were confirmed by the UEF-COM.

CONCLUSIONS

Compared with the conventional NHPT measure, the video-assisted composite measure of UEF detected additional patient-perceived improvement. This was less evident for mobility measures. Video-assisted composite measures may enhance the detection of treatment effects in MS clinical practice and trials.

Clinical assessment of patients with multiple sclerosis (MS) is notoriously difficult largely due to the heterogeneous clinical symptoms.1  With the availability of increasing numbers of effective disease-modifying therapies and expanding treatment targets, selecting appropriate outcome measures is increasingly important.2  For example, in a large phase 3 trial of patients with secondary progressive MS, the effect of natalizumab on progression was assessed using various measures. Although no treatment effect was found on Expanded Disability Status Scale (EDSS) score progression, a reduction in progression of upper extremity function (UEF) impairment was detected.3  The accurate assessment of disability is also relevant in the evaluation of symptomatic treatment options such as fampridine (also called dalfampridine).4  Fampridine improves walking speed and UEF in a subset of patients with MS, albeit patient-perceived improvement does not fully correspond with improvement measured using clinical instruments.57 

The EDSS is the most widely used outcome measure in MS trials, although it has several limitations (eg, high interrater and intrarater variability, disproportional impact of ambulation on the total score).8,9  To improve the assessment of commonly affected functional domains in MS, the Multiple Sclerosis Functional Composite is used to assess ambulation (Timed 25-Foot Walk test [T25FW]), UEF (Nine-Hole Peg Test [NHPT]), and cognition.10,11  Despite generally good psychometric properties, the individual components have several shortcomings.12,13  The T25FW and the NHPT measure only certain aspects of ambulation and UEF (walking speed and manual dexterity, respectively), which do not fully capture the broader aspects of functioning. Also, the T25FW may not be sensitive enough to detect abnormalities in patients with mild ambulatory impairment.14  Because of these shortcomings, clinically relevant treatment effects might be missed.

Video-assisted assessment of motor functioning has the potential to improve disability evaluation. Recording patients’ movements on video may improve certain psychometric properties, such as intrarater and interrater variability, and promotes data quality by allowing a visual check on compliance relative to the standardization of performance.15,16  Also, standardized activities of daily living (ADLs) can be added to the assessment and are valuable in measuring patients’ functioning.17 

We investigated the additional value of video-assisted composite measures, consisting of classic neurologic tests and ADLs, in relation to conventional measures and aimed to enhance the detection of changes in mobility and UEF in patients with MS.

Written informed consent was obtained from all patients before study participation, and the study was approved by the respective ethics committees. Data used in this study were part of a multicenter project to develop the ASSESS MS system, performed in 4 large European MS centers located in Amsterdam, the Netherlands, and in Basel, Bern, and Lucerne, Switzerland.18,19  Patients with MS were recorded using a three-dimensional depth-sensing and color camera (Kinect, Microsoft Corp) with the aim of training machine learning algorithms to automatically quantify motor functioning.

Participant Recruitment

Patients were recruited at the Amsterdam University Medical Center, Vrije Universiteit Medical Center between August 1, 2017, and April 30, 2018. Patients initiating treatment with fampridine were eligible for study participation because fampridine can have positive treatment effects on mobility and UEF.57  Inclusion criteria conformed to the official treatment label of fampridine, a diagnosis of MS according to the revised 2010 McDonald criteria,20  age older than 18 years, ambulatory impairment defined as an estimated EDSS score between 4.0 and 7.0, and no additional diseases that contributed to disability. Contraindications as stated in the fampridine product label were handled as exclusion criteria.21  Patients were assessed at baseline before fampridine treatment and at follow-up after a minimum of 2 weeks of fampridine treatment. Patient and disease characteristics were collected during the baseline visit: age, sex, EDSS score,8  disease type, and disease duration.

Conventional Measures

The T25FW was used to measure mobility, and the NHPT was used to measure UEF. Each test was performed twice during each visit, as implemented in the Multiple Sclerosis Functional Composite.10  For each visit, 2 trials were averaged for each patient. A decrease of at least 20% in baseline walking time was used to indicate significant improvement in mobility function.13,22,23  For the NHPT, trial results of the dominant and nondominant hands were averaged. Similar to the T25FW, a decrease in completion time of at least 20% between visits was used as the determinant of significant UEF improvement.22 

Video-Assisted Composite Measures

In addition to the conventional measures, the mobility composite (MOB-COM) consisted of 3 traditional neurologic tests and 4 ADLs (FIGURE 1A). The UEF composite (UEF-COM) consisted of 3 classic neurologic tests and 2 ADLs. The MOB-COM and UEF-COM movements were recorded using the Microsoft Kinect camera (FIGURE 1B). Each color video was rated by 2 neurologists (of C.E.P.vM., M.D.S., S.S.), who were blinded to the visit type (ie, before or during fampridine treatment) and to the patients’ performance on the conventional measures. The classic neurologic tests were quantified according to predetermined rating scales based on the Neurostatus EDSS functional system scoring definitions.24  A 5-point Likert scale was created ranging from 0 (normal) to 4 (unable to perform due to disability) for the ADLs. Videos were rated using reference videos.16  A reference video displays a person performing a test according to the predefined scale, the gold standard. A consensus score was calculated based on the video ratings of the 2 neurologists using an algorithm similar to that described by Sarkar et al,25  which takes into account individual rater bias. These scores were used in the statistical analyses. Videos of insufficient quality or inaccurately performed movements were excluded from the analysis. A more comprehensive description of the standardized movements, video rating, and score calculation can be found elsewhere.16,17  During follow-up visits, significant improvement on the MOB-COM and the UEFCOM was defined as improvement on 2 or more tests (ie, ≥1-point decrease) and a decline on no more than 1 test (ie, ≥1-point increase) compared with baseline.

FIGURE 1.

Video-Assisted Composite Measures.

FIGURE 1.

Video-Assisted Composite Measures.

Close modal

Clinically Relevant Improvement

Clinically relevant improvement was determined with the Global Rating of Change (GRC) scales.26  At the follow-up visit, patients rated their perceived change in mobility and in UEF. Patients with positive GRC scales scores were classified as having clinically relevant improvement, whereas negative GRC scales scores or scores of 0 were classified as no clinically relevant improvement.

Statistical Analysis

All statistical analyses were performed using IBM SPSS Statistics for Windows, version 24.0 (IBM Corp). Categorical variables are summarized as frequencies with percentages. The mean and SD are used to summarize continuous variables that were normally distributed; the median and interquartile range are used to summarize irregular distributions. The proportional agreement between the video-assisted composite and conventional measures was determined using 2×2 contingency tables for mobility (T25FW and MOB-COM) and UEF (NHPT and UEFCOM). The added value of the composite measures was determined and visualized using Venn diagrams showing the overlap in patients who improved on the various outcomes for mobility and UEF.

A total of 43 patients with MS initiating fampridine treatment were included and completed baseline measurements. Two patients discontinued treatment prematurely due to adverse effects, and 2 patients refused follow-up assessment because of time constraints. The baseline characteristics of the 39 patients who completed the study are summarized in TABLE S1, which is published in the online version of this article at IJMSC.org. No adverse events occurred while performing the tests.

FIGURE S1 shows the proportions of patients with significant mobility and UEF improvement, as assessed using the GRC scales, conventional measures, and video-assisted composite measures. The proportions of patients who improved on the classic neurologic tests and ADLs were similar to patients with improved MOB-COM (both 50.0%) and UEF-COM (54.5% and 45.5%, respectively).

Comparison of Conventional and Video-Assisted Composite Measures

TABLE S2 shows the distribution of improved mobility on the conventional and video-assisted composite measures. Three patients (7.7%) improved on the MOB-COM without improvement on the T25FW. Conversely, 5 patients (12.8%) improved on the T25FW without improvement on the MOB-COM. For the remaining patients (31 [79.5%]), the MOB-COM and T25FW outcomes were congruent.

Results of the UEF-COM and the NHPT, shown in TABLE S3, are congruent for 32 patients (82.1%). Four patients (10.3%) improved on the UEF-COM with no improvement on the NHPT. Conversely, 3 patients (7.7%) did not improve on the UEF-COM but improved on the NHPT.

Detection of Clinically Relevant Improvement

The overlap between improvement on the GRC scale and the conventional and video-assisted composite measures is depicted in FIGURE 2. In total, 26 of 39 patients (66.7%) reported improvement in mobility on the GRC scales. Of these patients, the number also showing improvement on the T25FW and the MOB-COM was similar (10 and 8 patients, respectively) and largely overlapped (6 of 12 patients). Concerning UEF, 13 patients (33.3%) reported improvement on the GRC scales. Although no overlap could be detected between these patients and those showing improvement on the NHPT (n = 3), an improved UEF-COM largely concurred with GRC scales improvement (n = 3 of 4).

FIGURE 2.

Venn Diagrams Depicting the Overlap Between Patient-Perceived Improvement and Detected Improvement.

FIGURE 2.

Venn Diagrams Depicting the Overlap Between Patient-Perceived Improvement and Detected Improvement.

Close modal

In this study we investigated the value of adding 2 video-assisted composite measures to conventional measures to improve detection of change in motor functioning. We found that the composite measures detected additional clinical improvement, as perceived relevant by the patient. The additional clinical value was most evident for UEF assessment.

The present findings suggest the importance of multidimensional assessment to detect changes in motor functioning. The composite measures used herein consisted of several classic neurologic tests and ADLs. In contrast, the conventional T25FW primarily tests walking speed which results in both ceiling and floor effects in patients who are more or less severely disabled, respectively.13,14  Similarly, a floor effect has been reported for the NHPT.27  In the present cohort, a ceiling effect is unlikely to have occurred because patients had relatively mild disabilities (highest EDSS score of 6.5).

The potential value of composite measures was most clearly present for UEF, as some overlap existed between the UEF-COM and the GRC scales. The inclusion of several ADLs in the composite measures might have contributed to this finding. In the present cohort, approximately 50% of all tests that contributed to improvement of the composite measure were ADLs. A previous study with similar methods also found that ADLs may be more valuable than classic neurologic tests.17  The NHPT and the T25FW also correlate to the ability to perform ADLs.12,28  However, these correlations are based on patient-reported outcome measures (ie, questionnaires), which differ from the performance tests used in the present study.

Video-assisted assessment of motor functioning has the potential to improve disability evaluation in several ways. By using reference videos,29  it may improve psychometric properties of outcome measures, such as intraobserver and interobserver variability. Digital video data could be used to build machine learning algorithms that automatically quantify motor performance. Video-assisted assessment could also shorten administration times and associated administrative burdens relative to other assessments, particularly the EDSS, which takes up to 30 minutes to obtain, including administration time. The tasks that made up the composite scores were shown on a screen integrated into the system and took 10 to 20 minutes to perform, depending on the level of disability. Ultimately, once determined safe for the patient, automated, video-assisted disability assessment has the potential to be performed by patients themselves at their preferred location and time instead of in a hospital or research clinic. To achieve this, machine learning algorithms of sufficient quality need to be developed.

This study has several limitations. The first is the small sample size. Another is, for the assessment of clinically relevant change, that potential placebo effects are not accounted for because no control group was available. Nevertheless, the goal of this study was not to assess the efficacy of fampridine but to detect change in performance. The present findings also provide limited insight into patients in the lower and upper ranges of EDSS scores because disability in this cohort of patients ranged from 3.5 to 6.5.

Another limitation is that the clinometric properties of the GRC scales scores as an assessment tool for clinically significant change are less established than those of the T25FW and the NPHT. Therefore, despite its use to determine relevant change from the clinicians’ or patients’ perspective, it is unclear whether the improvement of mobility on the GRC scale and the UEF truly reflects (subjective) improvement at the group level. Nonetheless, we believe that this 1-item patient-reported outcome stands more closely to the perceived treatment effect on the individual patient level.

Last, potential learning effects in performing the video-assisted composite at follow-up were not studied, although we assume that this is not a major bias because the composite measures were assessed once each visit with an interval of at least 2 weeks. Further improvement of the composite measures to detect changes in motor functioning might be achieved by adding ADLs with various difficulty levels to decrease floor effects or increase ceiling effects.

In this study, we demonstrated that the use of video-assisted composite measures may enhance the detection of motor functioning changes in patients with MS. This was most evident for the assessment of UEF. Using video-assisted composite measures as outcome measures in MS clinical practice and trials may enhance the detection of treatment effects, but further exploration is needed in a larger cohort of patients.

PRACTICE POINTS
  • » The clinical disease course of multiple sclerosis is heterogeneous, and, therefore, the assessment of treatment and rehabilitation effects is challenging. Adequate assessment of motor functioning is a crucial facet to monitor disease course.

  • » Video-assisted composite measures detected additional change in motor functioning. This is more evident for the assessment of upper extremity functioning than for mobility. Video-assisted composite measures may enhance detection of treatment and rehabilitation effects.

1.
Scalfari
A,
Neuhaus
A,
Daumer
M,
Deluca
GC,
Muraro
PA,
Ebers
GC.
Early relapses, onset of progression, and late outcome in multiple sclerosis
.
JAMA Neurol.
2013
;
70
:
214
222
.
doi:
2.
Faissner
S,
Plemel
JR,
Gold
R,
Yong
VW.
Progressive multiple sclerosis: from pathophysiology to therapeutic strategies
.
Nat Rev Drug Discov.
2019
;
18
(
12
):
905
922
.
doi:
3.
Kapoor
R,
Ho
PR,
Campbell
N,
et al.
Effect of natalizumab on disease progression in secondary progressive multiple sclerosis (ASCEND): a phase 3, randomised, double-blind, placebo-controlled trial with an open-label extension
.
Lancet Neurol.
2018
;
17
(
5
):
405
415
.
doi:
4.
Hayes
KC.
Impact of extended-release dalfampridine on walking ability in patients with multiple sclerosis
.
Neuropsychiatr Dis Treat.
2011
;
7
:
229
239
.
doi:
5.
Goodman
AD,
Brown
TR,
Schapiro
RT,
Klingler
M,
Cohen
R,
Blight
AR.
A pooled analysis of two phase 3 clinical trials of dalfampridine in patients with multiple sclerosis
.
Int J MS Care.
2014
;
16
(
3
):
153
160
.
doi:
6.
Savin
Z,
Lejbkowicz
I,
Glass-Marmor
L,
Lavi
I,
Rosenblum
S,
Miller
A.
Effect of Fampridine-PR (prolonged released 4-aminopyridine) on the manual functions of patients with Multiple Sclerosis
.
J Neurol Sci.
2016
;
360
:
102
109
.
doi:
7.
Klineova
S,
Farber
R,
Friedman
J,
Farrell
C,
Lublin
FD,
Krieger
S.
Objective and subjective measures of dalfampridine efficacy in clinical practice
.
Mult Scler J Exp Transl Clin.
2018
;
4
(
3
):
2055217318786742.
doi:
8.
Kurtzke
JF.
Rating neurologic impairment in multiple sclerosis: an expanded disability status scale (EDSS)
.
Neurology.
1983
;
33
(
11
):
1444
1452
.
doi:
9.
van Munster
CE,
Uitdehaag
BM.
Outcome measures in clinical trials for multiple sclerosis
.
CNS Drugs.
2017
;
31
(
3
):
217
236
.
doi:
10.
Cutter
GR,
Baier
ML,
Rudick
RA,
et al.
Development of a multiple sclerosis functional composite as a clinical trial outcome measure
.
Brain.
1999
;
122
(
pt 5
):
871
882
.
doi:
11.
Meyer-Moock
S,
Feng
YS,
Maeurer
M,
Dippel
FW,
Kohlmann
T.
Systematic literature review and validity evaluation of the Expanded Disability Status Scale (EDSS) and the Multiple Sclerosis Functional Composite (MSFC) in patients with multiple sclerosis
.
BMC Neurol.
2014
;
14
:
58
.
doi:
12.
Feys
P,
Lamers
I,
Francis
G,
et al.
The Nine-Hole Peg Test as a manual dexterity performance measure for multiple sclerosis
.
Mult Scler.
2017
;
23
(
5
):
711
720
.
doi:
13.
Motl
RW,
Cohen
JA,
Benedict
R,
et al.
Validity of the timed 25-foot walk as an ambulatory performance outcome measure for multiple sclerosis
.
Mult Scler.
2017
;
23
(
5
):
704
710
.
doi:
14.
Goldman
MD,
Motl
RW,
Scagnelli
J,
Pula
JH,
Sosnoff
JJ,
Cadavid
D.
Clinically meaningful performance benchmarks in MS: timed 25-foot walk and the real world
.
Neurology.
2013
;
81
(
21
):
1856
1863
.
doi:
15.
Steinheimer
S,
Dorn
JF,
Morrison
C,
et al.
Setwise comparison: efficient fine-grained rating of movement videos using algorithmic support: a proof of concept study
.
Disabil Rehabil.
2020
;
42
(
18
):
2640
2646
.
doi:
.
16.
D’Souza
M,
Steinheimer
S,
Dorn
J,
et al.
Reference videos reduce variability of motor dysfunction assessments in multiple sclerosis
.
Mult Scler J Exp Transl Clin.
2018
;
4
(
3
):
2055217318792399.
doi:
17.
van Munster
CE,
D’Souza
M,
Steinheimer
S,
et al.
Tasks of activities of daily living (ADL) are more valuable than the classical neurological examination to assess upper extremity function and mobility in multiple sclerosis
.
Mult Scler.
2019
;
25
(
12
):
1673
1681
.
doi:
18.
Morrison
C,
D’Souza
M,
Huckvale
K,
et al.
Usability and acceptability of ASSESS MS: assessment of motor dysfunction in multiple sclerosis using depth-sensing computer vision
.
JMIR Hum Factors.
2015
;
2
(
1
):
e11
.
doi:
19.
Kontschieder
P,
Dorn
JF,
Morrison
C,
et al.
Quantifying progression of multiple sclerosis via classification of depth videos
.
Med Image Comput Comput Assist Interv.
2014
;
17
(
pt 2
):
429
437
.
doi:
20.
Polman
CH,
Reingold
SC,
Banwell
B,
et al.
Diagnostic criteria for multiple sclerosis: 2010 revisions to the McDonald criteria
.
Ann Neurol.
2011
;
69
(
2
):
292
302
.
doi:
21.
Fampyra
.
European Medicines Agency.
22.
van Winsen
LM,
Kragt
JJ,
Hoogervorst
EL,
Polman
CH,
Uitdehaag
BM.
Outcome measurement in multiple sclerosis: detection of clinically relevant improvement
.
Mult Scler.
2010
;
16
(
5
):
604
610
.
doi:
23.
Hobart
J,
Blight
AR,
Goodman
A,
Lynn
F,
Putzki
N.
Timed 25-foot walk: direct evidence that improving 20% or greater is clinically meaningful in MS
.
Neurology.
2013
;
80
(
16
):
1509
1517
.
doi:
24.
Kappos
L.
Neurostatus scoring definitions (Version 04/10.2)
.
25.
Sarkar
A,
Morrison
C,
Dorn
JF,
et al.
Setwise comparison: consistent, scalable, continuum labels for computer vision
.
In:
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
.
San Jose, CA
:
Association for Computing Machinery
;
2016
:
261
271
.
26.
Jaeschke
R,
Singer
J,
Guyatt
GH.
Measurement of health status: ascertaining the minimal clinically important difference
.
Control Clin Trials.
1989
;
10
(
4
):
407
415
.
doi:
27.
Solaro
C,
Cattaneo
D,
Brichetto
G,
et al.
Clinical correlates of 9-hole peg test in a large population of people with multiple sclerosis
.
Mult Scler Relat Disord.
2019
;
30
:
1
8
.
doi:
28.
Kempen
JC,
de Groot
V,
Knol
DL,
Polman
CH,
Lankhorst
GJ,
Beckerman
H.
Community walking can be assessed using a 10-metre timed walk test
.
Mult Scler.
2011
;
17
(
8
):
980
990
.
doi:
29.
Burggraaff
J,
Dorn
J,
D’Souza
M,
et al.
Video-based pairwise comparison: enabling the development of automated rating of motor dysfunction in multiple sclerosis
.
Arch Phys Med Rehabil.
2020
;
101
(
2
):
234
241
.
doi:

FINANCIAL DISCLOSURES: Dr van Munster has received travel support from Novartis Pharma AG, Sanofi Genzyme, and Teva Pharmaceuticals and honoraria for lecturing and consulting from Biogen Idec and Merck Serono. Dr D’Souza has received travel support from Bayer AG, Teva, and Genzyme and research support from the University Hospital Basel. Dr Steinheimer has received travel support from Bayer and Merck and honoraria for consulting from Bayer, Merck, Roche, and Teva. Dr Kamm has received honoraria for lectures as well as research support from Biogen, Novartis, Almirall, Bayer Schweiz AG, Teva, Lilly, Merck, Genzyme, Roche, and the Swiss MS Society. Dr Burggraaff has received travel support from Novartis Pharma AG. Messrs Johnson and Zaykov are employees of Microsoft Research. Messrs Dorn and Dahlke are employees of Novartis Pharma AG. Dr Kappos’ institution, University Hospital Basel, has received funding for research support during the past 3 years from the Department of Neurology steering committee and advisory board; consultancy fees from Actelion, Alkermes, Almirall, Bayer, Biogen, df-mp, Excemed, GeNeuro SA, Genzyme, Merck, Minoryx, Mitsubishi Pharma, Novartis, Receptos, Roche, Sanofi-Aventis, Santhera, Teva, and Vianex; and royalties from Neurostatus. The hospital also received funding for departmental educational activities and honoraria from Allergan, Almirall, Bayer, Biogen, Excemed, Genzyme, Merck, Novartis, Pfizer, Sanofi-Aventis, Teva, and UCB. Dr Killestein has received speaker and consultancy fees from Merck, Biogen, Teva, Genzyme, Roche, and Novartis. Dr Uitdehaag has received consultancy fees from Biogen Idec, Genzyme, Merck Serono, Novartis, Roche, and Teva. Dr Lam has disclosed no relevant financial relationships.

FUNDING/SUPPORT: This research was funded by Novartis Pharma AG.

PRIOR PRESENTATION: Data from this study were presented as a poster presentation at the 34th Congress of the European Committee for Treatment and Research in Multiple Sclerosis; October 10-12, 2018; Berlin, Germany.

Author notes

*CEPvM and K-HL contributed equally to this paper.

Note: Supplementary material for this article is available at IJMSC.org.

Supplementary Material