Evidence-Based Rehabilitation for Multiple Sclerosis Made Easy: The Online Applying Evidence with Confidence (APPECO) Platform

With the current rate of growth in evidence for multiple sclerosis (MS) rehabilitation, systematic reviews that focus on available evidence of therapeutic interventions typically lag behind the most currently published research findings. The lag between evidence and clinical practice can be attributed to the enormous number of studies on the one hand and the complexity of work regarding thorough summary and meta-analysis of individual studies on the other hand. Thus, when systematic reviews and clinical guidelines based on available evidence are published, they are, in fact, already outdated. In addition, an update may not follow within the first 5 years or may even not occur at all for a specific intervention. Subsequently, this leads to a delay in knowledge translation between the addition and evaluation of new evidence, the systematic review, and subsequent clinical guidelines.

To illustrate, Amatya and colleagues¹  recently (2019) published an overview of Cochrane systematic reviews in the field of rehabilitation for MS. One aim was to provide a “guiding tool for evidence-based decisions on appropriate management approaches in people with MS.”¹  Based on a search in December 2017—which in itself already illustrates part of the delay—15 systematic reviews were identified (reporting on 164 randomized clinical trials [RCTs]). The median underlying search date of these 15 Cochrane reviews, despite strong recommendations to update Cochrane reviews regularly,²  was 2011 (range, 2003³  to 2015⁴ ). Hence, this indicates a gap of 4 to 16 years between the primary evidence on which each Cochrane review was built and publication of the overview of Cochrane reviews by Amatya et al.¹  A simple PubMed search (February 2020) on indexed clinical trials with respect to MS (Mesh) and rehabilitation (Mesh) shows that during this 5-year gap, approximately 70 RCTs on MS rehabilitation were published (and >372 since the earliest Cochrane review from 2003). This raises the question as to how “current” this guiding tool for evidence-based decisions on appropriate treatment approaches in people with MS is, a limitation that was rightly stressed and discussed by Amatya and colleagues.¹  However, where does this “flaw in evidence-based medicine” leave the time-constrained allied health care professional willing to practice evidence-based medicine?⁵ 

The next challenge for the allied health professional is to determine whether newly available evidence is actually the “best” evidence (ie, unbiased study results, applicability to context). In addition to time and resources, lack of knowledge and inadequate skills are reported as key barriers to evidence-based medicine across medical fields.⁵ 

When the therapist or physician has identified the latest evidence, and deemed it to be of sufficient quality and applicable to their specific context or patient, the subsequent third important challenge is the ability to implement that best evidence into clinical practice. Without a comprehensive published description of interventions, clinicians and patients are unable to reliably implement interventions that have been shown to be useful.⁶  The introduction and use of the Template for Intervention Description and Replication (TIDieR) checklist has addressed this challenge.⁶  In addition, the PROGRESS-plus recommendation—an acronym used to identify characteristics (place of residence, race/ethnicity/culture/language, occupation, gender/sex, religion, education, socioeconomic status, and social capital) that stratify health opportunities and outcomes—may also address some of the concerns in terms of generalizability and applicability (also called external validity) of the studied intervention.⁷ 

In this article, we describe the Applying Evidence with Confidence (APPECO) project, its development, and concise methods in the field of MS rehabilitation. APPECO is an innovative, freely available online platform that facilitates 1) intuitive evaluation of the latest evidence for the nonpharmacologic management of impairments, activity limitations, and participation restrictions in patients with MS, 2) easier clinical decision making informed by three relevant quality indicators, and 3) hands-on and structured tools to implement that best evidence into daily clinical practice at the click of a button. The APPECO platform (www.appeco.net) will assist each MS rehabilitation professional to apply evidence with confidence.

For APPECO to address the described challenges, three specific phases were developed and are described herein: 1) identify the latest evidence; 2) data extraction, appraisal, and synthesis; and 3) knowledge translation. In Figure S1, which is published in the online version of this article at ijmsc.org, an overview is provided of the development process and the concise methods behind key steps leading to the information available on www.appeco.net.

The APPECO platform was developed under the auspices of the Rehabilitation in Multiple Sclerosis (RIMS) European Network for Best Practice and Research as a proof-of-concept, with the vision to revolutionize evidence-based medicine in MS rehabilitation and beyond; a continuously up-to-date (ie, living) synthesis of evidence to equip health care professionals with the tools needed to practice evidence-based medicine. A stakeholder meeting, representing RIMS members, industry, and academia, was organized during the European Committee for Treatment and Research in Multiple Sclerosis (ECTRIMS) conference in Barcelona, Spain, in 2015. The purpose of this meeting was to discuss a core set of requirements and functionalities for the APPECO platform for it to be able to address the key challenges in evidence-based MS rehabilitation. A software developer experienced in the development of medically oriented online applications (everywhereIM, Amsterdam, the Netherlands) was contracted. A multidisciplinary advisory board was composed of key experts in the field of MS rehabilitation through the RIMS special interest groups. This advisory board was engaged at strategic moments in the development process to provide valuable feedback as to the functionality, layout, and user-friendliness of APPECO. Through multiple conferences (eg, RIMS special interest group meetings), the broader MS community was actively engaged in the concept and development of APPECO. The proof-of-principle version of APPECO, with 142 unique RCTs describing the effects of 162 different rehabilitation interventions on fatigue and/or cognitive function in patients with MS, was launched during the RIMS annual meeting in May 2018 in Amsterdam. After this launch, funding was acquired to 1) extend the platform to domains of mobility, pain, and depression; 2) update the information on fatigue and cognitive function; and 3) improve functionalities of the platform for editor, reviewer, and users.

Together with an expert medical librarian of the Vrije Universiteit Medical Center, a literature search strategy was developed (Appendices S1–S3) to identify studies meeting the following criteria: 1) randomized clinical trial, 2) patients with MS, and 3) at least one of the experimental interventions could be considered a rehabilitation intervention.

Rehabilitation was defined as any intervention that has the aim to improve or maintain body function, activity, and/or participation and involves the interference of an allied health care professional in the context of a medical setting (eg, hospital, rehabilitation clinic, physiotherapy clinic). Regular e-mail alerts were set up for MEDLINE (PubMed), Embase, and Web of Science and sent directly to the APPECO platform. Subsequently, the platform automatically derives vital study information from each e-mail alert, including title, source of publication, authors, and abstract. Any potential duplicates with or serial publications of previously included records are earmarked for verification by the editorial team to ensure that each patient sample is included in the platform only once. Studies that are identified through other sources (eg, word of mouth, systematic reviews, reference lists) can be added manually by the editor if applicable.

The administrative back-end of the platform facilitates the management of a pool of independent reviewers. This is a multidisciplinary expert panel that is provided role-based access rights. A notification is sent out to two independent reviewers who are tasked to verify that each new study (based on title and abstract) meets the APPECO inclusion and exclusion criteria. Noninferiority and equivalence trials are presently excluded due to the lack of contrast between the experimental arms (eg, aerobic training versus resistance training). The editor will be notified to make a final decision if the two reviewers disagree. When the study is included, the editor will be automatically requested to allocate the full text of that specific study for data extraction.

A third reviewer (with a similar interest as the subject of the specific study) is requested to conduct the data extraction of this specific study, and a fourth reviewer is requested to verify the data extraction of the third (ie, the four-eyes principle). Panel S1 shows all the study information that is extracted.^6,8 

Specific features have been implemented to reduce extraction error and improve reviewer convenience. For example, the APPECO system “learns” which outcome measures have been extracted previously and will auto-complete to avoid spelling errors. Subsequently, the domain (eg, fatigue) that a specific outcome relates to, and the conventional “direction” of the measurement instrument (eg, higher outcome is better result) linked to that specific outcome are “remembered.” Reviewers can decline to undertake a specific task in case of conflict of interest to avoid bias (eg, review own article).

Although the APPECO platform initially has been developed as a proof-of-concept with respect to fatigue and cognitive function, during the data extraction process, all other study outcomes were also extracted. Outcomes are structured according to the International Classification of Functioning model (Figure S2).

The study quality is assessed (reviewer 3) and verified (reviewer 4) using the Cochrane Risk of Bias tool (see Panel S1 for the seven items).⁸  Each item is appraised as having low, high, or unclear risk of bias. A study is considered to be of high methodological quality if all items, except blinding of participants and personnel (considered unfeasible in rehabilitation studies), are appraised as low risk.

Health care professionals are recommended to follow a stepwise approach in using APPECO to reach an informed clinical decision based on the scientific evidence (Table 1).

Figure S3 shows how results are presented: intervention main type (eg, cognitive training), intervention subtype (eg, attention/information processing speed training), and study level (eg, Charvet et al). A color scheme is used to quickly identify a significant positive result (green), a significant negative result (red), or a nonsignificant effect (gray). For each study, a maximum of two “effects” are plotted, the first to indicate the immediate postintervention effect and the second the “long-term” effect. These effects are plotted along the x-axis based on the duration of the therapy in weeks. Detail on the actual size of the effect can be obtained by hovering the pointer over the respective diamond at the aggregated main type of intervention level, at the intervention subtype level, and/or at the study level.

Each study is assigned a maximum of three stars, which can direct the health care professional to determine the best evidence. Star 1: This star is provided to studies that have a high study quality (ie, low risk of bias on all items except blinding of participants/personnel). Star 2: This star is provided when the primary outcome of the study matches the user-selected patient symptom domain. For example, a star is allocated if the user selects the domain “fatigue” and the primary outcome of the specific study was “fatigue.” Star 3: This star is provided if the patients in that specific study were included based on having a predefined level of severity with respect to the symptom or activity limitation. For example, a star is allocated if the user selects the domain “fatigue” and patients in the specific study had a predefined level of fatigue.

Stars 2 and 3 are flexible in a way that if the user selects a different patient symptom or activity limitation of interest, the stars allocated to the specific study will change as well. At the overarching intervention levels, an average appraisal is visualized by “filling” the star based on the percentage of underlying studies reaching that criterion.

The APPECO platform has included numerous technical as well as hands-on tools to assist with implementation of the best evidence of MS rehabilitation into clinical practice. First, APPECO has been developed for the tablet, laptop, and desktop. Second, each intervention has been extracted using the TIDieR checklist.⁶  A PDF print of this checklist can be obtained by the click of a button. Third, an APPECO summary of findings is automatically generated for each study, indicating the type of intervention studied, study quality, positive and negative effects found, outcomes for which no effect was found, and the type of study participants (ie, external validity). A PDF print of this summary can be obtained. Fourth, additional information that has been published open access with the original publication, that assists in implementation, is made available as attachments. Fifth, the APPECO study team proactively reaches out to corresponding authors of studies identified as being of high quality to provide hands-on tools that may assist implementation of their treatment protocols. An e-mail is sent from the APPECO platform providing the author access to a specific environment that will allow the author to 1) complement the data extraction and fill in the missing pieces, 2) notify the editor of errors in the data extraction if applicable, and 3) upload key documents and materials (.pdf, .docx, .ppt, .mp4, and others) that are valuable in the knowledge translation process. This can, for example, be a treatment guideline used during the conduct of the study, a recorded PowerPoint presentation, instructional video material for specific exercises, infographics, among others. To prevent bias, the quality of the materials and the change requests of study authors will be reviewed and verified by the editor before being made available.

The APPECO platform currently hosts detailed information from 250 RCTs, 293 interventions, and 1250 effect sizes on 53 patient complaints (as of February 2020). A total of 152 of the 293 interventions (52%) report on fatigue in patients with MS, 111 (38%) on mobility, 106 (36%) on depression, 58 (20%) on attention and information processing speed, 38 (13%) on memory and learning, 29 (10%) on executive functions, 12 (4%) on perceived cognitive problems, and 12 (4%) on other cognitive functions. In the process of extracting data on fatigue, cognition, mobility, pain, and depression, other domains were also included on the platform, of which quality of life (n = 108 interventions), balance (n = 51), and anxiety (n = 48) were most frequent (Table 2).

There are few studies with high quality available that were specifically designed to test the benefits of MS rehabilitation on fatigue (n = 6 [2.4%]),^9–14  cognitive function (n = 4 [1.6%]),^15–18  and depression (n = 2 [0.8%]).^14,19  No high-quality studies were included for pain and mobility. An appraisal of studies with a low risk of bias (star 1) was given in studies that included fatigue (n = 23 studies [9.2%]), attention and information processing speed (n = 8 studies [3.2%]), memory and learning (n = 8 studies [3.2%]), executive functions (n = 4 studies [1.6%]), perceived cognitive problems (n = 4 studies [1.6%]), and other cognitive functions (n = 3 studies [1.2%]).

The initial user statistics available in the first 2.5 years of being online (May 2017 through February 2020) indicate 2300 unique users who visited a total of 4200 sessions for, on average, 7.56 minutes. Of the users, 55.6% originated from sites not directly related to the study team developing the platform.

This paper describes the development, methods, and current content of a living documentation system to address key challenges in evidence-based rehabilitation for patients with MS, including 1) the time gap between evidence generation and uptake in guidelines and systematic reviews, 2) limited knowledge and resources for health care providers to appraise the quality and applicability of scientific research to their setting and/or patient, and 3) lack of hands-on tools that assist the implementation of research into clinical practice.

Under the assumption of a fully up-to-date platform with respect to MS rehabilitation, there are a couple of additional strengths that the APPECO platform may provide. First, in contrast to systematic reviews, which most often tend to be specific in terms of intervention and patient outcome, the APPECO platform provides an overview of multiple types of interventions compared with control interventions, and for a large number of patient outcomes. This “side-by-side” information could be used by clinicians to direct the patient to the most appropriate health care provider for the treatment most likely to lead to significant benefits. Second, information with respect to the duration and frequency of the intervention, and patient preferences, can be taken into account in making an informed, evidence-based, yet individualized decision with respect to the most appropriate and feasible treatment strategy. Contextualizing the evidence may increase adherence and uptake of evidence for rehabilitation.²⁰ 

The actuality and sustainability of APPECO in the provision of evidence-based MS rehabilitation depends on the platform staying up to date and the inclusiveness of all domains relevant to the rehabilitation of patients with MS. These two aspects (up to date, comprehensiveness) pose challenges, as with many conventional guidelines. The 250 RCTs currently included represent a significant, yet incomplete portion of the RCTs conducted in this field. It is going to require a substantial resource impulse (eg, time).

A method of assigning reviewers based on their special interest (in line with the RIMS special interest groups) has already been built in. Assignment of reviewers with a special interest to process new evidence that meets that special interest can be considered a viable strategy to reduce the perceived burden of maintaining APPECO by the academic community. A future perspective could be that APPECO is embraced by publishers and study authors themselves to provide detailed information at the time of publishing their study results. Obviously, this would then need to be verified by independent reviewers against the publication to ensure academic rigor. The process of filling the platform (≈2–3 hours per study for data extraction or verification) was conducted by a select group of researchers on the study team to address the sheer number of studies that needed to be included.

Until now, the study team has been conservative in the request for these materials, and only additional material for selected high-quality studies is currently available. Although knowledge translation in conventional guideline development is a deliberate process, the APPECO platform largely relies on the input from study authors. The level to which additional information is available that can really assist in implementing evidence into clinical practice remains to be evaluated and poses a second challenge for the APPECO platform. This can be further complicated by the potential multilingual nature and cross-cultural differences of implementation tools. The latter may require resources for translation of materials or limit the external validity for some of the studies. However, the online nature of the environment offers ample opportunity to integrate educational components and directives (eg, animations, instructional videos, a how-to-use guide) within APPECO that can assist the user in getting to the right information.

What if there is no (best) evidence for a specific intervention, patient symptom, or activity limitation? The methodological quality of the studies included was limited (star 1), and in the context of fatigue, cognitive function, depression, pain, and mobility, interventions were rarely designed to specifically and primarily address these domains as shown by star 2 and inclusion criteria (star 3) in these studies. The lack of (quality) evidence for some interventions albeit often used in clinical practice, therefore, poses an important third limitation. In conventional guideline development, in the absence of evidence, one formulates consensus-based recommendations based on extensive stakeholder and expert engagement. Including expert recommendations in the absence of evidence in APPECO is a step that needs to be carefully considered. Expert-based recommendations may be very context specific and may be biased or politically driven.

The APPECO platform aims to substantially improve the way knowledge is translated into clinical practice and to enable health care professionals to apply evidence with confidence in MS rehabilitation. Its potential depends on the ability to keep APPECO up to date, and the availability of practical materials that assist with implementation of effective treatments.

The authors acknowledge the multidisciplinary advisory board that provided invaluable input throughout the course of APPECO development.

The authors declare no conflicts of interest.

The development of APPECO has been financially supported by Sanofi Genzyme (phase 1; through RIMS). The Dutch MS Research Foundation (PP 18-001) and Roche financially supported the maintenance of the platform and data extraction of MS rehabilitation studies related to depression, pain, and mobility.