Abstract

With the current rate in growth of evidence in multiple sclerosis (MS) rehabilitation, systematic reviews and clinical guidelines typically lag behind the most currently published research findings. In addition, most health care professionals lack the time, resources, or adequate skills to identify and evaluate new evidence, and hands-on tools to implement the latest evidence into clinical practice are often lacking or not readily available. The Applying Evidence with Confidence (APPECO) platform is a newly developed online tool that addresses these key challenges. APPECO was initiated as a proof-of-concept for the common MS symptoms fatigue and cognitive function. Subsequently, intervention studies about depression, pain, and mobility were added. APPECO currently hosts detailed information from 250 randomized clinical trials, 293 interventions, and 1250 effect sizes on 53 patient outcomes. Few studies with high quality (ie, low risk of bias) are available specifically designed to test the benefits of MS rehabilitation on fatigue (n = 5 [2.4%]), cognitive function (n = 4 [1.6%]), and depression (n = 2 [0.8%]). No high-quality studies were included for pain and mobility. APPECO has the potential to address key challenges in evidence-based rehabilitation medicine for MS and to facilitate swift knowledge translation from evidence into clinical practice. Sustainability of APPECO depends on a continuous resource impulse (eg, financial, time, editorial management, platform maintenance) to ensure up-to-date information across all relevant MS symptoms and activity limitations (≈2–6 randomized controlled trials per month). Ways to optimize knowledge translation in the absence of high-quality evidence in APPECO need to be explored further.

With the current rate of growth in evidence for multiple sclerosis (MS) rehabilitation, systematic reviews that focus on available evidence of therapeutic interventions typically lag behind the most currently published research findings. The lag between evidence and clinical practice can be attributed to the enormous number of studies on the one hand and the complexity of work regarding thorough summary and meta-analysis of individual studies on the other hand. Thus, when systematic reviews and clinical guidelines based on available evidence are published, they are, in fact, already outdated. In addition, an update may not follow within the first 5 years or may even not occur at all for a specific intervention. Subsequently, this leads to a delay in knowledge translation between the addition and evaluation of new evidence, the systematic review, and subsequent clinical guidelines.

To illustrate, Amatya and colleagues1  recently (2019) published an overview of Cochrane systematic reviews in the field of rehabilitation for MS. One aim was to provide a “guiding tool for evidence-based decisions on appropriate management approaches in people with MS.”1  Based on a search in December 2017—which in itself already illustrates part of the delay—15 systematic reviews were identified (reporting on 164 randomized clinical trials [RCTs]). The median underlying search date of these 15 Cochrane reviews, despite strong recommendations to update Cochrane reviews regularly,2  was 2011 (range, 20033  to 20154 ). Hence, this indicates a gap of 4 to 16 years between the primary evidence on which each Cochrane review was built and publication of the overview of Cochrane reviews by Amatya et al.1  A simple PubMed search (February 2020) on indexed clinical trials with respect to MS (Mesh) and rehabilitation (Mesh) shows that during this 5-year gap, approximately 70 RCTs on MS rehabilitation were published (and >372 since the earliest Cochrane review from 2003). This raises the question as to how “current” this guiding tool for evidence-based decisions on appropriate treatment approaches in people with MS is, a limitation that was rightly stressed and discussed by Amatya and colleagues.1  However, where does this “flaw in evidence-based medicine” leave the time-constrained allied health care professional willing to practice evidence-based medicine?5 

The next challenge for the allied health professional is to determine whether newly available evidence is actually the “best” evidence (ie, unbiased study results, applicability to context). In addition to time and resources, lack of knowledge and inadequate skills are reported as key barriers to evidence-based medicine across medical fields.5 

When the therapist or physician has identified the latest evidence, and deemed it to be of sufficient quality and applicable to their specific context or patient, the subsequent third important challenge is the ability to implement that best evidence into clinical practice. Without a comprehensive published description of interventions, clinicians and patients are unable to reliably implement interventions that have been shown to be useful.6  The introduction and use of the Template for Intervention Description and Replication (TIDieR) checklist has addressed this challenge.6  In addition, the PROGRESS-plus recommendation—an acronym used to identify characteristics (place of residence, race/ethnicity/culture/language, occupation, gender/sex, religion, education, socioeconomic status, and social capital) that stratify health opportunities and outcomes—may also address some of the concerns in terms of generalizability and applicability (also called external validity) of the studied intervention.7 

In this article, we describe the Applying Evidence with Confidence (APPECO) project, its development, and concise methods in the field of MS rehabilitation. APPECO is an innovative, freely available online platform that facilitates 1) intuitive evaluation of the latest evidence for the nonpharmacologic management of impairments, activity limitations, and participation restrictions in patients with MS, 2) easier clinical decision making informed by three relevant quality indicators, and 3) hands-on and structured tools to implement that best evidence into daily clinical practice at the click of a button. The APPECO platform (www.appeco.net) will assist each MS rehabilitation professional to apply evidence with confidence.

Development and Structure of APPECO Platform

For APPECO to address the described challenges, three specific phases were developed and are described herein: 1) identify the latest evidence; 2) data extraction, appraisal, and synthesis; and 3) knowledge translation. In Figure S1, which is published in the online version of this article at ijmsc.org, an overview is provided of the development process and the concise methods behind key steps leading to the information available on www.appeco.net.

Development Process

The APPECO platform was developed under the auspices of the Rehabilitation in Multiple Sclerosis (RIMS) European Network for Best Practice and Research as a proof-of-concept, with the vision to revolutionize evidence-based medicine in MS rehabilitation and beyond; a continuously up-to-date (ie, living) synthesis of evidence to equip health care professionals with the tools needed to practice evidence-based medicine. A stakeholder meeting, representing RIMS members, industry, and academia, was organized during the European Committee for Treatment and Research in Multiple Sclerosis (ECTRIMS) conference in Barcelona, Spain, in 2015. The purpose of this meeting was to discuss a core set of requirements and functionalities for the APPECO platform for it to be able to address the key challenges in evidence-based MS rehabilitation. A software developer experienced in the development of medically oriented online applications (everywhereIM, Amsterdam, the Netherlands) was contracted. A multidisciplinary advisory board was composed of key experts in the field of MS rehabilitation through the RIMS special interest groups. This advisory board was engaged at strategic moments in the development process to provide valuable feedback as to the functionality, layout, and user-friendliness of APPECO. Through multiple conferences (eg, RIMS special interest group meetings), the broader MS community was actively engaged in the concept and development of APPECO. The proof-of-principle version of APPECO, with 142 unique RCTs describing the effects of 162 different rehabilitation interventions on fatigue and/or cognitive function in patients with MS, was launched during the RIMS annual meeting in May 2018 in Amsterdam. After this launch, funding was acquired to 1) extend the platform to domains of mobility, pain, and depression; 2) update the information on fatigue and cognitive function; and 3) improve functionalities of the platform for editor, reviewer, and users.

Phase 1: Identification of Latest Evidence and Allocating Resources

Identify New Evidence

Together with an expert medical librarian of the Vrije Universiteit Medical Center, a literature search strategy was developed (Appendices S1–S3) to identify studies meeting the following criteria: 1) randomized clinical trial, 2) patients with MS, and 3) at least one of the experimental interventions could be considered a rehabilitation intervention.

Rehabilitation was defined as any intervention that has the aim to improve or maintain body function, activity, and/or participation and involves the interference of an allied health care professional in the context of a medical setting (eg, hospital, rehabilitation clinic, physiotherapy clinic). Regular e-mail alerts were set up for MEDLINE (PubMed), Embase, and Web of Science and sent directly to the APPECO platform. Subsequently, the platform automatically derives vital study information from each e-mail alert, including title, source of publication, authors, and abstract. Any potential duplicates with or serial publications of previously included records are earmarked for verification by the editorial team to ensure that each patient sample is included in the platform only once. Studies that are identified through other sources (eg, word of mouth, systematic reviews, reference lists) can be added manually by the editor if applicable.

Inclusion/Exclusion of Studies

The administrative back-end of the platform facilitates the management of a pool of independent reviewers. This is a multidisciplinary expert panel that is provided role-based access rights. A notification is sent out to two independent reviewers who are tasked to verify that each new study (based on title and abstract) meets the APPECO inclusion and exclusion criteria. Noninferiority and equivalence trials are presently excluded due to the lack of contrast between the experimental arms (eg, aerobic training versus resistance training). The editor will be notified to make a final decision if the two reviewers disagree. When the study is included, the editor will be automatically requested to allocate the full text of that specific study for data extraction.

Phase 2: Data Extraction, Appraisal, and Synthesis

Data Extraction

A third reviewer (with a similar interest as the subject of the specific study) is requested to conduct the data extraction of this specific study, and a fourth reviewer is requested to verify the data extraction of the third (ie, the four-eyes principle). Panel S1 shows all the study information that is extracted.6,8 

Specific features have been implemented to reduce extraction error and improve reviewer convenience. For example, the APPECO system “learns” which outcome measures have been extracted previously and will auto-complete to avoid spelling errors. Subsequently, the domain (eg, fatigue) that a specific outcome relates to, and the conventional “direction” of the measurement instrument (eg, higher outcome is better result) linked to that specific outcome are “remembered.” Reviewers can decline to undertake a specific task in case of conflict of interest to avoid bias (eg, review own article).

Although the APPECO platform initially has been developed as a proof-of-concept with respect to fatigue and cognitive function, during the data extraction process, all other study outcomes were also extracted. Outcomes are structured according to the International Classification of Functioning model (Figure S2).

Quality Appraisal

The study quality is assessed (reviewer 3) and verified (reviewer 4) using the Cochrane Risk of Bias tool (see Panel S1 for the seven items).8  Each item is appraised as having low, high, or unclear risk of bias. A study is considered to be of high methodological quality if all items, except blinding of participants and personnel (considered unfeasible in rehabilitation studies), are appraised as low risk.

Phase 3: How to Use APPECO: Knowledge Translation

Health care professionals are recommended to follow a stepwise approach in using APPECO to reach an informed clinical decision based on the scientific evidence (Table 1).

Table 1.

Recommended stepwise approach for using APPECO to apply evidence with confidence

Recommended stepwise approach for using APPECO to apply evidence with confidence
Recommended stepwise approach for using APPECO to apply evidence with confidence

Presentation and Visualizing Effects of MS Rehabilitation Interventions

Figure S3 shows how results are presented: intervention main type (eg, cognitive training), intervention subtype (eg, attention/information processing speed training), and study level (eg, Charvet et al). A color scheme is used to quickly identify a significant positive result (green), a significant negative result (red), or a nonsignificant effect (gray). For each study, a maximum of two “effects” are plotted, the first to indicate the immediate postintervention effect and the second the “long-term” effect. These effects are plotted along the x-axis based on the duration of the therapy in weeks. Detail on the actual size of the effect can be obtained by hovering the pointer over the respective diamond at the aggregated main type of intervention level, at the intervention subtype level, and/or at the study level.

How to Find “Best-Evidence” Studies

Each study is assigned a maximum of three stars, which can direct the health care professional to determine the best evidence. Star 1: This star is provided to studies that have a high study quality (ie, low risk of bias on all items except blinding of participants/personnel). Star 2: This star is provided when the primary outcome of the study matches the user-selected patient symptom domain. For example, a star is allocated if the user selects the domain “fatigue” and the primary outcome of the specific study was “fatigue.” Star 3: This star is provided if the patients in that specific study were included based on having a predefined level of severity with respect to the symptom or activity limitation. For example, a star is allocated if the user selects the domain “fatigue” and patients in the specific study had a predefined level of fatigue.

Stars 2 and 3 are flexible in a way that if the user selects a different patient symptom or activity limitation of interest, the stars allocated to the specific study will change as well. At the overarching intervention levels, an average appraisal is visualized by “filling” the star based on the percentage of underlying studies reaching that criterion.

Assisting Implementation

The APPECO platform has included numerous technical as well as hands-on tools to assist with implementation of the best evidence of MS rehabilitation into clinical practice. First, APPECO has been developed for the tablet, laptop, and desktop. Second, each intervention has been extracted using the TIDieR checklist.6  A PDF print of this checklist can be obtained by the click of a button. Third, an APPECO summary of findings is automatically generated for each study, indicating the type of intervention studied, study quality, positive and negative effects found, outcomes for which no effect was found, and the type of study participants (ie, external validity). A PDF print of this summary can be obtained. Fourth, additional information that has been published open access with the original publication, that assists in implementation, is made available as attachments. Fifth, the APPECO study team proactively reaches out to corresponding authors of studies identified as being of high quality to provide hands-on tools that may assist implementation of their treatment protocols. An e-mail is sent from the APPECO platform providing the author access to a specific environment that will allow the author to 1) complement the data extraction and fill in the missing pieces, 2) notify the editor of errors in the data extraction if applicable, and 3) upload key documents and materials (.pdf, .docx, .ppt, .mp4, and others) that are valuable in the knowledge translation process. This can, for example, be a treatment guideline used during the conduct of the study, a recorded PowerPoint presentation, instructional video material for specific exercises, infographics, among others. To prevent bias, the quality of the materials and the change requests of study authors will be reviewed and verified by the editor before being made available.

Current Content

The APPECO platform currently hosts detailed information from 250 RCTs, 293 interventions, and 1250 effect sizes on 53 patient complaints (as of February 2020). A total of 152 of the 293 interventions (52%) report on fatigue in patients with MS, 111 (38%) on mobility, 106 (36%) on depression, 58 (20%) on attention and information processing speed, 38 (13%) on memory and learning, 29 (10%) on executive functions, 12 (4%) on perceived cognitive problems, and 12 (4%) on other cognitive functions. In the process of extracting data on fatigue, cognition, mobility, pain, and depression, other domains were also included on the platform, of which quality of life (n = 108 interventions), balance (n = 51), and anxiety (n = 48) were most frequent (Table 2).

Table 2.

Included number of interventions (if >20) per domain

Included number of interventions (if >20) per domain
Included number of interventions (if >20) per domain

Three-Star Studies

There are few studies with high quality available that were specifically designed to test the benefits of MS rehabilitation on fatigue (n = 6 [2.4%]),914  cognitive function (n = 4 [1.6%]),1518  and depression (n = 2 [0.8%]).14,19  No high-quality studies were included for pain and mobility. An appraisal of studies with a low risk of bias (star 1) was given in studies that included fatigue (n = 23 studies [9.2%]), attention and information processing speed (n = 8 studies [3.2%]), memory and learning (n = 8 studies [3.2%]), executive functions (n = 4 studies [1.6%]), perceived cognitive problems (n = 4 studies [1.6%]), and other cognitive functions (n = 3 studies [1.2%]).

APPECO Users

The initial user statistics available in the first 2.5 years of being online (May 2017 through February 2020) indicate 2300 unique users who visited a total of 4200 sessions for, on average, 7.56 minutes. Of the users, 55.6% originated from sites not directly related to the study team developing the platform.

Discussion

This paper describes the development, methods, and current content of a living documentation system to address key challenges in evidence-based rehabilitation for patients with MS, including 1) the time gap between evidence generation and uptake in guidelines and systematic reviews, 2) limited knowledge and resources for health care providers to appraise the quality and applicability of scientific research to their setting and/or patient, and 3) lack of hands-on tools that assist the implementation of research into clinical practice.

Additional Strengths of APPECO and Future Perspectives

Under the assumption of a fully up-to-date platform with respect to MS rehabilitation, there are a couple of additional strengths that the APPECO platform may provide. First, in contrast to systematic reviews, which most often tend to be specific in terms of intervention and patient outcome, the APPECO platform provides an overview of multiple types of interventions compared with control interventions, and for a large number of patient outcomes. This “side-by-side” information could be used by clinicians to direct the patient to the most appropriate health care provider for the treatment most likely to lead to significant benefits. Second, information with respect to the duration and frequency of the intervention, and patient preferences, can be taken into account in making an informed, evidence-based, yet individualized decision with respect to the most appropriate and feasible treatment strategy. Contextualizing the evidence may increase adherence and uptake of evidence for rehabilitation.20 

Challenges and Limitations

The actuality and sustainability of APPECO in the provision of evidence-based MS rehabilitation depends on the platform staying up to date and the inclusiveness of all domains relevant to the rehabilitation of patients with MS. These two aspects (up to date, comprehensiveness) pose challenges, as with many conventional guidelines. The 250 RCTs currently included represent a significant, yet incomplete portion of the RCTs conducted in this field. It is going to require a substantial resource impulse (eg, time).

A method of assigning reviewers based on their special interest (in line with the RIMS special interest groups) has already been built in. Assignment of reviewers with a special interest to process new evidence that meets that special interest can be considered a viable strategy to reduce the perceived burden of maintaining APPECO by the academic community. A future perspective could be that APPECO is embraced by publishers and study authors themselves to provide detailed information at the time of publishing their study results. Obviously, this would then need to be verified by independent reviewers against the publication to ensure academic rigor. The process of filling the platform (≈2–3 hours per study for data extraction or verification) was conducted by a select group of researchers on the study team to address the sheer number of studies that needed to be included.

Until now, the study team has been conservative in the request for these materials, and only additional material for selected high-quality studies is currently available. Although knowledge translation in conventional guideline development is a deliberate process, the APPECO platform largely relies on the input from study authors. The level to which additional information is available that can really assist in implementing evidence into clinical practice remains to be evaluated and poses a second challenge for the APPECO platform. This can be further complicated by the potential multilingual nature and cross-cultural differences of implementation tools. The latter may require resources for translation of materials or limit the external validity for some of the studies. However, the online nature of the environment offers ample opportunity to integrate educational components and directives (eg, animations, instructional videos, a how-to-use guide) within APPECO that can assist the user in getting to the right information.

What if there is no (best) evidence for a specific intervention, patient symptom, or activity limitation? The methodological quality of the studies included was limited (star 1), and in the context of fatigue, cognitive function, depression, pain, and mobility, interventions were rarely designed to specifically and primarily address these domains as shown by star 2 and inclusion criteria (star 3) in these studies. The lack of (quality) evidence for some interventions albeit often used in clinical practice, therefore, poses an important third limitation. In conventional guideline development, in the absence of evidence, one formulates consensus-based recommendations based on extensive stakeholder and expert engagement. Including expert recommendations in the absence of evidence in APPECO is a step that needs to be carefully considered. Expert-based recommendations may be very context specific and may be biased or politically driven.

Conclusions

The APPECO platform aims to substantially improve the way knowledge is translated into clinical practice and to enable health care professionals to apply evidence with confidence in MS rehabilitation. Its potential depends on the ability to keep APPECO up to date, and the availability of practical materials that assist with implementation of effective treatments.

PRACTICE POINTS
  • Health professionals need innovative tools to reduce the delay between the generation of evidence and the uptake of that evidence in clinical practice.

  • Applying Evidence with Confidence (APPECO) is an intuitive online platform that facilitates evidence-based rehabilitation in MS based on the latest evidence available.

  • The APPECO platform includes details for 250 studies (as of February 2020) on the management of fatigue, cognitive dysfunction, depression, pain, and mobility in patients with MS.

Acknowledgments

The authors acknowledge the multidisciplinary advisory board that provided invaluable input throughout the course of APPECO development.

Financial Disclosures

The authors declare no conflicts of interest.

Funding/Support

The development of APPECO has been financially supported by Sanofi Genzyme (phase 1; through RIMS). The Dutch MS Research Foundation (PP 18-001) and Roche financially supported the maintenance of the platform and data extraction of MS rehabilitation studies related to depression, pain, and mobility.

References

References
1.
Amatya
B,
Khan
F,
Galea
M.
Rehabilitation for people with multiple sclerosis: an overview of Cochrane Reviews
.
Cochrane Database Syst Rev.
2019
;
1
:
CD012732
.
2.
Garner
P,
Hopewell
S,
Chandler
J,
et al.
When and how to update systematic reviews: consensus and checklist
.
BMJ.
2016
;
354
:
i3507
.
3.
Steultjens
EE,
Dekker
JJ,
Bouter
LM,
et al.
Occupational therapy for multiple sclerosis [published online ahead of print July 21, 2003]
.
Cochrane Database Syst Rev
.
doi:
.
4.
Heine
M,
van de Port
I,
Rietberg
MB,
van Wegen
EEH,
Kwakkel
G.
Exercise therapy for fatigue in multiple sclerosis
.
Cochrane Database Syst Rev.
2015
;
9
:
CD009956
.
5.
Sadeghi-Bazargani
H,
Tabrizi
JS,
Azami-Aghdash
S.
Barriers to evidence-based medicine: a systematic review
.
J Eval Clin Pract.
2014
;
20
:
793
802
.
6.
Hoffmann
TC,
Glasziou
PP,
Boutron
I,
et al.
Better reporting of interventions: Template for Intervention Description and Replication (TIDieR) checklist and guide
.
BMJ.
2014
;
348
:
g1687
.
7.
O’Neill
J,
Tabish
H,
Welch
V,
et al.
Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health
.
J Clin Epidemiol.
2014
;
67
:
56
64
.
8.
Higgins
J,
Green
S.
Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]
.
The Cochrane Collaboration website.
http://handbook.cochrane.org.
9.
Heine
M,
Verschuren
O,
Hoogervorst
EL,
et al.
Does aerobic training alleviate fatigue and improve societal participation in patients with multiple sclerosis? a randomized controlled trial
.
Mult Scler.
2017
;
23
:
1517
1526
.
10.
van den Akker
LE,
Beckerman
H,
Collette
EH,
et al.
Cognitive behavioral therapy positively affects fatigue in patients with multiple sclerosis: results of a randomized controlled trial
.
Mult Scler.
2017
;
23
:
1542
1553
.
11.
Blikman
LJ,
van Meeteren
J,
Twisk
JW,
et al.
Effectiveness of energy conservation management on fatigue and participation in multiple sclerosis: a randomized controlled trial
.
Mult Scler.
2017
;
23
:
1527
1541
.
12.
Pöttgen
J,
Moss-Morris
R,
Wendebourg
J-M,
et al.
Randomised controlled trial of a self-guided online fatigue intervention in multiple sclerosis
.
J Neurol Neurosurg Psychiatry.
2018
;
89
:
970
976
.
13.
Thomas
S,
Thomas
PW,
Kersten
P,
et al.
A pragmatic parallel arm multi-centre randomised controlled trial to assess the effectiveness and cost-effectiveness of a group-based fatigue management programme (FACETS) for people with multiple sclerosis
.
J Neurol Neurosurg Psychiatry.
2013
;
84
:
1092
1099
.
14.
Grossman
P,
Kappos
L,
Gensicke
H,
et al.
MS quality of life, depression, and fatigue improve after mindfulness training: a randomized trial
.
Neurology.
2010
;
75
:
1141
1149
.
15.
Mäntynen
A,
Rosti-Otajärvi
E,
Koivisto
K,
et al.
Neuropsychological rehabilitation does not improve cognitive performance but reduces perceived cognitive deficits in patients with multiple sclerosis: a randomised, controlled, multi-centre trial
.
Mult Scler.
2014
;
20
:
99
107
.
16.
Campbell
J,
Langdon
D,
Cercignani
M,
et al.
A randomised controlled trial of efficacy of cognitive rehabilitation in multiple sclerosis: a cognitive, behavioural, and MRI study
.
Neural Plast.
2016
;
2016
:
4292585.
17.
Carr
SE,
das Nair
R,
Schwartz
AF,
et al.
Group memory rehabilitation for people with multiple sclerosis: a feasibility randomized controlled trial
.
Clin Rehabil.
2014
;
28
:
552
561
.
18.
Chiaravalloti
ND,
Moore
NB,
Nikelshpur
OM,
et al.
An RCT to treat learning impairment in multiple sclerosis: The MEMREHAB trial
.
Neurology.
2013
;
81
:
2066
2072
.
19.
Bombardier
CH,
Ehde
DM,
Gibbons
LE,
et al.
Telephone-based physical activity counseling for major depression in people with multiple sclerosis
.
J Consult Clin Psychol.
2013
;
81
:
89
99
.
20.
Kayes
NM,
Martin
RA,
Bright
FA,
Kersten
P,
Pollock
A.
Optimising the real-world impact of rehabilitation reviews: increasing the relevance and usability of systematic reviews in rehabilitation
.
Eur J Phys Rehabil Med.
2019
;
55
:
331
341
.

Author notes

From Amsterdam UMC, Vrije Universiteit Amsterdam, Rehabilitation Medicine, Amsterdam Movement Sciences, MS Center Amsterdam, Amsterdam, the Netherlands (MH, HB, VdG); Institute of Sport and Exercise Medicine, Faculty of Medicine and Health Sciences, Stellen-bosch University, Cape Town, South Africa (MH); and Finnish Neuro Society, Masku Neurological Rehabilitation Centre, Masku, Finland (PH).

Note: Supplementary material for this article is available at ijmsc.org.

Supplementary Material