Implementation Evaluation of an Online Peer-Mentor Training Program for Individuals With Spinal Cord Injury

A sequential cross-sectional design was used to assess the implementation context (objective 1) and to identify barriers and facilitators in the implementation of an evidence-based practice (objective 2). Finally, a pre-post design was used to evaluate the influence of the implementation process on implementation behaviour of peer mentors (objective 3).

We used an integrated KT mode22–24 to guide the implementation process. The research team worked closely with the knowledge users from SCI Canada to ensure the relevance and the utility of the process. A national project team (NPT) was created, which was comprised of researchers, community leads, and representatives from four SCI Canada provincial member associations who were willing to participate in the NPT (ie, SCI Alberta, SCI British Columbia, SCI Ontario, and the Canadian Paraplegic Association of Nova Scotia). The goal of the NPT was to support provincial teams during the implementation and ensure the realization of the study objectives. The promising practice (ie, online peer-mentor training) was selected by the members of the NPT and valeted within each provincial member association. Furthermore, an advisory committee led by a person from SCI Canada and composed of 12 persons with lived experience from across Canada critiqued existing peer-mentor training programs (notably the one from SCI Ontario), compared them with programs described (although briefly) in scientific literature,9,10,12 and extracted elements that fit the SCI Canada context, including regional differences related to building capacity and barriers thereto. The advisory committee then created a training curriculum from existing material. The module provided direct training to volunteers defining their role as a peer supporter/mentor and providing strategies for assisting others to adapt to barriers and to thrive through participation in their community (eg, social, physical activity, and economic). The online peer-mentor training modules, including theoretical content and video, were pilot-tested in January 2016 by members of the advisory committee, and appropriate modifications were made before the implementation started. The online peer-mentor training was made accessible to all SCI Canada member associations in February 2016. Peer mentors were enrolled by the employees of the provincial associations, and the NPT recorded commentaries from peer mentors. A revised version of the online peer-mentor training program was put online in August 2016, and the outline of the course including learning objectives is provided in  Appendix 1.

The implementation process was performed by the NPT using the five phases (project planning, pre-exploration, exploration, installation, and initial implementation)25 and two tools (Hexagon tool26 to evaluate the fit of innovation, evaluation of implementation drivers27) of the National Implementation Research Network (https://nirn.fpg.unc.edu/). The implementation process was structured around the five steps, and the tools were filed and discussed by the participants. The implementation process started in June 2014 and finished in August 2016.

The staff (directors, managers, and client and peer service specialists) of four provincial member associations of SCI Canada provided information about the context in which the implementation process was conducted (objective 1). More specifically, 18 persons were invited to complete the Evidence-Based Practice Attitude Scale (EBPAS) questionnaire and 11 persons were given the Organizational Readiness to Change Assessment (ORCA) questionnaire (five individuals answered both questionnaires). Then, the directors of SCI Canada members, who were deemed to have sufficient knowledge of the provincial association to be able to judge the readiness of their respective organizations for this implementation, identified facilitators of and barriers to the implementation of an online peer-mentorship program (objective 2). Except for one program manager who did not participate in objective 2, the individuals who participated in objective 2 were the same as those who completed the ORCA. For objectives 1 and 2, the entire staff in direct service and management positions of the four member associations were invited to complete the questionnaire, and all participated in the study. All candidates for the peer-mentor program (ie, former peer mentors, members of the SCI associations) were invited to participate in the evaluation of the influence of peer-mentor training using the Determinants of Implementation Behavior Questionnaire. Ethical approval was obtained from the Research Ethics Board of the Quebec City Institute of Rehabilitation in Physical Disability (IRDPQ) (no. 2014-382).

Staff of participating member associations completed a demographic questionnaire (eg, age, sex, position within SCI Canada). The EBPAS was sent via email by a research assistant. Participants with a managing position (directors and program managers) also completed the ORCA. Reminder emails were sent 1 week after the initial request.

Evidence-Based Practice Attitude Scale. The 15-item EBPAS was used to evaluate participants' global attitudes toward adopting a new evidence-based practice using four subscales: Requirements, Appeal, Openness, and Divergence.28 The items are formulated as positive (eg, “I am willing to try new types of interventions and practices even if I have to follow guidelines or written procedure”) and negative (eg, “I would not use guidelines or written procedure”) statements. Responses are given on a 5-point Likert scale ranging from 0 (not at all) to 4 (a very great extent). The EBPAS demonstrated good internal consistency reliability, with Cronbach's alpha ranging from .77 to .79.29 

Organizational Readiness to Change Assessment. ORCA can be used to identify and monitor organizational strengths and weaknesses in supporting the implementation of evidence-based practices.30 It consists of 74 questions with a 5-point Likert scale ranging from strongly disagree to strongly agree and an “I don't know/not applicable” choice. Questions may be grouped (mean of components) in 18 subscales, comprising between three and six items. The 18 subscales can also be aggregated into three scales: evidence (the nature and strength of the evidence), context (environment or setting), and facilitation (capacity or types of support needed). The ORCA demonstrated a good internal consistency (Cronbach's alpha 0.74–0.95).31 

The individuals in management positions at the provincial SCI Canada sites used a written questionnaire based on the Consolidated Framework for Implementation Research (CFIR)32 to document the perceived barriers and facilitators in the implementation of the peer-mentorship training. The questionnaire was sent by email by a research assistant. The questionnaire proposes to document both barriers and facilitators in the following five domains of the CFIR: (1) promising practice characteristics (ie, barriers and facilitators regarding online peer-mentor training); (2) outer settings (ie, the economic, political, and social contexts in which an organization resides); (3) inner settings (ie, features of structural, political, and cultural contexts through which the implementation process will proceed); (4) characteristics of individuals (eg, cultural, organizational, professional, and individual mindsets, norms, interests, and affiliations); and (5) the implementation process (ie, interrelated series of subprocesses that do not necessarily occur sequentially but which ideally are all targeted at achieving effective implementation). The five questions presented the CFIR domain along with examples and open-ended questions to allow for the documentation of the two categories of determinants.

Candidates for becoming a peer mentor completed the Determinants of Implementation Behavior Questionnaire before and after their online training on peer mentorship. They were recruited by the peer program regional leads of provincial association within their current service membership of persons with lived experience based on their willingness to undergo the training and to become a peer mentor within their association. All of them successfully completed the online training.

Determinants of Implementation Behavior Questionnaire. The determinants of implementation behaviour were assessed using the Determinants of Implementation Behavior Questionnaire.33 The original 92-item questionnaire assessed 18 domains, explaining 63.3% of the variance and with internal consistency reliability values ranging from .68 to .93.33 As recommended by Michie et al,34 the questionnaire was tailored to the implementation project in collaboration with SCI Canada representatives to elicit the determinants that were most relevant to the context. The adapted questionnaire had 32 questions that documented 11 distinct domains. The research assistant emailed the links to the participants including detailed instructions and the measurement tool before and after the training and sent a reminder message to those who had not completed the questionnaire within 1 week. The participants were informed that the provincial leads and a research assistant were present to answer any questions that they might have.

Descriptive statistics were used to describe the study samples relevant for each objective.

Statistical analyses were performed using SPSS version 24. Given the small number of participants (objective 1) and the distribution of the data (objective 3), we used nonparametric statistics with statistical significance (alpha = 0.05). The Wilcoxon signed-rank test was used in objective 3 to evaluate the difference in Determinants of Implementation Behavior Questionnaire scores from pre to post online training. Facilitators and obstacles gathered in objective 2 were analyzed using a thematic content analysis,35 and the frequency of each facilitator and barrier were documented.

The participants' sociodemographic profile (Table 1) and results are presented according to each study objective.

Eighteen participants from four provincial associations completed the EBPAS (12 females). Four were directors, eight were project managers, and six were client service specialists. They had 1 to 30 years of experience in their position (average 13±9 years). Background and previous training included administration degree (n = 4), social work degree (n = 5), health-related degree (n = 4), and work or lived experience with SCI but without a specialized degree (n = 5).

Participants reported an overall positive attitude toward evidence-based practice with a median score of 3 out of 4 for positive statements related to the subscale Openness to evidence. The median scores ranged from 0 to 2 out of 4 for the Divergence scale, suggesting that participants were not divergent (eg, valued research and would be willing to follow evidence-based guidelines). In the second half of the questionnaire, a high level of willingness to adopting a new practice for all described situations can be observed through a median score of 2.5 to 3 out of 4 for the items of the subscale Appeal and a median score of 3.0 to 4.0 on the subscale Requirement. Governmental jurisdiction, training, relevance of the evidence-based practice, and requirements for use by the employers were the determinants that were the most likely to influence participants' adoption.

Eleven participants with a managing position (5 directors, 6 program managers) completed this questionnaire (9 females). Table 2 presents median interquartile ranges for each of the subscale scores.

Participants evaluated that the strength of the evidence (peer mentoring) was strong, as was the perception of the organization's experts about it. Results of the three major scales (Evidence, Context, and Facilitation) were also rated strong by the participants (median of 4 or more).

Ten participants (8 females) with a managing position (5 directors, 5 program managers) from five provincial associations identified barriers and facilitators in relation to implementation. Table 3 summarizes the perceived facilitators and barriers in online peer-mentor training according to important characteristics of online peer-mentor training, outer settings, inner settings, characteristics of individuals, and the implementation process, and it provides details about the number of participants who indicated each response.

One of the two most prominent elements among the large number of items reported is the relevance of the practice chosen. In fact, 7 participants out of 10 reported that peer support is central to the mission of the member associations that make up SCI Canada and that this type of practice is very coherent with the organization's culture and values. Equally important, but as an obstacle, is the scarcity of associations' resources in general and, more specifically, resources devoted to the implementation of the program. The implementation of an online peer-mentorship program also competes with multiple priorities, making it difficult for the stakeholder to prioritize it over other activities.

Responses before and after online peer-mentorship training were obtained from 34 participants (12 females; 32 had SCI and 2 were family members; 18 had previous peer-mentor experience) from 6 provinces (Alberta = 12, British Columbia = 10, Nova Scotia = 5, Ontario = 4, Saskatchewan = 2, and Newfoundland = 1), of whom 18 had previous peer-mentorship experience. Forty-three other individuals only answered the pre or post training questionnaire and thus were not included in the analysis. Participants' ages were between 18 and 35 years (27%), 36 and 45 years (33%), 46 and 55 years (27%), and 56 and 65 years (12%).

Table 4 shows that the implementation process influences about half (5/11) of the determinants examined. In fact, the participants' perceptions of their knowledge about mentoring, organizational characteristics, and behavioral regulation showed an increase after the implementation process, whereas the changes about the organization and innovation strategy were also positively impacted but in a smaller way. No difference was observed in perceived skills, social role, beliefs, social influence, or innovation. In fact, a decrease was detected with regard to the goals.

Our study aimed to explore the context, process, and influence of a formal implementation process in SCI community-based organizations (ie, SCI Canada) to inform future implementation efforts. We found a positive community-based context for implementation that encompasses barriers and facilitators that are mainly related to characteristics of the online peer-mentor program and to the internal context of provincial associations. The implementation process was found to positively influence many determinants of the implantation for individuals who undergo the training.

Understanding the context in which a new intervention will be implemented is critical to the successful implementation thereof. The EBPAS provided useful insight about the current attitudes of the target members of community-based organizations, suggesting there is interest, willingness, and an overall positive attitude to adopting online peer-mentor training across SCI Canada. The EPBAS has been used to evaluate attitudes toward adopting evidence-based practices in a variety of settings, including health care,36,37 education,38 and social services.37 Compared to norms suggested by Aarons et al,37 participants in this study were slightly more open to implementation of evidence-based practices compared to public sector mental health service providers. It is possible that over the past 8 years, there has been an increase in the perceived value of KT in community-based settings.4 As a result, common knowledge about the importance of implementing best practices among community-based health service providers has grown. Furthermore, 12 of the 18 respondents were women, who tend to be more open to the implementation of evidence-based practices than men.37 Similarly, 9 of 11 females supported the implementation of online peer-mentor training according to the ORCA. All participants felt that senior leadership and a project champion would be instrumental to successful implementation. As with the addition of any new services, adequate resources and leadership were major contextual concerns.

Our exploration of the facilitators and obstacles to the online peer-mentor training program implementation is supported by previous literature and contributes to the value of our integrated KT approach. First, implementation of an online peer-mentoring training program was selected in close collaboration with the four provincial member associations of SCI Canada. The existing peer-mentorship program was deemed to be an important element within SCI Canada culture and philosophy, but the lack of formalization was noted as a limitation of existing peer-mentor programs. In fact, community peer support services such as peer mentorship are highly valued in the SCI community, as shared experiences among individuals with SCI can enhance credibility.13 There is a large body of evidence supporting the value of peer-led approaches for facilitating uptake of community-based practices.4 Alignment with the organization culture and philosophy was also the most frequent facilitator reported by Bach-Mortenson et al in a systematic review of determinants in the implementation of evidence-based interventions in third-sector organizations.39 Our experience clearly demonstrates that working with SCI organizations to choose a practice that fits their needs and values represents an important asset for implementation of evidence-based practice in third-sector organizations.

Also coherent with the literature,39 the lack of resources (time, human support, financial) was deemed to be the most frequent obstacle to the implementation of online peer-mentorship training. For example, insufficient staffing levels, lack of time,40,41 and deficient funding41 were reported as having a negative influence on practice change. On the other side, offering training and allocating enough time to change was perceived as positively influencing practice.41 Efforts toward implementation entail effecting changes in many areas of an organization's activities. They are often burdensome and seldom valued for their true worth. However, in our study, the presence of a large number of facilitators limited the number of changes required, including the presence of some already active peer mentors, existing material, and so on. Reducing the number of changes required to implement a practice tends to lower the resources needed to do so and to facilitate the implementation of said practice.

Finally, our study illustrates the influence of the implementation process on many determinants of implementation behavior, showing that the process can facilitate peer-mentors to adopt this practice. In fact, implementation of an online peer-mentor training program positively influenced participants' knowledge. However, knowledge alone is seldom sufficient to change behaviors in a meaningful way,42 and hence changes in perceived organizational characteristics (eg, presence of a formal training) require goal-driven and innovation strategies to reinforce the likelihood that individuals will adopt and sustain a new role as peer mentors. Involving participants in the entire research process, including development and evaluation of the research, can positively impact program success.43 Moreover, obtaining information through implementation evaluation about what works and what does not can provide the information needed to make improvements to existing programs.

To date there is no evidence of a threshold or cut-points indicating the magnitude of change required to determine whether implementation efforts succeed in producing meaningful change in behaviors related to evidence-based practice. Specific to the evidence-based practice examined in this study, knowledge about the determinants predicting the adoption of a peer-mentor role is still lacking. However, in other domains, there is growing recognition that environmental facilitators such as the presence of resources and organizational climate might influence the implementation of evidence-based practices.44,45 Our results suggest that implementation of evidence-based practices in community-based organizations could positively influence how individuals perceive some characteristics of their organization and may facilitate innovation strategies that in turn might create a positive climate for changing practice among community-based SCI organizations. Despite statistical significance, the changes observed in perceived knowledge, organizations, innovation strategy, and behavioral regulation remain small. Many factors contribute to explaining this situation, for example, the electronic format or the somewhat limited intensity of the online training modules that cannot be sufficient to induce major changes in individual determinants of implementation behavior. Although small, these changes might be sufficient to positively influence behavior over the long term. However, the amount of change required needs to be further investigated as there is no evidence related to the adoption of peer-mentorship activities. Finally, we observed a drop in the goal dimension, which relates to the priority of peer mentorship within personal agendas. It is possible that having more informed knowledge about the peer-mentorship process and effects resulted in more realistic expectations toward this activity and thus lowered its priority within individual agendas. The mechanisms by which the online training modules have affected the determinants of the implementation of a new behavior remain, however, to be explored more in-depth, in particular using qualitative interviews as a key strategy to this end.

Strengths and limitations should be considered when interpreting the findings of this study. One strength is that the research team collaborated very closely with the steering committee to ensure the relevance of the research process and results for SCI Canada. This collaboration likely increased the relevance of the results, but it may also have contaminated the interpretation of the findings. We considered provincial associations and participants to be representative of SCI staff and peer mentors across different provinces, thereby providing generalizable insight about the reality of the program's implementation across the country. However, the large geographic diversity may reduce the sensitivity of our findings about provincial variation and the distinctive characteristics of each setting (eg, not able to consider the characteristics of each province, such as climate and cultural variability). In addition, each Canadian province has its own health care organizations that were not considered in this study. The application of mixed-methods (qualitative and quantitative) and the use of robust questionnaires with strong psychometric properties and models (CFIR) strengthens the results of this study. Although input was obtained from multiple sources (eg, staff and managers of community-based organization, and peer mentors), mentees who received peer training from the peers who were trained online were not surveyed. Future implementation studies should evaluate attitudes of both peer mentors and mentees.

Community-based organizations provide a valuable opportunity for implementation of evidence-based practices. Overall, community-based SCI service providers express willingness and a positive attitude to implement online peer-mentor training, which is an evidence-based practice specifically targeted at community-based health service provision for individuals with SCI. Although there was an expressed need for implementation of evidence-based practices, contextual concerns related to lack of resources and leadership are a commonly perceived barrier by SCI Canada current staff. Therefore, successful implementation of online peer-mentor training will require that specific consideration be given to financial and human resources.

The authors declare no conflicts of interest.

Peer Mentor Training Online has been designed for individuals who have been pre-screened by the SCI organizations and will be volunteering as peer mentors within their communities. Through completion of this course, participants will learn about their role and responsibilities as peer mentors, the procedures they must follow as volunteers, and strategies they can use to establish positive, productive relationships with their peers.

The online course is self-paced and has a responsive design. This means that participants can take their training on a computer, mobile phone, or tablet. The only requirement is internet access. Patients progress at the pace that best matches their learning style and previous knowledge. Participants can log out whenever they like and log in later, returning automatically to the last viewed screen. Interactive screens break complex topics into smaller, more easily understood units of information, and scenario-based exercises allow participants to apply learned concepts in practical situations.

The content is presented in three modules. To move from one module to the next, participants must obtain 100% on end-of-module tests. Test questions are randomly selected from a bank of options to ensure that repeat tests will be unique.

• Describe the role of the peer mentor

• Identify the benefits of peer mentorship to mentors and mentees

• Work with peers to set appropriate and realistic expectations for the first meeting

• Use new communication techniques to improve interpersonal communication with a mentee

• Recognize the attitudes and behaviors of a good peer mentor

• Know the general safety precautions for volunteers

• Recognize acceptable and unacceptable peer mentor attitudes and behaviors

• Know the rules and procedures of the SCI organizations that apply to the peer mentorship program and how to protect the confidentiality of a peer

• Recognize and respond in an appropriate and supportive manner when a peer is in crisis or at risk.

The required learning material is in the course, including a Resource Library where participants may consult supplemental documents. After completion of the course, peer mentors continue to have full access to the course content and to all material within the library.