Alternate delivery platforms and implementation models for bringing evidence-based behavioral interventions to scale for youth facing adversity: a case study in West Africa
Implementation Science Communications volume 3, Article number: 16 (2022)
Youth Functioning and Organizational Success for West African Regional Development (Youth FORWARD) was launched as an implementation science collaboration focused on scaling out evidence-based mental health interventions for youth exposed to war and other adversities through novel delivery platforms. This implementation science case study examines the use of a collaborative team approach (CTA) as a scale-out strategy to foster the integration of an evidence-based group mental health intervention, the Youth Readiness Intervention, into youth employment programs tied to regional economic development in Sierra Leone.
A case study methodology is used to explore the feasibility and acceptability of integrating an evidence-based intervention, the Youth Readiness Intervention (YRI), into youth entrepreneurship programs (ENTR) in Sierra Leone, facilitated by the CTA. The authors analyzed field notes logged during program implementation, 8 weeks of supervision notes, 20 interviews with agency leaders and front-line staff delivering the YRI within this alternate delivery platform. Quantitative dissemination and implementation interviews administered to youth, facilitators, and agency leaders were analyzed using descriptive statistics and mixed linear models. A linked Hybrid Type II effectiveness-implementation cluster randomized trial is evaluating the clinical effectiveness of the YRI within this delivery platform.
Extant data indicate the strong feasibility and acceptability of integrating the YRI into the ENTR program. Facilitators of integration of the YRI into the ENTR include mission alignment of the organizations with the delivery of psychosocial interventions, shared commitment to serving vulnerable youth, support from local District Youth Councils, and high interest from the youth served. Barriers include perceived competition between frontline organizations seeking funding for psychosocial interventions, and challenges in flexibility between donors and implementation partners operating in a fragile/post-conflict setting. The CTA was a feasible and acceptable strategy to support fidelity and quality improvement while scaling out the YRI.
Youth entrepreneurship and livelihood programs offer a promising mechanism for expanding the reach of evidence-based interventions to youth in fragile and post-conflict settings. Quality improvement and sustainment of evidence-based interventions are novel concepts in such settings. The CTA strategy institutionalizes the integration of an evidence-based intervention into youth entrepreneurship programs.
The risk of long-term mental health problems is increased in settings where exposure to war and community violence compound other social problems. Studies in post-conflict settings have shown that more than one in five people suffer from mental health disorders, and nearly one in ten have a moderate to severe mental health disorder at any point in time . The Longitudinal Study of War-Affected Youth in Sierra Leone (LSWAY; 1R01HD073349-01), launched in 2002, found that youth exposed to war trauma demonstrated higher levels of psychosocial problems [2,3,4]. In particular, emotion regulation was identified as a major underlying mechanism that limited youth’s ability to participate successfully in opportunities such as employment and education programs. Adaptive and pro-social behaviors were associated with community acceptance and social support . LSWAY findings informed the establishment of a research hub, Youth Functioning and Organizational Success for West African Regional Development (Youth FORWARD), which is an integrative psychosocial initiative developed to respond to the mental health and emotion regulation needs of youth in Sierra Leone through development initiatives such as youth employment programming.
Sierra Leone has limited healthcare infrastructure and faces challenges in the delivery of mental health services. An 11-year civil war (1991–2002), Ebola outbreak (2014–2015), recent mudslide (August 2017), and now the COVID-19 pandemic (2020) have devastated the healthcare system and exacerbated the mental health treatment gap. The Government of Sierra Leone is committed to meeting some of the core needs of the country’s youth and its economic and health policy agendas align to support advancement of mental health interventions for youth exposed to compound adversities. Recently, the Ministry of Health and Sanitation implemented a multi-level, collaborative approach to treat mental health holistically and incorporate the formal and informal structures that provide mental health treatment across the country . Sierra Leone’s recent Economic Development plan includes policy actions to support youth entrepreneurship and skill development , and partnership with the World Bank prioritizes building human capital through education initiatives and job creation .
Given the current government’s prioritization of human capital formation, Youth FORWARD’s approach to integrated psychosocial programming is especially timely. Youth FORWARD has scaled out an evidence-based Youth Readiness Intervention (YRI) via an alternative delivery platform of a youth entrepreneurship program run by the German development agency, Gesellschaft für Internationale Zusammenarbeit (GIZ). GIZ’s multilevel model combines national-level policy and service sector approaches to support employment and income improvement for vulnerable, rural, and undereducated youth. Through a needs-oriented approach, GIZ’s Employment Promotion Programme ensures that young people gain qualifications for employment with training intended to strengthen labor market skills, increase income, and promote resilience to economic shocks [9, 10]. The YRI is a transdiagnostic intervention to assist youth facing complex problems using evidence-based best practices and underwent rigorous adaptation in-country to ensure cultural fit and appropriateness through the use of local parables, relevant language, and skills useful to the Sierra Leonean context . It is designed to be delivered by lay workers in a group format, which deepens social connections and enables peer-to-peer support long after the intervention ends. The YRI incorporates evidence-based components from cognitive behavioral therapy: psychoeducation, relaxation techniques, assertive communication strategies, cognitive restructuring, behavioral activation, goal setting, and sequential problem solving . The YRI was increased from 10 to 12 modules to respond to group needs and provide greater treatment for depression . A randomized controlled trial of the YRI demonstrated that youth assigned to the intervention reported significantly greater improvements in emotion regulation and prosocial attitudes and behaviors compared to control youth and were six times more likely to persist in school .
In this study, the YRI was integrated into the entrepreneurship training program (ENTR) of GIZ’s employment platform using a Collaborate Team Approach (CTA). While the YRI addressed youth’s mental health, interpersonal, and emotional functioning, the ENTR prepared them for employment by providing skills training and mentorship around starting a sustainable income generating activity. By integrating the two programs, Youth FORWARD aimed to improve youth’s daily functioning and interpersonal relationships while simultaneously developing critical livelihood skills. A feasibility pilot was conducted in Kailahun district, followed by a rigorous scale-out study implemented in the Kono, Koinadugu, and Kailahun districts in Sierra Leone, where GIZ operates. We adhered to the Standards for Quality Improvement Reporting Excellence Implementation Studies (SQUIRE) in preparaton of this manuscript .
This study assessed the feasibility and acceptability of integrating the YRI and ENTR, while using a CTA scale-out strategy. A Hybrid Type II Effectiveness-Implementation Cluster Randomized three-arm design allowed for the simultaneous study of implementation and clinical effectiveness . Implementation outcomes included indicators of feasibility, acceptability, and fidelity as well as YRI facilitator and youth satisfaction. Effectiveness outcomes, which will be reported elsewhere, concerned impacts on youth emotion regulation, interpersonal functioning, and participation in livelihood generating activities (manuscript submitted for publication, Desrosiers et al.; manuscript in preparation, Akram et al.; manuscript in preparation, Freeman et al.).
The Youth FORWARD CTA was intended to help address the issue of limited human resources by providing evidence-based mental health services while advancing goals shared with government and development actors (See Figs. 1 and 2). This approach increased opportunities for youth engagement in livelihood activities by training front-line lay workers to deliver the YRI to youth before participation in the ENTR.
The strategic use of the CTA was guided by the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework, which allowed for scaling and sustaining the YRI and integrating it into the ENTR . The EPIS framework accounts for the policy context within the country or at the local level as well as organizational service delivery conditions and has functioned well in health settings in low- and middle-income countries (LMIC). The EPIS framework is also helpful for assessing “fit” between an evidence-based practice and the intervention itself, considering innovation factors, bridging factors between the inner (organization) and outer (community) contexts, and the potential for sustainability of the intervention. For child and adolescent mental health interventions, bridging factors are particularly important because many services for youth facing adversity take place in public sector systems that span the social ecology .
The research team used the EPIS framework to develop an adapted model, demonstrating which factors were significant for the YRI+ENTR context (See Fig. 1). The framework was used to stage the project and assess progress while identifying challenges. Continuous quality improvement methods, such as integrated supervision and fidelity monitoring, Plan-Do-Study-Act (PDSA) cycles, and cross-site knowledge exchange, were utilized to ensure the YRI was delivered with fidelity and sites were able to benefit from lessons learned from other sites as they identified and addressed challenges . The partnership between universities and community agencies served to bridge the inner and outer contexts in Sierra Leone. YRI+ENTR’s focus on deployment served as an innovation factor that mitigated outer context challenges such as historical trauma, unemployment, and limited access to services.
Implementation models which rely heavily upon remote expertise for training and fidelity monitoring are a major obstacle to achieving sustainable results as they fail to develop local expertise . Therefore, Youth FORWARD’s pilot and scale-out study used a CTA to scale and sustain the YRI and integrate it into the ENTR in order to shift decision-making and ownership to community stakeholders. The CTA is modeled after the Interagency Collaborative Team model utilized in scaling up the SafeCare intervention for families involved in or at-risk for involvement in the child welfare system . Through this work, collaboration emerged as a key element for the implementation and sustainment of evidence-based practices into established service delivery systems. The Interagency Collaborative Team model is contingent upon a multiagency commitment and partnership with a range of stakeholders from the onset . The Youth FORWARD CTA included the YRI developers, YRI experts (members of the Caritas Freetown staff who were trained by the YRI developers and had prior experience implementing the YRI), YRI facilitators (local service providers contracted by GIZ to deliver the YRI and ENTR program), the Youth FORWARD research team, GIZ leadership, the contracted service provider agencies overseeing delivery of the YRI and ENTR program, and Caritas Freetown leadership. The CTA’s key activities included establishing a seed team, cross-site learning, collecting process data for monitoring quality and assessing barriers and facilitators, and communicating across stakeholders.
The seed team
The CTA catalyzed system-wide YRI sustainment through development of a local core unit of experts—a seed team—to provide leadership and guidance on the delivery of the intervention. The function of a seed team is to draw from their experience and contextual knowledge to monitor, supervise, and mentor new facilitators as they deliver the intervention while overseeing cross-site collaboration to expand institutional knowledge on best practices in program delivery . The seed team consisted of three YRI experts who provided ongoing training, coaching and support to YRI facilitators as the intervention moved from pilot to scale-out. Fidelity to evidence-based practices was achieved by meeting individually with YRI facilitators before each YRI session to ensure they were prepared and through weekly supervision meetings with YRI facilitators.
An intensive 2-week training course that was initially designed by the YRI developers was updated by the Youth FORWARD seed team members for the pilot and scale-out studies. The seed team members delivered the course and prepared new YRI facilitators to deliver the intervention. During the training, facilitators received an intervention manual to help them learn the YRI and guide them in its delivery. To become a YRI facilitator, the trainees committed to participating in enhanced supervision and reporting processes, working across intervention sites, collecting process data throughout the intervention, and engaging in problem-solving to create a feedback loop allowing for quality improvement of intervention delivery over time.
Scale out from feasibility pilot
The design of the YRI scale-out was first tested in a feasibility pilot from July through October 2018 in a rural district, Kailahun. In the pilot, youth were clustered and randomized to YRI+ENTR (n = 58) or ENTR-only (n = 57). A statistically matched comparison group (n = 60) was also recruited from Kailahun district. Due to logistical delays from the service provider, delivery of the YRI did not begin until the ENTR had completed, thereby undermining a study aim of investigating whether youth who received the YRI had better performance in the ENTR program. Findings from the pilot were further limited by a small sample size and non-randomized control group. However, the pilot allowed for pre-testing and refinement of study measures and testing of the CTA and communication structures when operating in rural Sierra Leone. In the scale-out study, modules were delivered in October and November of 2019 across 12 chiefdoms of Kailahun, Kono, and Koinadugu districts, with a total of 36 sites. The scale-out study increased engagement and collaborative problem solving with GIZ to ensure the YRI could be delivered before the ENTR and that adequate time was allowed for data collection given the larger sample size and expanded study area (See Table 1 for implementation and effectiveness study measures).
GIZ utilized community sensitization and local media advertisements to recruit youth to participate in the ENTR. Interested youth submitted an application and were consented and screened for study eligibility by Caritas research assistants. Clusters of eligible youth, stratified by geographic location and gender, were randomized into one of three study arms: control (n=387), ENTR-only (n=380), and YRI+ENTR (n=384). Youth were quantitatively assessed at baseline and post-ENTR. In order to isolate YRI treatment effects and strengthen the YRI evidence base, a subsample of 396 treatment youth (ENTR-only and YRI+ENTR) was selected to participate in a post-YRI follow-up assessment. Fundraising is underway to marshal resources to examine longer-term effects of the YRI, including economic impacts, as integrated into this alternate delivery platform.
The feasibility pilot was undertaken in two chiefdoms of Kailahun district, with two gender-segregated clusters per chiefdom. The pilot included a total of 175 youth (62% female, ages 18–30), 120 third-party reporters, 16 facilitators, 4 YRI experts, and two agency leaders. The scale-out study involved 1151 youth (47% female, ages 18–30), 618 third-party reporters, 12 YRI facilitators, and two agency stakeholders for a total sample size of 1783 participants across sites. In both the pilot and scale-out study, youth were eligible if they met study screening criteria which included elevated t-scores on assessments of functional impairment and emotional dysregulation, as measured by the WHO Disability Assessment Schedule (WHODAS)  and the Difficulties in Emotion Regulation Scale (DERS) , respectively. Youth also had to reside in our catchment area, be disengaged from school and formal employment, and for female youth, could not be pregnant given the higher-level support that would be required.
Given timing constraints and funding limitations, quantitative dissemination and implementation (D&I) surveys were not administered. Rather, scheduled qualitative interviews were restructured to include important implementation science domains. Sixteen YRI facilitators, four YRI experts, and two agency leaders participated in key informant interviews (See Table 2 for a summary) that assessed challenges and successes of the YRI and ENTR, facilitator preparation, perceptions regarding the YRI training, delivery, supervision, and its appropriateness in meeting participant needs and facilitators’ professional goals. The seed team completed a self-report quantitative assessment battery to assess cohesion, collaboration, and leadership [24,25,26,27]. Ongoing monitoring was conducted throughout the intervention delivery. The YRI facilitators documented youth attendance for each YRI session. To access YRI facilitator adherence to the intervention structure, sessions were observed in person by a seed team member or audio-recorded. When observing or listening, the seed team members evaluated YRI facilitators’ fidelity to evidence-based YRI practices using a fidelity monitoring checklist, informed by previous interventions using evidence-based components . The pilot data were then used to refine quantitative and qualitative measures administered in the larger scale-out study.
Youth FORWARD includes partnerships across a range of stakeholders in policy and investment, service delivery, research, and capacity building. As such, scale-out study data were collected to assess multi-level stakeholder engagement, implementation process and impact, and clinical effectiveness with 754 youth, 17 facilitators, and two agency leaders (See Table 1 for scale-out study measures and Table 3 for implementation science domain measures).
Qualitative data collected at multiple stakeholder levels assessed intervention satisfaction and acceptability, YRI relevance and impact, barriers to participation, and sustainability. Key informant interviews were done with a sample of youth (n=90, post-ENTR), YRI facilitators (n=17, baseline and post-ENTR), and agency leaders (n=2, baseline and post-ENTR).
A random subsample of 400 participants from the YRI+ENTR and the ENTR-only groups completed a reduced quantitative assessment battery just after completion of the YRI, to assess the effectiveness of the YRI training. Third-party reporters (peers, community members, work supervisors, ENTR facilitators) completed a quantitative assessment battery to evaluate changes in youth functioning at baseline (n=550), post-YRI (n=626), and post-ENTR (n=18, administered to ENTR facilitators only). Feasibility of the delivery approach and impact on sustainment of quality in the delivery of the YRI was assessed by the D&I survey to evaluate the core implementation science domains of adoption, acceptability, appropriateness, feasibility, reach/access, organizational climate, leadership in implementing, general leadership skills, and perceived sustainability . The survey was administered to youth assigned to the treatment arms at baseline (n=754) and post-ENTR (n=744), to the YRI facilitators at baseline (n=17) and post-ENTR (n=17), and to the agency leaders at baseline (n=2) and post-ENTR (n=2). During the scale-out study, a challenge emerged when the implementing partner expressed concern over the intent of the D&I measures. Given the competitive nature of contracts in Sierra Leone, questions regarding organizational climate and leadership were viewed as potentially jeopardizing future contracting opportunities. Several rounds of sensitization about the intent and use of the questions needed to occur before the D&I measures could be implemented.
The seed team completed the self-report quantitative assessment battery post-ENTR (n=3), which included a Perceived Cohesion Scale, Research Collaboration Survey, Levels of Collaboration Scale, and Seed team assessment questionnaire (see Table 4 for descriptive statistics) [24,25,26,27]. They also participated in a focus group discussion that included topics such as implementing seed team structure during YRI delivery, challenges, successes, experiences with supervision, teamwork, and cohesion.
The same monitoring procedures were applied for the scale-out intervention delivery. Youth attendance was documented, and the seed team members observed or audio-recorded the YRI sessions and then completed the fidelity monitoring checklists.
Seed team focuses on group discussions, notes from supervision sessions with facilitators, and agency, facilitator, and supervision key informant interviews were reviewed and coded inductively by the research team using MAXQDA . The seed team utilized audio recordings of YRI delivery in its pilot phase and directly observed sessions in the scale-out to assess fidelity. In the pilot, only nine of the 12 sessions were recorded and available for fidelity monitoring due to technical challenges.
Due to the small sample size, descriptive statistics were used to convey results of facilitator and agency leader D&I surveys and the seed team battery. Multi-level mixed linear effects models were used to predict the effect of different districts on D&I outcomes, controlling for gender and age of youth.
Pilot study data
Data collected during the pilot phase of Youth FORWARD’s scale-out into GIZ’s youth employment scheme demonstrated the feasibility and appropriateness of using a CTA to deliver an evidence-based mental health intervention like the YRI. Decisions and adaptations made throughout the scale-out process were guided by findings from pilot data.
Qualitative data from the pilot revealed insights on challenges and successes surrounding youth employment and psycho-social programming in the country (manuscript submitted for publication, Desrosiers et. al.). Facilitator surveys revealed a number of implementation challenges while delivering the YRI during the pilot (See Table 5). For example, facilitators described issues with providing youth with their travel allowance upfront “because if you give them all the money, they [participants] would just go and never come back” (male facilitator). Facilitators also explained an additional consequence of giving the allowance upfront; non-participants often showed up posing as participants, only to accept the travel allowance and disappear. Others faced delays due to participants arriving late or not coming at all. Many requested more time dedicated to training and to delivering each individual module. Finally, facilitators described language and educational barriers for youth, as a result of operating in rural, remote parts of Sierra Leone (manuscript submitted for publication, Desrosiers et al.).
Nonetheless, the uptake of the YRI by participants is demonstrated in the attendance records and was well-documented in qualitative interviews with YRI facilitators conducted at the conclusion of the pilot. During interviews, facilitators provided many examples of youth using YRI activities and strategies to solve everyday problems, providing evidence for the youth’s engagement with the YRI and the success of the intervention. Many YRI facilitators were also using the intervention to improve their own lives (manuscript submitted for publication, Desrosiers et al.).
Interviews with the YRI facilitators indicated that the YRI was considered culturally appropriate and relevant (manuscript submitted for publication, Desrosiers et al.). Facilitators spoke to the inner context of the YRI: training and supervision, embeddedness in local communities, and their perceptions regarding fit, relevance, and compatibility of the YRI+ENTR.
The scale-out study utilized a staggered approach to rolling out the YRI across study districts, which allowed for cost-effective resource sharing between study sites, cross-site learning between CTA stakeholders, and quality improvement through PDSA cycles .
As the YRI was scaled out, the CTA addressed initial logistical challenges identified in the pilot study. As facilitators were often not familiar with languages spoken by youth in rural areas of the country, peer translators were utilized in sessions with youth who did not speak Krio. Further, youth were provided with the transportation stiped at the end of the intervention rather than up-front.
As the scale-out study was capable of reaching more rural communities through the partnership with GIZ and their presence throughout Sierra Leone, the CTA structure was challenged to respond to emerging issues in a remote context. The supervision of facilitators across districts was originally intended to be done in-person by the seed team who were themselves having weekly teleconferences with the other CTA members to discuss progress and challenges. However, transportation was difficult or impossible in some areas of the country and the seed team was unable to travel between the YRI training sites as frequently as they intended. Thus, seed team members divided themselves amongst project sites and were hosted in a single community throughout the duration of YRI delivery. CTA Group meetings with the seed team and YRI facilitators shifted to phone calls, and weekly CTA meetings shifted to virtual meetings via Zoom. Poor network connectivity, limited service, and technology literacy made virtual knowledge-sharing challenging at times.
Scale out study data
The outer contexts, including contextual limitations, macro-level regulation, and intervention logistics and fidelity, were identified as challenges and barriers to YRI delivery and scale-out. Sierra Leone grapples with a health system overtaxed by war, disease outbreaks, and natural disasters, weak governance structures and limited policy supports, a fractured funding environment, prolonged elections that threaten project implementation, and a fragile context that contributes to reticence from donors.
Worsening intervention fidelity over time is a common obstacle to overall effectiveness. Traditionally, implementation models rely heavily on remote expertise and do not prioritize building local capacity. Organizations in Sierra Leone experience high levels of staff turnover without consistent funding to provide employees with long-term employment contracts. One facilitator noted: “My organization told us that [the job is a] 5-month contract. And when we came … all of a sudden they said six weeks so I was not okay” (male facilitator). This culture is not conducive to maintaining institutional knowledge or sustaining intervention delivery over time. Agency leaders at local organizations confirmed challenges in retaining newly trained staff when projects were short-term, citing funding structures as the chief barrier (Program Manager, Caritas). When delivering a program like Youth FORWARD that utilizes a CTA, considerable time and resources must be spent developing strategies to overcome this challenge.
A major challenge arose in 2017 when the elections resulted in a new government, creating uncertainty with government engagement and a smooth transition of power. As the incoming administration appointed new leadership, relationships initially built when designing the study had to be rebuilt. Another challenge arose when the World Bank, Youth FORWARD’s original partner, changed leadership and set back their Youth Employment Program by 2 years for re-design. As a result, PIs and the Scale-Out Study Team had to identify an alternative implementation partner. The team networked across multiple sectors and eventually partnered with GIZ. While GIZ’s programming provides a strong platform from which to deliver the YRI, a partnership with GIZ presented new challenges. Other adaptations were necessary as GIZ rolled out their employment programming, including the contracting delays and unanticipated acceleration of GIZ’s timeline during study scale-out. Overall, understanding and incorporating the Youth FORWARD research activities within GIZ’s more business-oriented style of operating has required ongoing negotiation and adaptations with implications for implementation research.
Youth FORWARD is part of a network of hubs delivering interventions in low-resource settings . Tremendous challenges lie in complying with regulatory processes, sometimes at odds with field realities. Current project oversight mimicked regulatory processes utilized in drug trials and relied on an external and independent Data and Safety Monitoring Board with a fixed meeting schedule and routine study monitoring from a contracted clinical research associate. This level of oversight was in place to ensure study compliance and participant safety but led to study delays given the fixed nature of the review and approval processes, which often did not align with the service delivery requirements of implementation partners. As a result, the need to align study procedures across several ethical review bodies resulted in increased study costs and implementation delays.
Interviews with the YRI facilitators and seed team members also revealed facilitators to implementation and successes of the intervention. Youth councils established by the Sierra Leonean government in 2012 are present in each district, with councils spanning from the district to the village level. These councils are positioned to work across relevant stakeholder groups to amplify youth presence in the policy making process, elevate youth perspectives, and illuminate challenges facing youth in Sierra Leone . As part of its youth employment platform, GIZ works closely with youth councils to support program coordination and implementation. Leveraging this established relationship was an asset for the YRI team given a limited presence in the more rural study districts. As a result, YRI experts and study research assistants relied heavily on youth council leadership to engage, identify, and contact youth. These councils represent an important innovation factor that supported YRI and ENTR implementation with the potential to influence long-term sustainability .
The CTA provided a bridging structure that allowed researchers from Boston College and the University of Georgia to learn from local stakeholders with significant programming expertise who are embedded in Sierra Leonean communities, which will contribute to long-term sustainability of the YRI. In addition, the CTA provided opportunities for Sierra Leonean agency members to learn from each other’s experience in mental health and youth programming. One seed team member appreciated how the CTA provided specific implementation guidance: “(The CTA) gives the team a road map to best practice. Clarity within the organization leaves less for assumption and allows all partners involved … to make the best decisions and strengthen the core of the program” (Male seed team member).
Two interviewed agency leaders expressed that the CTA allowed for sharing of knowledge and benefitting from the expertise of each member. As one agency leader described, “one organization might be an expert in one thing and the other might be an expert in the other thing. We’re able to kind of meet and discuss things and strengthen that model that we want to use for greater achievement in the communities. And that seems to be working really well” (Senior Program Manager, Restless Development). During regular meetings, both agency leaders appreciated how members across teams would work together to problem solve as issues arose during implementation. Communication and mission alignment improved throughout the process. Both agency leaders explained how multiple budgets and modes of operation were challenging at first, but ultimately led to increased collaboration. According to one agency leader, “it’s (the CTA) has given us a lot more insight to be able to have more team players in a small pitch. So that we, we coordinate more and, and we share a lot more information...which has been very good” (Program Manager, Caritas).
In supervision sessions, seed team members that comprised the CTA provided guidance to facilitators in a manner that empowered self-reflection and self-monitoring. One seed team supervisor described how he would begin supervision sessions by asking the facilitator what his own perceptions were on his delivery of the YRI before offering facilitation critiques and advice. This approach allowed for critical thinking and growth while maintaining fidelity of the intervention.
All interviewed facilitators expressed appreciation for supervision and provided examples of how group and individual meetings with supervisors assisted with problem solving and content challenges. One facilitator described her experience with supervision:
It is very helpful. My co-facilitator and I will be busy with other things in the manual, and maybe one session is not well explained. (Our supervisor) will be there to observe and she will tell us that you have to explain this area … she will tell us to probe very well so that the participants will get a better understanding. Supervision empowers you to be a good worker … it will empower you to become very self-sufficient in your job (female facilitator).
Mixed linear effects models revealed no significant differences in acceptability, adoption, appropriateness, feasibility, or reach between implementation districts, while accounting for gender and age of participants. These results indicate that the intervention functioned similarly across districts and suggests the effectiveness of the seed team in training and supervising YRI facilitators. D&I descriptive statistics from facilitator and agency leader surveys revealed that the perceptions of intervention adoption, appropriateness, feasibility, and organizational climate increased from baseline to endline, while acceptability, reach, implementation leadership, and general leadership scores decreased (See Tables 6 and 7 on D&I descriptive statistics).
The Government of Sierra Leone recognizes that addressing the lingering effects of trauma on the mental health and functioning of youth is a necessary first step in meeting their core needs. Recent policy describes the fundamental relationship between health and economic self-sufficiency, citing that health affects individual productivity and is a critical input for long-term in-country development . The current administration also identifies the benefits generated in the opposite direction, recognizing individual economic self-sufficiency as a major contributor to positive health outcomes, highlighting the potential for economic development initiatives to address health and psychosocial issues .
Increased investment in youth employment programs offers a cost-effective alternate system for delivery of mental health services . Embedding a mental health intervention into youth employment programming can add tremendous value since youth receiving psychosocial training will be better regulated and therefore better able to fully engage in the employment programming, perpetuating a positive feedback loop. By leveraging investments in youth and economic development programs and integrating mental health interventions therein, LMICs with limited mental health care infrastructure and personnel can build new capacity to address the mental health treatment gap. If proven feasible, integrated YRI+ENTR programs have high potential to increase access to evidence-based mental health services for thousands of high-risk youth in Sierra Leone and other post-conflict LMICs.
Future work will require a concerted effort to understand how emerging mental health and psychosocial programming can contribute to the development of longer term and sustainable systems of health and mental health. The outer context and inner context of interventions often change, and factors that bridge those contexts must be taken into account. The EPIS framework is particularly helpful in this regard. The challenges of scale-out and sustainment of mental health interventions are tremendous; especially in LMICs where the availability of specialists for implementation, training, and supervision is limited. However, this study reveals several exciting implementation science innovations useful in fragile settings. First, research and interventions should be contextualized with careful consideration of risk and protective factors across all levels of the social ecology. Second, psychosocial interventions should be based on locally identified needs, rather than externally imposed services or assumptions, identifying priorities through community-based approaches and collaboration with local service providers and community advisory boards. Educational and employment programs should be considered and explored as alternative delivery platforms to effectively deliver mental health services by community-based lay workers, addressing the limited human resource and capacity challenges in post-conflict settings. In addition, it may be that mental health interventions that are embedded within work and other daily activities can optimize outcomes. Finally, efforts to address the mental health treatment gap must include attention to implementation science questions on how to take evidence-based mental health services to scale. Achieving such goals will require innovations addressing limited human resources for health, consideration of incentives, training, supervision, cost, and ongoing professional development for intervention staff, financing, and policy structures, along with strategies for monitoring and improving quality.
An existing pilot study seed team was intended to train and monitor new seed teams in the larger scale-out study, creating a community of practice around the YRI. However, given the need for a competitive bidding process, a single-service provider was chosen for the scale-out study, creating a new cadre of facilitators. As such, the seed team for the scale-out study needed to be reconfigured to allow for the selected service provider to be trained and for supervision to be provided by the CTA, with support from Youth FORWARD leadership. This new seed team included Caritas YRI experts who had previously delivered the YRI and participated in the feasibility pilot, thus retaining the knowledge and skills.
Data demonstrate that GIZ’s youth employment scheme is an auspicious delivery platform for an evidence-based, mental health intervention in a low-resource, post-conflict setting. In fragile settings where youth face challenges accessing mental health care and livelihoods support, it is critical to develop interventions that can be delivered and maintained by local communities. Lay workers can be equipped to deliver evidence-based interventions when supported by a local seed team of expert facilitators. Finally, our findings provide further evidence that the Collaborative Team Approach promotes local ownership and capacity while supporting quality improvement and sustainability of evidence-based interventions.
Availability of data and materials
The datasets generated and/or analyzed during the current study are not publicly available due continued analysis by Youth FORWARD researchers but are available from the corresponding author on reasonable request.
Collaborative team approach
Difficulties in Emotion Regulation
Dissemination & Implementation
Exploration, Preparation, Implementation, and Sustainment
Deutsche Gesellschaft für Internationale Zusammenarbeit
Longitudinal Survey of War-Affected Youth
Low- and middle-income country
National Institute of Health
Randomized control trial
Research Program on Children and Adversity
World Health Organization
World Health Organization Disability Adjustment Scale
- Youth FORWARD:
Youth Functioning and Organizational Success for West African Regional Development
Youth Readiness Intervention
Charlson F, Ommeren M v, Flaxman A, Cornett J, Whiteford H, Saxena S. New WHO prevalence estimates of mental disorders in conflict settings: a systematic review and meta-analysis. Lancet. 2019;394(10194):240–8.
Betancourt TS, Brennan RT, Rubin-Smith J, et al. Sierra Leone's former child soldiers: a longitudinal study of risk, protective factors, and mental health. J Am Acad Child Adolesc Psychiatry. 2010;49(6):606–15.
Betancourt TS, McBain R, Newnham EA, Brennan RT. Trajectories of internalizing problems in war-affected Sierra Leonean youth: examining conflict and post conflict factors. Child Dev. 2013;84(2):455–70.
Betancourt TS, Borisova II, de la Soudière M, Williamson J. Sierra Leone's child soldiers: war exposures and mental health problems by gender. J Adolesc Health. 2011;49(1):21–8.
Sharma M, Fine SL, Brennan RT, Betancourt TS. Coping and mental health outcomes among Sierra Leonean war-affected youth: results from a longitudinal study. Dev Psychopathol. 2016;(1):1–13.
JRI Research and Training Institute, Inc. Strengthening access to mental health services in Sierra Leone. https://publications.jsi.com/JSIInternet/Inc/Common/_download_pub.cfm?id=19144&lid=3 Accessed 19 Dec 2020.
International Monetary Fund. Sierra Leone Economic Development Documents: National Development Plan 2019-23. https://www.imf.org/en/Publications/CR/Issues/2019/07/09/Sierra-Leone-Economic-Development-Documents-National-Development-Plan-2019-23-47099. Accessed 22 Dec 2020.
World Bank Group Launches New Country Partnership Framework for Sierra Leone. https://www.worldbank.org/en/news/press-release/2020/06/05/world-bank-group-launches-new-country-partnership-framework-for-sierra-leone. Accessed 22 Dec 2020.
GIZ. Employment promotion programme – needs oriented qualification for youth. Factsheet. https://www.giz.de/en/downloads/giz-Employment%20Prom-Progr%20Sierra-leone_191217_web.pdf Accessed 19 Dec 2020.
GIZ. Employment Promotion Programme (EPP III) Youth Development. https://www.giz.de/en/downloads/flyer%20capacity%20building%20for%20youth-WEB.pdf. Accessed 19 Dec 2020.
Betancourt T, Newnham E, Hann K, et al. Addressing the consequences of violence and adversity: the development of a group mental health intervention for war-affected youth in Sierra Leone. In: Raynaud J, Gau S, Hodes M, editors. From Research to Practice in Child and Adolescent Mental Health. Maryland: Rowman & Littlefield; 2014. p. 157–78.
Newnham EA, McBain RK, Hann K, Akinsulure-Smith AM, Weisz J, Lilienthal GM, et al. The Youth Readiness Intervention for war-affected youth. J Adolesc Health. 2015;56(6):606–11.
Betancourt TS, McBain R, Newnham EA, et al. A behavioral intervention for war-affected youth in Sierra Leone: a randomized controlled trial. J Am Acad Child Adolesc Psychiatry. 2014;53(12):1288–97.
Ogrinc G, Davies L, Goodman D, Batalden PB, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): Revised publication guidelines from a detailed consensus process. BMJ Qual Safety. 2016;25:986–92.
Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical care. 2012;50(3):217–26.
Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implementation Sci. 2019;14(1):1.
Stadnick NA, Brookman-Frazee L, Mandell DS, Kuelbs CL, Coleman KJ, Sahms T, et al. A mixed methods study to adapt and implement integrated mental healthcare for children with autism spectrum disorder. Pilot Feasibility Study. 2019;5(1):51.
Institute for Healthcare Improvement. Quality improvement essentials toolkit. Boston: Institute for Healthcare Improvement; 2017.
Franzen SRP, Chandler C, Lang T. Health research capacity development in low and middle income countries: reality or rhetoric? A systematic meta-narrative review of the qualitative literature. BMJ Open. 2017;7(1):e012332.
Aarons G, Fettes D, Hurlburt M, et al. Collaboration, negotiation, and coalescence for interagency-collaborative teams to scale-up evidence-based practice. J Clin Child Adolesc Psychol. 2014;43(6):915–28.
Hurlburt M, Aarons GA, Fettes D, Willging C, Gunderson L, Chaffin MJ. Interagency Collaborative Team model for capacity building to scale-up evidence-based practice. Children Youth Services Rev. 2014;39:160–8.
ÜstÜn TB. World Health Organization. Measuring health and disability: manual for WHO Disability Assessment Schedule WHODAS 2.0. Geneva: World Health Organization; 2010.
Gratz KL, Roemer L. Multidimensional assessment of emotion regulation and dysregulation: development, factor structure, and initial validation of the Difficulties in Emotion Regulation Scale. J Psychopathol Behav Assess. 2004;26:41–54.
Chin WW, WmD S, Pearson AW, Stollak MJ. Perceived cohesion in small groups: adapting and testing the perceived cohesion scale in a small-group setting. Small Group Res. 1999;30(6):751–66.
Seed Team Assessment Questionnaire. Content last reviewed. Agency for Healthcare Research and Quality. Rockville; 2012. http://www.ahrq.gov/teamstepps/instructor/reference/tmassess.html
Mâsse LC, Moser RP, Stokols D, Taylor BK, Marcus SE, Morgan GD, et al. Measuring collaboration and transdisciplinary integration in team science. Am J Prev Med. 2008;35(2 Suppl):S151–60.
Frey BB, Lohmeier JH, Lee SW, Tollefson N. Measuring collaboration among grant partners. Am J Eval. 2006;27(3):383–92.
Ward AM, Regan J, Chorpita BF, Starace N, Rodriguez A, Okamura K, et al. Research Network On Youth Mental Health. Tracking evidence based practice with youth: validity of the MATCH and standard manual consultation records. J Clin Child Adolesc Psychol. 2013;42(1):44–55.
Haroz EE, Bolton P, Nguyen AJ, Lee C, Bogdanov S, Bass J, et al. Measuring implementation in global mental health: validation of a pragmatic implementation science measure in eastern Ukraine using an experimental vignette design. BMC health services research. 2019;19(1):1–1.
Software VERBI. MAXQDA 2018 [computer software]. Berlin: VERBI Software; 2017. https://www.maxqda.com
Betancourt TS, Hansen N, Farrar J, Borg RC, Callands T, Desrosiers A, et al.. Youth functioning and organizational success for West African Regional Development (Youth FORWARD): Study Protocol. PS. 2020;appi.ps.2020000.
Ministry of Youth Affairs. A blueprint for youth development Sierra Leone’s National Youth Programme. 2014-2018. 2014. http://nationalyouthcommission.sl/pdf%20files/blue%20print.pdf. Accessed 1 Feb 2022.
Lengnick-Hall R, Willging C, Hurlburt M, Fenwick K, Aarons GA. Contracting as a bridging factor linking outer and inner contexts during EBP implementation and sustainment: a prospective study across multiple U.S. public sector service systems. Implementation Sci. 2020;15(1):43.
Government of Sierra Leone Ministry of Health and Sanitation. National Health Sector Strategic Plan. 2010-2015. November 2009. http://www.ministerial-leadership.org/sites/default/files/resources_and_tools/Abridged%20NHSSP.pdf. Accessed 19 Dec 2020.
National Youth Commission, Ministry of Youth Employment and Sports. Youth Development: Sierra Leone Youth Report. 2012. http://www.sl.undp.org/content/dam/sierraleone/docs/projectdocuments/povreduction/sl_status_ofthe_youth_report2012FINAL.pdf. Accessed 19 Dec 2020.
This feasibility pilot was funded by the Omidyar Network. Research reported in this publication was supported by the National Institute of Mental Health of the National Institutes of Health under Award Number U19MH109989. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Ethics approval and consent to participate
This study was approved by the Institutional Review Board at Boston College (protocol # 18.068) and the Sierra Leone Ethics and Scientific Review Committee. An independent Data and Safety Monitoring Board contracted by the study funder also provided ethical review and oversight. Study participants provided oral consent to participant in the study and were reconsented prior to each data collection time point.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Bond, L., Farrar, J., Borg, R.C. et al. Alternate delivery platforms and implementation models for bringing evidence-based behavioral interventions to scale for youth facing adversity: a case study in West Africa. Implement Sci Commun 3, 16 (2022). https://doi.org/10.1186/s43058-022-00259-5
- Mental health
- Low-resource setting
- Conflict-affected regions
- Collaborative team approach
- Alternate delivery platforms
- Sierra Leone
- Evidence-based intervention
- Scaling out
- Employment/entrepreneurship programs