Skip to main content
Download PDF
Download ePub

Individual-level associations between implementation leadership, climate, and anticipated outcomes: a time-lagged mediation analysis

Implementation Science Communications volume 4, Article number: 75 (2023) Cite this article

Abstract

Background

Leaders can improve implementation outcomes by developing an organizational climate conducive to the implementation of evidence-based practices (EBP). This study tested the lagged associations between individual-level perceptions of implementation leadership, implementation climate, and three anticipated implementation outcomes, that is EBP acceptability, appropriateness, and feasibility.

Methods

Screening tools and treatment methods for posttraumatic stress disorder were implemented in 43 Norwegian mental health services. A sample of 494 child and adult mental health care professionals (M = 43 years, 78% female) completed surveys addressing perceptions of first-level leaders’ (n = 47) implementation leadership and their clinics’ implementation climate. Single-level structural equation models estimating both direct, indirect, and total effects were used to investigate whether perceived implementation climate mediated the association between perceived implementation leadership and perceived acceptability, appropriateness, and feasibility of screening tools and treatment methods.

Results

Regarding the treatment methods, implementation leadership was associated with therapists’ perceptions of acceptability, appropriateness, and feasibility. Implementation climate also mediated between implementation leadership and the outcomes. Regarding the screening tools, implementation leadership was not associated with the outcomes. However, implementation climate mediated between implementation leadership and therapists’ perceptions of acceptability and feasibility, but not appropriateness. Analyses with the implementation climate subscales showed stronger associations for therapists’ perceptions of the treatment methods than of screening tools.

Conclusions

Leaders may promote positive implementation outcomes, both directly and through implementation climate. With regard to the effect sizes and explained variance, results indicated that both implementation leadership and implementation climate were more strongly associated with the therapists’ perceptions of the treatment methods, implemented by one group of therapists, than the screening tools, implemented by all therapists. This may imply that implementation leadership and climate may have stronger effects for smaller implementation teams within a larger system than for system-wide implementations or when the clinical interventions being implemented are more complex rather than simple ones.

Trial registration

ClinicalTrials NCT03719651, 25 October 2018.

Peer Review reports

Background

To improve care in mental health services, a major goal is to increase the use of evidence-based practices (EBPs). Although much attention has been paid to strategies that can lead to successful implementation [1], we have a poor understanding of how and why different implementation strategies work. That is, to understand the mechanisms that affect implementation outcomes [2,3,4]. More recent attention has focused on the concepts of perceived implementation leadership and implementation climate, which are believed to be important for successful EBP implementation in health services [5,6,7]. Aarons and colleagues [7] suggest in their theory that implementation leadership will improve implementation outcomes by creating a positive strategic organizational climate to support EBP implementation. In this study, we investigate whether perceived EBP implementation climate mediates the associations between perceived implementation leadership and mental health practitioners’ perceptions of EBP acceptability, appropriateness, and feasibility.

Leading through a strategic climate

Health care leaders, and especially first-level leaders (leaders who supervise and manage frontline employees), are believed to have an important, yet poorly understood role in the implementation of EBPs [8, 9]. There are several implementation theories offering explanations for how leaders can positively influence EBP implementation. Some theories emphasize the influence of leaders’ commitment to EBP implementation on implementation effectiveness [8]. Other theories emphasize behaviors leaders can take to develop a strategic climate for implementation by being supportive, knowledgeable, proactive, and perseverant about implementing specific EBPs [10]. While some differences exist between these leadership implementation theories, they do converge on the theoretical proposition that leaders can serve as change agents in their organizations to promote implementation effectiveness.

Implementation outcomes are defined as the effects of deliberate and purposive actions to implement new practices [11, 12] and can be divided into anticipated outcomes (i.e., the likelihood the practice will be adopted or delivered) and actual outcomes (i.e., the extent the practice is adopted or delivered) [13]. Anticipated outcomes, such as practitioners’ perceptions of EBP acceptability, appropriateness, and feasibility [12], are thought to be precursors and/or determinants of actual outcomes. Findings of direct relationships between implementation leadership and implementation outcomes are sparse and inconsistent [14]. Of two studies that have examined the relationship between implementation leadership and adoption of EBPs, one study found a positive relationship [15] whereas the other did not [16].

Organization theory posits that implementation climate is assumed to operate as a mechanism between implementation leadership and implementation outcomes [7]. This is consistent with theories of climate as a mediator of outcomes in the broader organizational literature [17]. Implementation climate is a type of strategic climate defined as employees’ shared perceptions of whether the specific EBP is expected, rewarded, and supported by the organization [18, 19]. A positive implementation climate means that employees clearly understand and experience that leaders support and facilitate the use of the practice [6]. There are several policies, practices, and procedures through which employees may perceive the importance of EBP implementation in their organization. Based on the strategic climate literature, relevant implementation climate dimensions thought to impact the implementation of EBPs are as follows: (1) whether the organization recruits and selects employees that are open for change, (2) whether staff are recognized for their EBP use, (3) whether staff are recruited and selected based on their experience with EBP, (4) whether the organization focuses on EBP use, (5) whether the organization provides EBP education and training opportunities, and (6) whether the organization rewards EBP use [18].

Organizational climate has been proposed as both an individual-level construct, referring to individuals’ own perceptions of a psychological organizational climate, and an organizational-level construct, referring to employees’ agreed perceptions of the climate [20]. Implementation climate at the unit-level is often favored [5, 21]. Some have argued that to fully understand the importance of organizational climate and how it relates to other constructs, it would be preferrable to examine both individual- and unit-level climate [22, 23]. Focusing on the individual level, organizational climate can be construed as a personal/individual level factor rather than a situational or unit level factor. From leaders’ perspective, who are often those who shape the climate [7], it can be expedient to aim their leadership initiatives both at the individual and the organizational level [22]. In this study, we examined how individual leaders’ behaviors relates to individual therapists’ perceptions of acceptability, appropriateness, and feasibility of interventions, both directly and through therapists’ perceived implementation climate. To test this, implementation climate was measured at the individual level.

Although the literature is still sparse, some cross-sectional [24,25,26] and longitudinal [15, 27] studies have shown that first-level leaders’ implementation leadership are linked to EBP implementation climate. There is also growing evidence supporting that implementation climate is related to implementation outcomes such as self-reported EBP use [15, 28,29,30] and fidelity to EBP [27, 31], but these relationships were also affected by third-variables such as molar organizational climate [28] and complexity of the practice being implemented or utilized [27].

To understand whether implementation climate operates as a mechanism between implementation leadership and implementation outcomes, mediation analyses with prospective or longitudinal designs may be used [3, 32]. Two longitudinal studies have examined implementation climate as a mediator between implementation leadership and implementation outcomes [15, 27]. Williams and colleagues’ [15] study included 30 outpatient children’s mental health clinics and showed that improvements in implementation leadership significantly increased the level of EBP implementation climate, which then significantly increased average employee-reported EBP use. In a similar study including 65 schools, Williams and colleagues [27] showed that principals’ increased implementation leadership predicted higher EBP implementation climate, which in turn predicted higher fidelity to school-based EBPs for autism.

Due to sparse empirical evidence currently supporting the theoretical assumption of a leadership-climate-outcome relationship, studies using prospective designs are needed [33, 34]. An empirical rigorous approach is to examine associations between variables measured at different time points (i.e., lagged associations). In this study, the Leadership and Organizational Change for Implementation (LOCI) strategy was administered to leaders in the participating clinics [35]. LOCI is a 12-month implementation strategy that aims to improve implementation leadership in order to create a positive strategic organizational climate that supports EBP implementation [7]. Accordingly, we wanted to investigate if perceived implementation leadership measured at the time the leaders received the LOCI strategy (T1) was associated with perceived implementation climate 4 months later (T2) and perceptions of the screening tools and treatment methods at the end of the implementation period (T3). The following hypotheses were examined:

  • H1: Implementation leadership (T1) is associated with EBP acceptability, appropriateness, and feasibility 8 months later (T3).

  • H2: Implementation leadership (T1) is associated with perceived implementation climate 4 months later (T2).

  • H3: Perceived implementation climate (T2) is associated with EBP acceptability, appropriateness, and feasibility 4 months later (T3).

  • H4: Perceived implementation climate at T2 mediates the association between implementation leadership at T1 and EBP acceptability, appropriateness, and feasibility at T3.

As we know little about how implementation climate affects implementation outcomes, we also investigated whether and how the subdimensions of perceived implementation climate mediated between implementation leadership and EBP acceptability, appropriateness, and feasibility. Since few have previously examined these relationships, exploratory analyses were conducted based on an expectation that the subdimensions might differ in how they relate to both implementation leadership and implementation outcomes, which in turn might produce differences in mediation.

Methods

Setting

This study was part of a national implementation of both screening tools for exposure to potentially traumatizing events and related posttraumatic stress symptoms and evidence-based treatment methods for posttraumatic stress disorder (PTSD) in Norwegian child and adult specialized mental health clinics [35].

Participants

The study sample included 494 therapists with an average age of 43 years, of whom most were female (Table 1), clinical psychologists, and psychiatrists. Therapists completed surveys addressing their perceptions of their clinics’ (n = 43) implementation climate and their first-level leaders’ implementation leadership. The leaders’ (n = 47) average age was 50 years. Approximately half were clinical psychologists, and the rest were clinical social workers, psychiatric nurses, and psychiatrists.

Table 1 Participant demographics (N = 494)
Full size table

Implementation strategy

The implementation took place between August 2018 and May 2020. First, therapists in all clinics were trained in screening of trauma exposure and posttraumatic stress symptoms and diagnosing of PTSD. Second, a subgroup of therapists received training and supervision in either Trauma-Focused Cognitive Behavioral Therapy (TF-CBT) (child clinics) or Eye Movement Desensitization and Reprocessing (EMDR) or Cognitive Therapy for PTSD (CT-PTSD) (adult clinics), three of the most well-documented EBPs for PTSD [36, 37].

To support the implementation, the clinics were randomly allocated into one of three cohorts that received the 12-months’ LOCI strategy in the active implementation phase [38] on overlapping time points (see Table 2). During LOCI, first-level leaders (i.e., those who directly supervise therapists) participated in 5 days of training (2 days of start-up, booster session after 4 and 8 months, and graduation after 12 months) spread over 1 year. They received feedback reports based on 360° assessments on their leadership and the clinic’s implementation climate and co-developed individualized leadership development plans with LOCI trainers/coaches. First-level leaders also participated in weekly coaching calls, and organizational strategy meetings to inform the development of an organizational climate development plan (see [39] for a detailed description of LOCI).

Table 2 Implementation and survey plan
Full size table

Procedure

Therapists completed surveys addressing their perceptions of first-level leaders’ implementation leadership (T1; August 2019), the clinics’ implementation climate (T2; December 2019), and whether they found the screening tools and treatment methods acceptable, appropriate, and feasible (T3; April 2020) (Table 2). The first survey (T1, implementation leadership, N = 360) was completed when leaders from all cohorts had participated in LOCI for a minimum of 3 months. The second survey (T2; implementation climate, N = 304) was completed when cohort I was graduating from LOCI and entering the sustainment phase. Cohorts II and III were still participating in LOCI. The third survey (T3; implementation outcomes, N = 254) was completed when cohort II was graduating from LOCI and entering their sustainment phase. Cohort III was still actively participating in LOCI. Across time points, 216 therapists had responded to both T1 and T2 surveys, 176 to both T1 and T3 surveys, and 171 to both T2 and T3 surveys. In sum, 139 therapists had responded to all three time points.

Measures

The implementation leadership scale (ILS) consists of 12 items measuring leadership behavior thought to impact the EBP implementation [10] across four subdimensions. The therapists were asked to evaluate their first-level leaders’ proactive, knowledgeable, supportive, and perseverant leadership towards the implementation of EBPs for PTSD on a scale between 0 (not at all) to 4 (to a very great extent). A higher score indicates more favorable implementation leadership. ILS demonstrated excellent internal consistency (α = 0.97), in addition to good psychometric properties in the main study [40].

The implementation climate scale (ICS) consists of 18 items measuring global and six subdimensions of implementation climate [18]. The therapists were asked to evaluate their clinic’s: (1) Focus on EBP, (2) Educational support for EBP, (3) Recognition for EBP, (4) Rewards for EBP, (5) Selection for EBP, and (6) Selection for openness on a scale between 0 (not at all) to 4 (to a very great extent). The questions were directed towards EBPs for PTSD, specifically. Items are scored on a scale between 0 (not at all) to 4 (to a very great extent). A higher score indicates more favorable implementation climate. The scale showed excellent internal consistency reliability in the current study [41]. However, due to psychometric challenges as a result of financial rewards/incentives and promotions being rare in Norwegian working conditions [41], the Rewards subscale and item 9 from the Recognition subscale (more likely to be promoted) were excluded from the ICS global scale (hereby called ICS-r), resulting in 14 items.

In the subscales analyses, the Rewards subscale was included. Item 9 was excluded from the Recognition subscale, hereby called the Recognition-r subscale. Recognition-r showed excellent internal reliability (α = 0.90).

EBP acceptability, appropriateness, and feasibility of screening tools and EBP treatment methods were measured using the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM), respectively [12]. These measures were developed to measure three implementation outcomes, as indicated by Proctor and colleagues [11]. The therapists were asked to evaluate separately whether they experienced (a) the PTSD screening tools and (b) the PTSD treatment methods as acceptable, appropriate, and feasible in their clinic. Items were scored from 1 (completely disagree) to 5 (completely agree). A higher score indicates more favorable acceptability, appropriateness, and feasibility. To account for potential measurement error due to few indicators, the outcome measures were included in the model as latent variables. Standardized factor loadings are shown in Table 3. All loadings were above 0.6 and statistically significant.

Table 3 Standardized factor loadings for the latent outcome variables
Full size table

Control variables in all models included LOCI cohort, age, sex, years of work experience, and organizational tenure. LOCI cohort was used as a control variable for both the leadership and the implementation outcomes and the implementation climate and the outcomes. As for age, years of work experience, and tenure, these may be indicative of the employees’ professional experience and managerial skills. Sex is included as women make up the majority of the mental health services workforce.

Statistical analyses

Study hypotheses were investigated with structural equation modeling (SEM) in Mplus 8.3 [42] adjusting for the nested data structure (TYPE=COMPLEX) using full information maximum likelihood estimation with robust standard errors (MLR). Measurement error may lead to both over- and underestimation of path coefficients in models investigating mediation [43]. As potential measurement error can be considerable when the number of indicators is low, we used latent variable modeling for the measures with few indicators (i.e., AIM, IAM, and FIM, and the ICS subscales). The ILS (12 items) and the ICS-r (14 items) both have many indicators and were included as manifest variables based on mean scores calculated by averaging across items to reduce model complexity. Since aggregation indices were below recommended thresholds for aggregating data, implementation climate was analyzed at the individual level [44]. We estimated a series of mediation models with direct, indirect (via perceived implementation climate), and total effects of implementation leadership on therapists’ perceptions of EBP acceptability, appropriateness, and feasibility (Fig. 1). Specifically, each outcome measure at T3 was regressed on implementation leadership at T1 (path c’, i.e., direct path) and perceived implementation climate at T2 (path b). Perceived implementation climate at T2 was also regressed on implementation leadership at T1 (path a). The indirect effect of implementation leadership was calculated as the product of paths a and b (indirect = a × b). We ran separate models for each of the three outcome measures, for each intervention (i.e., screening tools, treatment methods), and each mediator variable (i.e., perceived global and subscale implementation climate), resulting in six models for each mediator variable. Since EBP acceptability, appropriateness, and feasibility were only measured at the end of the implementation period, we were not able to control for baseline levels in the outcome variables. In models predicting perceptions of treatment methods, only therapists who were trained in one of the treatment methods were included in the analyses.

Fig. 1
figure 1

Conceptual model of relationships between implementation leadership, implementation climate, and anticipated implementation outcomes

Full size image

Distributions of indirect effects are known to violate the normality assumption on which frequentist methods are based. Additionally, upon inspection, distributions of all our outcome variables were skewed. Thus, a bootstrapping procedure—repeatedly drawing 1000 samples from the original sample to estimate sampling distributions—was used instead of traditional hypothesis testing. To test study hypotheses, we utilized 95% confidence intervals (CIs) based on 2.5. and 97.5 percentile values of the sampling distributions for each parameter. Parameters were considered significantly different from zero when their 95% CIs did not cross zero.

Results

On average, the therapists rated implementation leadership and perceived implementation climate at 2.51 and 2.37, respectively. Acceptability, appropriateness, and feasibility scores for both screening tools and treatment methods were generally very high (Table 4).

Table 4 Descriptive statistics for the study variables
Full size table

Associations between implementation leadership and implementation outcomes

Results from the main analyses with implementation climate as mediator are displayed in Table 5. Total effects of implementation leadership on acceptability, appropriateness, and feasibility only partially supported Hypothesis 1 that implementation leadership assessed at T1 would be associated with EBP acceptability, appropriateness, and feasibility assessed 8 months later (total parameters). Specifically, all estimates were positive but statistically significant only for acceptability, appropriateness, and feasibility of the treatment methods.

Table 5 Parameter estimates for direct effects, indirect effects (through implementation climate), and total effects of implementation leadership on anticipated outcomes of screening tools and treatment methods at T3
Full size table

Associations between implementation leadership and perceived implementation climate

The results indicated significant positive associations between implementation leadership and implementation climate among all therapists and in the subsample of therapists trained in the treatment methods (Table 5, a parameters). The results also indicated significant positive associations between implementation leadership and all subscales in the ICS, except the Rewards subscale where the estimates were close to zero and non-significant (Tables 6 and 7, a parameters). Thus, the results supported hypothesis 2 that implementation leadership assessed at T1 would be associated with perceived implementation climate assessed 4 months later (T2).

Table 6 Parameter estimates for direct and indirect effects (through implementation climate subscales) of implementation leadership on anticipated outcomes of screening tools at T3 (N = 494)
Full size table
Table 7 Parameter estimates for direct and indirect effects (through implementation climate subscales) of implementation leadership on anticipated outcomes of trauma treatment methods at T3 (N = 184)
Full size table

Associations between implementation climate and EBP acceptability, appropriateness, and feasibility

The results indicated significant positive associations between implementation climate and acceptability and feasibility of the screening tools and acceptability, appropriateness, and feasibility of the treatment methods (Table 5, b parameters). In the model predicting appropriateness of the screening tools, the estimate was positive but non-significant. Thus, the results provided full support for hypothesis 3 that implementation climate assessed at T2 would be associated with EBP acceptability, appropriateness, and feasibility of treatment methods assessed 4 months later (T3) and partial support regarding the screening tools.

Associations between specific aspects of perceived implementation climate and EBP acceptability, appropriateness, and feasibility

None of the specific aspects of perceived implementation climate (measured by subscales in the ICS) were consistently associated with the implementation outcomes (acceptability, appropriateness, and feasibility of screening tools and treatment methods) across all models. In models predicting perceptions of screening tools (Table 6, b parameters), associations were positive and significant between the Focus subscale and appropriateness and feasibility and between the Recognition-r subscale and acceptability and feasibility. The remaining estimates were non-significant.

In models predicting perceptions of treatment methods (Table 7, b parameters), the associations between the Focus subscale and acceptability were positive and significant, and Recognition-r was positively and significantly associated with all three outcomes. The remaining estimates were non-significant.

Mediated associations between implementation leadership and EBP acceptability, appropriateness, and feasibility through implementation climate

The results indicated that perceived implementation climate significantly mediated parts of the associations between implementation leadership and the acceptability and feasibility of screening tools and those between implementation leadership and the acceptability, appropriateness, and feasibility of treatment methods (Table 5, indirect parameters). In the model predicting appropriateness of screening tools, the estimate was positive but non-significant. Thus, the results provided support for hypothesis 4 that implementation climate (T2) would mediate the associations between implementation leadership (T1) and therapists’ acceptability, appropriateness, and feasibility of the treatment methods (T3) and therapists’ acceptability and feasibility of the screening tools. The effect sizes and the explained variance (Table 5, R-squared) were higher for the treatment methods than the screening tools.

Mediated associations between implementation leadership and EBP acceptability, appropriateness, and feasibility through specific aspects of perceived implementation climate

Regarding the screening tools, the results indicated that parts of the associations between implementation leaderships and outcomes were significantly mediated by the Focus subscale (appropriateness and feasibility) and the Recognition-r subscale (acceptability and feasibility) (Table 6, indirect parameters). Regarding the treatment methods, the results also indicated significant paths through the Focus subscale (acceptability) and the Recognition-r subscale (acceptability, appropriateness, and feasibility) (Table 7, indirect parameters). For the Recognition-r subscale, mediation was partial, with a significant direct path included. The results also indicated direct paths between implementation leadership and acceptability, appropriateness, and feasibility in the Educational support, Rewards, Selection for EBP, and Selection for openness models (Table 7, direct parameters). The Educational support, Rewards, Selection for EBP, and Selection for openness subscales did not significantly mediate associations between implementation leadership and any of the outcomes.

Discussion

All four study hypotheses were fully confirmed for the treatment methods but only partially for the screening tools. Implementation leadership at T1 was significantly associated with therapists’ perceptions of acceptability, appropriateness, and feasibility of treatment methods, but not screening tools 8 months later (hypothesis 1). Implementation leadership at T1 was also significantly associated with perceived implementation climate 4 months later (hypothesis 2). Moreover, perceived implementation climate at T2 was associated with therapists’ perceptions of acceptability and feasibility (but not appropriateness) of the screening tools, and acceptability, appropriateness, and feasibility of the treatment methods 4 months later (hypothesis 3). Overall, implementation climate mediated the parts of the associations between implementation leadership and therapists’ perceptions of acceptability and feasibility (but not appropriateness) of screening tools, and acceptability, appropriateness, and feasibility of treatment methods (hypothesis 4).

The mediating role of perceived implementation climate between implementation leadership and various implementation outcomes supports previous findings implying leaders have a key role in the implementation process [15, 27] and that leaders can shape the implementation climate to support implementation of EBPs [45]. Although implementation climate mediated the association between implementation leadership and implementation outcomes of both screening tools and treatment methods, there was a tendency for the climate to matter more for therapists’ perceptions of the treatment methods than of the screening tools, both with regard to the larger effect sizes and explained variance.

Analyses with the implementation climate subscales showed a similar pattern with implementation leadership, such that associations were stronger for therapists’ perceptions of treatment methods than of screening tools. Only a subgroup of therapists was trained in the treatment methods, while the screening tools were implemented by all therapists. It might be that as screening is implemented by everyone, a therapist can more easily observe or seek guidance from a peer to inform their implementation, compared to being a smaller number of therapists implementing a more complex treatment method. Having fewer opportunities to observe or seek guidance from peers may require more support from the leaders. Thus, their leaders’ adaptation to target each therapist’s individual needs, both directly and through the implementation climate, might have been more decisive for the therapists’ perceived acceptability, appropriateness, and feasibility of the treatment methods than the screening tools.

Further exploration of which implementation climate factors influence implementation outcomes can give leaders a better understanding of what to emphasize during an EBP implementation. In this study, there are two implementation climate subscales associated with therapists’ reports of acceptability, appropriateness, and feasibility of treatment methods and screening tools, namely Focus and Recognition-r. All factors were measured during an active implementation phase [38], where leaders through their LOCI follow-up worked a lot on focusing on the methods and tools being implemented and recognizing the therapists’ achievements. This may have influenced the therapists’ perceptions of Focus and Recognition as being important in the active implementation phase and perhaps more than the other subscales. At the same time, it is surprising that subscales such as Educational support did not obtain significant results. The results are also inconsistent for each of the implementation outcomes, except the Recognition-r subscale on treatment methods. However, the absence of findings does not necessarily mean that there is no connection. It might be that different implementation climate factors are more important than others depending on different implementation phases, outcomes, or what is being implemented. This should therefore be further investigated.

The study supports the implementation leadership model [10], hypothesizing that leaders can achieve better implementation outcomes when they are supportive, knowledgeable, proactive, and perseverant when implementing specific EBPs.

This study includes a complex design with many clinics. Although the clinics were not randomized based on the current factors, time-lagged analyses make it possible to describe the precise sequence of operations through which an effect may occur [3]. However, we only measured each concept at one timepoint and consequently could not investigate possible changes in implementation leadership, implementation climate, and anticipated implementation outcomes over time. The concepts may also have been affected by variables not measured in the study, such as leaders’ commitment and engagement in activities that promote implementation [8], molar climate [28], or the complexity of the practice [27]. Thus, results should be interpreted with caution and does not permit any definitive claims about causal relationships.

The implementation leadership and implementation climate scales were measured at the individual level. As perceptions of acceptability, appropriateness, and feasibility can be considered as individual-level phenomena, it seems reasonable that the individual’s understanding of the perceived implementation climate is related to the individuals’ understanding of the screening tools and the treatment methods [20]. Some argue that implementation climate is a unit-level construct, measuring collective perceptions of the climate in an organization [21]. Thus, relationships involving implementation climate should, ideally, be measured and analyzed in a multilevel fashion to capture both within- and between-unit effects. However, in our study, such models did not converge. Aggregation indices for implementation climate were also below recommended thresholds for aggregating data [44]. Thus, this study focused on studying associations on the individual-level. For future research, similar analyses with aggregated data or multi-level structural equation models that estimate both within- and between-effects could be useful.

We also acknowledge that the Rewards subscale and item 9 from the Recognition subscale were excluded from the ICS global scale due to lack of cultural relevance in the Norwegian health system context [41]. Comparisons with other studies should therefore be made cautiously.

In this study, we have measured anticipated implementation outcomes in the form of therapists’ perceived EBP acceptability, appropriateness, and feasibility. These are assumed to predict actual implementation outcome [13]. However, more research should be done on the leaders’ impact on actual outcomes.

Conclusion

Results from the study suggest that leaders can achieve better implementation outcomes both directly and through implementation climate. Implementation leadership and climate seemed to be of higher importance for the smaller group of therapists implementing more complex treatment methods, than for the whole group implementing somewhat simpler screening tools. More research is needed to understand which factors within the implementation climate affect different implementation outcomes, for whom, and under what conditions.

Availability of data and materials

The dataset used in the current study is available from the corresponding author on reasonable request.

Abbreviations

EBP:

Evidence-based practice

LOCI:

Leadership and organizational change for implementation

PTSD:

Posttraumatic stress disorder

References

  1. Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Kirchner JE. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10(1):1–14.

  2. Lewis CC, Boyd MR, Walsh-Bailey C, Lyon AR, Beidas R, Mittman B, et al. A systematic review of empirical studies examining mechanisms of implementation in health. Implement Sci. 2020;15(1):21.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Williams NJ. Multilevel mechanisms of implementation strategies in mental health: integrating theory, research, and practice. Adm Policy Ment Health Ment Health Serv Res. 2016;43:783–98.

  4. Vejnoska SF, Mettert K, Lewis CC. Mechanisms of implementation: An appraisal of causal pathways presented at the 5th biennial Society for Implementation Research Collaboration (SIRC) conference. Implement Res Pract. 2022;3. https://doi.org/10.1177/26334895221086271.

  5. Weiner BJ, Belden CM, Bergmire DM, Johnston M. The meaning and measurement of implementation climate. Implement Sci. 2011;6(1):78.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Aarons GA, Farahnak LR, Ehrhart MG, Sklar M. Aligning leadership across systems and organizations to develop strategic climate to for evidence-based practice implementation. Ann Rev Public Health. 2014;35:255–74.

    Article  Google Scholar 

  7. Aarons GA, Ehrhart MG, Moullin JC, Torres EM, Green AE. Testing the leadership and organizational change for implementation (LOCI) intervention in substance abuse treatment: a cluster randomized trial study protocol. Implement Sci. 2017;12(1):29.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Birken SA, Lee S-YD, Weiner BJ, Chin MH, Schaefer CT. Improving the effectiveness of health care innovation implementation: middle managers as change agents. Medical Care Res Rev. 2013;70(1):29–45.

    Article  Google Scholar 

  9. Stetler CB, Ritchie JA, Rycroft-Malone J, Charns MP. Leadership for evidence-based practice: strategic and functional behaviors for institutionalizing EBP. Worldviews Evid-Based Nurs. 2014;11(4):219–26.

    Article  PubMed  Google Scholar 

  10. Aarons GA, Ehrhart MG, Farahnak LR. The Implementation Leadership Scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci. 2014;9(1):45.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Proctor EK, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Mental Health Mental Health Serv Res. 2011;38(2):65–76.

    Article  Google Scholar 

  12. Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Damschroder LJ, Reardon CM, Opra Widerquist MA, Lowery J. Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): the CFIR Outcomes Addendum. Implement Sci. 2022;17(1):7.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Meza RD, Triplett NS, Woodard GS, Martin P, Khairuzzaman AN, Jamora G, et al. The relationship between first-level leadership and inner-context and implementation outcomes in behavioral health: a scoping review. Implement Sci. 2021;16(1):1–21.

    Article  Google Scholar 

  15. Williams NJ, Wolk CB, Becker-Haimes EM, Beidas RS. Testing a theory of strategic implementation leadership, implementation climate, and clinicians’ use of evidence-based practice: a 5-year panel analysis. Implement Sci. 2020;15(1):10.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Guerrero EG, Frimpong J, Kong Y, Fenwick K, Aarons GA. Advancing theory on the multilevel role of leadership in the implementation of evidence-based health care practices. Health Care Manag Rev. 2020;45(2):151–61.

  17. Ehrhart MG. Leadership and procedural justice climate as antecedents of unit-level organizational citizenship behavior. Personnel Psychol. 2004;57(1):61–94.

    Article  Google Scholar 

  18. Ehrhart M, Aarons GA, Farahnak LR. Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS). Implement Sci. 2014;9:157.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Klein KJ, Conn AB, Sorra JS. Implementing computerized technology: an organizational analysis. J Appl Psychol. 2001;86(5):811–24.

    Article  CAS  PubMed  Google Scholar 

  20. Schulte M, Ostroff C, Kinicki AJ. Organizational climate systems and psychological climate perceptions: a cross-level study of climate-satisfaction relationships. J Occup Organ Psychol. 2006;79(4):645–71.

    Article  Google Scholar 

  21. Ehrhart M, Schneider BH, Macey WH. Organizational climate and culture: an introduction to theory, research, and practice. New York: Routledge; 2014.

    Google Scholar 

  22. Schyns B, van Veldhoven M, Wood S. Organizational climate, relative psychological climate and job satisfaction. Leadersh Organ Dev J. 2009;30(7):649–63. https://doi.org/10.1108/01437730910991664.

  23. James LR, Choi CC, Ko C-HE, McNeil PK, Minton MK, Wright MA, et al. Organizational and psychological climate: a review of theory and research. Eur J Work Organ Psychol. 2008;17(1):5–32.

    Article  Google Scholar 

  24. Shuman CJ, Liu X, Aebersold ML, Tschannen D, Banaszak-Holl J, Titler MG. Associations among unit leadership and unit climates for implementation in acute care: a cross-sectional study. Implement Sci. 2018;13(1):1–10.

    Article  Google Scholar 

  25. Lyon AR, Cook CR, Brown EC, Locke J, Davis C, Ehrhart M, Aarons GA. Assessing organizational implementation context in the education sector: confirmatory factor analysis of measures of implementation leadership, climate, and citizenship. Implement Sci. 2018;13:1–14.

  26. Torres EM, Ehrhart MG, Beidas RS, Farahnak LR, Finn NK, Aarons GA. Validation of the Implementation Leadership Scale (ILS) with supervisors’ self-ratings. Commun Mental Health J. 2018;54(1):49–53.

    Article  Google Scholar 

  27. Williams NJ, Hugh ML, Cooney DJ, Worley JA, Locke J. Testing a theory of implementation leadership and climate across autism evidence-based interventions of varying complexity. Behav Ther. 2022;53(5):900–12.

  28. Williams NJ, Ehrhart MG, Aarons GA, Marcus SC, Beidas RS. Linking molar organizational climate and strategic implementation climate to clinicians’ use of evidence-based psychotherapy techniques: cross-sectional and lagged analyses from a 2-year observational study. Implement Sci. 2018;13(1):85.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Williams NJ, Ramirez NV, Esp S, Watts A, Marcus SC. Organization-level variation in therapists’ attitudes toward and use of measurement-based care. Adm Policy Mental Health Mental Health Serv Res. 2022;49(6):927–42.

    Article  Google Scholar 

  30. Turner K, Trogdon JG, Weinberger M, Stover AM, Ferreri S, Farley JF, et al. Testing the organizational theory of innovation implementation effectiveness in a community pharmacy medication management program: a hurdle regression analysis. Implement Sci. 2018;13(1):105.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Williams NJ, Becker-Haimes EM, Schriger SH, Beidas RS. Linking organizational climate for evidence-based practice implementation to observed clinician behavior in patient encounters: a lagged analysis. Implement Sci Commun. 2022;3(1):1–14.

    Article  Google Scholar 

  32. Kraemer HC, Kazdin AE, Offord DR, Kessler RC, Jensen PS, Kupfer DJ. Coming to terms with the terms of risk. Arch Gen Psychiatry. 1997;54(4):337–43.

    Article  CAS  PubMed  Google Scholar 

  33. Powell BJ, Fernandez ME, Williams NJ, Aarons GA, Beidas RS, Lewis CC, McHugh SM, Weiner BJ. Enhancing the Impact of Implementation Strategies in Healthcare: a research agenda [Perspective]. Front Public Health. 2019;7. https://doi.org/10.3389/fpubh.2019.00003.

  34. Lewis CC, Powell BJ, Brewer SK, Nguyen AM, Schriger SH, Vejnoska SF, et al. Advancing mechanisms of implementation to accelerate sustainable evidence-based practice integration: protocol for generating a research agenda. BMJ Open. 2021;11(10):e053474.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Egeland KM, Skar A-MS, Endsjø M, Laukvik EH, Bækkelund H, Babaii A, et al. Testing the leadership and organizational change for implementation (LOCI) intervention in Norwegian mental health clinics: a stepped-wedge cluster randomized design study protocol. Implement Sci. 2019;14(1):28.

    Article  PubMed  PubMed Central  Google Scholar 

  36. International Society for Traumatic Stress Studies. Posttraumatic Stress Disorder Prevention and Treatment Guidelines. Methodology and Recommendations. . Committee IG, editor. Oakbrook Terrace: ISTSS; 2018.

  37. National Institute for Health and Care Excellence. Post-traumatic stress disorder. NICE guideline. 2018.

    Google Scholar 

  38. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Mental Health Mental Health Serv Res. 2011;38(1):4–23.

    Article  Google Scholar 

  39. Skar AMS, Braathu N, Peters N, Bækkelund H, Endsjø M, Babaii A, Egeland KM. A stepped-wedge randomized trial investigating the effect of the Leadership and Organizational Change for Implementation (LOCI) intervention on implementation and transformational leadership, and implementation climate. BMC health Serv Res. 2022;22(1):1–15.

  40. Braathu N, Laukvik EH, Egeland KM, Skar A-MS. Validation of the Norwegian versions of the Implementation Leadership Scale (ILS) and Multifactor Leadership Questionnaire (MLQ) in a mental health care setting. BMC Psychol. 2022;10(1):25. https://doi.org/10.1186/s40359-022-00725-8.

  41. Peters N, Borge RH, Skar A-MS, Egeland KM. Measuring implementation climate: psychometric properties of the Implementation Climate Scale (ICS) in Norwegian mental health care services. BMC Health Serv Res. 2022;22(1):23.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Muthén LK, Muthén BO. Mplus User’s Guide. 8th ed. Los Angeles: Muthén & Muthén; 2017.

    Google Scholar 

  43. Cole DA, Preacher KJ. Manifest variable path analysis: potentially serious and misleading consequences due to uncorrected measurement error. Psychol Methods. 2014;19(2):300.

    Article  PubMed  Google Scholar 

  44. Bliese PD. Within-group agreement, non-indepenence, and reliability - implications for data aggregation and analysis. In: Klein KJ, Kozlowski SWJ, editors. Multilevel Theory, Research, and Method in Organizations. San Fransisco: Jossey-Bass; 2000. p. 349–81.

    Google Scholar 

  45. Schein EH. Organizational culture and leadership. San Fransisco: John Wiley & Sons; 2010.

    Google Scholar 

Download references

Acknowledgements

We thank all the clinics that participated and provided data for the study.

Funding

The current study is part of a national implementation of EBPs for posttraumatic stress disorder (PTSD) in Norwegian specialized mental health clinics for children and adults. The implementation project was funded by the Norwegian Ministry of Health and Care Services. The funders had no part in the study design, data collection and analysis, or in writing the manuscript.

Author information

Authors and Affiliations

  1. Norwegian Centre for Violence and Traumatic Stress Studies (NKVTS), Gullhaugveien 1, 0484, Oslo, Norway

    Karina Myhren Egeland, Randi Hovden Borge, Nadina Peters, Harald Bækkelund, Nora Braathu & Ane-Marthe Solheim Skar

  2. National Institute of Occupational Health, Gydas vei 8, 0363, Oslo, Norway

    Randi Hovden Borge

  3. Department of Psychiatry, University of California, San Diego, 9500 Gilman Drive (0812), La Jolla, CA, 92093-0812, USA

    Marisa Sklar & Gregory A. Aarons

  4. Diego ACTRI Dissemination and Implementation Science Center, UC San, 9500 Gilman Drive, La Jolla, CA, 92093, USA

    Marisa Sklar & Gregory A. Aarons

  5. Child and Adolescent Services Research Center, 3665 Kearny Villa Rd., Suite 200N, San Diego, CA, 92123, USA

    Marisa Sklar & Gregory A. Aarons

Authors
  1. Karina Myhren Egeland
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Randi Hovden Borge
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nadina Peters
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Harald Bækkelund
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nora Braathu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marisa Sklar
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Gregory A. Aarons
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ane-Marthe Solheim Skar
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

KME and AMSS were principal investigators. KME and RHB developed the manuscript outline. RHB conducted the analyses. KME drafted the manuscript. AMSS, RHB, HB, NB, NP, GAA, and MS edited and reviewed the manuscript. All authors participated in the conceiving of the study, and all read and approved the final manuscript.

Corresponding author

Correspondence to Karina Myhren Egeland.

Ethics declarations

Ethics approval and consent to participate

The current study is approved by the Norwegian Centre for Research Data (No. 60036/3/LH and 60059/3/OOS). All participants consented to participate in the study.

Consent for publication

Not applicable.

Competing interests

Gregory A. Aarons is a member of the journal's Editorial Board. The authors declare that they have no other competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Egeland, K.M., Borge, R.H., Peters, N. et al. Individual-level associations between implementation leadership, climate, and anticipated outcomes: a time-lagged mediation analysis. Implement Sci Commun 4, 75 (2023). https://doi.org/10.1186/s43058-023-00459-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s43058-023-00459-7

Keywords

Implementation Science Communications

ISSN: 2662-2211