Skip to main content
Download PDF
Download ePub

Implementation and modification of an organizational-level intervention: a prospective analysis

Implementation Science Communications volume 3, Article number: 59 (2022) Cite this article

Abstract

Background

Modifications to interventions can jeopardize intervention outcomes. Pre-existing perceived barriers and facilitators to the intervention arising in the implementation preparation phase may help explain why modifications to the intervention may occur during the implementation phase. This two-site comparative case study describes modifications made to a complex organizational-level intervention and examines how known implementation science factors may have enabled such changes to occur.

Methods

Northwestern Medicine’sTM Hispanic Kidney Transplant Program (HKTP) is a culturally competent transplant center-based intervention designed to reduce disparities in living donor kidney transplantation among Hispanics. In-depth qualitative interviews and discussions were longitudinally conducted with transplant stakeholders (i.e., physicians, administrators, clinicians) at two kidney transplant programs with large Hispanic populations during implementation preparation and implementation phases. The Consolidated Framework for Implementation Research (CFIR) guided interview design and qualitative analysis, and Stirman’s Framework for Reporting Adaptations and Modifications-Expanded (FRAME) guided modification classification.

Results

Across sites, 57 stakeholders participated in an interview, group discussion, and/or learning collaborative discussion. Site-B made more modifications than Site-A (n = 29 versus n = 18). Sites differed in the proportions of delaying/skipping (Site-A 50% versus Site-B 28%) and adding (Site-A 11% versus Site-B 28%) but had comparable substituting (Site-A 17% versus Site-B 17%) and tweaking (Site-A 17% versus Site-B 14%) modification types. Across sites, the transplant team consistently initiated the most modifications (Site-A 66%; Site-B 62%). While individuals initiated slightly more modifications at Site-B (21% versus Site-A 17%), institutions instigated proportionately slightly more modifications at Site-A (17% versus Site-B 10%). CFIR inner setting factors (i.e., structural characteristics, culture, available resources, implementation climate) that prominently emerged during the implementation preparation phase explained similarities and differences in sites’ modification numbers, types, and agents in the implementation phase.

Conclusion

Organizations implementing a culturally competent care intervention made modifications. CFIR inner setting factors emerging in the implementation preparation phase largely explained similarities and differences in study sites’ modifications. Identifying factors contributing to modifications may help institutions become better prepared to implement an intervention by addressing known factors in advance, which may foster greater fidelity leading to desired outcomes.

Trial registration

ClinicalTrials.govNCT03276390. We registered the study retrospectively on 9-7-17.

Peer Review reports

Background

Translating interventions into practice is challenging because intervention components may not fit within unique organizational contexts. The modifications made to better fit the intervention with the context can undermine intervention fidelity [1]. Implementation fidelity, the extent to which intervention is carried out as intended, is essential for evidence-based interventions to achieve their desired outcomes [2]. Determining an acceptable amount and type of intervention modifications remains a challenge of balancing fidelity with ensuring that interventions are implemented with the best fit [3].

Following Stirman et al., we define “modification” as planned or unintentional “changes to the design or delivery of an intervention.” [4] We distinguish modifications from adaptations, which are deliberately made before initiating intervention implementation in new settings or populations [5, 6]. Modifications can be classified in terms of who makes them, what intervention component is modified, the nature of the modification, and the reason for the modification [4, 7]. Examining modifications is important for understanding the potential for positive and negative effects on desired outcomes [8]. Modifications may be harmful or helpful to achieving the intervention’s outcomes, depending on whether intervention components are deemed necessary or proscribed [4, 8]. Assessing the types and levels of modifications can offer greater insights than fidelity monitoring alone. Such examination can demonstrate how modifications affect desired intervention outcomes and illuminate ways to enhance intervention fit within specific settings [4].

Relatively little is known about modifications made during intervention implementation. Studies have examined modifications by culturally adapting prevention interventions to improve fit with clients and optimize desired outcomes [6, 9,10,11]. The most common types of cultural adaptations are tailoring and adding elements [12]. Systematic reviews and other research on evidence-based interventions targeting the individual provider, staff coordination, and patient levels found that most modifications involved tailoring/tweaking/refining and adding components and were instigated by individual practitioners [4, 12]. One study found that changes or deletions were the most common modifications to a community-based intervention [13].

Identifying determinants of modifications may reveal which intervention components worked in achieving desired outcomes, the circumstances by which they worked [14], and help to prevent modifications from occurring in future research. Stirman’s Framework for Reporting Adaptations and Modifications-Expanded (FRAME) identifies reasons for modifications, which include both the intent of modifying (e.g., increase reach, improve feasibility, reduce costs) and contextual factors driving the decision to make the modifications (e.g., sociopolitical, organizational, provider, and recipient) [7].

However, little research has examined how organizational factors, identified by FRAME or other implementation science theories and frameworks, explain intervention modifications. In one of the few exceptions, Lau et al. examined therapist and practice-level factors that predict different adaptation types. The researchers examined whether the practice influenced augmenting or reducing/reordering adaptations but did not examine the features of those practices that explain differences [15]. Recently, an outcomes addendum was added to the Consolidated Framework for Implementation Research (CFIR) to account for both implementation and intervention outcomes [16]. Only a few studies have previously used the CFIR to examine outcomes [16]. For example, Hasson et al. evaluated factors affecting the fidelity of a complex care intervention for frail elderly involving collaboration between an emergency department nurse, hospital ward staff, and a multi-professional team [17]. Using CFIR, they found that high staff enthusiasm for the project and team responsiveness prompted many additions. Contextual factors, like the lack of financial resources, resulted in less staff attending care planning meetings, while staff with prior positive experiences in similar projects increased enthusiasm for the project [17].

While prior research on intervention modifications has focused primarily on simple interventions that intervene at the patient or physician level, few studies have examined modifications occurring at the organizational level [4]. Healthcare organizational-level interventions are fundamentally complex because they typically involve multiple components at the patient, provider, micro-system (healthcare team), and system levels [18,19,20]. US hospitals vary (e.g., hospital systems, hospital networks, academic, community) [21] and maintain different organizational structures, cultures, and priorities. Consequently, modifications made to complex organizational interventions raise uncertainty over whether intervention effects can be generalizable to other organizational settings [22]. Scholars have therefore called for a broader understanding of modifications made to “fit provider characteristics, organizational contexts, and service settings (e.g., historical, political, and economic contexts)” [23].

Although recently proposed by Damschoder and colleagues, studies have not yet evaluated how CFIR factors, identified in the preparation phase, engender modifications during the implementation phase [16]. Understanding perceived barriers and facilitators to the intervention arising in the pre-implementation period may explain why certain types of intervention modifications arise during the implementation process, as recommended [14]. A prospective, longitudinal assessment of modifications before implementing interventions can illuminate contextual precursor factors that may enable modifications to occur [14].

Objectives

This research is part of a large, multi-part 5-year study examining the implementation and effectiveness of Northwestern Medicine’sTM Hispanic Kidney Transplant Program. This paper uniquely examines the types and characteristics of modifications made to a complex, organizational-level, culturally competent care intervention. It describes how implementation science factors in the implementation preparation phase may have enabled modifications to occur during the first year of the implementation phase.

Methods

Frameworks

The CFIR [24, 25] was used to describe the contextual factors contributing to modifications and generate comparative site descriptions [26]. CFIR synthesizes myriad implementation science frameworks designed to understand and explain factors influencing implementation outcomes. CFIR comprises five domains: Intervention characteristics, Outer setting, Inner setting, Characteristics of individuals, and Process. Subsequently, the CFIR addendum conceptualized outcomes for use with the CFIR. This study focuses on the connection between CFIR factors and implementation outcomes. Implementation outcomes describe “the success or failure of the implementation” (p. 3) [16]. However, rather than examining implementation success, we take a more nuanced tact, examining modification as an implementation outcome.

Stirman’s FRAME was used to classify and determine the intent of modifications [4]. The FRAME is a coding system designed to classify intervention modifications to understand how modifications influence intervention implementation. The FRAME consists of eight categories: (1) when in the implementation process the adaptation occurred, (2) whether the adaptation was planned or unplanned, (3) who participated in the decision to modify, (4) what is modified, (5) level of delivery the adaptation was made, (6) the type or nature of context or content-level modifications (e.g., tweaking, adding, skipping), (7) whether the adaptation was fidelity consistent, and (8) the reasons for the adaptation, including (a) the goal or intent of the adaptation and (b) contextual factors that influenced the adaptation. We adapted the categories within the “Who participated in the decision to modify” to the HKTP intervention, as Stirman and colleagues recommended [7], by reclassifying the categories as “individual” (i.e., one stakeholder), “transplant team,” and “institution” (i.e., institutional leadership, organizational policies). However, we did not include Stirman’s categories “level of delivery the adaptation is made”, "training and evaluation", and subcategory “contextual factors that influenced the adaptation” because they did not apply to the intervention and CFIR covered contextual factors, respectively. Although the FRAME’s contextual factors overlap with CFIR constructs, we used CFIR because it accounts for a broader array of reasons for modifications that are not addressed by FRAME, including structural characteristics, culture, self-efficacy, and compatibility.

Overall study design

We conducted a type-2 hybrid effectiveness-implementation study design with a pre-post intervention evaluation to assess the impact of the HKTP at two intervention sites (Site-A and Site-B) and two matched control sites. The pre-implementation period spanned from January 1, 2011 to December 31, 2016. The intervention period spanned from January 1, 2017 to March 15, 2020, and ended prematurely due to the COVID-19 pandemic. Our previous research [27] evaluated the impact of the HKTP intervention, including fidelity, on the likelihood of receiving a living donor kidney transplant. The study used an implementation-effectiveness hybrid design and included 2063 recipients. Site-A exhibited greater fidelity to the intervention than Site-B. The HKTP improved the LDKT rates for Hispanic patients at Site-A by 47%. The intervention had no effect at Site-B. Additional findings are described elsewhere [27].

The HKTP intervention

Northwestern Medicine’s™ Hispanic Kidney Transplant Program (HKTP) is a complex healthcare organizational intervention [26] designed to increase live donor kidney transplant (LDKT) rates among Hispanics. The disproportionately lower LDKT rates by Hispanics and other minority groups are attributed to recipient donor, healthcare provider, and health system factors [28]. The HKTP addresses these factors, including Hispanics’ lack of kidney transplant knowledge, cultural beliefs about transplantation, shortage of bilingual staff, and lack of culturally competent care [29,30,31]. An additional file describes the HKTP components in detail [see Additional file 1]. A detailed description of procedures, materials, and processes used in the intervention can be found elsewhere [26, 32, 33].

Present study design

We extend this previous work [27] by conducting a comparative case study to examine the influence of CFIR pre-implementation factors on the types and characteristics of modifications made to a complex organization-level intervention during the first year of implementation. We assessed stakeholders’ shared perceptions of barriers and facilitators to the HKTP pre-implementation period by highlighting how stakeholders made sense of the intervention and their roles in implementing it within their broader organizational cultural context [34]. Qualitative data were valuable for generating in-depth insights into intervention implementation and modification characterization [34]. We used the Template for Intervention Description and Replication [35] and the Consolidated Criteria for Reporting Qualitative research for quality reporting [36]. Northwestern University’s Institutional Review Board granted study approval (STU00201331). Written and verbal informed consent was obtained from individuals participating in one-on-one interviews and group discussions, respectively.

Setting and participants

The HKTP intervention was implemented at two U.S. kidney transplant programs: in the South (Site-A) and the Southwest (Site-B). Sites were chosen because they perform fifty or more LDKTs per year, have a Hispanic bilingual transplant physician, and serve a large Hispanic patient population. Both were non-profit hospitals with LDKT disparities for Hispanics, as compared to non-Hispanic Whites in 2016 [32]. Site-A was a regionally based, academic affiliated medical center with a large-sized 1000-bed hospital and a level 1 trauma center. Site-B was part of a national academic medical center with a medium-sized 300-bed hospital and no trauma center. The implementation preparation phase occurred between April and December 2016 to prepare for a January 2017 launch. Eligible participants included transplant stakeholders involved in HKTP implementation: physicians (e.g., surgeons, nephrologists, urologists), administrators, clinical staff (e.g., nurses, social workers, schedulers), and others (e.g., IT, marketing, research staff). The study co-Principal Investigators (co-PIs) requested that stakeholders involved in HKTP implementation participate in one-on-one interviews and group discussions via email, phone, and/or in-person, in advance of site visits. Stakeholders were aware of study goals and knew they would engage with the study co-PI through research activities. Participants did not have a relationship with the interviewer (co-PI, EJG) prior to study commencement.

Implementation strategies

The CFIR guided implementation design [26]. We used the “Learning Collaborative” [37] method to help stakeholders design center-specific approaches to surmounting barriers to HKTP implementation to achieve healthcare quality improvement [38]. Learning collaborative discussions occurred via one teleconference call in September 2016 and one 2-day in-person meeting at Northwestern University in October 2016 (Fig. 1). Four teleconference calls occurred in 2017, in which stakeholders discussed successes and challenges to implementation strategies, modifications made, and strategized solutions for implementing modifications at their institutions. Additionally, weekly and biweekly telephone calls were conducted with each site’s Research Coordinators, and on occasion other stakeholders (e.g., Site PIs, administrators, outreach staff), to ascertain intervention progress and modifications.

Fig. 1
figure 1

Data collection timeline

Full size image

Data collection

Site visits were conducted in May 2016 and March–April 2017. In-depth, in-person, one-on-one and group interviews were conducted with transplant stakeholders by a trained female social scientist with a PhD, co-PI (EJG), using the CFIR Interview Guide [39]. An additional file presents the interview questions (see Additional file 2). Interviews assessed stakeholders’ attitudes about the implementation, perceived provider, operational, and center-level barriers and facilitators to HKTP implementation, and the intervention’s modifications. Additionally, group discussions, led jointly by EJG and surgeon co-PI (JCC), assessed operational logistics to intervention implementation and were not guided by CFIR. Interviews and discussions were audio-recorded and lasted 30–60 min. Learning collaborative discussions were audio-recorded. While the first learning collaborative telephone meeting lasted 2 h, the second learning collaborative meeting was a 2-day in-person workshop, and the remaining telephone meetings lasted 1 h. Handwritten field notes, including reflexive reflections, were taken by one co-PI (EJG) and/or Research Coordinators during research activities. Thus, modifications, as described by both sites, were prospectively documented.

Figure 2 illustrates 3 concentric circles, with each circle representing a different data collection activity, the frequency of which is denoted by the circle size. The weekly/biweekly meetings occurred most frequently, followed by learning collaborative meetings and site visits. The role and number of stakeholders overlapped across data collection activities. For example, site PIs, administrators, outreach staff, and research staff attended weekly/biweekly calls, learning collaborative meetings, and participated in site visits interviews.

Fig. 2
figure 2

Data collection approaches by frequency

Full size image

Qualitative analysis

2016 CFIR analysis

Transcriptions of 2016 data collected underwent a thematic analysis of barriers and facilitators to carrying out the HKTP during the pre-implementation phase, as described elsewhere [31]. Briefly, we used CFIR constructs as codes that guided the thematic analysis using the constant comparative [40], deductive, and inductive coding methods [41, 42]. In this paper, we only report the results of the deductive coding. After achieving inter-rater reliability (kappa > 0.80) [43], multiple teams of two coders discussed discrepancies and achieved consensus. The CFIR interview guide was used to conduct in-person, one-on-one, and group interviews during 2016 and 2017. However, we did not perform CFIR thematic analysis of the interview data collected in the first year of implementation (2017). Thus, the 2016 data were used to generate the comparative site descriptions.

2017 FRAME analysis

We performed a framework analysis using the FRAME by coding transcriptions of the audio-recordings and field notes from the implementation phase (2017) for the presence of modifications, their nature, initiator, intervention component, and intent. A codebook was based on Stirman’s FRAME. The research team (EJG, MS, ER, NA, JU) held multiple 1-h-long analytic retreats to review independently coded transcripts and revise the codebook for clarity, in an iterative process. The four-member coding team reached inter-rater reliability (kappa > 0.60), then resolved discrepancies between coders through discussion [31, 41, 42]. All transcripts were uploaded into qualitative analysis software (MAXQDA v.12).

Axial coding

During the second phase of analysis, axial coding techniques from grounded theory [44] were used to examine the relationship between CFIR factors and modifications. This process allowed us to theorize how CFIR contextual factors during the pre-implementation phase may have generated causal conditions prompting the implementation phase modifications. The research team had previously used thematic analysis [45, 46] to code all expressions of CFIR framework domains in transcripts of interviews conducted during the pre-implementation period [31]. Memo summaries of themes about each CFIR domain were examined for relationships with the modifications. We excluded the types of modifications that manifested ≤ 5 occurrences (i.e., repeating elements, shortening) from comparative analysis because of the insufficient number of examples. Both site PIs were provided the transcript of results for review.

Results

Demographics

Fifty-seven stakeholders (Site-A: n = 26, Site-B: n = 31; 100% recruitment rate) participated in one or more research activities, including 37 in-depth, one-on-one interviews (n = 28, some participated in interviews twice, once in 2016 and again in 2017), 34 group discussions (n = 45), and/or 6 learning collaborative discussions (n = 17) (Table 1). Most participants were female (61%), non-Hispanic (70%), and included non-physician clinicians (nurses, social workers) (25%), physicians (19%), administrators (14%), and staff (e.g., schedulers, marketing, IT, and research) (42%).

Table 1 Participants’ demographic characteristics by study site
Full size table

Comparative site descriptions

The following descriptions of the intervention sites are essential for obtaining valid measures of the effects of modifications on desired outcomes [23]. Table 2 compares both sites in light of CFIR constructs.

Table 2 Study site descriptions by consolidated framework for implementation research domains and constructsa
Full size table

Site-A

Site-A was part of an enterprise system whereby surgeons practiced at one of two of the institution’s locations within the state, with Site-A serving as the larger, dominant location. While Site-A performed kidney transplant surgeries and provided post-transplant care, all pre-kidney evaluations were outsourced to a community nephrology organization, with which Site-A has had a long-standing collaborative relationship. Accordingly, intervention implementation required collaboration between staff, administrators, and surgeons from Site-A and the community nephrology organization.

Site-A is comprised of experienced surgeons following a team-based approach, whereby collaboration from all team members was expected. Decision making was centralized at the transplant division level, which enabled quick changes and approvals. The HKTP was led by the Site-A lead administrators and the site PI. Because the community nephrology organization was outside of Site-A’s charge, challenges arose in ensuring the HKTP’s successful implementation. While the HKTP generally fit well within Site-A, and there was ample space to deliver the education sessions, hiring a bicultural/bilingual outreach staff was a protracted process.

Site-B

Site-B was also part of an enterprise system involving three integrated campuses across the country, but Site-B was not the leading location. All institutional changes required formal review and approval by the leading institutional location before changes could be implemented across campuses. Further, Site-B had different structural characteristics—marked by greater complexity due to the collaboration between two different clinical departments to enable a Hispanic physician to deliver the HKTP education sessions in Spanish.

Site-B can be characterized as having an egalitarian culture, whereby physicians and administrators were expected to co-lead institutional initiatives. Decisions are commonly protracted because they are made through multiple committee meetings involving physicians and staff. While the HKTP generally fit well within Site-B, finding the space to deliver the education sessions remained an ongoing challenge. Although there was a bilingual outreach staff to deliver the dialysis lobby days, this staff member was constrained by other institutional responsibilities that limited time for delivering the intervention.

Modifications

Site-B made nearly twice as many modifications than Site-A (n = 29 versus n = 18) (Table 3). Sites differed in the proportions of delaying/skipping (Site-A 50% versus Site-B 28%) and adding (Site-A 11% versus Site-B 28%), but were comparable in substituting (Site-A 17% versus Site-B 17%) and tweaking (Site-A 17% versus Site-B 14%) modification types.

Table 3 Frequency of modification initiator, type, and goal by study site
Full size table

Across sites, the transplant team consistently initiated most modifications (Site-A 66%; Site-B 62%). Individuals instigated proportionately slightly more modifications at Site-B (21% versus Site-A 17%). In contrast, institutions instigated proportionately slightly more modifications at Site-A (17% versus Site-B 10%). At Site-B, both the institution and individuals initiated two modifications (7%).

The goals of making modifications slightly varied by site (Table 3). Sites were comparable in making modifications that aimed to improve effectiveness/outcomes (Site-A 11%; Site-B 10%). However, the goal of addressing cultural factors drove proportionately more modifications at Site-A (11% versus Site-B 7%), whereas the goals of increasing reach and improving feasibility drove proportionately more modifications at Site-B (10% versus Site-A 6%). The goals of increasing satisfaction and retention or reducing costs did not motivate any modifications across sites.

Modification similarities between sites

Modification types

Skipping and delaying

Both sites made similar skipping-type modifications, primarily due to their institutions’ structural characteristics. For example, both sites did not establish a direct, Spanish-speaking telephone line in the transplant division, per the protocol. Because Site-A worked in collaboration with a community nephrology organization that made and received all HKTP-related intake calls in which “everybody who answers the phone is bilingual,” Site-A did not perceive any problems in providing access to the HKTP to their Spanish-speaking patients. Site-B did not implement a Spanish telephone line because doing so would require a large infrastructural change that would challenge the institution’s “philosophy… [to] provide the same care at every site no matter what door you are walking into.” As a stakeholder from Site-B explained, setting up the Spanish line would be a “big ask for the institution” because “the Spanish portion of them having access to someone who speaks Spanish has never been an option at the institution”. Thus, if a Spanish-speaking telephone line were implemented in the transplant division, the institution would have to implement one in other clinical divisions.

Both sites experienced delays in publishing the Spanish HKTP information on their institutional websites due to structural characteristics. Site-A encountered delays in translating the HKTP website content because its merger with another healthcare institution posed other institutional priorities. Site-B encountered delays because it had not received approval from institutional enterprise leadership. As one interviewee explained, “Well, I know that our website needs to be translated and unfortunately I have no control over that. This is something that needs to be done by [the enterprise]”.

Both institutions delayed the translation of HKTP patient-facing materials (i.e., HKTP brochures, patient letters, flyers) into Spanish. Both institutions attributed these delays, in part, to their lack of Spanish-speaking staff or time available by such staff to dedicate to this task (available resources). As one interviewee from Site-B explained, “I mean we have international services that does translation for our patients, and I’ve asked them before if they would translate information. I think in the past when they are more heavily staffed they were able to, but now they’re short staffed, and they’re not able to.” Delays were also due to conflicts in coordinating with the transplant team, translation services, and marketing department. The intervention’s 16 components, involving different institutional departments and different organizations, made skipping and delays more likely to arise, given the challenges of coordination between these organizational units.

Both sites also skipped monitoring their progress and/or success with the intervention, which coincides with the CFIR construct of implementation climate in terms of goals and feedback systems. Thus, sites were equally unaware of their HKTP participants’ status in terms of numbers, progress through the LDKT evaluation process, and outcomes. Stakeholders at both sites acknowledged early on that they did not evaluate their program by patient ethnicity or race. One interviewee from Site-A explained, “I wish I had a number, I should, I know how much you guys have a better idea of your numbers, and our numbers than we did, but I think… just for me, I look around the community, and I’m like this is obviously a huge need.” An interviewee at Site-B explained further “I don’t think ever looked at 'Oh let’s look at our Hispanic population of recipients!' How many living donors do they have? Did anyone call in? At what point? How far do they make through the process? Did they withdraw? Were they excluded? Like I think that historically, we never really looked at that.”

Adding

Both sites added an intervention component of providing laptops to potential living donors during the HKTP clinic to facilitate Spanish-speaking potential living donors’ access to and completion of the English-language living donor online medical history questionnaire, which was not available in Spanish.

Substituting

Both sites made similar substituting type of modifications to the intervention, which appeared to compensate for the lack of available resources. To increase the HKTP reach to the Hispanic patient population, the bilingual/bicultural research coordinators at both sites occasionally assisted the non-Hispanic white outreach staff when visiting dialysis centers with a large Hispanic population. Further, at Site-A, there was originally no bilingual/bicultural outreach staff; therefore, they relied on an English-speaking outreach staff. For example, at Site-A, one stakeholder stated “…we’ve had a position posted for essentially 9 months, and while we have had many, many applicants, none, we’ve interviewed several people, and it has not been a good match. … And so [Ms. X], who is here with us today too, is our outreach coordinator, and she’s absolutely promoting this, and [the bicultural/bilingual research coordinator] is helping some. But we still don’t have this full-time dedicated person.” At Site-B, a bilingual/bicultural transplant nurse delivered the HKTP education sessions when bilingual/bicultural transplant physicians were unavailable.

Tailoring, Tweaking, Refining

Both sites performed tweaking-type modifications to the HKTP to better meet the needs of their patient population. For example, both transplant teams modified the schedulers’ script because the schedulers had expressed that patients perceived the script’s delivery as forced and rehearsed, rather than natural and organic.

Spreading

At both sites, scheduling conflicts arose due to limited staffing availability and patients’ needs. At Site-A, the absence of enough dietitians on HKTP clinic days made scheduling patients with HKTP physicians a challenge. At Site-B, multiple HKTP patients were seen by nephrologists who simultaneously needed an interpreter, but there were insufficient numbers of interpreters and the nephrologists did not like using the interpreter app on iPads. These staffing conflicts resulted in patients' appointments with nephrologists to be spread out, which led to some HKTP patient evaluation clinics being extended over 2 days, instead of occurring in 1 day, as recommended in the intervention protocol.

Modification differences between sites

CFIR Inner Setting factors in the implementation preparation phase explained many differences in the number and types of modifications made across sites during the implementation phase. Details including illustrative quotations about each modification made by site are presented in the Supplemental File 3. Site-A’s prior experience and success with implementing other interventions increased the institution’s confidence in implementing the HKTP (i.e., their collective self-efficacy). Site-A had previously expanded their services to living donors, modeled after a program they adopted from another institution. “[This other institution has] a red phone and when donors call, it’s always answered. [So we implemented], a phone number that is always answered also, but the focus was, we value you as a donor, and we are going to take excellent care of you … I mean that project, we implemented and brought it in 2½ years ago.” Site-A’s previously successful implementation experience led them to make fewer overall modifications. The greater number of intervention modifications at Site-B was due to the HKTP being less compatible with existing workflows, structural characteristics, culture, and available resources. One interviewee explained, “I think that what it’s gonna make it hard to implement is that nothing happens at [Site-B] without 20 meetings and 20 people dotting the I and crossing the T. I think the biggest barrier is just kind-of going through the process, and then redoing the process, and making sure that everyone is part of the process and I think that it’s kind of like the nail in the hay.” For example, Site-B’s HKTP outreach staff had multiple work commitments, which prevented them from performing the minimum 16 h of outreach per week for a total of 832 h for the year, as outlined in the intervention protocol. Further, Site-B could not schedule the target number of patients to attend the HKTP education sessions because of limits imposed by the size of the education room.

Modification types

Skipping and delaying

Site-A delayed or skipped proportionately more intervention components than Site-B, which can be explained by the CFIR inner factor of structural characteristics. Site-A had to coordinate the implementation of HKTP components with their partnering community nephrologist organization, which had their own rules and leadership. As one interviewee explained, “I think that one thing that makes it a little more difficult is that our recipient [and] donor teams are in two different locations with two different staff, so it’s almost like we have to have double staff if, you will. So I think in general that makes it a little more difficult, because of the contractual agreement that we have with [second organization] for the recipient portion, while we collaborate and we pay them contractually, we don’t manage them, and so [they] have recipients, I have donors.” Thus, despite their good working relationship, Site-A was not able to ensure that the external community nephrology organization could translate patient letters into Spanish or mail letters to nephrologists on time. Moreover, Site-A’s merger with another healthcare organization delayed the translation of HKTP information to be posted on the institutional website.

By contrast, Site-B’s egalitarian culture resulted in diffused leadership engagement in the intervention. One interviewee noted concerns that Site-B’s PI did not speak Spanish and that another doctor from the Urology department would be delivering the Spanish education sessions. Site-B’s PI did not exert their upward influence (also known as the “Pelz effect” [47]) on institutional committees to expedite their review and approval of essential HKTP components. Stakeholders at Site-B attributed delayed approvals of intervention components, including the HKTP Spanish website content and educational materials’ approval, to their institution’s time-consuming decision-making process.

Adding

Site-B added proportionately more intervention components than Site-A. Most of Site-B’s modifications involved adding elements to the intervention, which were related to the institution’s culture. One interviewee explained, “You don’t really have power. You don’t really go around telling people what to do. You just find ways to identify key people and stakeholders and help motivate them to do the job. We don’t really run an institution of authoritative dictatorship. It’s not like a traditional academic environment where the Chair says, 'Okay, this is what you all are going to do' and people say, 'Okay [the Chair] told me to do that.' It doesn’t work that way here.” The team-based approach to decision making placed a high value on obtaining input from all levels within Site-B’s social structure. Stakeholders involved in HKTP implementation behaved as though they felt empowered to make additions that they believed would improve the HKTP that bypassed the prolonged team-based approach to decision making. Additions included the bilingual outreach staff creating a flyer that was posted in dialysis centers to advertise the upcoming lobby days, and the bilingual/bicultural research staff creating a folder containing educational materials to give to potential living donors and family members attending the HKTP clinic. By contrast, decision making at Site-A was concentrated at the transplant division level, which facilitated approvals with implementing the HKTP with greater efficiency.

Site-A’s culture featuring strong leadership engagement by the transplant division facilitated approvals with implementing the HKTP with greater efficiency, leading to proportionately fewer components of the intervention being added. One interviewee explained, “We are organized in a fashion, so for us, we do whatever we want. There is no one who will oppose us. I mean no one will say no. If we say this is what we want to do, then everybody else says, 'Yes, sir.' That’s the way it works.” The strong leadership engagement from operation administrators at Site-A fostered momentum in intervention implementation.

Modification initiator

The modifications made by individuals at Site-B were primarily related to the institutional structural characteristics, culture, and absence of sufficient leadership engagement. As described above, decision making typically required multiple team meetings to come to a consensus. For example, the HKTP flyer developed for distribution to dialysis patients during outreach visits had to undergo an extensive chain of review before receiving authorization from the institutional enterprise’s “mothership.” Site-B’s team-based culture and the need to rapidly implement the intervention enabled individual stakeholders to expedite modifications by bypassing the typical lengthy approval process through “workarounds.” By contrast, decisions at Site-A were made internally by the transplant team. For example, the transplant team decided to host their usual Spanish Town Hall Question and Answer discussions for the public to promote the HKTP, adjusted the day and time of HKTP clinics to solve scheduling conflicts with healthcare providers, and cancelled HKTP clinics when too few patients were scheduled.

Discussion

In this study of a complex, culturally competent care intervention, we found that both institutions made numerous modifications. Modifications occurred after the intervention had been implemented and as new organizational issues emerged. By contrast, Stirman’s review found that most studies described modifications made proactively, early in the implementation process [4]. Thus, our findings underscore the importance of assessing modifications throughout the implementation phase.

This study is among the first to identify the CFIR factors that influence implementation outcomes, a new area of CFIR research [16]. CFIR Inner Setting factors identified during the pre-implementation period helped explain the types of modifications made. The CFIR factor, structural characteristics, appeared to contribute to intervention delays/skipping in Site-A. By contrast, the lack of leadership engagement appeared to contribute to delays/skipping in Site-B. These examples illustrate how institutional-level factors are relevant to modifications made to complex interventions.

Modifying interventions is the subject of much debate because they potentially interfere with implementation fidelity [14]. Our findings suggest that CFIR factors influenced the type and number of modifications that institutions made to the HKTP intervention, which is an implementation outcome. Based on our previous research [27], a greater number of modifications was associated with reduced intervention effectiveness. Thus, this research connects CFIR factors, implementation, and intervention effectiveness. Our findings may help explain what different organizational factors lead to modifications in intervention implementation, which may help institutions to adapt the intervention before implementation and to more effectively foster its implementation [48].

Most modifications involved delaying/skipping, adding, or tailoring/tweaking intervention components, consistent with other research [23]. Our findings shed light on CFIR factors that explain modification types that may arise during implementation. Specifically, we found that Inner Setting factors such as culture, available resources, compatibility, and goals and feedback largely explain these modifications. The sites’ nested layers of institutional bureaucracy (i.e., structural characteristics) promoted skipping and delaying modification types.

Most modifications were made by the transplant team (64%), suggesting a unified or thoughtful approach. However, 19% of modifications were made by individuals, which highlights the need for greater oversight of the intervention implementation process by study champions. Modifications that originated from individuals were most common when there was an egalitarian culture, suggesting that such intervention sites require greater oversight than sites with greater leadership engagement.

Practical implications

Our findings have practical implications for leaders intending to implement complex interventions. In particular, our study sheds light on Pawson and Tilley’s pivotal question, “What works, for whom, in what circumstances, in what respects and why?” [49] We found that when there is greater compatibility between an intervention and the institution, and when the institution possesses ample resources to support the intervention, fewer modifications were made. Similarly, when the institution maintains an egalitarian form of culture, more additions were made to the intervention. Moreover, when an institution has complex organizational structures (structural characteristics), greater delays to implementation occurred. Practically, these findings can help prepare interventionists to expect certain modifications in certain settings.

Strengths

Our study’s strengths include a prospective analysis of a complex intervention. We evaluated the implementation at the organizational level at two geographically distinct healthcare organizations. Given that the HKTP was devised as a culturally competent intervention, our evaluation of modifications to the HKTP largely controls for concerns raised by the cultural adaptation field within implementation science, modifications based on clients’ cultural backgrounds, and enables a broad analysis of organizational factors influencing modification. Involving the co-PI (EJG), who had an intimate knowledge of the intervention and what comprised minor versus moderate modifications, helped ensure appropriate modification classification in the coding and analysis processes.

Limitations

Our study has limitations. Detailed descriptions of modifications depended on self-report, which may have been limited by social desirability bias. However, triangulation through interviews with multiple stakeholders and member checking through weekly phone calls with Site-A and Site-B research staff and quarterly learning collaborative calls served to validate and clarify the nature of modifications. This analysis did not evaluate the impact of modifications on implementation fidelity and on HKTP outcomes. The relative weight of modifications for their impact on fidelity to different intervention components remains to be determined [17].

We subsequently evaluated the intervention’s sustainability in light of modifications made over time [50, 51]. Sustainability, as opposed to effectiveness, describes the degree to which the intervention is continued over time. Sustainability is particularly challenging for complex interventions, like the HKTP [52].

Conclusions

Complex interventions require the coordination of multiple departments and individuals. The implementation of complex interventions creates many opportunities for modifications to occur in various locations within the organization. This research demonstrates that identifiable factors from the CFIR explain the frequency, type, and source of those modifications. In doing so, it guides how organizational actors are likely to modify the intervention, either to better fit the unique context or in ways that undermine intervention fidelity.

Availability of data and materials

The datasets generated and/or analyzed during the current study are not publicly available due to the sensitive nature in that individual privacy could be compromised.

Abbreviations

CFIR:

Consolidated Framework for Implementation Research

Co-PI:

Co-Principal Investigator

FRAME:

Framework for Reporting Adaptations and Modifications-Expanded

HKTP:

Hispanic Kidney Transplant Program

LDKT:

Living Donor Kidney Transplantation

NHW:

Non-Hispanic White

NM:

Northwestern Medicine®

References

  1. Cohen DJ, Crabtree BF, Etz RS, et al. Fidelity versus flexibility: translating evidence-based research into practice. Am J Prev Med. 2008;35(5 Suppl):S381–9. https://doi.org/10.1016/j.amepre.2008.08.005.

    Article  PubMed  Google Scholar 

  2. Breitenstein SM, Gross D, Garvey CA, Hill C, Fogg L, Resnick B. Implementation fidelity in community-based interventions. Res. Nurs. Health. 2010;33(2):164–73. https://doi.org/10.1002/nur.20373.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Carvalho ML, Honeycutt S, Escoffery C, Glanz K, Sabbs D, Kegler MC. Balancing fidelity and adaptation: implementing evidence-based chronic disease prevention programs. J. Public Health Manag. Pract. 2013;19(4):348–56. https://doi.org/10.1097/PHH.0b013e31826d80eb.

    Article  PubMed  Google Scholar 

  4. Stirman SW, Miller CJ, Toder K, Calloway A. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. 2013;8:65. https://doi.org/10.1186/1748-5908-8-65.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Aarons GA, Sklar M, Mustanski B, Benbow N, Brown CH. "Scaling-out" evidence-based interventions to new populations or new health care delivery systems. Implement Sci. 2017;12(1):111. https://doi.org/10.1186/s13012-017-0640-6.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Bernal G, Adames C. Cultural adaptations: conceptual, ethical, contextual, and methodological issues for working with ethnocultural and majority-world populations. Prev Sci. 2017;18:681–8.

    Article  Google Scholar 

  7. Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. 2019;(1):58. https://doi.org/10.1186/s13012-019-0898-y.

  8. Chambers DA, Norton WE. The adaptome: advancing the science of intervention adaptation. Am J Prev Med. 2016;51(4 Suppl 2):S124–31.

    Article  Google Scholar 

  9. Castro FG, Barrera M Jr, Martinez CR Jr. The cultural adaptation of prevention interventions: resolving tensions between fidelity and fit. Prev Sci. 2004;5(1):41–5.

    Article  Google Scholar 

  10. Parra-Cardona JR, Bybee D, Sullivan CM, et al. Examining the impact of differential cultural adaptation with Latina/o immigrants exposed to adapted parent training interventions. J Consult Clin Psychol. 2017;85(1):58–71.

    Article  Google Scholar 

  11. Castro FG, Yasui M. Advances in EBI development for diverse populations: towards a science of intervention adaptation. Prev Sci. 2017;18:623–9.

    Article  Google Scholar 

  12. Escoffery C, Lebow-Skelley E, Haardoerfer R, et al. A systematic review of adaptations of evidence-based public health interventions globally. Implement. Sci. 2018;13:125.

    Article  Google Scholar 

  13. Cooper BR, Shrestha G, Hyman L, Hill L. Adaptations in a community-based family intervention: replication of two coding schemes. J. Prim. Prev. 2016;37(1):33–52. https://doi.org/10.1007/s10935-015-0413-4.

    Article  PubMed  Google Scholar 

  14. Rabin BA, McCreight M, Battaglia C, et al. Systematic, multimethod assessment of adaptations across four diverse health systems interventions. Front. Public Health. 2018;6:102. https://doi.org/10.3389/fpubh.2018.00102.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Lau A, Barnett M, Stadnick N, et al. Therapist report of adaptations to delivery of evidence-based practices within a system-driven reform of publicly funded children's mental health services. J. Consult. Clin. Psychol. 2017;85(7):664–75. https://doi.org/10.1037/ccp0000215.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Damschroder LJ, Reardon CM, Opra Widerquist MA, Lowery J. Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): the CFIR Outcomes Addendum. Implement Sci. 2022;17(1):7. https://doi.org/10.1186/s13012-021-01181-5.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Hasson H, Blomberg S, Dunér A. Fidelity and moderating factors in complex interventions: a case study of a continuum of care program for frail elderly people in health and social care. Implement Sci. 2012;7:23. https://doi.org/10.1186/1748-5908-7-23.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655. https://doi.org/10.1136/bmj.a1655.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Paskett E, Thompson B, Ammerman AS, Ortega AN, Marsteller J, Richardson D. Multilevel interventions to address health disparities show promise in improving population health. Health Aff. 2016;35(8):1429–34.

    Article  Google Scholar 

  20. Chin MH, Clarke AR, Nocon RS, et al. A roadmap and best practices for organizations to reduce racial and ethnic disparities in health care. J. Gen. Intern. Med. 2012;27(8):992–1000.

    Article  Google Scholar 

  21. American Hospital Association. Hospitals Fast Facts on U.S, 2019. https://www.aha.org/statistics/fast-facts-us-hospitals#system. Accessed on 30 July 19.

  22. Coley A, Parry G. Evaluating complex health interventions: a guide to rigorous research designs. Academy Health; 2017. https://academyhealth.org/sites/default/files/AH_Evaluation_Guide_FINAL.pdf. Accessed 18 Apr 22.

  23. Bauman AE, Cabassa LJ, Stirman SW. In: Brownson RC, Colditz G, Proctor EK, editors. Adaptation in dissemination and implementation science: Dissemination and Implementation Research in Health. Oxford University Press; 2017. p. 285–300.

    Chapter  Google Scholar 

  24. Brownson R, Colditz G, Proctor E. Dissemination and Implementation Research in Health. Oxford University Press; 2012.

    Google Scholar 

  25. Damschroder L, Aron D, Keith R, Kirsh S, Alexander J, Lowery J. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

    Article  Google Scholar 

  26. Gordon EJ, Lee J, Kang RH, et al. A complex culturally targeted intervention to reduce Hispanic disparities in living kidney donor transplantation: an effectiveness-implementation hybrid study protocol. BMC Health Serv. Res. 2018;18(1):368.

    Article  Google Scholar 

  27. Gordon EJ, Uriarte J, Lee J, et al. Effectiveness of a culturally competent care intervention in reducing disparities in Hispanic live donor kidney transplantation: A hybrid trial. Am J Transplant. 2022;22(2):474–88.

    Article  Google Scholar 

  28. Purnell TS, Hall YN, Boulware LE. Understanding and overcoming barriers to living kidney donation among racial and ethnic minorities in the United States. Adv. Chronic Kidney Dis. 2012;19(4):244–51.

    Article  Google Scholar 

  29. Gordon EJ, Mullee J, Ramirez D, et al. Hispanic/Latino concerns about living kidney donation: a focus group study. Prog. Transplant. 2014;24(2):152–62.

    Article  Google Scholar 

  30. Gordon EJ, Caicedo JC, Ladner DP, Reddy E, Abecassis MM. Transplant center provision of education and culturally and linguistically competent care: a national study. Am. J. Transplant. 2010;10(12):2701–7.

    Article  CAS  Google Scholar 

  31. Gordon E, Romo E, Amortegui D, et al. Implementing culturally competent transplant care and implications for reducing health disparities: a prospective qualitative study. Health Expect. 2020;23(6):1450–65.

    Article  Google Scholar 

  32. Gordon EJ, Lee J, Kang R, et al. Hispanic/Latino disparities in living donor kidney transplantation: role of a culturally competent transplant program. Transplant. Direct. 2015;1(8):e29.

    Article  Google Scholar 

  33. Alhalel N, Francone NO, Salazar A, et al. Perceptions of a culturally targeted hispanic kidney transplant program: a mixed methods study. Clin. Transplant. 2019;33(6).

  34. Gordon EJ, Wolder LB. Contextualizing ethical dilemmas: ethnographic methods in bioethics. In: Jacoby L, Siminoff L, editors. Empirical Methods for Bioethics: A Primer, vol. 11. Emerald Group Publishing Limited; 2008. p. 83–116.

  35. Hoffmann TC, Glasziou PP, Boutron I, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. Bmj. 2014;348:g1687. https://doi.org/10.1136/bmj.g1687.

    Article  PubMed  Google Scholar 

  36. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int. J. Qual. Health Care. 2007;19(6):349–57.

    Article  Google Scholar 

  37. The Commonwealth Fund. Improving health care delivery: the "learning collaborative" approach. 2005; Accessed at http://www.commonwealthfund.org/Content/Innovations/Tools/2005/Jun/Improving-Health-Care-Delivery%2D%2DThe%2D%2DLearning-Collaborative%2D%2DApproach.aspx.

  38. Becker DRDR, Bond GR. The IPS supported employment learning collaborative. Psychiatr Rehabil J. 2014;37(2):79–85.

    Article  Google Scholar 

  39. CFIR. Interview Guide. CFIR Interview Guide. http://www.cfirguide.org/guide/app/index.html.

  40. Lincoln Y, Guba E. Naturalistic Inquiry. Sage Publications; 1985.

    Book  Google Scholar 

  41. Guest G, MacQueen K, Namey E. Applied Thematic Analysis. Sage Publications; 2012.

    Book  Google Scholar 

  42. Bradley E, Curry L, Devers K. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv. Res. 2007;42(4):1758–72.

    Article  Google Scholar 

  43. Campbell JL, Quincy C, Osserman J, Pedersen OK. Coding in-depth semistructured interviews: problems of unitization and intercoder reliability and agreement. Sociol. Methods Res. 2013;42(3):294–320. https://doi.org/10.1177/0049124113500475.

    Article  Google Scholar 

  44. Strauss A, Corbin J. Basics of qualitative research: Techniques and procedures for developing grounded theory. Thousand Oaks: Sage Publications; 1998.

  45. Bernard H, Ryan G. Text analysis: Qualitative and quantitative methods. In: Bernard H, editor. Handbook of Methods in Cultural Anthropology. Altamira Press; 1998. p. 595–646.

    Google Scholar 

  46. Luborsky M. The identification and analysis of themes and patterns. In: Gubrium JS, A, ed. Qualitative Methods in Aging Research. Sage Publications; 1994:189–210.

  47. Anderson LR, Tolson J, Fields MW, Thacker JW. Extension of the Pelz effect: the impact of leader's upward influence on group members' control within the organization. Basic Appl. Soc. Psychol. 1990;11(1):19–32.

    Article  Google Scholar 

  48. Rabin BA, McCreight M, Battaglia C, et al. Systematic, multimethod assessment of adaptations across four diverse health systems interventions. Front. Public Health. 2018;6:102.

    Article  Google Scholar 

  49. Pawson R, Tilley N. Realistic Evaluation. Sage Publications; 1997.

    Google Scholar 

  50. Schoenwald SK, et al. Toward the effective and efficient measurement of implementation fidelity. Adm Policy Ment Health. 2011;38(1):32–43.

    Article  Google Scholar 

  51. Gordon E, Uriarte J, Anderson N, Smith J, Caicedo J, Shumate M. Sustainment of a complex culturally competent care intervention for hispanic living donor kidney transplantation: a longitudinal analysis of adaptations. J Clin Transl Sci. 2020;1–34. https://doi.org/10.1017/cts.2022.378.

  52. Shelton RC, Cooper BR, Stirman SW. The sustainability of evidence-based interventions and practices in public health and health care. Annu Rev Public Health. 2018;39:55–76. https://doi.org/10.1146/annurev-publhealth-040617-014731.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Special thanks go to the research staff at Site-A and Site-B. We thank the faculty, staff, and administrators from both study sites for participating in interviews and implementing the HKTP intervention.

Funding

NIDDK funded this research (1R01DK104876 to EJ Gordon and JC Caicedo, co-PIs).

Author information

Authors and Affiliations

  1. Department of Surgery - Division of Transplantation, Center for Health Services and Outcomes Research, Center for Bioethics and Medical Humanities, Feinberg School of Medicine, Northwestern University, 633 N. St. Clair, 20th Floor, Chicago, IL, 60611, USA

    Elisa J. Gordon

  2. Center for Health Services and Outcomes Research, Feinberg School of Medicine, Northwestern University, Chicago, IL, USA

    Jefferson J. Uriarte

  3. School of Education and Policy, Northwestern University, Evanston, IL, USA

    Naomi Anderson

  4. Department of Neurology – Biological Sciences Division, University of Chicago, Chicago, IL, USA

    Elida Romo

  5. Department of Surgery - Division of Transplantation, Feinberg School of Medicine, Northwestern University, Chicago, IL, USA

    Juan Carlos Caicedo

  6. Department of Communication Studies, Northwestern University, Evanston, IL, USA

    Michelle Shumate

Authors
  1. Elisa J. Gordon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jefferson J. Uriarte
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Naomi Anderson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Elida Romo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Juan Carlos Caicedo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Michelle Shumate
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conception and design: EJG, MS. The acquisition of data: EJG. Collection and assembly of data: EJG. Analysis and interpretation of the data: EJG, MS, JU, NA, ER. Drafting of the article: EJG, MS, JU. Critical revision of the article for important intellectual content: EJG, MS, JU, ER, NA. Administrative, technical, or logistic support: EJG. Final approval of the article: EJG, JU, NA, ER, JCC, MS. Obtaining of funding: EJG, JCC. All authors have approved the submitted version and have agreed both to be personally accountable for the author’s contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature.

Corresponding author

Correspondence to Elisa J. Gordon.

Ethics declarations

Ethics approval and consent to participate

Northwestern University’s Institutional Review Board granted study approval (STU00201331). Written and verbal informed consent was obtained from individuals participating in one-on-one interviews and group discussions, respectively.

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

HKTP Implementation Components (same as protocol table). This file describes all 16 HKTP intervention components.

Additional file 2.

2017 Site Visit Interview Guide for the Site PI and Administrators. This file provides the interview guide used for the 2017 site visits.

Additional file 3.

Modification Description, Reason, Illustrative Quotation, Modification Type, and Initiator by Study Site. This file presents a description of modifications, the reasons why they occurred, modification type, and illustrative quotations by study site.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gordon, E.J., Uriarte, J.J., Anderson, N. et al. Implementation and modification of an organizational-level intervention: a prospective analysis. Implement Sci Commun 3, 59 (2022). https://doi.org/10.1186/s43058-022-00296-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s43058-022-00296-0

Keywords

Implementation Science Communications

ISSN: 2662-2211