Skip to main content
Download PDF
Download ePub

Community and partner engagement in dissemination and implementation research at the National Institutes of Health: an analysis of recently funded studies and opportunities to advance the field

Implementation Science Communications volume 4, Article number: 77 (2023) Cite this article

Abstract

Background

As the focus has grown in recent years on both engaged research and dissemination and implementation (D&I) research, so too has federal funding to support these areas. The purpose of this analysis is to provide an overall perspective about the range of practices and approaches being used to engage partners in D&I research, with special attention to disparities-relevant research, and to identify gaps and opportunities in research funded by the US National Institutes of Health (NIH) in this space.

Methods

This analysis examined a portfolio of active D&I research grants funded in fiscal years 2020 and 2021 across the NIH. Grant applications were deductively coded and summary statistics were calculated. Cross-tabulations were used to identify trends by engagement and disparities foci.

Results

There were 103 grants included in the portfolio, of which 87% contained some form of community or partner engagement, and 50% of engaged grants were relevant to health disparities. Engagement was planned across the research continuum with each study engaging on average 2.5 different partner types. Consultation was the most common level of engagement (56%) while partnership was the least common (3%). On average, each study used 2.2 engagement strategies. Only 16% of grants indicated formally measuring engagement. Compared to non-disparities studies, disparities-relevant studies were about twice as likely to engage partners at the higher levels of partnership or collaboration (19% vs. 11%) and were also more likely to be conducted in community settings (26% vs. 5%).

Conclusions

Based on this portfolio analysis, D&I research appears to regularly integrate engagement approaches and strategies, though opportunities to deepen engagement and diversify who is engaged remain. This manuscript outlines several gaps in the portfolio and describes opportunities for increasing engagement to improve the quality of D&I research and application to advancing health equity. In addition, opportunities for leveraging the consistent and systematic application of engagement approaches and strategies to advance the science of engagement are discussed.

Peer Review reports

Background

In recent years, it has become increasingly clear that for efficacious health care and public health interventions to be effectively implemented, broadly scaled up, and sustained over time, there needs to be increased appreciation for and investment in areas of scientific inquiry that can provide insight and perspectives as to how to best reach these goals. Specifically, leveraging the behavioral and social sciences, community and partner engagement, health equity, and implementation science is critical to advance such efforts.

Dissemination and Implementation (D&I) research at the US National Institutes of Health (NIH), under the umbrella of implementation science, has evolved over the past few decades. As one of the largest funders of D&I research, NIH now has a standing set of funding opportunities [1,2,3], a standing study section [4], an annual conference [5], dedicated D&I-focused staff, and ongoing training programs for extramural investigators (e.g., Training in D&I Research in Cancer [TIDIRC]) [6]. In addition, most NIH Institutes, Centers, and Offices feature some aspect of D&I research in strategic plans or initiatives as there is an increasing recognition of the value of implementation science approaches to facilitating effective uptake, sustainability, and scale up of efficacious medical and behavioral interventions to improve population health.

Several recent efforts have recognized the importance of integrating the methods and approaches of community engagement with D&I research to enhance rigor and relevance and to address health disparities. A special issue of Translational Behavioral Medicine highlighted research gaps and opportunities and included a reflection and historical perspective on NIH efforts in this space [7, 8]. There appears to be broad interest in bringing not just community, but also other partner engagement into D&I research. This is evidenced by an increasing number of perspective pieces and commentaries highlighting the potential benefits of this integration to both research and practice, outlining examples of federal efforts to support this research, and offering of potential frameworks to accomplish that integration [9,10,11,12,13]. There have also been a few published case studies documenting interventions that worked to integrate engagement in D&I research for a diverse range of health and related outcomes including violence, heart disease, and cancer [14,15,16]. A recent scoping review by Triplett and colleagues identified 103 child mental health evidence-based treatment implementation projects that included community or partner engagement [17]. Triplett et al. [17], and others [18], have suggested that integration of engagement methods and approaches into D&I research could lead to improved implementation and health outcomes, but this has yet to be documented in many cases.

One of the major lessons learned from the experiences during the COVID-19 pandemic is that even with the best scientific advancements (such as vaccines), these life-saving interventions will not be widely taken up and scaled without a concerted effort to understand and deploy implementation strategies that match each unique setting, context, and population. A primary goal of D&I research is to achieve broad, sustainable use of an evidence-based program, policy, or practice, which requires inclusivity and shared ownership and buy-in from a range of partners and community members. As a result of this understanding, the NIH launched two major COVID efforts designed to focus on community engagement, partnerships, and D&I approaches in order to engage populations most at risk and who simultaneously face the most barriers to access and participation in COVID testing (Rapid Acceleration of Diagnostics-Underserved Populations) [19] and vaccine uptake (Community Engagement Alliance against COVID-19) [20]. Further, in 2022, a new NIH Common Fund initiative was launched, Community Partnerships to Advance Science for Society (ComPASS), which focuses on advancing health equity research through community-driven, multisectoral structural interventions to reduce health disparities. Though not an implementation science initiative per se, the focus on sustainability and structural level change over time means that a number of implementation models, processes, and outcomes are likely to be incorporated [21].

The examples above notwithstanding, there remain clear and notable gaps in community and partner engagement in D&I research. There do not appear to be many “standard” D&I studies that routinely and rigorously integrate community or partner engagement approaches into their work, and engagement is not well-integrated into most D&I theories, models, or frameworks [22]. Further, implementation scientists may not have the competencies necessary to lead engaged or participatory research [23]. Given increased recent public attention to health disparities following the highly publicized racial/ethnic COVID-19 disparities, it seems timely and imperative that D&I research be done in partnership with the communities that are most in need and the organizations that serve them. Yet, D&I research often fails in this regard. Indeed, a recent reflection on “promises and pitfalls in implementation science” identifies this lack of connection with community partners and lack of focus on health equity as fundamental to the field’s challenges [24]. These identified pitfalls include re-creation of the research to practice gap, misalignment of timelines, incentives, and priorities with community partners, and complexity of strategies that may not match community needs. These authors offer several ways forward, many of which involve increased attention to better integration of engagement approaches to inform D&I efforts since community engaged research is rooted in social justice and advancing health equity [25,26,27]. Similarly, in their scoping review, Triplett et al. found that the engagement across 103 projects was often shallow and lacking shared power in decision-making [17]. They called for increased effort to improve engagement to proactively address barriers and increase likelihood of successful implementation.

Given the increased attention to health disparities and health equity in recent years, particularly since the start of the COVID-19 pandemic in 2020, and calls to increase engagement in D&I research, several initiatives across multiple NIH Institutes, Centers, and Offices over the last several years have attempted to stimulate the field. These initiatives invest in programs of research that seek to bring community and partner engagement together with D&I research to tackle entrenched health disparities and challenges across topics such as cancer [28,29,30], HIV [31, 32], opioid addiction [33, 34], and environmental health [35]. In addition, the Clinical and Translational Science Award (CTSA) Program has long prioritized engaging communities in the research process, but has recently increased their emphasis on diversifying who is being engaged and expanded their integration of D&I research into their infrastructure and function [36, 37]. These efforts are promising, yet we currently lack a comprehensive understanding of how NIH investments are working to support this integration, whether they are in fact successfully stimulating the field to conduct studies that increasingly incorporate and integrate engaged research approaches in D&I research more broadly. Such an understanding would allow us to identify gaps and opportunities to further stimulate and support the field and ultimately to produce more rigorous, relevant, and impactful science.

The purpose of this analysis is to provide insight into the current range of practices and approaches being used to engage communities and partners in NIH-sponsored D&I research, with special attention to disparities-relevant research, and to identify gaps and opportunities. This analysis focuses on new research funded in fiscal years 2020 and 2021, the most current complete data available at the time of analysis, to evaluate whether recent efforts to promote community engagement in research across NIH through specific initiatives and conversations such as those mentioned above have extended into the general NIH-funded D&I portfolio, which given its size may be a proxy for the D&I field more broadly. To that end, the research questions we sought to answer were as follows:

  1. 1

    To what extent and how are community members and other partners engaged in recent NIH-funded D&I research?

  2. 2

    Are there common themes that emerge with respect to engagement across implementation phases, strategies, or other key domains?

    1. a

      To what extent are NIH-funded engaged D&I research studies relevant to health disparities?

    2. b

      How do NIH-funded D&I research studies relevant to health disparities involve community or partner engagement?

Methods

This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guideline for reporting observational studies since it was examining a cross-section of grants that met specified eligibility criteria [38, 39]. This analysis included active research grants funded in fiscal years 2020 and 2021 across all NIH Institutes, Centers, and Offices that were funded through the dedicated D&I funding opportunities [1,2,3] or reviewed in the Science of Implementation in Health and Healthcare (SIHH) [4] (Dissemination and Implementation Research in Health (DIRH) through 10/2020) study section. An internal NIH portfolio analysis platform, iSearch, was used, which provides comprehensive, easy-to-use access to carefully curated, extensively linked datasets of funded and unfunded grant applications, historic and current.

The study team developed a codebook directly stemming from the research questions through an iterative process that included review of other NIH-specific portfolio analysis codebooks and publications, a review of the literature on D&I and community and partner engagement, and discussions among the study team and collaborators. Table 1 shows data extracted from grants. Grant characteristics were extracted from the metadata in iSearch. Variables coded captured information about the condition under study and details regarding the intervention under study including the social ecological level(s) targeted [40, 41] and where and by whom they are delivered. Codes also recorded details about the study design and methods [42,43,44], implementation phase, implementation strategies used [45, 46], and use of theories, models, and frameworks [47, 48]. Grants were coded for where on the research continuum (e.g., intervention design, evaluation) and through what approaches (e.g., practice-based research [49], community-based participatory research (CBPR) [50]) engagement was undertaken. Grants were also coded for the planned audience for engagement according to the 7Ps framework proposed by Concannon et al. [51] with definitions adapted from the Patient Centered Outcomes Research Institute (PCORI) [52], the highest level of engagement described according to the continuum proposed by Sanders Thompson et al. [53], and any indicators of equity in engagement the proposal described, drawing on concepts discussed by Key et al. [54] (see Table 1 for definitions). Grants were also coded as disparities-relevant if they related to minority health or health disparities, as defined by the National Institute on Minority Health and Health Disparities [55]. The codebook was programmed into iSearch’s curation tool and piloted on two grants by two members of the study team.

Table 1 Data extraction
Full size table

Full-text grant proposals were accessible through iSearch. Each grant proposal (research summary, human subjects, and budget justifications sections) was independently coded directly in iSearch by two assigned coders. Coders (N = 21) were generally assigned to grants from their own Institutes, with additional support provided by the NIH Office of the Director’s Division of Program Coordination, Planning, and Strategic Initiatives. Training for coders on how to use the iSearch platform was provided by the NIH Office of Portfolio Analysis. A subsequent training was held by the study team via videoconference to orient coders to the study purpose and review codebook variables and definitions. The recording of the training was made available to all coders to refer to throughout the coding period. The study team held optional “office hours” via video conference for coders three times over the course of the coding period to answer questions and offer clarifications about code definitions to facilitate reliability. The office hours recordings were distributed to all coders via emails that also included notes on important coding clarifications and examples reviewed during the office hour discussions. Coding discrepancies were resolved through discussion to consensus between the coding pairs and amended in iSearch. If intercoder consensus could not be reached, a third coder from the study team reviewed the grants and assisted in resolving disagreements, though this occurrence was rare given the clarifications provided via office hours and emails. Summary analyses, including descriptive statistics, were used to answer the first research question and bivariate cross-tabulations were used to answer the second research question. All analyses were conducted in a Microsoft Excel database exported from iSearch.

Results

A total of 103 grants met the inclusion criteria and were funded by 16 different NIH Institutes, Centers, and Offices (Table 2). A majority (57%) of grants were funded through the DIRH funding opportunities [1,2,3], 87% were reviewed by the SIHH/DIRH study section, and most were R01s (64%) or R21s (22%). Ten grants did not involve interventions and rather were solely focused on developing databases, conducting modeling, developing or validating measures, retrospectively studying policies or grants portfolios, or were in early pre-implementation exploratory phases prior to developing an intervention. Over two-thirds of grants (69%) targeted two or more levels of the social-ecological model; predominantly the individual (66%) and organizational (62%) levels, with the fewest targeting the societal level (6%). The most common intervention settings included healthcare organizations (67%) and the community (18%). Accordingly, common intervention implementers included healthcare providers (63%), lay health personnel (17%), clinic support staff (15%), and clinic administrators (13%). Less common implementers included policymakers (4%), social service workers (5%), public health professionals (7%), and implementation support practitioners like knowledge brokers, technical assistance providers, or practice facilitators (8%).

Table 2 Characteristics of included grants (N = 103)
Full size table

Table 3 shows that most studies (87%, n = 90) included some form of community or partner engagement. Engagement was planned across the research continuum, with formative research (61%) and implementation strategy design (52%) being the most common and theory development (6%) being least common. On average, each study engaged 2.5 different partner types with direct service providers (77%) and patients/survivors (54%) being the most common, and product makers (3%) and purchasers or payers (7%) being the least common. The highest level of engagement ranged Sanders Thompson et al.’s [53] continuum with consultation being the most common (56%) and partnership (3%) and outreach (4%) being the least common. Practice-based research was a common approach (23%), as was CBPR (16%), while one-third of studies did not specify any particular approach to engagement. On average, each study used 2.2 engagement strategies, with the most common being key informant data collection (77%) and project advisory groups (39%). Nearly half (48%) of studies did not describe any indicators for equitable engagement, but for those that did, influence (28%) and mutual benefit (26%) were most clearly articulated. Finally, only 16% of engaged research grants reported formally measuring engagement.

Table 3 Characteristics of the engagement in included grants
Full size table

In looking at themes that emerged with respect to engagement and key implementation domains, studies that were categorized as D&I or de-implementation all involved engagement at some level, whereas 14% (n = 11) of the grants categorized as implementation-only studies had no engagement (Table 4). Similarly, 12% (n = 11) of studies coded as involving an intervention reported no engagement (data not shown in table). Compared to grants without community or partner engagement, engaged research projects were more likely to be focused on implementation phases of testing strategies (61% vs. 31%), scaling up (20% vs. 8%), and/or sustainability (19% vs. 8%). Grants involving engagement were more likely to use a case study (3% vs. 0%) or observational design (37% vs. 23%) and non-engaged grants were more likely to use modeling (15% vs. 7%); there was no substantial difference in the use of experimental or quasi-experimental designs. All grants set in the community or within a community-based organization involved some level of engagement (data not shown in table). Grants involving community or partner engagement appeared more likely to leverage less-common D&I theories, models, and frameworks or those from other disciplines compared to grants without engagement (60% vs. 23%). Compared to grants without community or partner engagement, engaged research projects were more likely to apply the following implementation strategies: adapt and tailor to context (50% vs. 38%), develop interrelationships (36% vs. 23%), engage consumers (40% vs. 8%), provide interactive assistance (32% vs. 23%), support clinicians/service providers (29% vs. 8%), train and educate (48% vs. 23%), and use evaluative and iterative strategies (52% vs. 23%).

Table 4 Characteristics of implementation domains for all included grants, and by whether or not they were engaged research
Full size table

In looking at themes that emerged with respect to engagement in the subset of 47 grants focused on populations with health disparities, all but two had some level of engagement (96%, n = 45). Compared to non-disparities studies, disparities-relevant studies were almost twice as likely to engage partners at the level of partnership or collaboration (19% vs. 11%) (Fig. 1). About 26% of disparities-focused studies were conducted in the community or at a community-based organization compared to just 5% of non-disparities studies. CBPR was the most common engagement approach described in studies with populations experiencing disparities at 26% compared with only 4% in studies not focused on disparities.

Fig. 1
figure 1

Comparison of how NIH-funded D&I research grants involve community or partner engagement by disparities relevance

Full size image

Discussion

This analysis illuminated gaps in the application of engagement in NIH-funded D&I research and opportunities to study engagement processes and outcomes in the context of this research (Fig. 2).

Fig. 2
figure 2

Summary of gaps and opportunities in community engaged implementation science

Full size image

For our primary research question of to what extent and how are community members and partners engaged in recent NIH-funded D&I research, it was not surprising to see that nearly nine in ten grants reviewed involved engagement in some way, given the increased attention to engaged research in recent years. However, engagement scientists might argue that the more than half of studies involving partners at the level of consultation are not truly meeting the spirit of engagement when partners are simply serving as research participants, key informants, or advisory board members. Level of engagement did not appear to vary by type of partner, though that was difficult to examine since multiple partners were often engaged in each grant. Full partnership may not be warranted or necessary in every research study, but the rigor and relevance of D&I research would likely benefit if investigators were engaging communities and partners at the higher levels of collaboration and partnership more often. This opportunity goes beyond engagement just in formative research and implementation strategy design. To fully appreciate the potential benefits of such engagement, opportunities across the research continuum will likely be necessary. For example, community and partner engagement in theory development could result in theories, models, and frameworks that are more relevant and applicable to the populations of interest. Likewise, engagement in intervention design and evaluation may lead to more effective and sustainable interventions and assessment of outcomes that matter most to communities and partners.

It was encouraging to see that multiple diverse groups of partners, on average, were engaged in each study. For intervention implementers, we found opportunities remain to increase engagement of policymakers, social service workers, public health professionals, and implementation support practitioners (e.g., knowledge brokers, technical assistance providers, practice facilitators). Further, engagement with product makers, purchasers or payers, and health equity experts could be considered to potentially increase outcomes related to “designing for dissemination” including acceptability, cost-effectiveness, scalability, and sustainability [57, 58]. Our findings were largely consistent with the findings from the review of 103 child mental health implementation studies by Triplett et al. [17]. Providers were the most commonly engaged and both analyses highlighted gaps in terms of limited depth of engagement (skewed toward consultation) and opportunities to increase engagement with policymakers, private payers, and patients/clients.

Whereas an engagement approach was not specified in one in three grants that had engagement activities, it was encouraging to see that nearly one quarter of the 90 grants with engagement were employing a practice-based research approach (23%, n = 22), and more than a quarter were conducting CBPR or following a generic community-engaged research approach (27%, n = 24). Grant applications that did not identify a specific approach were most commonly operating at the consultation level, reinforcing critiques that they are likely not meeting the true spirit of engagement. Opportunities remain for incorporating strategies that involve deeper engagement approaches with both end-user implementers and beneficiaries of innovations such as human or user centered design [59], process mapping [60], systems mapping [61], intervention mapping [62], or implementation mapping [63], and other longitudinal and hands-on strategies. Future research can explore the impact of these increasing levels of engagement and various strategies on relevant implementation outcomes.

Indicators of equitable engagement were challenging to extract and infer from the proposals, despite having access to the full narrative and related budget documents. Where indicators were apparent, partners’ influence on the research and mutual benefit of engagement were most commonly evident. Little to no description of issues of ownership, responsibilities, or power and control over the data, findings, or otherwise was found. It is possible that issues of equitable engagement emerged and were addressed over the course of the project but were not included in the proposal for several reasons: not part of grant evaluation criteria; lack of space; not yet defined with partners; thought to be irrelevant to the science. Regardless, the issues around the influence of power, responsibility, ownership, and control in engaged research and as they relate to dismantling structural drivers of social determinants of health (SDOH) are ripe for future research. Exploring power dynamics in engaged research and proactively co-creating acceptable engagement structures could enhance community participatory research [64]. In 2021, Shelton et al. [65] outlined recommendations to the D&I research field for addressing structural racism that included leveraging engagement as an essential method, assessing and addressing power differentials, and applying multi-level approaches for implementing interventions, policies, and strategies to address structural discrimination and advance health equity. In 2022, Stanton et al. [66] proposed a typology of 3 types of power that appear in implementation efforts. Discursive, epistemic, and material power respectively relate to how narratives about health and its determinants are formed, what forms of evidence are valued, and how resources are distributed. Stanton et al. [66] outlined a number of research avenues to investigate how power operates and influences implementation and health equity-related outcomes. Consistent with these recommendations, the findings of the present analysis suggest that opportunities remain to explicitly address equitable engagement in D&I research as it could influence how these types of power are established and leveraged by ensuring that a diversity of voices and experiences are included and elevated.

D&I research offers promise for more systematically applying engagement and in so doing, providing a venue for advancing the science of engagement. Distinct from participatory research that uses engagement techniques, engagement science examines the methods and outcomes of engagement in order to develop an evidence base for why engagement matters and how to do it well [67,68,69]. Triplett et al.’s review found that implementation outcomes most likely to be related to engagement were least reported and few of the more than 100 projects reviewed reported the impacts of engagement [17]. Certainly, studying engagement and relating it to D&I strategies or outcomes is not possible without measuring or documenting engagement in some way. Given that most grants reviewed here did not formally assess engagement, there remains a significant opportunity for D&I scientists to fully embrace engagement approaches and rigorously assess and measure these processes to ideally make an impact on the sciences of D&I and of engagement moving forward.

Our second research question strived to uncover any trends in engagement variables based on key study characteristics. There were noted missed opportunities for engaging key community members and partners in a portion of studies focused solely on implementation. Furthermore, there was a notable lack of engagement described in studies that involved delivery of an intervention; future studies could likely benefit from more active attempts to engage the implementers, decision makers, and intervention beneficiaries more substantially and to examine the impact that may have on study design, execution, and outcomes. Although there were fewer grants in total that were both D&I (n = 23), they all involved engagement of some sort, possibly due to the increased challenge of studying and successfully accomplishing both dissemination and implementation within a single time-limited grant. Similarly, grants involving testing strategies, scaling up, or studying sustainability were more likely to involve community or partner engagement, possibly due to the active and complex nature of these phases of implementation. In their review, Triplett et al. also found deeper engagement with more advanced phases of implementation [17]. Many implementation efforts require engagement with a decision maker to facilitate the test of change, and studying scale up and sustainability are intuitively enhanced with the involvement of partners responsible for expanding and maintaining program implementation. Similarly, case study and observational study designs appeared to be more common in engaged grants, likely because they required access through partners to collect data. In contrast, grants without engagement were more likely to utilize modeling approaches, which may not obviously require input from partners. Yet, partner engagement in modeling approaches to interventions remains a relatively unexplored area of inquiry that has the potential to make such efforts more responsive and relevant to the communities they aim to serve. Indeed, an initial effort to develop a “participatory systems dynamics modeling” approach has shown promise [70]. Logically, the implementation strategies employed in engaged grants tended to be strategies that stemmed from that engagement (e.g., engaging consumers, developing interrelationships, training). Similarly, community-situated studies all involved engagement and there was greater indication of decision making and power and control in studies at higher levels of engagement.

It was encouraging to find that nearly half of grants overall were relevant to disparities in some way. Given the potential of D&I research and engagement approaches to address SDOH and advance health equity among underserved populations who experience disparities [24, 71, 72], opportunities remain for growing the amount of disparities-related and equity-focused engaged D&I studies. In particular, there was limited engaged research with some populations experiencing health disparities such as underserved rural populations, sexual and gender minorities, and others (e.g., disability communities). With respect to engagement themes among disparities-focused grants, nearly all these grants involved engagement at some level, and they engaged partners more deeply at the levels of collaboration or partnership as compared to non-disparities focused grants. This may be in part due to the longitudinal commitment necessitated by equity-focused work as well as the noted increasing federal support for engaged research approaches to advance health equity. The disparities-focused grants were more likely to be conducted in a community setting or at a community-based organization and were more likely to involve multi-level interventions as compared to non-disparities grants. These findings perhaps indicate that D&I researchers are beginning to answer the calls to truly partner with communities and devise multi-pronged strategies that address complex problems, build capacity among D&I researchers, and incorporate equity across implementation focus areas [65, 73].

NIH has been a leader in building the field of D&I research, co-founding the Annual Conference on the Science of Dissemination and Implementation in 2007 and growing a dedicated D&I grants portfolio over nearly 20 years that is now supported by most of the Institutes, Centers, and Offices. Our analysis was limited to grants funded through the three NIH-wide D&I funding opportunities and/or reviewed by the SIHH/DIRH study section. These are investigator-initiated research awards, not necessarily related to a targeted funding opportunity requiring engagement, so this sample of grants should be representative of the range of current investigator-initiated research in the D&I field. With more than approximately $1 billion in D&I funding in fiscal year 2022 according to NIH’s categorical spending report [74], we are confident that NIH is one of the largest funders of D&I research and therefore the gaps and opportunities identified in this portfolio (Fig. 2) may extend to the broader D&I field. Consistent with gaps previously noted in the literature [17, 22, 24], we identified engagement in most grants analyzed but engagement was often not deep or rigorous, lacked a diversity of partners, and failed to evaluate engagement processes or outcomes. NIH and other funders of D&I research may wish to conduct internal analyses of initiatives that have promoted community and partner engagement generally, and specifically in D&I research, to identify successful strategies to use in future funding opportunities. In addition, other funding organizations, such as PCORI, may offer lessons learned about funding opportunity language, review criteria, investigator training, and other practices that address noted gaps in the application of engagement science within D&I research.

Strengths and limitations

This portfolio analysis had several strengths, including the large sample of grants, the use of dual coders, and a structured codebook. In addition, as an internal NIH project, coders had access to the full grant proposal narrative including the research summary and budget documents to be able to code more accurately.

Yet, important limitations were present as well. The inclusion criteria were limited to a specific set of funding opportunities and a single study section; this likely omitted some D&I research funded across NIH through other mechanisms and study sections. However, the included grants do incorporate the single largest portion of individual investigator-initiated D&I research grants supported by the NIH. Another limitation is that NIH is not the only funder of D&I research, although possibly the primary one, and because this analysis was limited to a selection of NIH-funded grants, we cannot speak to the work funded by others. Given the literature cited in the introduction and discussion has similar themes to our findings, it is likely that this portfolio analysis is fairly representative to the state of the field at large.

Another limitation is that coders reviewed grant proposals, which may or may not be a true reflection of study activities implemented once funded. Some engagement variables coded, such as approach and equity indicators, are elements that are not required to be specified or described in proposals, potentially leading to an overcount of “not specified/described.” Finally, despite training coders, developing a thorough codebook with definitions, and providing support throughout the project, coding is subject to human judgement and manual data entry errors, though this was mitigated by dual coding to agreement.

Conclusions

Based on this portfolio analysis, the D&I research field appears to regularly integrate community and partner engagement approaches and strategies, though opportunities remain to deepen engagement and diversify partners. Further, to meaningfully advance health equity, there are critical opportunities for D&I research that leverages community and partner engagement to address health disparities, especially among underserved populations. Through this increased attention to engagement within D&I research, new opportunities to study engagement-related research questions may be pursued that will likely increase the rigor, relevance, and impact of both engagement science and D&I research.

Availability of data and materials

The datasets generated and analyzed during the current study are not publicly available due to the confidential nature of grant proposals but a limited dataset is available from the corresponding author on reasonable request.

Abbreviations

CBPR:

Community-Based Participatory Research

COVID-19:

Coronavirus-2019

D&I:

Dissemination and implementation

DIRH:

Dissemination and Implementation Research in Health

NIH:

National Institutes of Health

SDOH:

Social determinants of health

SIHH:

Science of Implementation in Health and Healthcare

TIDIRC:

Training in Dissemination and Implementation Research in Cancer

References

  1. PAR-19-274: Dissemination and Implementation Research in Health (R01 Clinical Trial Optional). Cited 2022 Nov 28. Available from: https://grants.nih.gov/grants/guide/pa-files/PAR-19-274.html.

  2. PAR-19-276: Dissemination and Implementation Research in Health (R03 Clinical Trial Not Allowed). Cited 2022 Nov 28. Available from: https://grants.nih.gov/grants/guide/pa-files/PAR-19-276.html.

  3. PAR-19-275: Dissemination and Implementation Research in Health (R21 Clinical Trial Optional). Cited 2022 Nov 28. Available from: https://grants.nih.gov/grants/guide/pa-files/PAR-19-275.html.

  4. SIHH | NIH Center for Scientific Review. Cited 2022 Nov 28. Available from: https://public.csr.nih.gov/StudySections/DABP/HSS/SIHH.

  5. 2022 D&I - About | AcademyHealth. Cited 2022 Nov 28. Available from: https://academyhealth.org/page/2022-di-about.

  6. Training Institute for Dissemination and Implementation Research in Cancer (TIDIRC) OpenAccess | Division of Cancer Control and Population Sciences (DCCPS). Cited 2022 Nov 28. Available from: https://cancercontrol.cancer.gov/is/training-education/training-in-cancer/TIDIRC-open-access.

  7. Holt CL, Chambers DA. Opportunities and challenges in conducting community-engaged dissemination/implementation research. Transl Behav Med. 2017;7(3):389–92.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Blachman-Demner DR, Wiley TRA, Chambers DA. Fostering integrated approaches to dissemination and implementation and community engaged research. Transl Behav Med. 2017;7(3):543–6.

  9. Hursting LM, Chambers DA. Practitioner engagement in implementation science: initiatives and opportunities. J Public Health Manag Pract. 2021;27(2):102–4.

    Article  PubMed  Google Scholar 

  10. Glandon D, Paina L, Alonge O, Peters DH, Bennett S. 10 Best resources for community engagement in implementation research. Health Policy Plan. 2017;32(10):1457–65.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Mensah GA, Curry JS, Engelgau MM, Johnson LE. Stakeholder engagement in late-stage translation research and implementation science: perspectives from the National Heart, Lung, and Blood Institute. Glob Heart. 2019;14(2):191–4.

    Article  PubMed  Google Scholar 

  12. Hudon C, Chouinard MC, Bisson M, Danish A, Karam M, Girard A, et al. Case study with a participatory approach: rethinking pragmatics of stakeholder engagement for implementation research. Ann Fam Med. 2021;19(6):540–6.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Ramanadhan S, Davis MM, Armstrong R, Baquero B, Ko LK, Leng JC, et al. Participatory implementation science to increase the impact of evidence-based cancer prevention and control. Cancer Causes Control. 2018;29(3):363–9.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Stalker KC, Brown ME, Evans CBR, Hibdon J, Telep C. Addressing crime, violence, and other determinants of health through community-based participatory research and implementation science. Am J Commun Psychol. 2020;66(3–4):392–403.

    Article  Google Scholar 

  15. Ramanadhan S, Donaldson ST, Siqueira CE, Rackard-James C, Miller E, Tappin J, et al. Connecting implementation science, community-engaged research, and health promotion to address cancer inequities in Massachusetts: The UMB/DF-HCC U54 Outreach Core. Am J Health Promot. 2022;36(4):597–601.

    Article  PubMed  Google Scholar 

  16. Skolarus LE, Sales AE, Zimmerman MA, Corches CL, Landis-Lewis Z, Robles MC, et al. Stroke Ready: a multi-level program that combines implementation science and community-based participatory research approaches to increase acute stroke treatment: protocol for a stepped wedge trial. Implement Sci. 2019;14(1):24.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Triplett NS, Woodard GS, Johnson C, Nguyen JK, AlRasheed R, Song F, et al. Stakeholder engagement to inform evidence-based treatment implementation for children’s mental health: a scoping review. Implement Sci Commun. 2022;3(1):82.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Adam MB, Minyenya-Njuguna J, Karuri Kamiru W, Mbugua S, Makobu NW, Donelson AJ. Implementation research and human-centred design: how theory driven human-centred design can sustain trust in complex health systems, support measurement and drive sustained community health volunteer engagement. Health Policy Plan. 2020;35(Supplement_2):ii150–62.

    Article  PubMed  PubMed Central  Google Scholar 

  19. RADx Programs. National Institutes of Health (NIH). 2020. Cited 2022 Nov 28. Available from: https://www.nih.gov/research-training/medical-research-initiatives/radx/radx-programs .

    Google Scholar 

  20. Community Engagement Alliance (CEAL) Against COVID-19 Disparities | Community Engagement Alliance. Cited 2022 Nov 28. Available from: https://covid19community.nih.gov/.

  21. Community Partnerships to Advance Science for Society (ComPASS). 2022. Cited 2023 May 10. Available from: https://commonfund.nih.gov/compass.

  22. Pinto RM, Park S (Ethan), Miles R, Ong PN. Community engagement in dissemination and implementation models: a narrative review. Implement Res Pract. 2021;2:2633489520985305.

    PubMed  PubMed Central  Google Scholar 

  23. Shea CM, Young TL, Powell BJ, Rohweder C, Enga ZK, Scott JE, et al. Researcher readiness for participating in community-engaged dissemination and implementation research: a conceptual framework of core competencies. Transl Behav Med. 2017;7(3):393–404.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Beidas RS, Dorsey S, Lewis CC, Lyon AR, Powell BJ, Purtle J, et al. Promises and pitfalls in implementation science from the perspective of US-based researchers: learning from a pre-mortem. Implement Sci. 2022;17(1):55.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Organizing Committee for Assessing Meaningful Community Engagement in Health & Health Care Programs & Policies. Assessing meaningful community engagement: a conceptual model to advance health equity through transformed systems for health. NAM Perspectives. 2022; Cited 2022 Nov 28. Available from: https://nam.edu/assessing-meaningful-community-engagement-a-conceptual-model-to-advance-health-equity-through-transformed-systems-for-health/.

  26. Haapanen KA, Christens BD. Community-engaged research approaches: multiple pathways to health equity. Am J Commun Psychol. 2021;67(3–4):331–7.

    Article  Google Scholar 

  27. Payán D, Zawadzki M, Song A. Advancing community-engaged research to promote health equity: considerations to improve the field. Perspect Public Health. 2022;142(3):139–41.

    Article  PubMed  Google Scholar 

  28. PE CGS. About PE-CGS. Participant Engagement and Cancer Genome Sequencing. Cited 2023 May 10. Available from: http://pe-cgs.containers.it.osu.edu/.

  29. Implementation Science Centers in Cancer Control. ISC3. Cited 2023 May 10. Available from: https://iscentersincancercontrol.org/.

  30. ACCSIS - Home Page. Cited 2023 May 10. Available from: https://accsis.rti.org/.

  31. July 1, 2022 - Volume 90 - Issue S1 : JAIDS Journal of Acquired Immune Deficiency Syndromes. Cited 2023 May 10. Available from: https://journals.lww.com/jaids/toc/2022/06001.

  32. Tucker JD, Iwelunmor J, Abrams E, Donenberg G, Wilson EC, Blachman-Demner D, et al. Accelerating adolescent HIV research in low-income and middle-income countries: evidence from a research consortium. AIDS. 2021;35(15):2503.

    Article  PubMed  Google Scholar 

  33. Ducharme LJ, Wiley TRA, Mulford CF, Su ZI, Zur JB. Engaging the justice system to address the opioid crisis: The Justice Community Opioid Innovation Network (JCOIN). J Subst Abuse Treat. 2021;128:108307.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Chandler R, Clarke T, Walters S. The HEALing Communities Study: an integrated approach to addressing opioid misuse, addiction, and overdose. Drug and Alcohol Dependence. 2021 Cited 2023 May 10. Available from: https://www.sciencedirect.com/journal/drug-and-alcohol-dependence/special-issue/10B0L1KKVKK.

  35. Community Engagement Cores - Environmental Health Sciences Core Centers. National Institute of Environmental Health Sciences. Cited 2022 Nov 28. Available from: https://www.niehs.nih.gov/research/supported/centers/core/coe/index.cfm.

  36. Leppin AL, Mahoney JE, Stevens KR, Bartels SJ, Baldwin LM, Dolor RJ, et al. Situating dissemination and implementation sciences within and across the translational research spectrum. J Clin Transl Sci. 2019;4(3):152–8.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Mahoney JE, Stevens KR, Mehta T. Cross walk between consensus recommendations and new NCATS PAR-21-293 requirements for D&I in CTSA hubs. J Clin Transl Sci. 6(1):e17.

  38. Von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007;370(9596):1453–7.

    Article  Google Scholar 

  39. Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. Epidemiology. 2007;18(6):805–35.

    Article  PubMed  Google Scholar 

  40. NIMHD Research Framework Details. NIMHD. Cited 2023 May 10. Available from: https://www.nimhd.nih.gov/about/overview/research-framework/research-framework.html.

  41. McLeroy KR, Bibeau D, Steckler A, Glanz K. An Ecological Perspective on Health Promotion Programs. Health Educ Quarterly. 1988;15(4):351–77.

    Article  CAS  Google Scholar 

  42. Brown CH, Curran G, Palinkas LA, Aarons GA, Wells KB, Jones L, et al. An overview of research and evaluation designs for dissemination and implementation. Ann Rev Public Health. 2017;38(1):1–22.

    Article  Google Scholar 

  43. Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health. 2011;38(1):44–53.

    Article  PubMed  Google Scholar 

  44. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation Hybrid Designs: Combining Elements of Clinical Effectiveness and Implementation Research to Enhance Public Health Impact. Med Care. 2012;50(3):217–26.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10(1):21.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10(1):109.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging Research and Practice: Models for Dissemination and Implementation Research. Am J Prev Med. 2012;43(3):337–50.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10(1):53.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Potter MA, Quill BE, Aglipay GS, Anderson E, Rowitz L, Smith LU, et al. Demonstrating excellence in practice-based research for public health. Public Health Rep. 2006;121(1):1–16.

    Article  Google Scholar 

  50. Wallerstein N, Duran B. Community-based participatory research contributions to intervention research: the intersection of science and practice to improve health equity. Am J Public Health. 2010;100(Suppl 1):S40–6.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise JM, Santa J, et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med. 2012;27(8):985–91.

    Article  PubMed  PubMed Central  Google Scholar 

  52. PCORI’s Stakeholders. PCORI’s Stakeholders | PCORI. 2014 Cited 2023 Jan 13. Available from: https://www.pcori.org/about-us/our-programs/engagement/public-and-patient-engagement/pcoris-stakeholders.

  53. Sanders Thompson VL, Ackermann N, Bauer KL, Bowen DJ, Goodman MS. Strategies of community engagement in research: definitions and classifications. Transl Behav Med. 2021;11(2):441–51.

    Article  PubMed  Google Scholar 

  54. Key KD, Furr-Holden D, Lewis EY, Cunningham R, Zimmerman MA, Johnson-Lawrence V, et al. The continuum of community engagement in research: a roadmap for understanding and assessing progress. Progress Commun Health Partnerships. 2019;13(4):427–34.

    Article  Google Scholar 

  55. HDPulse - Health Disparities and Minority Health Resources (NIMHD). Cited 2022 Nov 28. Available from: https://hdpulse.nimhd.nih.gov/index.html.

  56. Norton WE, Chambers DA, Kramer BS. Conceptualizing de-Implementation in cancer care delivery. J Clin Oncol. 2019;37(2):93–6.

    Article  PubMed  Google Scholar 

  57. Brownson RC, Jacobs JA, Tabak RG, Hoehner CM, Stamatakis KA. Designing for dissemination among public health researchers: findings from a national survey in the United States. Am J Public Health. 2013;103(9):1693–9.

    Article  PubMed  PubMed Central  Google Scholar 

  58. Mora Pinzon MC, Myers S, Renken J, Eggert E, Chewning B, Mahoney JE. Essential elements to “design for dissemination” within a research network—a modified Delphi study of the Community-Academic Aging Research Network (CAARN). Implement Sci Commun. 2021;2(1):18.

    Article  PubMed  PubMed Central  Google Scholar 

  59. Beres LK, Simbeza S, Holmes CB, Mwamba C, Mukamba N, Sharma A, et al. Human-centered design lessons for implementation science: improving the implementation of a patient-centered care intervention. J Acquir Immune Defic Syndr. 2019;82 Suppl 3(3):S230–43.

    Article  PubMed  Google Scholar 

  60. Kononowech J, Landis-Lewis Z, Carpenter J, Ersek M, Hogikyan R, Levy C, et al. Visual process maps to support implementation efforts: a case example. Implement Sci Commun. 2020;1(1):105.

    Article  PubMed  PubMed Central  Google Scholar 

  61. Kiekens A, de Casterlé BD, Vandamme AM. Qualitative systems mapping for complex public health problems: A practical guide. PLOS One. 2022;17(2):e0264463.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  62. Bighetti N. Intervention Mapping. Intervention Mapping. Cited 2022 Nov 28. Available from: https://www.interventionmapping.com/

  63. Fernandez ME, ten Hoor GA, van Lieshout S, Rodriguez SA, Beidas RS, Parcel G, et al. Implementation mapping: using intervention mapping to develop implementation strategies. Front Public Health. 2019. Cited 2022 Nov 28;7. Available from: https://www.frontiersin.org/articles/10.3389/fpubh.2019.00158.

  64. Roura M. The social ecology of power in participatory health research. Qual Health Res. 2021;31(4):778–88.

    Article  PubMed  Google Scholar 

  65. Shelton RC, Adsul P, Oh A. Recommendations for addressing structural racism in implementation science: a call to the field. Ethnic Dis. 2021;31(Suppl):357–64.

    Article  Google Scholar 

  66. Stanton MC, Ali SB, Team the SC. A typology of power in implementation: building on the exploration, preparation, implementation, sustainment (EPIS) framework to advance mental health and HIV health equity. Implement Res Practice. 2022;3:250.

    Article  Google Scholar 

  67. Meissner P, Cottler LB, Eder M “Mickey”, Michener JL. Engagement science: The core of dissemination, implementation, and translational research science. J Clin Transl Sci. 4(3):216–8.

  68. AcademyHealth. Engagement Science: Introducing Inclusive Research Practices & Potential Impacts. Cited 2023 Jan 13. Available from: https://academyhealth.org/blog/2019-01/engagement-science-introducing-inclusive-research-practices-potential-impacts.

  69. Science of Engagement PCORI Funding Announcement -- Cycle 2 2022. 2022. Cited 2023 Jan 13. Available from: https://www.pcori.org/funding-opportunities/announcement/science-engagement-pcori-funding-announcement-cycle-2-2022.

  70. Zimmerman L, Lounsbury DW, Rosen CS, Kimerling R, Trafton JA, Lindley SE. Participatory system dynamics modeling: increasing stakeholder engagement and precision to improve implementation planning in systems. Adm Policy Ment Health. 2016;43(6):834–49.

    Article  PubMed  PubMed Central  Google Scholar 

  71. Baumann AA, Cabassa LJ. Reframing implementation science to address inequities in healthcare delivery. BMC Health Serv Res. 2020;20(1):190.

    Article  PubMed  PubMed Central  Google Scholar 

  72. Emmons KM, Chambers DA. Commentary: Policy implementation science - an unexplored strategy to address social determinants of health. Ethnic Dis. 2021;31(1):133–8.

    Article  Google Scholar 

  73. Adsul P, Chambers D, Brandt HM, Fernandez ME, Ramanadhan S, Torres E, et al. Grounding implementation science in health equity for cancer prevention and control. Implement Sci Commun. 2022;3(1):56.

    Article  PubMed  PubMed Central  Google Scholar 

  74. National Institutes of Health Research Portfolio Online Reporting Tools. Estimates of Funding for Various Research, Condition, and Disease Categories (RCDC) 2023 [updated March 31, 2023]. Available from: https://report.nih.gov/funding/categorical-spending#/.

Download references

Acknowledgements

The authors would like to thank their colleagues who helped with coding: Brenda Adjei, Fernando Bruno, Lori Ducharme, Kalina Duncan, Andrea Horvath Marques, Marion Koso-Thomas, Keith Mintzer, Elizabeth Neilson, Gila Neta, April Oh, Suzanne Pollard, Laura Prakash, Jenelle Walker, Jackie Wallace, Miya Whitaker, and Tisha Wiley. The authors also appreciate the thoughtful feedback on earlier drafts of this manuscript by David Chambers, Christine Hunter, and Tilda Farhat.

Funding

Open Access funding provided by the National Institutes of Health (NIH). This study was conducted by NIH staff members, trainees, and contractors as part of their official duties. No additional funding was used to support this research.

Author information

Authors and Affiliations

  1. Division of Cancer Control and Population Sciences, National Cancer Institute, 9609 Medical Center Dr, Bethesda, MD, 20892, USA

    Aubrey Villalobos & Manami Bhattacharya

  2. Office of Behavioral and Social Sciences Research, Office of the Director, National Institutes of Health, 31 Center Dr, Bethesda, MD, 20892, USA

    Dara Blachman-Demner & Deshiree Belis

  3. Office of Science Policy, Planning, Evaluation and Reporting, National Institute on Minority Health and Health Disparities, 6707 Democracy Blvd, Bethesda, MD, 20817, USA

    Antoinette Percy-Laurry

Authors
  1. Aubrey Villalobos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dara Blachman-Demner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Antoinette Percy-Laurry
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Deshiree Belis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Manami Bhattacharya
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

AV, DBD, and APL conceptualized the study’s research questions and methods. AV led the project administration and supervision of coders, development of the codebook, data cleaning and analysis, and manuscript writing and editing. DB was a major contributor to coding, data cleaning, and analysis. APL and DBD were also major contributors in writing the manuscript and editing. MB contributed to advanced data analysis and manuscript review and editing. All authors read and approved the final manuscript. The views expressed in this article are those of the authors and do not necessarily represent the official position of the National Institutes of Health.

Corresponding author

Correspondence to Aubrey Villalobos.

Ethics declarations

Ethics approval and consent to participate

Not applicable. Ethics approval and informed consent to participate were not required as this manuscript does not report on human subject research.

Consent for publication

Not applicable

Competing interests

Other than being affiliated with the National Institutes of Health, whose funded research portfolio is the subject of this analysis, the authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Villalobos, A., Blachman-Demner, D., Percy-Laurry, A. et al. Community and partner engagement in dissemination and implementation research at the National Institutes of Health: an analysis of recently funded studies and opportunities to advance the field. Implement Sci Commun 4, 77 (2023). https://doi.org/10.1186/s43058-023-00462-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s43058-023-00462-y

Keywords

Implementation Science Communications

ISSN: 2662-2211