Skip to main content
Download PDF
Download ePub

A pragmatic evaluation of a public health knowledge broker mentoring education program: a convergent mixed methods study

Implementation Science Communications volume 3, Article number: 18 (2022) Cite this article

Abstract

Background

Public health professionals are expected to use the best available research and contextual evidence to inform decision-making. The National Collaborating Centre for Methods and Tools developed, implemented, and evaluated a Knowledge Broker mentoring program aimed at facilitating organization-wide evidence-informed decision-making in ten public health units in Ontario, Canada. The purpose of this study was to pragmatically assess the impact of the program.

Methods

A convergent mixed methods design was used to interpret quantitative results in the context of the qualitative findings. A goal-setting exercise was conducted with senior leadership in each organization prior to implementing the program. Achievement of goals was quantified through deductive coding of post-program interviews with participants and management. Interviews analyzed inductively to qualitatively explain progress toward identified goals and identify key factors related to implementation of EIDM within the organization.

Results

Organizations met their goals for evidence use to varying degrees. The key themes identified that support an organizational shift to EIDM include definitive plans for participants to share knowledge during and after program completion, embedding evidence into decision-making processes, and supportive leadership with organizational investment of time and resources. The location, setting, or size of health units was not associated with attainment of EIDM goals; small, rural health units were not at a disadvantage compared to larger, urban health units.

Conclusions

The Knowledge Broker mentoring program allowed participants to share their learning and support change at their health units. When paired with organizational supports such as supportive leadership and resource investment, this program holds promise as an innovative knowledge translation strategy for organization wide EIDM among public health organizations.

Peer Review reports

Introduction

Evidence-informed decision-making (EIDM) in public health integrates the best available evidence from research and the local context to optimize decision-making. Achieving EIDM requires, in part, finding, synthesizing, and applying the best available evidence from research sources, community data and local contexts, societal and political preferences, and available resources [1,2,3]. EIDM can help maximize population health outcomes given finite public health resources by implementing strategies with known effectiveness [4, 5]. The Public Health Agency of Canada outlines core competencies for the public health workforce including finding, analyzing, and applying evidence from research and community sources [6]. In Ontario, Canada, EIDM has been identified as a foundational standard for practice [2].

Despite these expectations, barriers to achieving EIDM persist. Public health practitioners do not universally have the knowledge and skills for finding, appraising, and using evidence in practice; some public health organizations are unable or unwilling to support evidence-informed practice [7,8,9]. Strategies that focus solely on building individuals’ skills for EIDM have been effective in increasing the practitioners’ capacity but have not had organization-wide impact [10,11,12]. To achieve an organizational shift to EIDM, strategies must facilitate organizational change whilst building individual capacity [5, 13, 14]. In this context, the term “capacity” refers to the knowledge, skill, and ability to apply EIDM in practice, reflecting both competence and situational support [5, 15, 16].

Reviews of interventions to increase research use in public health have found that strategies should be adapted to the needs and priorities of the organization and individuals [17,18,19]. Knowledge broker (KB) roles are inherently well-suited to meet this requirement, as the scope of a KB’s role is linked to the organizational context [20]. KBs are responsible for knowledge management (obtaining relevant evidence, creating tailored knowledge products, supporting evidence sharing), knowledge linkage and exchange (facilitating collaboration, developing and maintaining networks), and capacity building (helping develop analytic skills, facilitating and evaluating change) [20]. In Canada, KBs have long been a component of efforts to support EIDM [21]. KBs in Canada are found across diverse contexts, such as facilitating stakeholder relationships for income assistance policies for people who use drugs [22], bridging the research-practice gap for uptake of measurement tools in children’s rehabilitation clinics [23], and partnering with secondary schools to facilitate the uptake of interventions to promote and improve student health [24]. While results of studies evaluating KB interventions in public health settings are mixed, KBs show promise in supporting organization wide EIDM [19, 25,26,27,28,29,30,31,32].

This pragmatic study sought to determine the effect of an intensive KB mentoring program delivered by the National Collaborating Centre for Methods and Tools (NCCMT) to facilitate organizational change for EIDM including organizational level use of evidence, perceptions of the value of the KB mentoring program, and public health units’ success or challenges in integrating EIDM.

Methods

Study design

A convergent mixed methods design was used to evaluate two sequential cohorts of the KB mentoring program, with individuals from five public health units participating in each cohort. In the quantitative phase of the study, senior leadership at each health unit participated in a focus group to set goals for research use. Following program completion interviews with participants and their managers were deductively coded to quantify the extent to which the goals were achieved. In the next phase, a descriptive qualitative study [33] investigated why some participating health units were more or less successful in implementing EIDM through inductive analysis of in-depth, semi-structured interviews. Findings from the quantitative and qualitative phase were merged and used to evaluate the overall impact of the KB mentoring program, as well as how and why health units achieved their identified goals. The study is reported following the Good Reporting of a Mixed Methods Study (GRAMMS) checklist [34]. Ethics approval was obtained (Hamilton Integrated Research Ethics Board Project #15-016), and participants provided written informed consent.

Sample

Organizations

Any Canadian public health organization involved in front-line service delivery was eligible to participate in the KB mentoring program. The NCCMT issued an open call through its newsletter and contacted organizations with previously expressed interest in the program. As a result of these efforts, ten public health organizations confirmed their participation in the program over two sequential cohorts. The first cohort ran from January 2015 to December 2016, and the second cohort ran from January 2017 to June 2018.

Individuals

Organizations identified five-to-six staff members to participate in the program by seeking volunteers or strategic selection. Diverse staff positions were eligible to participate including public health nurses, health promoters, dietitians, public health inspectors, librarians, managers, or others.

Knowledge broker mentoring program

The program’s objectives were to assess and assist public health organizations to develop organizational capacity for EIDM, as well as to build capacity among selected staff to function as “internal” knowledge brokers in evidence-informed practice. The program is described according to the TIDieR (Template for Intervention Description and Replication) Checklist; a completed TIDieR Checklist is included in Additional file 1: Appendix 1 [35]. Participants enrolled in the program were given opportunities to develop their knowledge about the EIDM process including the types of evidence used in public health decision-making and skills related to searching for, appraising, synthesizing, and applying evidence to practice. The program, co-designed with public health professionals, was also informed by collective decades of experience delivering EIDM education to public health. The four NCCMT knowledge translation specialist mentors who delivered the program held relevant graduate degrees and a wealth of experience in public health and EIDM. Mentors were assigned 1–3 organizations in the first cohort and two organizations in the second cohort. Intervention content was identical for each cohort.

The program began with an organizational assessment with the senior leadership team of each health unit. Assessments identified the leadership team’s EIDM goals for the program specifically and the organization generally.

Participants took part in a 5-day in-person workshop, followed by a 3-day workshop at 6 months, and a 2-day workshop at 12 months held at McMaster University in Hamilton, ON. A printed course pack containing the course syllabus and copies of the required readings were given to participants prior to the start of the Program. Workshops were held daily from 9 AM to 4 PM, consisting of a small number of didactic lectures, with most time spent in small-group problem-based sessions; content focused on a systematic approach to EIDM [1]. Between workshops, participants convened virtually each month to share progress and continue practice-based learning through group critical appraisals of research evidence. Groups at each public health organization met with mentors every 2–4 weeks for a 30-to-60-min consultation via teleconference to address emerging questions and strategize organizational change for EIDM. During the final 6 months of the program, participants completed a rapid review [36]. Participants selected synthesis topics with senior management to ensure findings would benefit the public health organization. Completion of these rapid reviews reinforced the EIDM skills participants gained during the program, including defining an answerable research question, searching for literature, critically appraising included studies, and synthesizing results for a final report. Examples of topics include workplace interventions for mental health, community interventions for testing and treatment of chlamydia, and characteristics of natural environments that affect youth mental health and wellbeing. A full list of research questions is included in Table 1.

Table 1 Characteristics of participating health units
Full size table

Phase 1: Quantitative study component

Characteristics of each participating health unit and the populations they serve were collected from government websites [37]. Participant demographic questionnaires were completed at baseline. Descriptive statistics were used to summarize participant demographics.

Organizational goals for EIDM and program participation were captured using the validated organizational self-assessment tool Is Research Working for You? [38]. Implementation of EIDM was assessed through telephone interviews between 6 and 18 months following program completion. The third-party interviewer had extensive experience in qualitative evaluations in public health, leadership development, and knowledge translation. All interviews were recorded and transcribed verbatim with identifying information removed. Data management and coding was facilitated with the use of NVivo 12 Plus. Interviewees included a purposeful sample of program participants, managers, and senior decision makers to obtain perspectives from participants and non-participants at various levels of each health unit.

Interviews probed the program’s organizational impact in terms of overall staff capacity and system and process changes toward realizing the organization’s EIDM goals. Interview questions are included in Additional file 2: Appendix 2. Goals set by each participating public health organization during organizational assessments at the program start were used to deductively code interviews [39]. In quantifying evidence for having met goals, there was substantial evidence for achieving a goal if the goal condition was described by at least three interviewees, including at least one manager; and there was some evidence if the goal condition was described by some interviewees but not a manager.

Phase 2: Qualitative study component

Following the quantification of goals achieved by each health unit, a descriptive qualitative study [33] was conducted to identify how EIDM was implemented within each health unit. Descriptive qualitative studies explore phenomena or processes from the perspectives of those involved, and findings can often be integrated in a mixed methods analysis [33]. The qualitative component of this study allowed for insight into why some participating health units were more or less successful in reaching goals for implementing EIDM.

Data were also analyzed using a conventional content analysis approach to describe processes related to EIDM implementation [40]. Categories that emerged from this conventional analysis were organized into meaningful clusters [41].

Phase 3: Data integration

Quantitative and qualitative data were merged in a mixed methods data analysis [42, 43]. Health units’ overall success in EIDM implementation was measured through quantitative analysis of evidence for meeting goals, and their successes were explained through integration of results of an inductive analysis of interviews. Specifically, categories identified in the qualitative analysis were linked to each organization’s stated goals, such that categories provided insight into factors that impacted EIDM implementation.

Results

Participant characteristics

Public health organizations

Ten public health units in the province of Ontario, Canada, participated in two cohorts of the program. Participating health units served diverse populations and geographies (Table 1). Two health units invited staff to volunteer, five selected staff to participate, and three used a combination of these approaches.

Individuals

Fifty-five participants took part in the program, ranging from four to eight participants per public health unit. Five participants left the program due to role changes within their public health unit or leaves of absence and were replaced by new participants. Demographic data were collected from 51 participants (Table 2). Two participants declined to complete the demographic questionnaire, and one was absent when questionnaires were completed. One participant was added to the program following the first workshop based on interest in the program (Fig. 1).

Table 2 Participant demographics
Full size table
Fig. 1
figure 1

Participant flow diagram

Full size image

Organizational change for EIDM

Organizational goals for EIDM were grouped into themes, with specific priorities per organization identified in Table 3. One public health unit has been omitted from this analysis after not providing post-program data. Levels of success in reaching identified EIDM goals varied (Table 3). In some cases, there was a disconnect between interviewees perceptions at the same public health unit. For example, managers reported that staff were allocated sufficient protected time for EIDM work, while staff noted they needed more time.

Table 3 Achievement of EIDM goals by health units
Full size table

Further inductive analysis of interviews explored factors and themes that may have contributed to public health units’ success and challenges in integrating EIDM.

Increased organizational capacity for EIDM

Each health unit prioritized improving staff skills for EIDM and using research more often. Most prioritized providing staff with time and resources for EIDM. While most reported staff had increased their skills for EIDM, some indicated there was insufficient time and resources to support applying EIDM. Findings reveal strategies for increasing skills, including participants stepping into EIDM support or teaching roles and training additional staff. Reasons why EIDM capacity goals were not achieved included limitations of the program and participants’ transition out of current roles.

Participants consulting or teaching peers

In seven health units, participants entered teaching roles, in some cases, acting as informal consultants within teams or departments. An executive noted that participants, “have, to a certain extent, become champions [of EIDM] within their teams” (Health Unit 7, Senior Executive Z). Some participants were formally assigned with providing orientation and guidance for EIDM. Other participants became informal sources of knowledge and guidance for colleagues, “…[participant] walked our staff through how to apply this approach and so as a result, a brief review was done of the intervention and literature related to it.” (Health Unit 6, Manager V).

At four health units, participants provided formal workshops to the broader organization. One health unit had participants deliver a series of presentations on EIDM, while three others implemented monthly journal clubs where participants led appraisals of journal articles with colleagues.

Overall, management and peers recognized program participants as resources for EIDM. Various strategies allowed participants to share their expertise, reinforcing participants’ learning and developing other staff’s skills.

Limitations of program

Some participants felt the program curriculum did not directly or easily transfer to their work. A participant noted little room to apply EIDM, “our programs are very clearly scripted about what we have to do, and how and why.” (Health Unit 3, Participant O).

Others noted that the program’s focus on research evidence did not equip them for applying other types of evidence,

“When you look at the model about [EIDM] we focused a lot on the research bubble. And so things like the political climate and resources, those other bubbles that are in that model weren’t really touched on but yet I know from my own job that they’re extremely important.” (Health Unit 2, Participant G)

These criticisms of the program curriculum related to its applicability to participants’ work and whether the program adequately prepared participants to apply EIDM in different contexts.

Participant transitions to new roles

Most health units encountered staff turnover during the program. At one health unit, half of the participants had transitioned to new roles by the time the interviews were conducted. It was noted that participants were typically high achievers, which led to their promotion to new roles, “what happened was probably partly because the individuals we selected were high performers and were leaders in and of their own right, but over time they went on to other positions” (Health Unit 1, Manager A). Other participants left roles for personal reasons. Loss of these participants hindered implementation of EIDM, especially as they had become champions for EIDM.

Integration of EIDM into processes

Health units prioritized using evidence more consistently in decision-making. Most sought to assess and adapt research evidence more often while fewer prioritized considering quality of evidence, systematically integrating evidence use and holding management accountable for research use.

Thematic analysis uncovered that some health units concretely integrated EIDM into decision-making processes by developing structures, processes or templates to support EIDM, creating or adapting staff positions for EIDM and pursuing additional learning opportunities.

Structures, processes, or templates to support EIDM

Six health units described embedding EIDM into reporting templates or processes to prompt providing an evidence base for program planning recommendations. One health unit revised its briefing note template, “We’ve got a template and a process for doing briefing notes [that includes] what level of literature research are you using” (Health Unit 1, Manager B). Some health units integrated resources supporting EIDM, such as guidelines and tools, into various processes. Others described guides or resource hubs to support all staff at the organization to engage in EIDM. A resource library was shared through a health unit’s internal network to help staff apply EIDM in program planning. These various strategies helped reinforce and remind staff to use EIDM in all processes.

Staff positions dedicated to EIDM

Three health units noted that roles had changed to integrate EIDM work, “because with the training we were given dedicated time, dedicated FTE [full-time equivalent], all of us, and I think the dedicated FTE really helped us” (Health Unit 4, Participant R).

At two health units, additional staff were hired in specialist roles for EIDM support. On the growing number of research analysts at a health unit, “when I first started, I was one of the first [research analysts] and there were only about three at the time, and now I think there's four or five of them” (Health Unit 4, Participant Q). A dedicated specialist role had been created and filled at another health unit, to further educate frontline staff and guide them through rapid review processes.

Staff noted that dedicated time and roles to EIDM not only facilitated implementation, but it reinforced the prioritization of EIDM at the organization.

Lack of plan for participants

Five health units reported specific goals that had not been achieved. Issues included lack of direction for participants,

“We really had to … remind people that we have this training. It wasn’t necessarily, okay, you guys have this training now, we’re going to get you to do this, this and this or this is, kind of, the plan to have it sustainable. It’s, kind of, been us that’s been advocating for the sustainability of it.” (Health Unit 2, Participant H)

Plans for participants at these health units were absent. Newly acquired knowledge and skills were not applied or shared more broadly within the health unit, which led to frustration from participants.

Lack of protected time

Interviewees at six public health units expressed frustration at not dedicating enough time for EIDM practices. Noting that it was difficult to balance staff time to meet requirements while also using evidence,

“I think the challenge is always to sort of protect the time of those people so that they’re able to act as sort of a knowledge broker for the team and that they’re not struggling to sort of meet that need while continuing to do 100 percent of their normal job.” (Health Unit 10, Manager PP)

Interviewees noted that decision makers needed to allocate time dedicated to EIDM work to ensure that EIDM remained a priority and the work sustainable. Similar to lacking a concrete plan for participants to apply their EIDM skills, participants were frustrated at not having the opportunity to apply the knowledge and skills gained from the program.

Changes at the health unit

Some health units faced other factors that limited their focus on EIDM. Three had unexpected large-scale changes, such as in leadership, organizational structure or a merger with another health unit. Changes significantly overshadowed EIDM as a priority, “We’ve had a lot of change in upper leadership… and just because of all that flux and change we haven't really had a chance to really dig in [to integrating EIDM]” (Health Unit 2, Participant I). Similarly, merging with another health unit delayed progress with EIDM,

“We released our rapid review just as the merger was happening and, you know, we were without phones and fax machines and computers, and the knowledge broker piece was very much lost in that shuffle, and it's just picked up momentum again.” (Health Unit 9, Participant LL).

While the changes described by interviewees were varied, the effect was consistently a lack of focus on EIDM and lower prioritization of EIDM implementation.

Culture

Fewer goals were set for staff attitudes and organizational culture. Many health units prioritized expanding work with external partners. Health units commented on varying extents of culture shift. Culture changes included leadership support for EIDM, expectations that decisions would use EIDM, acceptance of time for learning and doing EIDM, and peer learning.

Leadership support

Six public heath units’ leadership set expectations for the transition to an EIDM approach. Actions of Medical Officers of Health were specifically cited, “EIDM and the use of the knowledge brokers within the office of the Medical Officer of Health [supported] ongoing involvement and leadership from the MOH office in creating a culture of [EIDM]” (Health Unit 3, Senior Executive K). Noted of another Medical Officer of Health,

“She’s always wanting, you know, to know like where does this come from, you know, and it’s not to kind of, you know, be testing you or whatnot, but it’s really wanting to be grounded in the evidence and that like all the – everything that we do, our practice is to be grounded in evidence.” (Health Unit 7, Manager AA)

Leadership was described as especially impactful for helping generate organization-wide interest in EIDM and an appreciation of its value, beyond participants and their teams.

Expectations for using EIDM

Some health units described a profound shift in expectations for EIDM. Compared to before the program, “I find that staff are more apt to say ‘well, we should look at the research.” (Health Unit 10, Manager OO). At another public health unit, “[EIDM} is definitely the status quo now. It's present everywhere” (Health Unit 4, Participant Q).

At another health unit, not only is there an expectation to use evidence, but to use evidence appropriately, “before, staff would say, ‘We really want to do this, here’s the evidence to support it, let's go’, and we’re able to say, ‘Actually I think that came in the wrong order, what's our question, let's find the evidence and then let's make a decision’” (Health Unit 4, Participant T).

Overall, these health units described environments where applying EIDM had become an expectation. Staff had become accustomed to preparing evidence and managers consistently required evidence to support decision-making.

EIDM across public health roles

Participants represented a variety of core public health roles, including typically office-based such as policy analysts and community-based positions, such as public health inspectors (Table 2). Participants in diverse roles described contributing to EIDM implementation. For example, a public health inspector described:

“I think another big way that it’s also impacted me is in my work as a health inspector. I would say … I’ve definitely increased in our peer review literature that I’m looking at and also working with some of the committees that I work on here to disseminate that to the health inspectors.” (Health Unit 7, Participant DD)

Participants who worked in the community were not limited in applying EIDM compared to participants who worked in offices with more consistent computer access. Participating public health inspectors in particular described championing EIDM among their teams.

Acceptance of time for EIDM

Three public health units allocated time specifically for staff to develop skills for EIDM, and managers were accepting and encouraging of this practice.

“People are just really taking the time [for EIDM] and, supervisors and, managers are providing the time. They're very supportive of giving employees time to learn about this, which is a shift in itself.” (Health Unit 7, Participant BB)

While a lack of protected time to apply EIDM hindered implementation, the provision of time for staff to develop EIDM skills was described as valuable. Acceptance of this time allowed participants and other staff to develop their skills without time pressures.

Lack of staff buy-in

Four public health units found staff on teams that did not have a participant from the KB mentoring program embedded in the team were reluctant to change their processes. For some, this was attributed to the time required for EIDM.

“… there probably still is the tendency for some of the areas to want to find the quick answer … they would be more likely to just search for something that supports [their opinion] as opposed to taking the time to ask it in a more objective way.” (Health Unit 3, Participant M)

Reluctance to adopt EIDM in some departments was due to biases, specifically to participants’ perceived lack of career experience,

“But my team – I had people on my team going ‘I’m not listening to her. She’s only been here for like three years!’ … she does have all this great knowledge, but it just doesn’t work that way.” (Health Unit 10, Manager OO)

In some cases, staff reluctance was directly related to EIDM processes, or to external factors, such as interpersonal relationships.

Lack of EIDM understanding among management

Some participants’ immediate supervisors did not appreciate how EIDM would affect day-to-day work and did not set appropriate expectations. Training of management would be beneficial,

“… engaging a few more at the management level would have resulted in perhaps a few of those glitches that we experienced, like mitigating those a little bit in terms of how we supported more of our frontline staff … and how to best support the team in terms of sharing their learning and building on it more systematically than we were.” (Health Unit 4, Manager P)

There was frustration that workloads and time were not managed effectively. While EIDM work was added, there were no other projects removed from portfolios, leading to participants becoming overwhelmed.

Summary of themes for organizational implementation of EIDM

Interviews provided insight into the successes and challenges faced by program participants in implementing EIDM. Thematic analysis uncovered several major factors for the implementation of EIDM on an organizational level, including building capacity for EIDM, integrating EIDM into processes, and organizational culture.

Discussion

Over two cohorts, ten public health units in Ontario participated in an intensive KB mentoring program. Public health units reported varying degrees in meeting their identified EIDM goals. These results show the real-world attainment of organizational EIDM goals through such a program and the factors that influenced EIDM implementation.

Achievement of EIDM goals varied across organizations. Interviewees at some organizations reported culture shifts to widespread adoption of EIDM; others saw limited implementation. Organizations that demonstrated readiness through strong leadership supportive of resource investment and a culture accepting and enthusiastic for change achieved greater success in realizing EIDM goals. Similar readiness factors have been shown to be important factors for EIDM [5, 13, 19, 26, 28, 44]. This study’s findings illustrated that leaders who established clear expectations for EIDM from staff also helped facilitate resource investments, such as dedicated EIDM support staff and clear plans for program participants to champion EIDM within the organization. These findings are consistent with previous studies in public health settings, where supportive leadership, organizational readiness, and integration of EIDM processes predicted success [19, 26, 45, 46].

Smaller, rural public health units were successful in implementing EIDM, demonstrating that size and location of organizations do not limit EIDM practices. Several of the smallest and most remote participating organizations were successful in achieving their EIDM goals, despite their own expectations of limited success due to relatively fewer available resources. As noted by participants, the impact of program participants was perceived to be greater in smaller organizations than in larger ones. Previous EIDM implementation studies in health-related rural and remote settings identified perceived barriers to EIDM, such as gaps in knowledge, unfavorable attitudes, and lack of resources [47,48,49,50,51]. This study’s findings provide an alternative perspective to these barriers and may enable smaller, rural organizations to overcome perceived barriers to EIDM.

Participants’ professional roles did not limit contributions to EIDM practices, consistent with findings reported by others [26, 45]. This study found that public health inspectors contributed to advancing EIDM to similar extents as colleagues in other professional roles that are based in offices for their work rather than mobile in the community. Front-line environmental health staff have previously reported tension between mandated practices and emerging evidence, lack of evidence for emerging topics, perception that only medical research evidence should be used and challenges in measuring environmental health-related outcomes [52]. In this study, public health inspectors overcame many previously identified barriers for implementing EIDM, suggesting a variety of public health roles can engage in EIDM. Others have noted the important role managers play in supporting public health inspectors to engage in EIDM [52].

One limitation of the KB mentoring program was the focus on using research evidence, rather than data from other sources such as local context. Local evidence has been previously found to be more influential than external evidence [30]. Further training on the EIDM steps of implementation and evaluation were also identified. This is consistent with reports of EIDM training in public health, noting that staff value evidence from local contexts but lack skills to effectively gather, appraise, and synthesize other forms of evidence, and that evaluation is not well-supported [7]. To integrate participate feedback, future program iterations should provide a broader curriculum, including finding and assessing non-research evidence, synthesizing multiple types of evidence, adapting evidence to local contexts, and developing and evaluating implementation plans.

The study did not include a control group and as such it is not possible to distinguish the program’s effects from changes that may have occurred without program participation. In future iterations, a time series or stepped wedge design, where organizations are observed prior to intervention, may help further determine the program’s specific effects [53].

KB roles can pose challenges, both for the individuals acting as KBs and the organizations they serve. The KB role is not well-defined, so contributions vary greatly in different contexts [20, 54]. Since KBs function between the worlds of evidence and practice, they may not be considered experts or accepted by colleagues in either community [55]. To minimize exclusion by public health colleagues, this program built capacity of staff already practicing and experienced within public health. In some cases, more junior participants’ attempts to share knowledge were dismissed by senior staff. It is important to carefully consider how staff chosen to participate in such programs will be accepted by peers [26]. The use of dedicated KBs may limit organizational capacity for knowledge translation, as evidence bottlenecks can form at the KB level [55]. This study found that participants transferred newly acquired knowledge and skills to colleagues. Evidence bottlenecks may be avoided with multiple people within the same organization in the role [48].

While this study offers new insights into a KB mentoring program to advance EIDM in public health, there are aspects to explore further. Enhanced focus on non-research evidence and implementation should be included in future program iterations. Participation by smaller, rural, or remote organizations should be considered to further explore EIDM implementation in settings typically perceived as lacking capacity. Exploration of EIDM implementation for diverse public health roles will optimize approaches for organization wide EIDM.

Conclusions

This KB mentoring program was promising at an individual level, indicated by reported practice changes by program participants. Shifting organizational culture to EIDM is complex, but this program shows potential for effective integration of EIDM within organizations. A KB mentoring program can support organizations to initiate and progress toward identified EIDM goals. Important organizational factors that are key to progress toward EIDM goals include supportive leadership and invested resources.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

EIDM:

Evidence-informed decision-making

KB:

Knowledge broker

NCCMT:

National Collaborating Centre for Methods and Tools

References

  1. National Collaborating Centre for Methods and Tools. Evidence-Informed Public Health 2021 [Available from: https://www.nccmt.ca/tools/eiph.]

  2. Ministry of Health and Long-Term Care. Ontario Public Health Standards: requirements for programs, services, and accountability; 2018.

    Google Scholar 

  3. Rycroft-Malone JS, Seers K, Titchen A, Harvey G, Kitson A, McCormack B. What counts as evidence in evidence-based practice? Nursing and Health Care Management and Policy. 2004;47(1):81–90.

    Google Scholar 

  4. Brownson RC, Baker EA, Deshpande AD, Gillespie KN. The need for evidence-based public health. 3rd ed. New York: Oxford University Press; 2018. p. 1–28.

    Google Scholar 

  5. Brownson RC, Fielding JE, Green LW. Building Capacity for evidence-based public health: reconciling the pulls of practice and the push of research. Annu Rev Public Health. 2018;39:27–53.

    Article  PubMed  Google Scholar 

  6. Public Health Agency of Canada. Core competencies for public health in Canada. 1 ed; 2008.

    Google Scholar 

  7. Martin W, Wharf Higgins J, Pauly BB, MacDonald M. “Layers of translation” - evidence literacy in public health practice: a qualitative secondary analysis. BMC Public Health. 2017;17(1):803.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Van der Graaf P, Forrest LF, Adams J, Shucksmith J, White M. How do public health professionals view and engage with research? A qualitative interview study and stakeholder workshop engaging public health professionals and researchers. BMC Public Health. 2017;17(1):892.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Peirson L, Ciliska D, Dobbins M, Mowat D. Building capacity for evidence informed decision making in public health: a case study of organizational change. BMC Public Health. 2012;12:137.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Morshed AB, Ballew P, Elliott MB, Haire-Joshu D, Kreuter MW, Brownson RC. Evaluation of an online training for improving self-reported evidence-based decision-making skills in cancer control among public health professionals. Public Health. 2017;152:28–35.

    Article  CAS  PubMed  Google Scholar 

  11. Jones K, Armstrong R, Pettman T, Waters E. Knowledge Translation for researchers: developing training to support public health researchers KTE efforts. J Public Health (Oxf). 2015;37(2):364–6.

    Article  Google Scholar 

  12. Dreisinger M, Leet TL, Baker EA, Gillespie KN, Haas B, Brownson RC. Improving the public health workforce: evaluation of a training course to enhance evidence-based decision making. J Public Health Manag Pract. 2008;14(2):138–43.

    Article  PubMed  Google Scholar 

  13. Jacob RR, Baker EA, Allen P, Dodson EA, Duggan K, Fields R, et al. Training needs and supports for evidence-based decision making among the public health workforce in the United States. BMC Health Serv Res. 2014;14:564.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Kislov R, Waterman H, Harvey G, Boaden R. Rethinking capacity building for knowledge mobilisation: developing multilevel capabilities in healthcare organisations. Implement Sci. 2014;9:166.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Pfund C, Spencer KC, Asquith P, House SC, Miller S, Sorkness CA. Building national capacity for research mentor training: an evidence-based approach to training the trainers. CBE Life. Sci Educ. 2015;14(2):14:ar24.

    Google Scholar 

  16. Leeman J, Calancie L, Hartman MA, Escoffery CT, Herrmann AK, Tague LE, et al. What strategies are used to build practitioners' capacity to implement community-based interventions and are they effective?: a systematic review. Implement Sci. 2015;10:80.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Campbell DM, Moore G. Increasing the use of research in population health policies and programs: a rapid review. Public Health Res Pract. 2018;28(3):e2831816.

  18. DeCorby-Watson K, Mensah G, Bergeron K, Abdi S, Rempel B, Manson H. Effectiveness of capacity building interventions relevant to public health practice: a systematic review. BMC Public Health. 2018;18(1):684.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Dobbins M, Hanna SE, Ciliska D, Manske S, Cameron R, Mercer SL, et al. A randomized controlled trial evaluating the impact of knowledge translation and exchange strategies. Implement Sci. 2009;4:61.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Bornbaum CC, Kornas K, Peirson L, Rosella LC. Exploring the function and effectiveness of knowledge brokers as facilitators of knowledge translation in health-related settings: a systematic review and thematic analysis. Implement Sci. 2015;10:162.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Canadian Health Services Research Foundation. The theory and practice of knowledge brokering in Canada’s health system; 2003.

    Google Scholar 

  22. Mendell J, Richardson L. Integrated knowledge translation to strengthen public policy research: a case study from experimental research on income assistance receipt among people who use drugs. BMC Public Health. 2021;21(1):153.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Russell DJ, Rivard LM, Walter SD, Rosenbaum PL, Roxborough L, Cameron D, et al. Using knowledge brokers to facilitate the uptake of pediatric measurement tools into clinical practice: a before-after intervention study. Implement Sci. 2010;5:92.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Brown KM, Elliott SJ, Robertson-Wilson J, Vine MM, Leatherdale ST. Can knowledge exchange support the implementation of a health-promoting schools approach? Perceived outcomes of knowledge exchange in the COMPASS study. BMC Public Health. 2018;18(1):351.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Carter EJ, Hessels A, Cato K, Sun C, Cohen B, Rivera RR, et al. Evaluation of the joint nurse scientist role across academia and practice. Nurs Outlook. 2020;68(3):261–9.

    Article  PubMed  Google Scholar 

  26. Dobbins M, Traynor RL, Workentine S, Yousefi-Nooraie R, Yost J. Impact of an organization-wide knowledge translation strategy to support evidence-informed public health decision making. BMC Public Health. 2018;18(1):1412.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Langeveld K, Stronks K, Harting J. Use of a knowledge broker to establish healthy public policies in a city district: a developmental evaluation. BMC Public Health. 2016;16:271.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Traynor R, DeCorby K, Dobbins M. Knowledge brokering in public health: a tale of two studies. Public Health. 2014;128(6):533–44.

    Article  CAS  PubMed  Google Scholar 

  29. Yost J, Dobbins M, Traynor R, DeCorby K, Workentine S, Greco L. Tools to support evidence-informed public health decision making. BMC Public Health. 2014;14:728.

    Article  PubMed  PubMed Central  Google Scholar 

  30. White J, Grant K, Sarkies M, Haines T. Evidence Translation in Allied Health G. Translating evidence into practice: a longitudinal qualitative exploration of allied health decision-making. Health Res Policy Syst. 2021;19(1):38.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Sarkies MN, Robins LM, Jepson M, Williams CM, Taylor NF, O'Brien L, et al. Effectiveness of knowledge brokering and recommendation dissemination for influencing healthcare resource allocation decisions: a cluster randomised controlled implementation trial. PLoS Med. 2021;18(10):e1003833.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Minian N, Ahad S, Ivanova A, Veldhuizen S, Zawertailo L, Ravindran A, et al. The effectiveness of generic emails versus a remote knowledge broker to integrate mood management into a smoking cessation programme in team-based primary care: a cluster randomised trial. Implement Sci. 2021;16(1):30.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Bradshaw C, Atkinson S, Doody O. Employing a qualitative description approach in health care research. Glob Qual Nurs Res. 2017;4:2333393617742282.

    PubMed  PubMed Central  Google Scholar 

  34. O'Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy. 2008;13(2):92–8.

    Article  PubMed  Google Scholar 

  35. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.

    Article  PubMed  Google Scholar 

  36. Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. Evidence summaries: the evolution of a rapid review approach. Syst Rev. 2012;1:10.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Association of Local Public Health Agencies. Local public health units. [Accessed from: alphaweb.org/page/PHU2018]. Accessed 10 Aug 2020.

  38. Kothari A, Edwards N, Hamel N, Judd M. Is research working for you? Validating a tool to examine the capacity of health organizations to use research. Implement Sci. 2009;4:46.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Yates J, Leggett T. Qualitative research: an introduction. Radiol Technol. 2016;88(2):225–31.

    PubMed  Google Scholar 

  40. Chapman AL, Hadfield M, Chapman CJ. Qualitative research in healthcare: an introduction to grounded theory using thematic analysis. J R Coll Physicians Edinb. 2015;45(3):201–5.

    Article  CAS  PubMed  Google Scholar 

  41. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

    Article  PubMed  Google Scholar 

  42. Shorten A, Smith J. Mixed methods research: expanding the evidence base. Evid Based Nurs. 2017;20(3):74–5.

    Article  PubMed  Google Scholar 

  43. Love HRC, C. Integrating without quantitizing: two examples of deductive analysis strategies within qualitatively driven mixed methods research. J Mixed Methods Res. 2021;16(1):64–87.

    Article  Google Scholar 

  44. Ward M, Mowat D. Creating an organizational culture for evidence-informed decision making. Healthc Manage Forum. 2012;25(3):146–50.

    Article  PubMed  Google Scholar 

  45. Bryant B, Ward M. A strategic approach to workforce development for local public health. Can J Public Health. 2017;108(4):e403–e8.

    Article  PubMed  Google Scholar 

  46. Cheetham M, Wiseman A, Khazaeli B, Gibson E, Gray P, Van der Graaf P, et al. Embedded research: a promising way to create evidence-informed impact in public health? J Public Health (Oxf). 2018;40(suppl_1):i64–70.

    Article  CAS  Google Scholar 

  47. Hisham R, Liew SM, Ng CJ, Mohd Nor K, Osman IF, Ho GJ, et al. Rural doctors’ views on and experiences with evidence-based medicine: the FrEEDoM qualitative study. PLoS One. 2016;11(3):e0152649.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Mudderman J, Nelson-Brantley HV, Wilson-Sands CL, Brahn P, Graves KL. The effect of an evidence-based practice education and mentoring program on increasing knowledge, practice, and attitudes toward evidence-based practice in a rural critical access hospital. J Nurs Adm. 2020;50(5):281–6.

    Article  PubMed  Google Scholar 

  49. Prakash V, Hariohm K, Balaganapathy M. Barriers in implementing evidence-informed health decisions in rural rehabilitation settings: a mixed methods pilot study. J Evid Based Med. 2014;7(3):178–84.

    Article  CAS  PubMed  Google Scholar 

  50. Kitto SC, Peller JC, Villanueva EV, Gruen RL, Smith JA. Rural surgeons’ attitudes towards and usage of evidence-based medicine in rural surgical practice. J Eval Clin Pract. 2011;17(4):678–83.

    Article  PubMed  Google Scholar 

  51. Parsons JE, Merlin TL, Taylor JE, Wilkinson D, Hiller JE. Evidence-based practice in rural and remote clinical practice: where is the evidence? Aust J Rural Health. 2003;11(5):242–8.

    PubMed  Google Scholar 

  52. Powers A, Pelletier T, Ray R, Reynolds A, Howarth C, Dobbins M. Participation in the National Collaborating Centre for Methods and Tools’ Knowledge Broker Mentoring Program: a public health inspector perspective. Environ Health Rev. 2021;64(1).

  53. Miller CJ, Smith SN, Pugatch M. Experimental and quasi-experimental designs in implementation research. Psychiatry Res. 2020;283:112452.

    Article  PubMed  Google Scholar 

  54. Lomas J. The in-between world of knowledge brokering. BMJ. 2007;334(7585):129–32.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Kislov R, Wilson P, Boaden R. The ‘dark side’ of knowledge brokering. J Health Serv Res Policy. 2017;22(2):107–12.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge Susan Snelling for her role as a program mentor, Jennifer Yost and Kara DeCorby-Watson for their role in developing the KB mentoring program materials, and Susan Jack for her support in preparing this manuscript.

Funding

The NCCMT is hosted by McMaster University and funded by the Public Health Agency of Canada. The views expressed herein do not necessarily represent the views of the Public Health Agency of Canada. The funder had no role in the design of the study, collection, analysis, or interpretation of data or in writing the manuscript.

Author information

Authors and Affiliations

  1. National Collaborating Centre for Methods and Tools, McMaster University, McMaster Innovation Park, 175 Longwood Rd S, Suite 210a, Hamilton, ON, L8P 0A1, Canada

    Emily C Clark, Bandna Dhaliwal, Donna Ciliska, Sarah E Neil-Sztramko & Maureen Dobbins

  2. School of Nursing, McMaster University, Health Sciences Centre, 2 J20, 1280 Main St W, Hamilton, ON, L8S 4 K1, Canada

    Donna Ciliska & Maureen Dobbins

  3. Department of Health Research Methods, Evidence & Impact, McMaster University, McMaster University Medical Centre, 2C Area, 1280 Main St W, Hamilton, ON, L8S 4 K1, Canada

    Sarah E Neil-Sztramko

  4. Vancouver, Canada

    Marla Steinberg

Authors
  1. Emily C Clark
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bandna Dhaliwal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Donna Ciliska
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sarah E Neil-Sztramko
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Marla Steinberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Maureen Dobbins
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

E.C., D.C., and M.D. designed and implemented the study. M.S. planned and conducted interviews. E.C. and B.D. analyzed study results. E.C. wrote the manuscript in consultation with D.C and S.N-S. D.C. and M.D. provided feedback on all drafts and the final version of the manuscript. The authors read and approved the final manuscript.

Authors’ information

Dr. Maureen Dobbins is the Scientific Director of the National Collaborating Centre for Methods and Tools (NCCMT). The NCCMT’s mandate is to support EIDM among public health professionals in Canada. She is also a Professor in the School of Nursing at McMaster University. Dr. Donna Ciliska is a Professor Emeritus in the School of Nursing at McMaster University and holds decades of experience conducting research on EIDM among public health decision makers.

Corresponding author

Correspondence to Maureen Dobbins.

Ethics declarations

Ethics approval and consent to participate

Ethics approval was obtained via the Hamilton Integrated Research Ethics Board (Project #15-016). Participants were informed of the purpose, potential risks, and benefits and requirements of the study and provided written informed consent.

Consent for publication

N/A

Competing interests

The authors declare that they have no competing interests. The program was delivered by staff employed by the NCCMT. Each participating public health organization contributed nominal funds to participate in the program, primarily to gauge program commitment and readiness for implementing the EIDM process.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Appendix 1.

The TIDieR (Template for Intervention Description and Replication) checklist.

Additional file 2: Appendix 2.

Interview structure.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Clark, E.C., Dhaliwal, B., Ciliska, D. et al. A pragmatic evaluation of a public health knowledge broker mentoring education program: a convergent mixed methods study. Implement Sci Commun 3, 18 (2022). https://doi.org/10.1186/s43058-022-00267-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s43058-022-00267-5

Keywords

Implementation Science Communications

ISSN: 2662-2211