The method for this study was previously described in a published study protocol [9]. Herein we summarize the method and provide additional details about its actual execution, but readers should refer to [9] for a more fully detailed description. Additional file 1 contains a checklist of reporting guidelines for mixed-method research (supplemented with specific items for concept mapping) that we completed for the study.
Recruitment and participants
To ensure our participants had appropriate expertise and constituted an internationally representative sample, recruitment used a combination of purposive and snowball sampling [33] in which we sent invitation emails to experts in implementation and/or UCD. Purposive sampling targeted experts from research centers and professional organizations that were centers of excellence for research in implementation and/or UCD; snowball sampling involved nominations from participants who completed the study. Interested participants contacted the study coordinator (second author) and were given login information for Concept Systems Global MAX (CSGM [34];), the web-based software platform that we used to conduct concept mapping. Once they logged into CSGM, the participants read and electronically signed the informed consent form, completed a short demographic questionnaire, and then began the concept mapping exercise.
The 56 participants were implementation experts (n = 34; 61%) and UCD experts (n = 22; 39%). Expertise was self-reported based on experience in research, practice/industry, and/or education over the past 5 or more years. We did not ask participants to identify specific areas of expertise, but we believe many had both research and applied experience in their discipline based on our recruitment methods and our interactions with participants during the study. Participants averaged 10.3 years of professional experience (SD = 6.7, range = 5–35). When asked how often their work involved interdisciplinary collaboration, half of participants indicated 80–100% of the time (top fifth), with increasingly smaller proportions endorsing 61–80%, 41–60%, 21–40%, and 0–20% of the time (21%, 16%, 11%, and 2% endorsement, respectively). Most participants (88%) reported focusing on health care in their work, but many also reported working with the prevention and health promotion (36%), education (18%), or human services (e.g., justice, child welfare, housing) (16%) sectors. When asked which CFIR domains they seek to improve through their work, most participants endorsed the individual (88%) and intervention/innovation (84%) levels, a smaller majority indicated the inner setting (70%), and the smallest proportion indicated the outer setting (34%). Finally, because the concept mapping software program limited the number of demographic questions that we could ask participants, we collected gender and race data in a follow-up Qualtrics survey which was completed by 51 participants (9% missing). Demographic data indicated that the sample was 59% female (n = 33; another 18 [32%] were male) and 73% white (n = 41; another six [11%] were Asian and the remaining four [8%] were other races).
We originally aimed to recruit 30 experts from each discipline [9], but more participants self-reported expertise in implementation than anticipated at enrollment (which filled slots originally intended for UCD experts), and several recruited UCD experts did not complete the study. Nevertheless, our sample size was still adequate for concept mapping as it exceeded the recommended sample size of n ≥ 15 per group [35].
Procedures
Concept mapping
We used concept mapping [32] to systematically capture the relationships that participants perceived between different concepts or ideas (i.e., implementation strategies and UCD strategies). This method guides participants through a structured conceptualization process where they sort ideas into related groups and then rate the ideas on key dimensions. It is a self-contained mixed-method approach (i.e., incorporating both qualitative and quantitative data collection and analysis approaches) consisting of four phases: (1) idea generation, (2) sorting, (3) rating, and (4) analysis.
Idea generation. As detailed in [9], our research team generated the ideas/concepts for participants to sort and rate by using existing resources that documented implementation and UCD strategies. For implementation, we selected a subset of 36 strategies from the full list of ERIC [7] strategies, with strategies chosen to maximize representativeness across (i) CFIR domains, (ii) categories of implementation strategies from a previous concept mapping study [8], and (iii) importance ratings (also collected by [8]). For UCD, we included all 30 strategies from our aforementioned compilation [22]. We uploaded each strategy (name and brief definition) into CSGM as a separate “statement” for subsequent sorting and rating by participants.
Sorting and rating. The middle two phases of concept mapping, sorting and rating, were completed in tandem through the CSGM platform. CSGM allows participants to complete tasks in any order, and participants could also stop and start the activities as often as they wished. Our instructions and rating dimensions were adapted from ERIC [8].
For the sorting step, participants sorted each of the 66 implementation and UCD strategies into groups based on their view of the strategies’ meaning or theme. The order of strategy presentation was randomized, with no distinction between implementation versus UCD strategies. For the rating step, participants rated each strategy on its importance and feasibility on a scale ranging from 1 (least important/feasible) to 5 (most important/feasible). Ratings for importance and feasibility were completed separately.
Post-survey
After participants completed all steps in CSGM, the system displayed a link to the post-survey in Qualtrics which collected additional demographic information; questions about challenges in collaboration between implementation experts and UCD experts (which were not yet analyzed for this initial study); and snowball sampling nominations. Upon completion, participants received a unique link for a $20 electronic gift card.
Analytic strategy
The final step of concept mapping, data analysis [32], involved using multidimensional scaling techniques (embedded in CSGM [34]) to identify clusters of implementation and UCD strategies that were generated most consistently across participants. We retained and analyzed data provided by all participants, including those who did not complete all study steps, although usable data were available from most participants (98% for sorting; 96% for rating).
CSGM can empirically generate any number of clusters, so the research team reviewed the results for conceptual clarity and credibility before selecting which set of clusters to report. To guide our thinking, we examined cluster maps produced by CSGM, which represent the relatedness of concepts within and between clusters in terms of visual distance. We also considered the extent to which clusters were consistent with or expanded upon the (1) clusters of implementation strategies identified in the ERIC study [8]; (2) CFIR domains [10]; and (3) the Integrated Promoting Action on Research Implementation in Health Services (i-PARIHS) framework [36], which describes the process of facilitating EBP use in practice settings by attending to characteristics of the EBP, recipients, and context (i-PARIHS is a process framework, which complements the determinant-focused nature of CFIR [37]). We began with a 13-cluster solution, which is one SD above the mean number of clusters in a typical concept mapping solution [35], and examined splitting and merging of clusters in a stepwise fashion. Once we selected the final set of clusters, we calculated average importance and feasibility ratings for each cluster and strategy. We used unweighted averages because weighting by subsample size (to account for the different number of implementation vs. UCD experts in the sample) resulted in very small changes to the average values, with no changes to study conclusions. We also examined ladder graphs, which provide a visual representation of the relationship between dimensions (e.g., importance and feasibility) within and across clusters. In addition, we explored the number and types (i.e., by discipline) of strategies in each cluster.
Following the initial analyses of concept mapping data from all participants, we also examined results separately by subgroup (i.e., implementation vs. UCD experts). We applied the same analytic approach described previously with data separated by discipline, and we evaluated whether there were observed differences in the number, content, or ratings of the clusters. We also used multivariate general linear models to test for differences in ratings of each cluster’s perceived importance and feasibility across disciplines.