Skip to main content
Download PDF
Download ePub

Implementing the DEcision-Aid for Lupus (IDEAL): study protocol of a multi-site implementation trial with observational, case study design

Implementing the DEcision-Aid for Lupus

Implementation Science Communications volume 2, Article number: 30 (2021) Cite this article

Abstract

Objective

To provide the details of the study protocol for an observational, case study design, implementation trial.

Methods

Implementing the DEcision-Aid for Lupus (IDEAL) study will put into practice a shared decision-making (SDM) strategy, using an individualized, culturally appropriate computerized decision-aid (DA) for lupus patients in 15 geographically diverse clinics in the USA. The overarching frameworks that guide this implementation study are the Consolidated Framework for Implementation Research (CFIR) and Powell’s typology of implementation strategies. All 15 clinics will receive standardized capacity-building activities for lupus DA implementation in the clinic, including education, training, technical assistance, re-training, and incorporation of a clinic champion in the core team of each site. In addition, clinics will also choose among clinic-targeted activities to integrate the DA into existing work processes and/or patient-targeted activities to raise awareness and educate patients about the DA. These activities will be chosen to stimulate participant recruitment and retention activities that support the implementation of the DA at their clinic. In study aim 1, using surveys and semi-structured interviews with clinic personnel in 15 lupus clinics, we will assess stakeholder needs and identify clinic and contextual characteristics that inform the implementation strategy component selection and influence implementation effectiveness. Study aim 2 is to implement and assess the effectiveness of the IDEAL (standardized and tailored) strategy in 15 lupus clinics by examining the changes in our primary outcome of penetration, i.e., the proportion of all eligible patients in the clinic that receive the lupus DA, and secondary outcomes include DA appropriateness, acceptability, success, permanence, and feasibility. Study aim 3 is to identify ways to sustain and disseminate our lupus DA via semi-structured debriefing interviews with key clinic personnel and patients.

Discussion

The study will enroll at least 500 patient participants with lupus across all 15 sites and assess the effectiveness in implementing the DA in various clinic settings across the USA.

Trial registration

ClinicalTrials.gov, NCT03735238. Protocol version number: 15, date 6/8/2020

Introduction

Systemic lupus erythematosus (SLE) is a multi-system autoimmune disease that typically affects young women. SLE, commonly referred to as lupus, is a rare disease that affects 0.1% of the general population but causes 2% of the end-stage kidney disease in the USA [1]. Decision-making for the use of medications is frequently poor. Many lupus patients decline life-saving immunosuppressive medications for their treatment [2,3,4,5], at least partly due to the lack of recognition of associated benefits and a fear of side effects [6,7,8,9,10]. Significant health disparities in outcome in lupus have been described [11,12,13,14], with 2.4 times higher mortality than age-matched controls [15,16,17,18].

In a PCORI-funded, multicenter randomized trial, 301 high-risk adult women with lupus kidney disease, including racial/ethnic minorities with low socio-economic status, either received an individualized, evidence-based, patient-centered, computerized lupus decision-aid (DA) [19,20,21,22,23,24], or the American College of Rheumatology (ACR) lupus paper pamphlet with information on treatments [25]. Compared to the ACR paper pamphlet, people who used the lupus DA had a much greater decrease in decisional conflict (uncertainty in choosing options) for immunosuppressive drugs and were more likely to choose the treatment option most consistent with their values, having viewed information that mattered the most for the treatment decision [26]. Compared to the pamphlet group, more patients rated the information in lupus DA to be excellent for understanding the impact of lupus (49% vs. 33%), risk factors (43% vs. 27%), medication options (50% vs. 33%), and evidence about medications (47% vs. 24%) and rated the ease of use of materials higher (51% vs. 38%; p values ≤ 0.006 for all) [26]. The RCT demonstrated the effectiveness of the lupus DA [26]. However, implementation of effective interventions is difficult and does not always succeed. Therefore, the focus needs to be on the implementation in order to ensure the lupus DA reaches as many people as possible.

The Implementing the DEcision-Aid for Lupus (IDEAL) study will use an observational case study design to evaluate the effectiveness of different implementation strategies to implement this computerized decision-aid (DA) for lupus patients in 15 clinics located across the USA, using an observational case study design.

Methods

Study setting

We strategically chose 15 US clinics from diverse settings to permit evaluation of success (and failure) in different contexts. Clinics are diverse with regard to the location (urban vs. suburban), region of the country, number of patients seen, type of clinic (general rheumatology vs. lupus vs. rheumatology/renal clinic), and practice type (private vs. academic vs. private/academic hybrid; Table 1). The study is registered at ClinicalTrials.gov (ID: NCT03735238).

Table 1 Participating sites, practice type, clinic type (general rheum, lupus), location, # lupus patients, site PIs, and region
Full size table

Study-specific aims, study eligibility criteria, and the conceptual model

Study-specific aims and study inclusion and exclusion criteria are summarized in Table 2. The overarching frameworks that guide this implementation study are the Consolidated Framework for Implementation Research (CFIR) [27] and Powell’s typology of implementation strategies [28] (Fig. 1). The CFIR organizes the factors that influence implementation into five domains, which will guide the semi-structured interview protocols for aims 1 and 3 (Fig. 1). Aim 2 will use Powell’s typology to thematically group implementation activities.

Table 2 Study-specific aims and study inclusion and exclusion criteria
Full size table
Fig. 1
figure 1

Conceptual and evaluation logic model. Boxes with snip corners indicate the outcome of each aim; empty boxes without color indicate intermediate steps to achieve aims; aim 2 activities are in the grey colored boxes

Full size image

Interventions

All 15 clinics will receive standardized capacity-building activities prior to starting lupus DA implementation in the clinic, including the following (Fig. 1):

  • Education: a series of 60-min seminars aimed at all 15 clinics that educates clinic personnel about the decision-aid including its purpose, content, and supporting evidence of effectiveness

  • Implementation coaching: a series of webinars, offered 1–2 months after each sites’ formative evaluation, that describes the clinic-specific findings of the formative evaluation, reviews the implementation strategies available to the clinic (both standardized and tailored), and jointly identifies the preferred strategies for the clinic

  • Technical assistance: ongoing, ad hoc technical support on the use and maintenance of an electronic tablet (e.g., trouble launching the DA, problems navigating screens)

  • Clinic champion: designated member of the clinic who is dedicated to supporting the implementation of the DA in the clinic

  • Refresher training course: a webinar conducted every 6 months or as needed that describes the implementation strategies and common barriers being confronted across the participating clinics and shares best practices of how these strategies are being deployed

In addition, clinics will choose to engage in either or both of the following two activities designed to stimulate DA implementation in their clinic: (1) clinic-targeted activities to further integrate the DA into existing work processes and (2) patient-targeted activities to raise awareness and educate patients about the DA (Table 3). At the end of the study, all sites will be reviewed to determine which strategies were used throughout the study (Fig. 1).

Table 3 Implementation strategy component, based on barriers and facilitators from our pilot work with target patient population and clinics
Full size table

Outcomes

Primary outcome measure

  1. 1.

    Penetration: this is measured using study records and/or electronic health records (# of patients who viewed the DA/# of eligible patients) [time frame 24 months].

Secondary outcome measures

  1. 1.

    Perceived acceptability of intervention measure (AIM): clinic personnel’s perception of the acceptability of the decision-aid, measured using a validated scale with four (4) items with responses ranging from 1 (“completely disagree”) to 5 (“completely agree”). The four items will be averaged to create one composite mean scale score (range 1–5), where higher scores reflect perceptions of greater acceptability (i.e., better outcome) [time frame 12 months].

  2. 2.

    Perceived DA implementation success: clinic personnel’s perception of the implementation success of the decision-aid, measured using a validated scale with three (3) items with responses ranging from 1 (“disagree”) to 5 (“agree”). These three items will be averaged to create one composite mean scale score (range 1–5), where higher scores reflect perceptions of greater implementation success (i.e., better outcome) [time frame: 12 months].

  3. 3.

    Perceived DA permanence: clinic personnel’s perception of the permanence of the decision-aid, measured using one validated item that is scored ranging from 1 (“not at all permanent”) to 5 (“extremely permanent”). This item will be examined by itself, where higher scores indicate perceptions of greater permanence of the decision-aid in the clinic (i.e., better outcome) [time frame 12 months].

  4. 4.

    Patient perception of DA usefulness: patient perception of the effect of the decision-aid on preparing the patient for decision-making measured using the preparation for decision-making (PDM), a validated scale consisting of 10 questions scored on an ordinal scale from not at all = 1 to a great deal = 5. For scoring, sum the score of the 10 items and divide by 10. Scores can then be converted to a 0–100 scale by subtracting 1 from this summed score and multiplying by 25 [time frame 0 months (baseline)].

  5. 5.

    Patient satisfaction with DA: patient satisfaction with the ease of the use of the decision-aid measured using a validated single-item scale scored on an ordinal scale from strongly disagree = 1 to strongly agree = 5. This item will be examined by itself, where higher scores indicate greater patient satisfaction with the decision-aid (i.e., better outcome). This is a single-item scale, and there are no subscales. It was adapted from another study that assessed satisfaction with iPad or interactive voice response [time frame 0 months (baseline)].

  6. 6.

    Perceived intervention appropriateness measure (IAM): clinic personnel’s perception of the appropriateness of the decision-aid, measured using a validated scale with four (4) items with responses ranging from 1 (“completely disagree”) to 5 (“completely agree”). The four items will be averaged to create one composite mean scale score (range 1–5), where higher scores reflect perceptions of greater appropriateness (i.e., better outcome). [time frame 12 months].

  7. 7.

    Perceived feasibility of intervention measure (FIM): clinic personnel’s perception of the feasibility of the decision-aid, measured using a validated scale with four (4) items with responses ranging from 1 (“completely disagree”) to 5 (“completely agree”). The four items will be averaged to create one composite mean scale score (range 1–5), where higher scores reflect perceptions of greater feasibility (i.e., better outcome) [time frame 12 months].

Study outcomes that will be assessed at the level of research team, clinic/practice personnel and patients are shown in Table 4. These include #activities pursued at each clinic, change in penetration, decision-aid acceptability, appropriateness and feasibility [29, 30], decision-aid success [31], and decision-aid permanence [32]. Patient-centered outcomes include decisional conflict [33,34,35,36,37,38], patient-physician care process [39], patient involvement in decision-making [40,41,42,43], patient acceptability [44] and feasibility [45, 46], decision-aid review time and patient satisfaction [45], patient perception of decision-aid usefulness [47] and patient-physician communication [48] (Table 4).

Table 4 Summary of Aim 2 data collection from the research team (orange), all practice members (blue) and patients (purple)
Full size table

Participant timeline

The project and participant timeline is shown in Fig. 2. The participant timeline for aim 2 is summarized in Table 4.

Fig. 2
figure 2

Summary of study-specific aims and project timeline. Each site will get the capacity-building strategy and chose among a set of clinic-targeted or patient-targeted activities, which map to our conceptual model

Full size image

Sample size

No sample size calculations were done for specific aims 1 and 3, which were convenience samples and done for descriptive purposes. We targeted at least 4 semi-structured phone interviews and surveys per site for aim 1 (average 6–8/site); repeat for aim 1 duirng the study follow-up, and in addition interview 2 patients per site.

A power analysis for the quantitative portion of specific aim 2 was based on the following assumptions for a sample size of 500 participants: (1) 15 clinics and (2) an intraclass correlation coefficient of 0.05. This provided us with a total confidence interval width of 5.3% for 10% of the total clinic population that is administered the DA to a width of 8.1%, if 40% of patients are administered the DA. For outcomes on a 5-point Likert scale, the corresponding 95% CI widths for a standard deviation of 0.5 and 2.0 on the mean score is estimated to be 0.09 and 0.35, respectively. For individual-level outcomes measured as a rate (e.g., # visits/1000) or as means (e.g., Likert scale), we anticipate small margins of error. A minimum number of proposed sites (n = 15) are needed to comprehensively assess how different implementation strategies are influenced by and interact with different contextual conditions (different geographic areas, clinic type, and patient diversity) to affect implementation effectiveness and, thus, determine the generalizability of the study findings. However, given the proven effectiveness of the DA, and in the spirit of implementation research, we will continue to recruit to our proposed maximum sample size target of 4000.

Recruitment and retention

Key clinic personnel recruitment and retention

Semi-structured phone interviews with 6–8 key clinic personnel. Clinic personnel surveys of up to 15 clinic personnel per site. End of study semi-structured phone interviews with the same 6–8 key clinic personnel as in aim 1, and 2 patients per site selected by the PI.

Patient participant recruitment and retention

We will recruit and enroll up to 4000 adults (≥ 18 years of age; from the 15 sites) who have been diagnosed with lupus during a regular clinic visit. Adult men and women of all races/ethnicities with lupus will be recruited by the study coordinator/site PI. Paper flyer advertisements with study information will be placed in clinics and waiting rooms. Other recruitment strategies will be implemented at some sites, such as DA information offered on the patient portal, a paper-based version of the DA mailed to the patient, clinic posters, and/or study information through waiting room TV or online kiosk.

Data collection methods

This study will be coordinated by the University of Alabama at Birmingham (UAB). All study visits and procedures will be performed at the 15 sites. Enrollment will be competitive. Participants will be seen according to the details that follow:

Specific aim 1: Clinic personnel survey and semi-structured interviews

Clinic personnel will be emailed a link to an online Qualtrics® survey. An automatic email will be generated in the Qualtrics® system and will go out to all non-responders every 7–8 days.

A 45 to 60-min semi-structured phone interview will assess stakeholder needs and identify clinic characteristics (e.g., policy environment, physician, clinic, patient barriers) that influence DA implementation and choice of the implementation strategy. The interview protocol will be adapted to fit different key personnel. A link to the brief video animation introducing the decision-aid use and content along with the interview protocol and copy of the decision-aid will be emailed to clinic personnel prior to being interviewed. All interviews will be recorded and transcribed verbatim for accuracy.

Specific aim 2: Implementation

Patient baseline visit

The baseline visit will take approximately 1 h to complete. All adults with lupus, who understand English or Spanish, are eligible. Once the potential participant checks in either at a kiosk or with a front desk clerk at their regularly scheduled lupus clinic appointments, they will be made aware of the lupus DA by either the clinic clerk or the study coordinator. After informed consent and privacy assurance, a participant number will be assigned. During the baseline visit, the study coordinator will assess the inclusion/exclusion criteria.

Participants will watch a short informational video on an electronic tablet, review the computerized lupus DA, and complete the questionnaire on Qualtrics® using an electronic tablet. Each patient participant will be reminded of the 3- and 6-month follow-up phone calls prior to the visit departure.

Specific aim 2: Follow-up patient participants

At 3 and 6 months post baseline visit, patient participants will be administered a 15-min survey via phone, email, or phone text with a link. Patient participants will receive a $10 payment for completion. Follow-up reminders will be done for any/all methods. Non-responders will be administered the survey on their next regularly scheduled lupus clinic visit. If requested, surveys will be mailed to patient participants via US mail and will include a pre-stamped return envelope.

Specific aim 2: Clinic personnel survey

Up to 15 clinic personnel, the same clinic personnel as surveyed in aim 1, will be emailed a link to complete an online Qualtrics® survey at 0, 6, 12, and 24 months to assess their perception of appropriateness, acceptability, success, permanence, and feasibility of the decision-aid. This survey will take between 10 and 15 min to complete. A reminder email will be sent to all non-responders every 7–8 days thrice.

Specific aim 3: Semi-structured interviews

Clinic personnel interviewed for specific aim 1, as well as 2 participants at each site (selected by the site PI), will be administered a 45–60-min semi-structured phone interview. This interview will identify opportunities for sustaining and disseminating a lupus DA and help to develop a manual that provides a step-by-step implementation guide for incorporating lupus DA into regular clinical visits.

Data management

Dr. Beasley, lead statistician, will supervise and oversee all quantitative data management at the UAB IDEAL Data Coordinating Center (DCC) and analysis for IDEAL DCC. Questionnaires will be tested before being uploaded into Qualtrics®. Data entered into Qualtrics® by participants will be captured into the IDEAL electronic Data Entry System (eDES; Birmingham, AL) directly using iPads for seamless data management and auditing across the IDEAL sites. The DCC will ensure that the data collected are of the highest quality possible and will be accomplished in part by having thorough edit checks as close to the collection in time as possible, and updated as needed to guarantee high-quality data through quality control and quality assurance. Edit checks will be reviewed by Dr. Beasley on an ongoing basis to evaluate whether any checks need to be added or any existing checks need to be modified. All quantitative analyses will be conducted using SAS (Cary, NC) version 9.4 or higher or R-routines for specialty programs as needed. All qualitative analyses will be conducted with NVivo 12, available through the UAB School of Medicine.

Statistical methods

Specific aim 1

The purpose is to identify and quantify clinic and contextual characteristics (e.g., readiness for change, physician attitudes, patient barriers) to inform strategy component selection and influence implementation effectiveness of lupus DA to be evaluated in specific aim 2. Thus, aim 1 is largely descriptive and qualitative, and there are no formal statistical hypotheses (Additional file 1: Appendix 1). Overall rates and 95% confidence intervals will be computed. We will also perform qualitative analyses of semi-structured interview data by first coding the data using the major domains of the Consolidated Framework for Implementation Research (CFIR) as codes (innovation characteristics, individual characteristics, inner setting, outer setting, process), investigators discussing differences in domain definitions, reconciling, and then analyzing for key themes related to how readiness to implement change varied as a function of clinic characteristics, i.e., we will focus on themes pertaining to the inner setting.

Specific aim 2

For individual-level outcomes, initial analyses will focus on how clinic personnel and patient perceptions change over time. The statistical null hypothesis is the assumption that there is no change over time vs. the alternative that there is a change over time. For the analytic approach, we will examine the mean levels (total change) and quantify with means (or proportions, as appropriate) and 95% confidence intervals. We will also examine the patterns of change (growth trajectories) while controlling for differences between clinics using multivariable models that account for repeated measures and clinic clustering of patients (i.e., generalized estimating equations (GEE) with robust standard errors). Additional analyses may examine by-group comparisons of clinics determined by clinical characteristics (e.g., geographic areas, clinic type, and patient diversity) observed in specific aim 1. Any such by-group analyses will be conducted separately. A primary independent variable of interest is the type of strategy used for implementation. Following the completion of the study, each sites’ strategies will be evaluated, and sites will be grouped accordingly. A covariate allowing for testing group differences will be included in the multivariable models to assess whether trajectories differ by implementation strategy.

The second component of our evaluation of specific aim 2 will use fuzzy set qualitative comparative analysis (fsQCA) [49, 50] to assess the effectiveness of implementation strategies for clinic-level outcomes. For patient-physician communication analysis, question-asking, assertive responses, and expressions of concern will be coded as patient participation; supportive talk and partnership building by provider will be coded as active physician communication. Other outcomes will be obtained from the electronic health records, including healthcare utilization outcomes (e.g., inpatient visits and ER/urgent care visits) and, missed/canceled appointments.

Specific aim 3

The analytic approach is largely descriptive, and there are no statistical hypotheses. Specific qualitative analyses are described in Additional file 1: Appendix 1. Qualitative methods will be similar to those described above for aim 1, except that the focus will be to identify opportunities for sustaining and disseminating a lupus DA.

Data monitoring

This study poses no more than minimal risk to participants, and therefore, there will be no Data Safety Monitoring Board (DSMB). No clinical monitoring will be required for this study.

Quality control (QC) procedures have been implemented beginning with the Qualtrics® data entry system and data QC checks that will be run on the database once enrollment begins. All personal identifiers collected will be stored on a secure UAB server and backed up nightly. Survey data will be merged with baseline enrollment data to create master SAS datasets. Logic and range checks in SAS 9.4 (Cary, NC) will ensure the identification of data accuracy (see section 9.4). Any missing data or data anomalies will be communicated to the site(s) for clarification/resolution.

Potential Benefits and Harms

A potential benefit from this study is that the knowledge gained will provide patients valuable information about lupus and its treatment, as well as to inform clinicians about the best way to implement the decision-aid into normal clinic workflow. We will take the following precautions to avoid/minimize risks to participants.

We will make it a top priority to protect patient privacy, confidentiality, and autonomy for all study procedures, summarized as follows. Patient participant initial data and responses to questionnaires will be captured during regularly scheduled lupus clinic visits in a database behind the UAB firewall. Follow-up surveys will be conducted via phone, email, or in the clinic visit depending on the preference of the patient. All clinic personnel surveys and interviews will be conducted during general work hours (even before or after their shift). All semi-structured in-depth interviews will be captured via phone and transcribed for accuracy as soon as possible. All data (except data from semi-structured interviews) will be entered into coded electronic case report forms (CRF) and will be checked by study personnel daily for accuracy. On a quarterly basis, study investigators and the research team will review GCP, human subject’s protections/confidentiality, and study procedures. The research team will meet biweekly to review recruitment, enrollment, source documents, and electronic case report forms; in the event an adverse event occurs, this will be reported to the UAB IRB at the time of continuing review. All serious adverse events (SAEs) will be reported to the IRB and PCORI within 48 h of the principal investigator becoming aware of the event. All research team members will be informed by the PIs about any unanticipated problems involving risks to the participant. If any protocol changes are needed, the PIs will submit a modification request to the IRB. Protocol changes will not be implemented prior to IRB approval unless necessary to eliminate apparent immediate hazards to the research participant. In such cases, the IRB will be promptly informed of the change following implementation (within 1 week).

Considering the aim of this study is to determine the best practices for implementation of a treatment decision-aid, it poses no more than minimal risk to participants. We will use a secured database to minimize the risk of disclosure of personal health identifiers (PHI); therefore, no Data Safety Monitoring Board (DSMB) will be convened.

Auditing

There is no formal auditing planned for this study.

Research ethics approval

No study procedures will be performed before the approval of the institutional review board (IRB) at the University of Alabama at Birmingham (UAB) and at all other clinical sites. Each site will also perform the continuing annual review of the study. A copy of site ethics committee approvals and the approval of protocol amendments will be stored at each site. Copies of the ethics committee approvals for the UAB coordinating center will be provided to each site, and additional approvals will be obtained at the site as needed.

Protocol amendments

All protocol amendments for the UAB coordinating center will be provided to each site promptly. These will also be discussed and reviewed (as needed) during the monthly project calls with each site.

Consent

A non-signature consent form will be provided to all participants (both clinic personnel and patient participants), as permitted by Institutional Review Boards. The non-signature consent will describe the study in detail and payment procedures, as well as give participants important contact information for study personnel. Participants will be informed that participation is voluntary and that they may withdraw from the study at any time. Study coordinators will review the consent with the patient and answer any questions that may arise. Limited personal health identifiers (PHI) may be used (audio recording with physician, etc.), so all patient participants will be asked to sign a HIPPA form and will be given a copy for their records and a copy will be kept in the study binder. The reason for non-participation will be recorded. Some study sites will use consent forms with patient's signature, as required by their Institutional Review Boards.

Clinic staff participating in the study will be informed about the study by their nurse manager/supervisor and that participation is voluntary and they may withdraw from the study at any time. Staff will have the opportunity to carefully review the information sheet and HIPPA form prior to signing the HIPPA document. Clinic staff participating in the clinic survey and the semi-structured interviews will be given the information sheet and HIPPA form. A copy of the information sheet and HIPPA form will be given to all staff for their records.

Confidentiality

Participant confidentiality and privacy will be strictly held in trust by the participating investigators, their staff, and their interventions. Therefore, the study protocol, documentation, data, and all other information generated will be held in strict confidence. All research activities will be conducted in as private a setting as possible. The information obtained during the conduct of this study is confidential, and disclosure to third parties other than those noted below is prohibited. The results of the research study may be published, but study participant’s names or identities will not be revealed.

Declaration of interests

Each academic institution’s conflict of interest review board has established policies and procedures for all study group members to disclose all conflicts of interest and has mechanisms for the management of such conflicts.

Access to data

Paper and computer files will be safeguarded from unauthorized access and stored in secure locations. Information collected electronically via the computerized decision-aids will be stored in a secure central location. Information collected on paper forms will be sent to UAB and stored in a secure location. Computer records of study data will be stored in a central database, controlled by a system of user identification and passwords. Patient-physician conversation audio recordings (aim 2) will be kept controlled by unique usernames and passwords, with access to a limited number of study personnel.

Data entered into computerized files will be accessible only to authorized study personnel and will be coded. Touchscreen computers will be password-protected and encrypted by the UAB Informatics, so that in case of a loss of a unit, no patient information can be retrieved.

Ancillary and post-study care

Study documents will be retained for a minimum of 7 years after the study completion. These documents will be retained for a longer period, however, if required by the IRB or PCORI. No records will be destroyed without prior approval from UAB/CORI.

Dissemination policy

Dissemination will be ensured with a multi-faceted plan: (1) partner institution/practice leaders have pledged strong support; (2) stakeholder partners from the Arthritis Foundation (AF) and the Lupus Foundation of America (LFA), two key US patient advocacy organizations, will help develop complementary patient messages to drive lupus DA dissemination; (3) institutional resources from three UAB centers, Center for Clinical and Translational Studies (CCTS), Center for Outcomes Effectiveness and Education (COERE), and Comprehensive Arthritis, Musculoskeletal, Bone and Autoimmunity Center (CAMBAC) will allow continuation of technical support to sites for an additional 6 months to facilitate transition to a patient education and QI initiative; (4) study implementation strategies emphasize capacity building in clinics that extend beyond the study period; (5) this study aims to provide a QI package (i.e., a lupus DA implementation manual) which will facilitate spread and sustainability; and (6) ongoing 3-monthly meeting with a multistakeholder panel (patient, clinician, researcher, patient advocacy leaders on stakeholder committee; Additional file 1: Appendix 2) and a steering committee with active engagement on study conduct, troubleshooting, and protocol. A number of the implementation activities emphasize building capacity and overcoming change barriers, which will foster a continuous learning environment [51, 52] where clinics can continue to adapt the existing DA to changing circumstances. Data will be made available to colleagues after obtaining permission from UAB IRB and PCORI.

Discussion

The purpose of this mixed methods study is to evaluate the impact of different combinations of implementation strategies in promoting the use of an evidence-based decision-aid (DA) for patients with lupus and the conditions that may support or attenuate their effectiveness. The study is significant for a number of reasons. First, lupus is a rare autoimmune disease with significant health outcome disparities, with 2.4 times higher mortality than age-matched controls [15,16,17,18]. It affects 0.1% of the population, but lupus nephritis (kidney disease) accounts for 2% of all end-stage renal disease in the USA [1]. Medication decision-making in lupus is challenging [6], with many declining life-saving immunosuppressive medications. The identification of effective strategies for implementing an evidence-based DA can help patients overcome these problems and take advantage of effective treatment options. Second, our study takes a contextually driven approach to assessing implementation strategies that allows participants to choose from a menu of strategies that are believed to be most effective for their unique circumstances (e.g., geography, resources). In doing so, our study explicitly acknowledges that healthcare providers and patients are often situated in unique organizational and community contexts and empirically assesses how these different contexts shape the decisions around the most appropriate strategies for implementing our evidence-based DA [53]. Finally, the study leverages multi-level, multi-method, longitudinal data over a 3-year period from 15 rheumatology clinics from across the USA. These data enable us to examine the study relationships across a variety of contexts and understand how the implementation strategies and their effects on implementation outcomes change over time and why.

The study also faces several challenges. First, despite the inclusion of 15 different clinics from across the USA, our ability to conduct organizational-level analyses will be limited, even with our use of small-N analytic techniques such as fsQCA and qualitative case comparison methods. Likewise, our ability to assess questions of change is predicated on our ability to collect data from the same respondents over time, a non-trivial issue given potential clinical staff and patient turnover [54]. Compounding this second issue is the fact that the study was funded in the pre-COVID-19 pandemic era. We anticipate that unique challenges in patient care, healthcare delivery changes that are impacting outpatient and inpatient care of patients in general, will have both a transient and permanent effect on our study implementation. We are planning to examine this by comparing data in the pre-COVID-19 period to the COVID-19 period. However, these analyses will be limited since the number of patients enrolled during the pandemic is low due to a severe slow down and cessation of patient enrollment during the first 4–6 months of the COVID-19 pandemic in the USA. In response to the COVID-19 pandemic, we modified the patient enrollment with the addition of a virtual patient enrollment in the study for people having their telemedicine clinic visits virtually using telephone or video visits. Patients can now enroll with an e-consent and complete the viewing of the decision-aid and survey virtually, using the links sent to them via email or phone text by using their phone or computer devices. Similarly, the impact of the COVID-19 pandemic on the morale of clinic personnel in general, and specifically towards healthcare quality improvement and patient education efforts, is unknown.

In sum, our study presents a significant opportunity to better understand the most effective strategies for implementing an evidence-based shared decision-aid for patients with lupus and the contextual circumstances that may determine why some strategies are more effective than others. Such insights are especially important given the significant cost and quality consequences of inappropriate treatment for patients with lupus.

Availability of data and materials

The study PI, biostatistician (Dr. Beasley), and key study investigators (Drs. Hearld, Hall, and Qu) will have access to the final trial dataset. There are no contractual agreements that limit such access for investigators.

Abbreviations

UAB:

University of Alabama at Birmingham

IDEAL:

Implementing the DEcision-Aid for Lupus

SDM:

Shared decision-making

DA:

Decision-aid

ACR:

American College of Rheumatology

References

  1. Maisonneuve P, Agodoa L, Gellert R, Stewart JH, Buccianti G, Lowenfels AB, et al. Distribution of primary renal diseases leading to end-stage renal failure in the United States, Europe, and Australia/New Zealand: results from an international comparative study. Am J Kidney Dis. 2000;35(1):157–65.

    Article  CAS  PubMed  Google Scholar 

  2. Appel GB, Contreras G, Dooley MA, Ginzler EM, Isenberg D, Jayne D, et al. Mycophenolate mofetil versus cyclophosphamide for induction treatment of lupus nephritis. J Am Soc Nephrol. 2009;20(5):1103–12.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Hahn BH, McMahon MA, Wilkinson A, Wallace WD, Daikh DI, Fitzgerald JD, et al. American College of Rheumatology guidelines for screening, treatment, and management of lupus nephritis. Arthritis Care Res (Hoboken). 2012;64(6):797–808.

    Article  Google Scholar 

  4. Chen W, Liu Q, Chen W, Tang X, Fu P, Liu F, et al. Outcomes of maintenance therapy with tacrolimus versus azathioprine for active lupus nephritis: a multicenter randomized clinical trial. Lupus. 2012.

  5. Dooley MA, Jayne D, Ginzler EM, Isenberg D, Olsen NJ, Wofsy D, et al. Mycophenolate versus azathioprine as maintenance therapy for lupus nephritis. N Engl J Med. 2011;365(20):1886–95.

    Article  CAS  PubMed  Google Scholar 

  6. Chambers SA, Raine R, Rahman A, Isenberg D. Why do patients with systemic lupus erythematosus take or fail to take their prescribed medications? A qualitative study in a UK cohort. Rheumatology (Oxford). 2009;48(3):266–71.

    Article  CAS  Google Scholar 

  7. Garcia Popa-Lisseanu MG, Greisinger A, Richardson M, O'Malley KJ, Janssen NM, Marcus DM, et al. Determinants of treatment adherence in ethnically diverse, economically disadvantaged patients with rheumatic disease. J Rheumatol. 2005;32(5):913–9.

    PubMed  Google Scholar 

  8. Kumar K, Gordon C, Barry R, Shaw K, Horne R, Raza K. It's like taking poison to kill poison but I have to get better': a qualitative study of beliefs about medicines in Rheumatoid arthritis and Systemic lupus erythematosus patients of South Asian origin. Lupus. 2011;20(8):837–44.

    Article  CAS  PubMed  Google Scholar 

  9. Mosley-Williams A, Lumley MA, Gillis M, Leisen J, Guice D. Barriers to treatment adherence among African American and white women with systemic lupus erythematosus. Arthritis Rheum. 2002;47(6):630–8.

    Article  PubMed  Google Scholar 

  10. Sutanto B, Singh-Grewal D, McNeil HP, O'Neill S, Craig JC, Jones J, et al. The experiences and perspectives of adults living with systemic lupus erythematosus (SLE): Thematic synthesis of qualitative studies. Arthritis Care Res (Hoboken). 2013.

  11. Barr RG, Seliger S, Appel GB, Zuniga R, D'Agati V, Salmon J, et al. Prognosis in proliferative lupus nephritis: the role of socio-economic status and race/ethnicity. Nephrol Dial Transplant. 2003;18(10):2039–46.

    Article  PubMed  Google Scholar 

  12. Bernatsky S, Boivin JF, Joseph L, Manzi S, Ginzler E, Urowitz M, et al. Race/ethnicity and cancer occurrence in systemic lupus erythematosus. Arthritis Rheum. 2005;53(5):781–4.

    Article  CAS  PubMed  Google Scholar 

  13. Helmick CG, Felson DT, Lawrence RC, Gabriel S, Hirsch R, Kwoh CK, et al. Estimates of the prevalence of arthritis and other rheumatic conditions in the United States. Part I. Arthritis Rheum. 2008;58(1):15–25.

    Article  PubMed  Google Scholar 

  14. Krishnan E, Hubert HB. Ethnicity and mortality from systemic lupus erythematosus in the US. Ann Rheum Dis. 2006;65(11):1500–5.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Bernatsky S, Boivin JF, Joseph L, Manzi S, Ginzler E, Gladman DD, et al. Mortality in systemic lupus erythematosus. Arthritis Rheum. 2006;54(8):2550–7.

    Article  CAS  PubMed  Google Scholar 

  16. Cervera R, Khamashta MA, Font J, Sebastiani GD, Gil A, Lavilla P, et al. Morbidity and mortality in systemic lupus erythematosus during a 10-year period: a comparison of early and late manifestations in a cohort of 1,000 patients. Medicine (Baltimore). 2003;82(5):299–308.

    Article  Google Scholar 

  17. Krishnan E. Hospitalization and mortality of patients with systemic lupus erythematosus. J Rheumatol. 2006;33(9):1770–4.

    PubMed  Google Scholar 

  18. Ward MM, Pyun E, Studenski S. Mortality risks associated with specific clinical manifestations of systemic lupus erythematosus. Arch Intern Med. 1996;156(12):1337–44.

    Article  CAS  PubMed  Google Scholar 

  19. Qu H, Shewchuk RM, Alarcon G, Fraenkel L, Leong A, Dall'Era M, et al. Mapping Perceptions of Lupus Medication Decision-Making Facilitators: The Importance of Patient Context. Arthritis Care Res (Hoboken). 2016;68(12):1787–94.

    Article  PubMed  Google Scholar 

  20. Singh JA, Qu H, Yazdany J, Chatham W, Shewchuk R. Minorities with lupus nephritis and medications: a study of facilitators to medication decision-making. Arthritis Res Ther. 2015;17:367.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Singh JA, Hossain A, Kotb A, Wells G. Risk of serious infections with immunosuppressive drugs and glucocorticoids for lupus nephritis: a systematic review and network meta-analysis. BMC Med. 2016;14(1):137.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  22. Singh JA, Hossain A, Kotb A, Oliveira A, Mudano AS, Grossman J, et al. Treatments for Lupus Nephritis: A Systematic Review and Network Metaanalysis. J Rheumatol. 2016;43(10):1801–15.

    Article  CAS  PubMed  Google Scholar 

  23. Singh JA, Qu H, Yazdany J, Chatham W. Dall'era M, Shewchuk RM. Barriers to Medication Decision Making in Women with Lupus Nephritis: A Formative Study using Nominal Group Technique. J Rheumatol. 2015;42(9):1616–23.

    Article  PubMed  Google Scholar 

  24. Singh JA, Hossain A, Kotb A, Wells GA. Comparative effectiveness of immunosuppressive drugs and corticosteroids for lupus nephritis: a systematic review and network meta-analysis. Syst Rev. 2016;5(1):155.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Singh JA, Shah N, Green C. Individualized patient decision-aid for immunosuppressive drugs in women with lupus nephritis: study protocol of a randomized, controlled trial. BMC Musculoskelet Disord. 2017;18(1):53.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Singh JA, Fraenkel L, Green C, Alarcon GS, Barton JL, Saag KG, et al. Individualized decision aid for diverse women with lupus nephritis (IDEA-WON): A randomized controlled trial. PLoS Med. 2019;16(5):e1002800.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012;69(2):123–57.

    Article  PubMed  Google Scholar 

  29. Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Aarons GA. Mental health provider attitudes toward adoption of evidence-based practice: the Evidence-Based Practice Attitude Scale (EBPAS). Ment Health Serv Res. 2004;6(2):61–74.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Tucker AL, Nembhard IM, Edmondson AC. Implementing new practices: An empirical study of organizational learning in hospital intensive care units. Manag Sci. 2007;53(6):894–907.

    Article  Google Scholar 

  32. Goodman RM, McLeroy KR, Steckler AB, Hoyle RH. Development of level of institutionalization scales for health promotion programs. Health Educ Q. 1993;20(2):161–78.

    Article  CAS  PubMed  Google Scholar 

  33. Cohan D, Gomez E, Greenberg M, Washington S, Charlebois ED. Patient perspectives with abbreviated versus standard pre-test HIV counseling in the prenatal setting: a randomized-controlled, non-inferiority trial. PLoS One. 2009;4(4):e5166.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  34. O'Connor AM. Validation of a decisional conflict scale. Med Decis Mak. 1995;15(1):25–30.

    Article  CAS  Google Scholar 

  35. Brehaut JC, O'Connor AM, Wood TJ, Hack TF, Siminoff L, Gordon E, et al. Validation of a decision regret scale. Med Decis Mak. 2003;23(4):281–92.

    Article  Google Scholar 

  36. O'Connor AM, Bennett CL, Stacey D, Barry M, Col NF, Eden KB, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev. 2009;(3):CD001431.

  37. Stacey D, Bennett CL, Barry MJ, Col NF, Eden KB, Holmes-Rovner M, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev. 2011;(10):CD001431.

  38. Sun Q. Predicting Downstream Effects of High Decisional Conflict: Meta-analyses of the Decisional Conflict Scale. Unpublished MSc thesis. Demonstrates the predictive validity of the DCS on option knowledge, decision delay, discontinuance, and regret. 2005.

  39. Stewart AL, Napoles-Springer AM, Gregorich SE, Santoyo-Olsson J. Interpersonal processes of care survey: patient-reported measures for diverse groups. Health Serv Res. 2007;42(3 Pt 1):1235–56.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Singh JA, Sloan JA, Atherton PJ, Smith T, Hack TF, Huschka MM, et al. Preferred roles in treatment decision making among patients with cancer: a pooled analysis of studies using the Control Preferences Scale. Am J Manag Care. 2010;16(9):688–96.

    PubMed  PubMed Central  Google Scholar 

  41. Degner LF, Russell CA. Preferences for treatment control among adults with cancer. Res Nurs Health. 1988;11(6):367–74.

    Article  CAS  PubMed  Google Scholar 

  42. Degner LF, Kristjanson LJ, Bowman D, Sloan JA, Carriere KC, O'Neil J, et al. Information needs and decisional preferences in women with breast cancer. JAMA. 1997;277(18):1485–92.

    Article  CAS  PubMed  Google Scholar 

  43. Adams JR, Drake RE, Wolford GL. Shared decision-making preferences of people with severe mental illness. Psychiatr Serv. 2007;58(9):1219–21.

    Article  PubMed  Google Scholar 

  44. O’Connor AM. User Manual – Acceptability [document on the Internet]. © 1996 [modified 2002]. Ottawa: Ottawa Hospital Research Institute; 1996. Available from http://decisionaid.ohri.ca/docs/develop/User_Manuals/UM_Acceptability.pdf

    Google Scholar 

  45. Mudano AS, Gary LC, Oliveira AL, Melton M, Wright NC, Curtis JR, et al. Using tablet computers compared to interactive voice response to improve subject recruitment in osteoporosis pragmatic clinical trials: feasibility, satisfaction, and sample size. Patient Preference Adherence. 2013;7:517–23.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Mullan RJ, Montori VM, Shah ND, Christianson TJ, Bryant SC, Guyatt GH, et al. The diabetes mellitus medication choice decision aid: a randomized trial. Arch Intern Med. 2009;169(17):1560–8.

    Article  PubMed  Google Scholar 

  47. Bennett C, Graham ID, Kristjansson E, Kearing SA, Clay KF, O'Connor AM. Validation of a preparation for decision making scale. Patient Educ Couns. 2010;78(1):130–3.

    Article  PubMed  Google Scholar 

  48. Street RL Jr, Millay B. Analyzing patient participation in medical encounters. Health Commun. 2001;13(1):61–73.

    Article  PubMed  Google Scholar 

  49. Schneider CQ, Wagemann C. Set-theoretic methods for the social sciences: A guide to qualitative comparative analysis: Cambridge University Press; 2012.

  50. Sheehy TJ, Thygeson NM. Physician organization care management capabilities associated with effective inpatient utilization management: a fuzzy set qualitative comparative analysis. BMC Health Serv Res. 2014;14:582.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Kolodny O, Edelman S, Lotem A. The evolution of continuous learning of the structure of the environment. J R Soc Interface. 2014;11(92):20131091.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Tracey JB, Tannenbaum SI, Kavanagh MJ. Applying trained skills on the job: The importance of the work environment. J Appl Psychol. 1995;80(2):239–52.

    Article  Google Scholar 

  53. Waltz TJ, Powell BJ, Fernandez ME, Abadie B, Damschroder LJ. Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implement Sci. 2019;14(1):42.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Collini SA, Guidroz AM, Perez LM. Turnover in health care: the mediating effects of employee engagement. J Nurs Manag. 2015;23(2):169–78.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The views in this article are solely the responsibility of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee, or the Department of Veterans Affairs.

Funding

The research reported in this article was funded through a Patient-Centered Outcomes Research Institute (PCORI) Award (SDM-2017C2-8224). Dr. Singh is supported by the resources and use of facilities at the Birmingham VA Medical Center, Birmingham, Alabama, USA.

The funding bodies did not play any role in the design; in the collection, analysis, and interpretation of the data; in the writing of the manuscript; and in the decision to submit the manuscript for publication. The statements in this article are solely the responsibility of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee.

Author information

Authors and Affiliations

  1. Medicine Service, VA Medical Center, 700 19th St S, AL, Birmingham, 35233, USA

    Jasvinder A. Singh

  2. Department of Medicine at School of Medicine, University of Alabama at Birmingham, 1720 Second Ave South, Birmingham, AL, 35294-0022, USA

    Jasvinder A. Singh

  3. Division of Epidemiology at School of Public Health, University of Alabama at Birmingham, 1720 Second Ave South, Birmingham, AL, 35294-0022, USA

    Jasvinder A. Singh

  4. University of Alabama at Birmingham, Faculty Office Tower 805B, 510 20th Street S, Birmingham, AL, 35294-0022, USA

    Jasvinder A. Singh, Larry R. Hearld & Allyson G. Hall

  5. Department of Biostatistics, University of Alabama at Birmingham; Birmingham/Atlanta VA Geriatric Research, Education, & Clinical Center, Department of Veteran’s Affairs, 510 20th Street S, Birmingham, AL, 35294-0022, USA

    T. Mark Beasley

Authors
  1. Jasvinder A. Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Larry R. Hearld
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Allyson G. Hall
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. T. Mark Beasley
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JAS is responsible for the study concept and design; obtaining funding, study supervision and conduct; data acquisition; analysis and interpretation of the results; administrative, technical, or material support; and writing the first draft of the manuscript. LH is responsible for the study design, data acquisition, analysis and interpretation of the results, and critical revision of the manuscript for important intellectual content. AH is responsible for the study design, data acquisition, analysis and interpretation of the results, and critical revision of the manuscript for important intellectual content. MB is responsible for the data acquisition, analysis and interpretation of the results, and critical revision of the manuscript for important intellectual content. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Jasvinder A. Singh.

Ethics declarations

Ethics approval and consent to participate

The study was approved by the Institutional Review Board (IRB) at the University of Alabama at Birmingham (UAB, 300002272; UAB Coordinating Center 300002554). Each study site’s Institutional Review Board (IRB) also reviewed and approved the study. All investigations were conducted in conformity with ethical principles of research.

Consent for publication

Not applicable.

Competing interests

There are no financial conflicts related directly to this study. JAS has received consultant fees from Crealta/Horizon, Medisys, Fidia, UBM LLC, Trio Health, Medscape, WebMD, MedIQ, Two labs Inc, Adept Field Solutions, Clinical Care options, Clearview healthcare partners, Putnam associates, Spherix, Practice Point Communications, the National Institutes of Health, and the American College of Rheumatology. JAS owns stock options in TPT Global Tech, Vaxart pharmaceuticals and Charlotte’s Web Holdings, Inc. JAS previously owned stock options in Amarin and Moderna pharmaceuticals and Viking therapeutics. JAS is on the speaker’s bureau of Simply Speaking. JAS is a member of the executive of OMERACT, an organization that develops outcome measures in rheumatology, and receives arms-length funding from 12 companies. JAS serves on the FDA Arthritis Advisory Committee. JAS is a member of the Veterans Affairs Rheumatology Field Advisory Committee. JAS is the editor and the Director of the UAB Cochrane Musculoskeletal Group Satellite Center on Network Meta-analysis. JAS previously served as a member of the following committees: member, the American College of Rheumatology’s (ACR) Annual Meeting Planning Committee (AMPC) and Quality of Care Committees, the Chair of the ACR Meet-the-Professor, Workshop and Study Group Subcommittee, and the co-chair of the ACR Criteria and Response Criteria subcommittee. Drs. Hearld, Hall, and Beasley have no financial conflicts.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Appendix 1.

Data Collection and Analyses for Specific Aim 1 and Specific Aim 3. Appendix 2. Members of the various IDEAL committees.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Singh, J.A., Hearld, L.R., Hall, A.G. et al. Implementing the DEcision-Aid for Lupus (IDEAL): study protocol of a multi-site implementation trial with observational, case study design. Implement Sci Commun 2, 30 (2021). https://doi.org/10.1186/s43058-021-00118-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s43058-021-00118-9

Keywords

Implementation Science Communications

ISSN: 2662-2211