Skip to main content
Download PDF
Download ePub

Implementation for Sustained Impact in Teleophthalmology (I-SITE): applying the NIATx Model for tailored implementation of diabetic retinopathy screening in primary care

Implementation Science Communications volume 2, Article number: 74 (2021) Cite this article

Abstract

Background

Teleophthalmology provides evidence-based, telehealth diabetic retinopathy screening that is underused even when readily available in primary care clinics. There is an urgent need to increase teleophthalmology use in the US primary care clinics. In this study, we describe the development of a tailored teleophthalmology implementation program and report outcomes related to primary care provider (PCP) adoption.

Methods

We applied the 5 principles and 10 steps of the NIATx healthcare process improvement model to develop and test I-SITE (Implementation for Sustained Impact in Teleophthalmology) in a rural, the US multi-payer health system. This implementation program allows patients and clinical stakeholders to systematically tailor teleophthalmology implementation to their local context. We aligned I-SITE components and implementation strategies to an updated ERIC (Expert Recommendations for Implementing Change) framework. We compared teleophthalmology adoption between PCPs who did or did not participate in various components of I-SITE. We surveyed PCPs and clinical staff to identify the strategies they believed to have the highest impact on teleophthalmology use.

Results

To test I-SITE, we initiated a year-long series of 14 meetings with clinical stakeholders (n=22) and met quarterly with patient stakeholders (n=9) in 2017. Clinical and patient stakeholder groups had 90.9% and 88.9% participant retention at 1 year, respectively. The increase in teleophthalmology use was greater among PCPs participating in the I-SITE implementation team than among other PCPs (p < 0.006). The proportion of all PCPs who used the implementation strategy of electing diabetic eye screening for their annual performance-based financial incentive increased from 0% (n=0) at baseline to 56% (n=14) following I-SITE implementation (p = 0.004). PCPs and clinical staff reported the following implementation strategies as having the highest impact on teleophthalmology use: reminders to ask patients about diabetic eye screening during clinic visits, improving electronic health record (EHR) documentation, and patient outreach.

Conclusions

We applied the NIATx Model to develop and test a teleophthalmology implementation program for tailored integration into primary care clinics. The NIATx Model provides a systematic approach to engaging key stakeholders for tailoring implementation of evidence-based telehealth interventions into their local context.

Peer Review reports

Background

Telehealth has the potential to greatly increase access to clinical care, especially for populations with limited access to healthcare due to distance, time, and financial constraints [1,2,3,4]. Successful telehealth programs have been largely implemented in single-payer health systems both in the USA and abroad [5,6,7,8]. Single-payer health systems have been successful in implementing telehealth because they provide highly-coordinated care through a shared electronic health record system and have unique funding structures [6]. Yet, the majority of the US patients obtain care within multi-payer health systems, which face greater financial and logistical challenges to implementing complex telehealth programs [6]. Few studies in dissemination and implementation science provide a stepwise process for overcoming these barriers using stakeholder input to tailor integration of interventions into their local context [9, 10].

There is an urgent need to facilitate widespread adoption of telehealth in multi-payer health systems to improve detection of diabetic retinopathy. Diabetic retinopathy remains the leading cause of blindness in the USA largely due to lack of screening [11]. Blindness from diabetes is projected to continue increasing dramatically due to our rapidly rising diabetes population [11]. Teleophthalmology can greatly increase screening and reduce blindness from diabetes through early detection and treatment [5, 12, 13]. It is an evidence-based method for diabetic eye screening endorsed by the American Diabetes Association [14] in which patients obtain retinal photos using specialized cameras located conveniently in the primary care clinic. However, a 5-year randomized controlled trial found a lack of sustained effectiveness beyond 18 months when implemented in multi-payer primary care clinics [15]. Of the multiple patient and provider-related barriers to teleophthalmology use, [16,17,18] our work and that of others have identified that the major barrier is lack of teleophthalmology integration into primary care workflows.

Few clinical interventions have been proposed to increase teleophthalmology use, and the majority have focused on patient-level educational interventions [9, 19, 20]. We previously identified multiple patient and primary care provider (PCP) barriers and facilitators to teleophthalmology use in a rural, multi-payer health system through individual interviews [17]. Based on our results from that qualitative study, we postulated that health systems-level implementation strategies would be effective for increasing teleophthalmology use in the same rural health system. We identified the NIATx Model as a model that could increase teleophthalmology use by allowing patients and clinical stakeholders (e.g., PCPs, staff, and administrators) to systematically tailor teleophthalmology implementation to the unique needs and resources of their clinic. The NIATx Model is a healthcare process improvement model originally developed by systems engineers for behavioral health [21]. We adapted the NIATx Model to develop and test a teleophthalmology implementation program, I-SITE: Implementation for Sustained Impact in Teleophthalmology [22].

I-SITE allows patients and clinical stakeholders to systematically tailor teleophthalmology implementation to their local context. We previously reported patient-level clinical outcomes and found that I-SITE implementation at a single primary care clinic (Main) sustained a significant overall increase in teleophthalmology use and diabetic eye screening rates among all 5 primary care clinics within a rural multi-payer health system over 2 years [23]. Yearly teleophthalmology use increased following I-SITE implementation by more than 4-fold (from 41 to 196), with a greater increase at the Main (odds ratio [OR] 10.0) versus the other 4 Outreach clinics (OR 1.69). Overall, diabetic eye screening rates within the same health system were sustained at 36% above baseline at 2-years post-implementation.

In this work, we describe the development of I-SITE by applying the NIATx Model and assessed provider adoption from the same study. We aligned I-SITE components and implementation strategies with those from an updated ERIC framework [24]. We tested the hypothesis that teleophthalmology adoption would be higher among PCPs who did versus those who did not participate in various components of I-SITE. We also report implementation strategies that PCPs and clinical stakeholders believed to be the most impactful for increasing teleophthalmology use.

Methods

Research setting and design

We partnered with the Mile Bluff Medical Center, a rural US multi-payer health system, where a teleophthalmology program was established in 2015. Mile Bluff is a federally designated Rural Health Clinic located in Juneau County, WI, where the population is 83.5% rural and ranks in the lowest quartile of Wisconsin counties based on both health outcomes and socioeconomic determinants [25]. The teleophthalmology program was developed based on the 2011 American Telemedicine Association Telehealth Practice Recommendations for Diabetic Retinopathy [17, 26].

In this type of store-and-forward telemedicine program, PCPs place orders for patients to obtain teleophthalmology imaging at their convenience (usually same-day). A single-field, a 45-degree photograph of the disc and macula, along with an anterior segment photograph, is obtained for each eye using a Topcon NW400 non-mydriatic fundus camera (Topcon Medical Systems, Inc., Oakland, NJ, USA). The digital images are evaluated remotely by the University of Wisconsin eye specialists who send reports to the PCP and patient within 1 week. Expedited scheduling of a follow-up clinical exam with a local eye care provider is performed when further in-person care is needed. The teleophthalmology service was billed using CPT 92227 (remote imaging for detection of retinal disease) [27]. However, teleophthalmology was underutilized in the first 2 years of the program, prior to the introduction of I-SITE, with fewer than 20 patients imaged per quarter among approximately 2000 patients with diabetes.

We applied the NIATx Model to develop Implementation for Sustained Impact in Teleophthalmology (I-SITE) (free online toolkit available at: http://hipxchange.org/I-SITE) and tested I-SITE at Mile Bluff. We used a skilled primary care practice facilitator who had training in the NIATx Model to serve as the I-SITE Coach and recruited a local implementation team to develop and test implementation strategies. The I-SITE Coach was responsible for setting meeting agendas, preparing presentation materials, summarizing prior meeting minutes, facilitating discussions, and assigning tasks for next steps. The I-SITE implementation team was comprised of a mixed group of clinical stakeholders from Mile Bluff including PCPs, patient care staff, information technology/electronic health records (IT/EHR) staff, and clinic administrators recruited as described below. Our ideal size for the implementation team was proposed to be 8–12 members to allow for input from all team members. We also planned to recruited a patient stakeholder group of similar size to further refine patient-targeted implementation strategies.

The first 3 meetings of the implementation team were designed to be 1-h in length and conducted in-person to facilitate in-depth discussion regarding: (1) flowcharting the teleophthalmology workflow, (2) discussion of barriers to teleophthalmology use, and (3) strategies to overcome barriers to teleophthalmology use. Subsequent meetings were designed as a series of brief 15–30 min teleconferences to accommodate team members participating from different locations. These I-SITE in-person meetings and teleconferences incorporated the following components from the NIATx Model 10 steps: assemble the change team, plan the change, assign roles and tasks, and rapid cycle testing (Fig. 1). The teleconference meetings also encompass the following ERIC implementation strategies: implementation facilitation, organize implementation teams and team meetings, assess and redesign workflow, obtain feedback, use iterative strategies, and provide ongoing consultation, as well as audit and feedback. The Nominal Group Technique is used to generate ideas and to make decisions regarding the top barriers and strategies to test [17, 28]. With this technique, each individual on the implementation team sequentially offers one idea until no new ideas are generated. Each individual then votes for 2–3 ideas, after which the top 2–3 ideas endorsed by the group are identified through tallying the votes. The top strategies selected are then implemented by clinical providers and staff.

Fig. 1
figure 1

Implementation for Sustained Impact in Teleophthalmology (I-SITE) and patient stakeholder meeting goals and participants

Full size image

Given the busy clinical schedules of PCPs and clinical staff, it can be challenging to schedule meetings so we used teleconferences and regularly scheduled staff meetings whenever possible to facilitate participation. We also note that while the implementation team designed various strategies that were offered to all PCPs, the health system’s organizational culture empowered PCPs to independently select which strategies to implement in their clinical practice.

We measured I-SITE’s impact on teleophthalmology adoption by PCPs and surveyed PCPs and clinical staff to identify implementation strategies they perceived to have the greatest impact on increasing diabetic eye screening rates. The final intervention was reported according to guidance from TiDieR (Supplementary Material) [29].

Patient and clinical stakeholder recruitment

Patients and clinical stakeholders were recruited in March 2017 from Mile Bluff to participate in two separate stakeholder groups to develop, test, and refine their teleophthalmology implementation with guidance from the I-SITE Coach. For our patient stakeholder group, we invited adult patients with diabetes who either had previous teleophthalmology imaging or had expressed interest in participating in research during a prior diabetic eye screening rate survey [17]. All patients were first mailed an invitation letter from the Chief Medical Officer at Mile Bluff, including the opportunity to opt-out. This was followed by a phone call from a member of our research team who provided a detailed description of the project and participant expectations. Patient stakeholder group participants received $50 for each meeting attended. For recruitment of our clinical stakeholder group, PCPs were provided with an introduction to the I-SITE program and invited to participate during an in-person presentation at a regularly scheduled staff meeting. Patient care staff who regularly worked with each participating PCP joined the clinical stakeholder team as well. Additional primary care staff and administrators (e.g., clinic management, quality improvement, and IT/EHR staff) were selected for participation by Mile Bluff leadership. No compensation was provided for clinical stakeholder group members as they were considered to be participating in quality improvement activities within the responsibilities of their employment.

Outcome measures and statistical analysis

Data on teleophthalmology use was obtained using medical records abstraction. Patient and clinical stakeholder characteristics were obtained through self-report, medical records review, and review of personnel records at the Mile Bluff Medical Center. We used the Wilcoxon rank sum test to evaluate the change pre- and post-I-SITE implementation in teleophthalmology referrals among PCPs on the I-SITE implementation team versus PCPs not on the implementation team. We used McNemar’s test to assess the change in the number of PCPs who elected diabetic eye screening for their performance-based financial bonus each year. We also used t tests to compare yearly teleophthalmology referrals among PCPs who did versus PCPs who did not elect diabetic eye screening for their performance-based financial incentive. We identified implementation strategies perceived by PCPs and clinic staff to have the greatest impact on teleophthalmology use through a written survey conducted at a regularly scheduled staff meeting 2 years following I-SITE implementation, which was timed to align with our primary clinical effectiveness outcome reported previously [23]. All statistical analyses were performed using Microsoft® Excel (Microsoft Corp., Redmond, WA, USA).

Ethics and institutional review board review

Our study was reviewed by the University of Wisconsin Health Sciences Institutional Review Board (IRB) staff and was determined to be exempt from full IRB review due the activities being consistent with providing quality improvement for implementing an evidence-based clinical practice within a health system. All research activities were conducted in accordance with the Declaration of Helsinki and all federal and state laws.

Results

Application of the NIATx Model in the development of Implementation for Sustained Impact in Teleophthalmology (I-SITE)

We applied the 5 principles and 10 steps of the NIATx Model, a systematic healthcare process improvement model, to develop I-SITE (Table 1). I-SITE allows for patients and clinical stakeholders to tailor teleophthalmology implementation through a series of meetings over the course of 1 year. These principles and steps from the NIATx Model and I-SITE align with several key implementation strategies selected from an updated ERIC framework (Table 1) [24]. The final version of I-SITE (Fig. 1) was developed as a modification of the process we applied in this study. The main modifications in the final version were that (1) the Introduction and Invitation to participate will be delivered as a live webinar during a regularly scheduled staff meeting for efficiency, rather than in-person, and that (2) the same implementation team, comprised of mixed group of clinical stakeholders, would gather for all I-SITE meetings. In this study, we had a larger clinical stakeholder group (n=22) present for the in-person I-SITE meetings, due to a greater-than-expected level of interest in participation. We subsequently met with a subset of this group (n=11) for implementation team teleconference meetings to achieve the desired group size to ensure input from all team members.

Table 1 Alignment between the ERIC framework [24] with NIATx Model and Implementation for Sustained Impact in Teleophthalmology (I-SITE)
Full size table

Patient and clinical stakeholder meetings

A total of 9 patients with diabetes and 22 clinical stakeholders participated in separate meetings from May 2017 to October 2018 to develop and iteratively test strategies to increase teleophthalmology use at Mile Bluff (Table 2). We held separate patient and clinical stakeholder groups due to differences in power dynamics, relationships, and terminology used among participants, which could limit patient stakeholder input. In addition, by dividing into two groups, we were able to include a greater total number of participants to lend their diverse perspectives. Information feedback and exchange between the two groups was facilitated by a practice facilitator (e.g., I-SITE Coach).

Table 2 Demographics of participants in Implementation for Sustained Impact in Teleophthalmology (I-SITE)
Full size table

The patient stakeholder group met four times approximately quarterly over a period of 1 year. Each patient stakeholder meeting was 2 h in length and held in the evenings with a light meal served beginning 30 min before the meeting. Each patient stakeholder meeting began with a community-building exercise (i.e., ice breaker question), followed by a brief review of the purpose and importance of obtaining the group’s input for increasing teleophthalmology use and diabetic eye screening in their community. Information was then presented on how patient stakeholder feedback from the last meeting was used to further the project’s goals. Meetings were facilitated by a research team member (Bachelor’s-level Research Specialist), and discussion topics included identifying top patient barriers and strategies for increasing teleophthalmology use and diabetic eye screening. In addition, patient stakeholder input was used to refine patient-targeted implementation strategies to increase teleophthalmology use, including patient education handouts, example scripts for PCPs and patient care staff to discuss teleophthalmology with patients, and the patient outreach process by which patients were reminded to obtain yearly teleophthalmology imaging. For patient outreach, reminder letters along with an enclosed patient education handout were mailed to patients who had obtained teleophthalmology 10 months previously to remind them to obtain their yearly eye screening. This was followed by a phone call to help assist with scheduling when needed.

The clinical stakeholder group was composed of 22 members who participated in three in-person meetings led by the I-SITE Coach. The first meeting of the clinical stakeholder group focused on creating a step-by-step flowchart of the primary care clinic workflow for identifying patients due for diabetic eye screening and placing orders for teleophthalmology. The second and third meetings employed the Nominal Group Technique to identify the top barriers to teleophthalmology use and the top implementation strategies based on those strategies participants expected to have the highest impact and be the most feasible to test within the next 2–3 weeks [17].

We subsequently brought together a subset of the clinical stakeholders to form a local implementation team to iteratively test implementation strategies. The implementation team (n = 11) consisted of 3 PCPs, 3 medical assistants (each whom works in the clinic with one of the 3 PCPs on the team), and 5 clinic administrative staff involved in quality improvement, IT/EHR support, and personnel management. This group met for a total of 11 meetings every 1–2 months for 15–30-min teleconferences led by the research team leader (Y.L.), a physician knowledgeable about primary care workflows, to review data and obtain feedback on implementation strategies. The I-SITE Coach did not lead these sessions in our study due to logistical reasons related to scheduling. The implementation team made decisions regarding whether to continue, abandon, or further refine each implementation strategy, as well as to propose new implementation strategies for testing.

We found that I-SITE had high retention of clinical and patient stakeholders at 12 months (90.9% (n=10) and 88.9% (n=9)), respectively). Patient stakeholders reported that they were motivated to continue participation because they received updates at each meeting about on how their feedback was used to make meaningful changes to the clinic’s implementation strategies. Other major factors patients reported for their continued participation included that they felt they were helping their community by contributing towards increasing diabetic eye screening. They also appreciated learning more about teleophthalmology and diabetic eye screening during these meetings, which allowed them to share that knowledge with family and friends with diabetes.

Teleophthalmology adoption by primary care providers (PCPs)

The majority of PCPs (n=25) referred at least one patient for teleophthalmology in the year prior to and in the year following initiation of I-SITE implementation (n=18 (72.0%) and n=20 (80.0%), respectively). PCPs on the implementation team (n=3) had a greater increase in the number of teleophthalmology referrals pre- and post-I-SITE than other PCPs (n=22) (p < 0.006).

Of note, the PCP with the highest number of teleophthalmology referrals in the first year after initiation of I-SITE implementation was the first PCP to elect, on his own initiative, to tie his performance-based financial bonus to diabetic eye screening in that year (i.e., early adopter of this implementation strategy). This provider, who was on the implementation team, developed this implementation strategy during I-SITE implementation. In this health system, PCPs were able to choose to have their performance-based financial bonus tied to any major healthcare quality measure (e.g., hemoglobin A1c testing, mammography, colonoscopy) based on the proportion of their patients who had fulfilled that particular measure. We were unable to obtain details from the health system regarding the precise structure of this financial bonus as employee compensation is confidential.

In subsequent years, there was an increase in the number of PCPs who adopted this implementation strategy by electing diabetic eye screening as a performance-based financial incentive (up to 56%, n=14, of PCPs in 2020) (p = 0.004) (Table 3). PCPs who elected to have diabetic eye screening as a performance-based financial incentive referred more patients for teleophthalmology than those who did not make this election in 2018 and 2019 (p = 0.03 and p = 0.005, respectively). In 2020, we note that there was significant confounding related to the decline in teleophthalmology use beginning in quarter 2 due to reductions in primary care patient clinic visits at the Mile Bluff Medical Center related to safety concerns from the COVID-19 pandemic. In addition, the increase in overall referrals over time was not explained by an increase in eligible patients since the total numbers of patients with diabetes cared for by primary care providers at the Mile Bluff Medical Center was similar across all years of our study (average: 2293, range from 2119 to 2497).

Table 3 Teleophthalmology referrals among primary care providers (PCPs) based on election of diabetic eye screening performance-based financial incentive
Full size table

Teleophthalmology implementation strategies

The teleophthalmology implementation strategies tested are summarized in Table 4. All have been continued with the exception of the implementation team, which has been put on hold due to competing clinical demands related to the COVID-19 pandemic. We were not able to fully capture the duration or adaptation of every implementation strategy tested because the health system’s organizational culture empowered each PCPs to individually select some of the implementation strategies to use and when, as well as adapt the strategies and components of those strategies to meet each provider’s needs.

Table 4 Primary care provider (PCP) and clinical staff perceptions of implementation strategies’ impact on teleophthalmology use
Full size table

However, we were able to obtain feedback on which strategies clinical stakeholders believed to have a high impact on teleophthalmology use using a written survey. The written survey was distributed to PCPs and clinical staff at regularly scheduled clinic meetings 2 years following initial I-SITE implementation, which was timed to align with our primary clinical effectiveness outcome that was reported previously [23]. At the time of the survey, there were a total of 25 PCPs and a total of 35 primary care clinical staff in the organization who would have been eligible to participate. At the meeting in which the survey was administered, there were 14 PCPs and 27 primary care clinical staff present who were invited to participate. Therefore, the response rate among those invited to participate was 78.6% (n=11) and 51.9% (n=14) among PCPs and clinical staff, respectively.

Strategies perceived by the majority of PCPs and clinical staff to have a high impact on increasing teleophthalmology use were reminders to ask patients about diabetic eye screening during clinic visits, improving the ease and consistency of diabetic eye screening documentation in the EHR (e.g., results from teleophthalmology imaging or from a dilated eye exam performed by the patient’s eye doctor), and outreach to contact patients due for diabetic eye screening (Table 4). Most clinical staff (n=9, 64.3%) believed that providing the patient education handout during a patient’s clinic visit had a high impact on increasing teleophthalmology use, but this strategy was endorsed by fewer PCPs (n=4, 36.4%).

Implementation strategies that were not considered to be high impact by the majority of PCPs and clinical staff included bi-monthly and quarterly emails with audit and feedback reporting of group- and individual-level data on teleophthalmology use (Table 4). PCP participation in the implementation team and PCP performance-based financial incentives also were not considered to be high impact by most clinical stakeholders. Notably, one PCP wrote in the word “No!” next to the survey item regarding whether financial incentives had a high impact. Only a minority of PCPs (n=2, 18.2%) and clinical staff (n=2, 14.3%) perceived teleophthalmology presentations providing education regarding the importance of teleophthalmology for diabetic screening and reporting yearly teleophthalmology use and diabetic eye screening rates at regularly scheduled provider and staff meetings to have a high impact on teleophthalmology use.

Discussion

We adapted the NIATx Model, a healthcare process improvement model originally designed by systems engineers for behavioral health, to develop a systematic implementation program (I-SITE) to guide tailored teleophthalmology implementation in primary care clinics by patient and clinical stakeholders. We found that I-SITE increased provider teleophthalmology use, particularly among PCPs that participated in the I-SITE implementation team and among those that elected diabetic eye screening for their performance-based financial incentive. Data on strategies that increased provider teleophthalmology use complements our clinical outcomes demonstrating that I-SITE effectively sustained increased diabetic eye screening rates in a primary care-based teleophthalmology program in our prior work [23]. This study provides a model for adapting NIATx to guide implementation of evidence-based telehealth interventions to increase their adoption in primary care settings.

We aligned NIATx Model and I-SITE components to selected strategies from an updated ERIC framework developed by Perry et al. [24] based on ERIC utilization in a large-scale dissemination and implementation research program in primary care clinics. Among the suggested changes by these authors, they proposed an updated grouping of ERIC strategies as follows: (1) build health information technology to support data-informed quality improvement, (2) build quality improvement capacity and improve outcomes, (3) enhance clinician and practice member knowledge, and (4) build community connections and patient involvement [24]. Notably, I-SITE’s components and implementation strategies encompass all 4 of these categories, which may help explain why I-SITE was effective for increasing teleophthalmology use among PCPs. A team in Ireland also recently developed a proposed intervention to increase uptake of teleophthalmology diabetic eye screening by combining theory (i.e., Theoretical Domains Framework) with stakeholder engagement in a primary care setting [9]. All of the implementation strategies included in their final intervention are also operationalized within I-SITE, which was developed independently. According to Proctor et al.’s [32] recommendations for specifying implementation strategies, we note that the actors and target in I-SITE are primarily the practice facilitator and health system administrators (actors) working in concert to make systems-level changes to the clinical practice (target). While ERIC provides a valuable, standardized nomenclature for describing implementation strategies, the NIATx Model provides systematic guidance for the process by which implementation strategies are selected and utilized. The application of NIATx may be generalizable to integrating other telehealth interventions in primary care through tailoring implementation to fit the local context by key stakeholders.

The NIATx Model can be applied to engage local implementation teams across a broad range of settings and highlights the importance of practice facilitation. While NIATx was originally designed for increasing behavioral health services utilization, Pankow et al. [33] modified the NIATx Model to engage local implementation teams in correctional facilities to increase provision of HIV services. In this study, we successfully engaged patient and clinical stakeholders using I-SITE to sustain increased teleophthalmology use and diabetic eye screening rates by applying the NIATx Model. For our clinical stakeholder meetings, we invited individuals with varying roles within the organization to fill in critical information that may only be known to certain members of the organization, such medical assistants, EHR staff, and schedulers, to ensure that the meetings could move forward more effectively and efficiently. While sustaining engagement among clinical stakeholders may not always be practical in a primary care setting [34], we took advantage of previously scheduled meetings and used a teleconference format whenever possible to limit the burden of additional meetings. In addition, the NIATx Model provides a clearly defined role for a practice facilitator, which may be especially important for successful implementation within primary care clinics. A 2014 randomized controlled trial found that having a practice facilitator work in conjunction with a clinical process improvement team resulted in a significantly greater increase in traditional in-person diabetic eye screening, as well as other measures of diabetes care quality, compared to teams without a practice facilitator [35].

The Technology Acceptance Model, while not used in our work a priori, does however provide a useful model to help understand which implementation strategies clinical stakeholders identified to have the highest impact on teleophthalmology use. This model theorizes that two factors affect acceptance of technology: its perceived usefulness and perceived ease of use [36]. Rho et al. [37] found that both perceived usefulness and perceived ease of use were significantly associated with physicians’ intention to use telemedicine. Interestingly, implementation strategies identified by clinical stakeholders in our study as having the highest impact on teleophthalmology use were all related to increased perceived ease of use by clinical stakeholders (i.e., reminders to ask patients about diabetic eye screening during clinic visits, improving electronic health record (EHR) documentation, and patient outreach). In addition, we believe “perceived usefulness” was already high among PCPs since the implementation strategy of ‘Provider and Staff Education’ was considered by a minority of PCPs (n=2 (18.2%)) to have a high impact on teleophthalmology use. In addition, the majority of PCPs had referred patients for teleophthalmology prior to I-SITE and it would be highly unlikely for a busy PCP to take the time to explain, recommend, and order this type of test for a patient if he or she did not perceive that it would be useful. Implementation strategies targeting clinical stakeholders to increase teleophthalmology use in primary care may be most effective, therefore, if they primarily focus on improving perceived ease of use rather than perceived usefulness. Similar strategies may be effective for other types of evidence-based screening programs managed by primary care providers such as mammography, as well as for other store-and-forward telemedicine programs such as teledermatology.

Multiple studies have assessed the effectiveness of provider-based financial incentives to increase preventive screening. Levin-Scherz et al. [38] found that when physician bonuses were tied to quality measures, diabetic eye screening rates increased significantly. Similar results were found with increased hemoglobin A1c testing in a federally qualified health clinic where physicians could earn back withheld portions of their salaries by meeting certain diabetes quality measure targets [39]. However, a 2011 Cochrane review found insufficient evidence to either support or discourage the use of provider financial incentives to improve quality metrics in primary care [40]. Cultural attitudes may also lead providers and healthcare staff to ambivalently or negatively perceive the influence of provider financial incentives [41]. This may explain why few PCPs and clinical staff endorsed provider financial incentives as having a high impact on teleophthalmology use despite our data supporting its effectiveness in this study.

In addition, PCPs who volunteered to participate on the I-SITE implementation team referred significantly more patients for teleophthalmology than other PCPs following I-SITE implementation. PCPs on the implementation team were actively associated with supporting the intervention, a role defined by the Consolidated Framework for Implementation Research (CFIR )[42] as being a “champion.” Prior studies have demonstrated the importance of champions or peer leaders for increasing adoption of telehealth and evidence-based interventions at the organizational level, but have not examined the effect of being a champion on individual provider behavior [43, 44]. Of note, the provider champions in our study did not demonstrate higher teleophthalmology use prior to I-SITE implementation compared to other PCPs. It is possible that participation as a champion in the I-SITE implementation team resulted in a higher “dose” of provider engagement and buy-in that led to increased teleophthalmology use among these PCPs. We note that since participation on the implementation team was voluntary, these PCPs likely had individual characteristics (e.g., level of personal interest, self-efficacy, and motivation) that differed from other PCPs. Therefore, it seems unlikely that either random assignment or formal appointment of PCPs to the implementation team would have resulted in the same effect on an individual provider’s teleophthalmology use.

Limitations of our study included that we used a pre-post study design in a single rural health system with a small sample size of primary care providers serving a predominantly white, non-Hispanic population, as reflected in the demographics of our patient stakeholders (Table 2). Additional studies testing I-SITE across multiple health systems, including urban health systems and those serving non-white populations, are needed to further assess the generalizability of our findings. In addition, the health system’s organizational culture empowered each PCPs to individually select some of the implementation strategies to use and when. Therefore, multiple strategies were tested across overlapping time periods, which made it difficult to precisely quantify the relative contributions of each implementation strategy towards increasing teleophthalmology use, particularly for such strategies as patient rooming checklists (i.e., each PCP used their own, rather than a clinic-standardized, checklist), consistent documentation of diabetic eye screening in the EHR, patient outreach, as well as bimonthly and quarterly emailed performance reports. However, we did survey clinical stakeholders to ascertain their perceptions regarding high impact implementation strategies, with excellent response rates of 50–80% [45]. Finally, we did not ask patient stakeholders about which implementation strategies they perceived to have the highest impact on teleophthalmology use. The majority of tested strategies were directed at clinical stakeholders, rather than patients, based on our prior qualitative work that identified systems-based workflow strategies as being the most likely to be effective for increasing teleophthalmology use in rural primary care clinics [17].

Conclusions

We applied the NIATx Model to develop and test a teleophthalmology implementation program that allows for tailored integration into primary care clinics. Our data suggest that the NIATx Model provides a systematic and effective approach to engaging key stakeholders for tailoring implementation of evidence-based telehealth interventions to fit their local context.

Availability of data and materials

The dataset supporting the conclusions of this article is available by request to those who obtain approval following the guidelines of the University of Wisconsin Health Sciences Institutional Review Board.

Abbreviations

CFIR:

Consolidated Framework for Implementation Research

EHR:

Electronic health record

ERIC:

Expert Recommendations for Implementing Change

I-SITE:

Implementation for Sustained Impact in Teleophthalmology

IT:

Information technology

NIATx Model :

NIATx Model of healthcare improvement

PCP:

Primary care provider

MA:

Medical assistant

References

  1. Hale NL, Bennett KJ, Probst JC. Diabetes care and outcomes: disparities across rural America. J Community Health. 2010;35(4):365–74. https://doi.org/10.1007/s10900-010-9259-0.

    Article  PubMed  Google Scholar 

  2. Jani PD, Forbes L, Choudhury A, Preisser JS, Viera AJ, Garg S. Evaluation of Diabetic Retinal Screening and Factors for Ophthalmology Referral in a Telemedicine Network. JAMA Ophthalmol. 2017;135(7):706–14. https://doi.org/10.1001/jamaophthalmol.2017.1150.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Krishna S, Gillespie KN, McBride TM. Diabetes burden and access to preventive care in the rural United States. J Rural Health. 2010;26(1):3–11. https://doi.org/10.1111/j.1748-0361.2009.00259.x.

    Article  PubMed  Google Scholar 

  4. Liu Y, Zupan NJ, Shiyanbola OO, Swearingen R, Carlson JN, Jacobson NA, et al. Factors influencing patient adherence with diabetic eye screening in rural communities: A qualitative study. Plos One. 2018;13(11):e0206742. https://doi.org/10.1371/journal.pone.0206742.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Scanlon PH. The English national screening programme for sight-threatening diabetic retinopathy. J Med Screen. 2008;15(1):1–4. https://doi.org/10.1258/jms.2008.008015.

    Article  CAS  PubMed  Google Scholar 

  6. Hussey P, Anderson GF. A comparison of single- and multi-payer health insurance systems and options for reform. Health Policy. 2003;66(3):215–28. https://doi.org/10.1016/S0168-8510(03)00050-2.

    Article  CAS  PubMed  Google Scholar 

  7. Lynch MG, Maa AY. Diagnostic Time for Teleophthalmic Care. JAMA Ophthalmol. 2019;137(7):808–9. https://doi.org/10.1001/jamaophthalmol.2019.1202.

    Article  PubMed  Google Scholar 

  8. VA Telehealth Services: Fact Sheet. In: Affairs DoV, editor. Washington DC: Office of Public Affairs, Media Relations.

  9. Riordan F, Racine E, Phillip ET, Bradley C, Lorencatto F, Murphy M, et al. Development of an intervention to facilitate implementation and uptake of diabetic retinopathy screening. Implement Sci. 2020;15(1):34. https://doi.org/10.1186/s13012-020-00982-4.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Powell BJ, Beidas RS, Lewis CC, Aarons GA, McMillen JC, Proctor EK, et al. Methods to Improve the Selection and Tailoring of Implementation Strategies. J Behav Health Serv Res. 2017;44(2):177–94. https://doi.org/10.1007/s11414-015-9475-6.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Centers for Disease Control and Prevention. National diabetes statistics report: estimates of diabetes and its burden in the United States, 2014. Atlanta: Department of Health and Human Services; 2014.

  12. Liew G, Michaelides M, Bunce C. A comparison of the causes of blindness certifications in England and Wales in working age adults (16-64 years), 1999-2000 with 2009-2010. BMJ Open. 2014;4(2):e004015. https://doi.org/10.1136/bmjopen-2013-004015.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Early photocoagulation for diabetic retinopathy. ETDRS report number 9. Early Treatment Diabetic Retinopathy Study Research Group. Ophthalmology. 1991;98(5 Suppl):766-85.

  14. American Diabetes Association. 11. Microvascular Complications and Foot Care: Standards of Medical Care in Diabetes-2020. Diabetes Care. 2020;43(Suppl 1):S135–SS51. https://doi.org/10.2337/dc20-S011.

  15. Mansberger SL, Sheppler C, Barker G, Gardiner SK, Demirel S, Wooten K, et al. Long-term Comparative Effectiveness of Telemedicine in Providing Diabetic Retinopathy Screening Examinations: A Randomized Clinical Trial. JAMA Ophthalmol. 2015;133(5):518–25. https://doi.org/10.1001/jamaophthalmol.2015.1.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Bouskill K, Smith-Morris C, Bresnick G, Cuadros J, Pedersen ER. Blind spots in telemedicine: a qualitative study of staff workarounds to resolve gaps in diabetes management. BMC Health Serv Res. 2018;18(1):617. https://doi.org/10.1186/s12913-018-3427-9.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Liu Y, Zupan NJ, Swearingen R, Jacobson N, Carlson JN, Mahoney JE, et al. Identification of barriers, facilitators and system-based implementation strategies to increase teleophthalmology use for diabetic eye screening in a rural US primary care clinic: a qualitative study. BMJ Open. 2019;9(2):e022594. https://doi.org/10.1136/bmjopen-2018-022594.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Ramchandran RS, Yilmaz S, Greaux E, Dozier A. Patient perceived value of teleophthalmology in an urban, low income US population with diabetes. Plos One. 2020;15(1):e0225300. https://doi.org/10.1371/journal.pone.0225300.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Lawrenson JG, Graham-Rowe E, Lorencatto F, Rice S, Bunce C, Francis JJ, et al. What works to increase attendance for diabetic retinopathy screening? An evidence synthesis and economic analysis. Health Technol Assess. 2018;22(29):1–160. https://doi.org/10.3310/hta22290.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Lee SJ, McCarty CA, Sicari C, Livingston PM, Harper CA, Taylor HR, et al. Recruitment methods for community-based screening for diabetic retinopathy. Ophthalmic Epidemiol. 2000;7(3):209–18. https://doi.org/10.1076/0928-6586(200009)731-VFT209.

    Article  CAS  PubMed  Google Scholar 

  21. Hoffman KA, Ford JH 2nd, Choi D, Gustafson DH, McCarty D. Replication and sustainability of improved access and retention within the Network for the Improvement of Addiction Treatment. Drug Alcohol Depend. 2008;98(1-2):63–9. https://doi.org/10.1016/j.drugalcdep.2008.04.016.

    Article  PubMed  PubMed Central  Google Scholar 

  22. NIATx. The NIATx Model 2021. Available from: https://chess.wisc.edu/niatx/Content/ContentPage.aspx?PNID=1&NID=8. [Accessed 20 Nov 2020]

  23. Liu Y, Carlson JN, Torres Diaz A, Lock LJ, Zupan NJ, Molfenter TD, et al. Sustaining Gains in Diabetic Eye Screening: Outcomes from a Stakeholder-Based Implementation Program for Teleophthalmology in Primary Care. Telemed J E Health. 2020. https://doi.org/10.1089/tmj.2020.0270.

  24. Perry CK, Damschroder LJ, Hemler JR, Woodson TT, Ono SS, Cohen DJ. Specifying and comparing implementation strategies across seven large implementation interventions: a practical application of theory. Implement Sci. 2019;14(1):32. https://doi.org/10.1186/s13012-019-0876-4.

    Article  PubMed  PubMed Central  Google Scholar 

  25. U.S. Census Bureau. QuickFacts: Juneau County, Wisconsin. Available from: https://www.census.gov/quickfacts/fact/table/juneaucountywisconsin/PST045216. [Accessed 20 Nov 2020]

  26. Li HK, Horton M, Bursell SE, Cavallerano J, Zimmer-Galler I, Tennant M, et al. Telehealth practice recommendations for diabetic retinopathy, second edition. Telemed J E Health. 2011;17(10):814–37. https://doi.org/10.1089/tmj.2011.0075.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Carter K, Edgar J, Glasser DB, McNett C, Repka MX, Vicchrilli S. Telehealth Retinal Codes 2019. Available from: https://www.aao.org/practice-management/news-detail/teleheath-retinal-codes. [cited 2020 November 20]

  28. Nominal Group Technique: CHESS/ NIATx; 2020. Available from: https://chess.wisc.edu/niatx/content/contentpage.aspx?NID=147. [cited 2020 November 20]

  29. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348(mar07 3):g1687. https://doi.org/10.1136/bmj.g1687.

    Article  PubMed  Google Scholar 

  30. U.S. Census Bureau. ACS Income Data Tables. Available from: https://www.census.gov/topics/income-poverty/income/data/tables/acs.html. [Accessed 20 Nov 2020].

  31. Morris NS, MacLean CD, Chew LD, Littenberg B. The Single Item Literacy Screener: Evaluation of a brief instrument to identify limited reading ability. BMC Family Practice. 2006;7(1).

  32. Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8(1):139. https://doi.org/10.1186/1748-5908-8-139.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Pankow J, Willett J, Yang Y, Swan H, Dembo R, Burdon WM, et al. Evaluating Fidelity to a Modified NIATx Process Improvement Strategy for Improving HIV Services in Correctional Facilities. J Behav Health Serv Res. 2018;45(2):187–203. https://doi.org/10.1007/s11414-017-9551-1.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Cabassa LJ, Gomes AP, Meyreles Q, Capitelli L, Younge R, Dragatsi D, et al. Using the collaborative intervention planning framework to adapt a health-care manager intervention to a new population and provider group to improve the health of people with serious mental illness. Implement Sci. 2014;9(1):178. https://doi.org/10.1186/s13012-014-0178-9.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Dickinson WP, Dickinson LM, Nutting PA, Emsermann CB, Tutt B, Crabtree BF, et al. Practice facilitation to improve diabetes care in primary care: a report from the EPIC randomized clinical trial. Ann Fam Med. 2014;12(1):8–16. https://doi.org/10.1370/afm.1591.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Davis FD, Bagozzi RP, Warshaw PR. User Acceptance of Computer Technology: A Comparison of Two Theoretical Models. Management Sci. 1989;35(8):902–1028.

    Article  Google Scholar 

  37. Rho MJ, Choi IY, Lee J. Predictive factors of telemedicine service acceptance and behavioral intention of physicians. Int J Med Inform. 2014;83(8):559–71. https://doi.org/10.1016/j.ijmedinf.2014.05.005.

    Article  PubMed  Google Scholar 

  38. Levin-Scherz J, DeVita N, Timbie J. Impact of pay-for-performance contracts and network registry on diabetes and asthma HEDIS measures in an integrated delivery network. Med Care Res Rev. 2006;63(1 Suppl):14S–28S. https://doi.org/10.1177/1077558705284057.

    Article  PubMed  Google Scholar 

  39. Coleman K, Reiter KL, Fulwiler D. The impact of pay-for-performance on diabetes care in a large network of community health centers. J Health Care Poor Underserved. 2007;18(4):966–83. https://doi.org/10.1353/hpu.2007.0090.

    Article  PubMed  Google Scholar 

  40. Scott A, Sivey P, Ait Ouakrim D, Willenberg L, Naccarella L, Furler J, et al. The effect of financial incentives on the quality of health care provided by primary care physicians. Cochrane Database Syst Rev. 2011;9:CD008451.

    Google Scholar 

  41. Martin B, Jones J, Miller M, Johnson-Koenke R. Health Care Professionals’ Perceptions of Pay-for-Performance in Practice: A Qualitative Metasynthesis. Inquiry. 2020;57:46958020917491.

    PubMed  Google Scholar 

  42. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50. https://doi.org/10.1186/1748-5908-4-50.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Yakovchenko V, Miech EJ, Chinman MJ, Chartier M, Gonzalez R, Kirchner JE, et al. Strategy Configurations Directly Linked to Higher Hepatitis C Virus Treatment Starts: An Applied Use of Configurational Comparative Methods. Med Care. 2020;58(5):e31–e8. https://doi.org/10.1097/MLR.0000000000001319.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Li J, Talaei-Khoei A, Seale H, Ray P, Macintyre CR. Health Care Provider Adoption of eHealth: Systematic Literature Review. Interact J Med Res. 2013;2(1):e7. https://doi.org/10.2196/ijmr.2468.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Babbie ER. Survey research methods. Second edition. ed. Belmont, Calif.: Wadsworth Pub. Co.; 1990. xx, 395 pages p.

Download references

Acknowledgements

The authors would like to acknowledge Mark Banghart for his assistance with statistical analysis. We also appreciate the Wisconsin Network for Research Support (WINRS), Wisconsin Research and Education Network (WREN), and the Primary Care Academics Transforming Healthcare (PATH) Writing Collaborative for their feedback on stakeholder recruitment methodology and the development of implementation program materials. We thank the Mile Bluff Medical Center Diabetes Patient Advisory Council and Diabetes Quality Care Team for providing feedback to refine our research results.

Funding

This work was supported by the NIH/NEI K23 EY026518 and a Wisconsin Partnership Program New Investigator Award. It was also supported, in part, by an institutional grant from Research to Prevent Blindness, New York City, New York, USA, to the University of Wisconsin School of Medicine and Public Health, Department of Ophthalmology and Visual Sciences.

The funders had no influence on the design or results of the study.

Author information

Authors and Affiliations

  1. Department of Ophthalmology and Visual Sciences, University of Wisconsin School of Medicine and Public Health, 2870 University Ave., Ste. 206, Madison, WI, 53705, USA

    Alejandra Torres Diaz, Loren J. Lock & Yao Liu

  2. Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, WI, USA

    Todd D. Molfenter

  3. Department of Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA

    Jane E. Mahoney

  4. Department of Family Medicine and Community Health, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA

    Deanne Boss

  5. Mile Bluff Medical Center, Mauston, WI, USA

    Timothy D. Bjelland

Authors
  1. Alejandra Torres Diaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Loren J. Lock
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Todd D. Molfenter
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jane E. Mahoney
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Deanne Boss
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Timothy D. Bjelland
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Y.L., T.M., and J.M. contributed to developing the implementation program. Y.L., A.T.D., and D.B. conducted the stakeholder meetings. Y.L., A.T.D., and L.L. gathered and analyzed the data. J.M., T.M., D.B., and T.B. contributed to the discussion and reviewed/edited the manuscript. Y.L., A.T.D., and L.L. wrote the manuscript. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Yao Liu.

Ethics declarations

Ethics approval and consent to participate

This study was reviewed by the UW Human Subjects Institutional Review Board staff and was determined to be exempt from full IRB review because the activities were consistent with providing evidence-based quality improvement within a health system. All research activities were conducted in accordance with the Declaration of Helsinki and all federal and state laws.

Consent for publication

Not applicable.

Competing interests

None to declare.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Torres Diaz, A., Lock, L.J., Molfenter, T.D. et al. Implementation for Sustained Impact in Teleophthalmology (I-SITE): applying the NIATx Model for tailored implementation of diabetic retinopathy screening in primary care. Implement Sci Commun 2, 74 (2021). https://doi.org/10.1186/s43058-021-00175-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s43058-021-00175-0

Keywords

Implementation Science Communications

ISSN: 2662-2211