A mixed-methods approach to assessing implementers’ readiness to adopt digital health interventions (RADHI)

Abstract

Background

Despite being the primary setting for HIV prevention among men who have sex with men (MSM) since the start of the epidemic, community-based organizations (CBOs) struggle to reach this historically stigmatized and largely hidden population with face-to-face interventions. HIV researchers have readily turned to the internet to deliver critical HIV education to this group, with evidence of high effectiveness and acceptability across studies. However, implementation outside of research contexts has been limited and not well studied. We aimed to assess HIV CBOs’ readiness to adopt digital health interventions and identify contextual factors that may contribute to differing levels of readiness.

Methods

We recruited 22 CBOs across the US through a pragmatic request-for-proposals process to deliver Keep It Up! (KIU!), an evidence-based eHealth HIV prevention program. We used mixed methods to examine CBO readiness to adopt digital health interventions (RADHI). Before implementation, CBO staff completed a 5-item RADHI scale (scored 0–4) that demonstrated concurrent and predictive validity. We interviewed CBO staff using semi-structured questions guided by the Consolidated Framework for Implementation Research and compared RADHI score groups on determinants identified from the interviews.

Results

Eighty-five staff (range = 1–10 per CBO) completed the RADHI. On average, CBOs reported moderate-to-great readiness (2.74) to adopt KIU!. High RADHI CBOs thought KIU! was a top priority and an innovative program complementary to their existing approaches for their clients. Low RADHI CBOs expressed concerns that KIU! could be a cultural mismatch for their clients, was lower priority than existing programs and services, relied on clients’ own motivation, and might not be suitable for clients with disabilities. Value, appeal, and limitations did not differ by RADHI group.

Conclusions

While HIV CBOs are excited for the opportunities and advantages of digital interventions, additional pre-implementation and implementation support may be needed to increase perceived value and usability for different client populations. Addressing these limitations is critical to effective digital prevention interventions for HIV and other domains such as mental health, chronic disease management, and transitions in care. Future research can utilize our novel, validated measure of CBOs’ readiness to adopt digital health interventions.

Trial registration

NCT03896776, clinicaltrials.gov, 1 April 2019.

Contributions to the literature

• This study is among the first to identify contextual factors associated with community-based organizations’ readiness to adopt an evidence-based digital health intervention. These factors can be targeted by strategies to facilitate adoption and implementation of such programs, which have thus far seen limited delivery in the real world.

• Even among organizations that have formally adopted a digital health intervention, staff perceptions of readiness vary.

• The novel, brief, and domain-agnostic Readiness to Adopt Digital Health Interventions (RADHI) scale can be used by digital health researchers and potentially implementation decision-makers to identify organizations that may have greater likelihood of implementation success.

Introduction

More than 1.2 million persons in the U.S. are living with HIV [1]. Despite existing biomedical technologies with very high efficacy at preventing onward transmission and acquisition, HIV continues to affect approximately 32,100 new individuals each year [2]. It remains a significant concern among key minoritized populations, particularly cisgender men who have sex with men (MSM), who suffer the greatest burden of the disease [2]. Since the beginning of the epidemic, local community-based organizations (CBOs) have been essential in the delivery of HIV prevention and treatment services (e.g., condom distribution, linkage-to-care, sexual education, stigma reduction) [3]. These grassroots agencies that formed among historically stigmatized groups (e.g., sexual and gender minorities) were, for a time, the only providers to address HIV in their communities [4]. With the emergence of advancements in HIV testing, antiretroviral therapies, and pre-exposure prophylaxis (PrEP), CBOs have continued to play a crucial role in spreading essential information, debunking misconceptions, and facilitating access to these innovations.

Despite their critical function in brokering and providing a broad range of services, CBOs encounter various implementation challenges, such as scheduling and transportation barriers, limited resources, and staff constraints that hinder their ability to reach individuals at scale through face-to-face interventions [5, 6]. In response to these challenges, HIV researchers have increasingly turned to internet-based and digital approaches to provide necessary HIV education to this population. Significant funding from the National Institutes of Health and other organizations has enabled the development and evaluation of many digital tools to support HIV prevention and treatment [7]. These studies consistently demonstrate the effectiveness and positive reception of these interventions [8], expanding the range of programs available for CBOs to choose from and implement, regardless of their specific context. However, there remains a substantial gap between the number of digital HIV interventions created and those widely adopted in public health practice.

One possible reason for this gap could be the lack of structural support from the US Centers for Disease Control and Prevention (CDC), the largest funder of domestic HIV prevention programs [6, 9,10,11] While the CDC's collection of evidence-based HIV interventions does include various digital HIV interventions [12], none of them specifically receive support from CDC grant mechanisms for dissemination or implementation. Furthermore, interventions developed for research may not be "implementation ready," as they are developed and assessed in controlled settings that do not account for the multilevel determinants that may impede implementation [13], such as CBO staff and leadership training, real-world usability, or available resources [14]. CBOs and their staff may also face challenges in terms of technological capacity, infrastructure, resources, and strategies to effectively integrate digital tools into their practice [15]. These challenges can create significant barriers and impede successful dissemination.

Existing research on digital HIV interventions has primarily overlooked the perspective of implementers, specifically the CBOs that play a crucial role in HIV education, testing, and linkage services [14]. Although some studies demonstrate CBO enthusiasm for integrating digital tools to enhance their in-person services and expand their reach, [15, 16] there remains a lack of understanding about CBOs' readiness to adopt and implement these technologies as well as about pragmatic methods to assess CBO readiness. It is also imperative for researchers to gain a better understanding of how CBOs decide on whether to adopt a digital health intervention. To address these knowledge gaps, we measured and described the readiness of HIV CBOs adopting an evidence-based digital program. Specifically, we developed and validated a novel scale, called RADHI (Readiness to Adopt Digital Health Interventions), to quantitatively assess CBOs' readiness, and we examined differences in contextual factors among CBOs at various levels of readiness using the Consolidated Framework for Implementation Research (CFIR) [17].

Methods

Study design

The data for this study came from a larger implementation trial of Keep It Up! (KIU!), an evidence-based digital HIV prevention program designed for MSM aged 18–29 [18]. To understand how to best implement KIU!, our research team conducted an effectiveness–implementation hybrid type 3 trial comparing two overarching implementation approaches for KIU!: a direct-to-consumer (DTC) model in which KIU! was delivered from a central implementation site (i.e., the research team) and a CBO model in which 22 CBOs from around the US delivered KIU! as part of their standard suite of HIV testing and prevention services [19]. The current study focused on the readiness to adopt KIU! within the CBO arm. We used a convergent mixed-methods design: surveys and interviews were collected independently and later integrated and analyzed together [20]. All research was approved by the Northwestern University Institutional Review Board.

Keep it up! Intervention

KIU! is a mobile-compatible, multimedia web application that presents HIV risk reduction messaging embedded within young MSM’s lived experiences. In a multi-city effectiveness trial, KIU! demonstrated effectiveness at changing biomedical HIV risk outcomes [18] and received a “best evidence” rating from the CDC [12]. Thus, KIU! is a prime digital intervention to study implementation. KIU! is structured around a central video drama series that contains 7 context-based modules (e.g., seeking partners online, going out to bars, spending time with friends) presented across 3 core episodes and 2 “booster” episodes that reinforce prior content. Progression through the intervention is self-directed such that deliverers (i.e., CBOs) only need to recruit, remind, and retain their users.

Procedures and measures

Participant selection

As described elsewhere [19], we selected the 22 CBOs from 44 counties with high estimated numbers of young men who have sex with men (YMSM) through a competitive Request for Proposals process that mimicked how organizations typically apply for public health funding. Each agency designated staff—typically HIV testers and/or prevention counselors—to participate in KIU! implementation. CBOs ranged in size from 8 full-time staff to more than 100 full-time staff. On average, they dedicated 2 staff to implementation of KIU! (range = 1–6). CBOs had a range of 3 to 36 years of providing services to YMSM, with 63% having 20 or more years of experience. Prior to starting KIU! delivery, CBO staff completed a self-guided online training about the program content, recruitment strategies for KIU!, account setup, and monitoring participant progress.

For survey and interview sampling, the CBO Project Director responsible for the contract chose which staff members participated in the study. As a result, CBO staff held a range of roles including managers, executive leadership, outreach coordinators, community health educators, advocates, specialists, PrEP navigators, and more (see Table 1). CBO staff were in their then-current roles for an average of 2.7 years (range=2 to 14 years). Interview sampling focused on “information saturation;” that is, sampling for those with the greatest amount of information regarding the topic at hand [21, 22]. Thus, staff in charge of KIU! were recruited to participate in interviews, as they directly experienced the on-the-ground barriers and facilitators to its implementation. Overall, 85 staff members from the 22 CBOs (median = 2.5 staff; range = 1–10 staff) participated in this study.

Table 1 CBO staff roles and tenure

Surveys and other quantitative data

After completing the KIU! training module, CBO staff were asked to complete a survey in REDCap about their attitudes toward implementing KIU! and baseline implementation metrics (e.g., general organizational readiness). We found no existing quantitative assessments of readiness specific to client-facing behavioral intervention technologies [23] in the literature, so we drafted an initial set of nine items based on key concepts from several hypothesized frameworks [24,25,26], including familiarity with digital/eHealth, perceived benefit of technology, individual and organizational proficiency in using new and digital technologies, (mis)trust in technology’s use of data, willingness to use eHealth, and self-efficacy for implementing a digital intervention. Each item was rated on a 5-point scale from 0 (not at all) to 4 (to a very great extent). Using exploratory factor analysis, we identified a single-factor, 5-item solution for the final Readiness to Adopt Digital Health Interventions (RADHI) scale:

• Our organization is ready to implement an eHealth intervention.

• Our organization has experience with implementing eHealth interventions.

• Our organization plans to use other eHealth interventions in the future if they are available.

• Our organization adopts new technology and computer systems easily.

• eHealth interventions can be as effective as in-person/face-to-face interventions.

Confirmatory factor analysis of this solution demonstrated acceptable fit (RMSEA = .07, CFI = .985, TLI = .971, SRMS = .033). The measure also had good internal consistency (alpha=.825). To assess concurrent and predictive validity of the RADHI, we compared scale scores to reviewers’ scores of the CBOs’ funding proposals, CBO staff’s ratings of general organizational readiness to implement change (using the Organizational Change Recipients’ Belief Scale), CBO staff’s acceptability of KIU!, the number of MSM recruited by CBOs in the first 3 months of implementation, and the number of CBO requests for technical assistance during the first 3 months of implementation. Additional details regarding scale development and validation procedures are presented in Additional file 1.

Interviews

After CBO staff were trained to use KIU! but prior to enrollment of YMSM, three qualitative researchers interviewed staff members from each CBO. To minimize potential biases, research staff from the DTC arm of the larger trial conducted the interviews; research staff supporting implementation in CBOs did not conduct the interviews. Interviews were conducted over Zoom, audio recorded, and transcribed using Otter.ai software. Each interviewer used the same interview guide that included open ended questions based on CFIR [27]. While an updated CFIR 2.0 now exists [17], interview guides were developed using CFIR 1.0, and interviews were conducted prior to the recent release of the updated framework. Interviewers asked CBO staff questions about the relative advantage of KIU! in comparison to their ongoing programming and services (including any other online HIV prevention interventions), the relative priority of KIU!, and KIU!’s compatibility with their existing workflow. Interviews ranged from 53 minutes to over 3 hours (mean=1.2 hours). Most CBOs (18/22) had 2 staff members participate. Three CBOs had only 1 staff member participate, and one CBO did not participate, resulting in a total sample of 37 staff members. In the qualitative analysis section, we describe elements of our analysis of interview data in line with existing standards for reporting qualitative research [28].

Analysis

Quantitative scoring

We calculated mean RADHI scores for each CBO using responses from all participating staff and from just staff who completed both the survey and the interview. Due to our sampling procedure, mean RADHI scores were calculated using a range of 1 to 10 staff scores. For comparisons within our sample, we categorized CBOs into quartiles and compared the top and bottom 25% of RADHI scores to the middle 50%. This approach aims to explain variation in an outcome by identifying characteristics that differ in maximally contrasting groups [29, 30]. These categories were based off the 37 survey respondents who also participated in qualitative interviews. The group cutoffs were above 3.17 for high, between 2.33 and 3.17 for middle, and below 2.33 for low readiness to adopt.

Qualitative coding

Two researchers with PhD-level qualifications, authors az and JPZ, and a research assistant, conducted deductive coding in Dedoose software. They developed a codebook based on CFIR constructs and used these to code the transcripts. Under the guidance and training of author az, all three individuals coded the same set of four transcripts until they achieved a kappa threshold of 0.7 [31]. In addition to the CFIR codes, the coders developed three codes to capture the perceptions of CBO staff members regarding the value, appeal, and limitations of KIU!. "Value" pertained to staff or organizational benefits perceived from the intervention, while "appeal" referred to clients’ interest in and potential benefits from the intervention. “Limitations” was defined as negative attributes of KIU! as well as client challenges that KIU! was not designed to or able to address (e.g., PrEP referrals, housing). Interrater reliability demonstrated substantial agreement among raters (k = 0.7).

Authors az and JPZ analyzed the coded data with a focus on seven key codes: relative priority, relative advantage, compatibility, tension for change, KIU! value, KIU! appeal, and KIU! limitations. As defined by CFIR, relative priority was an inner setting construct that assessed the level of priority CBO staff would assign to KIU! in comparison to other organizational activities. In contrast, relative advantage was an innovation-level construct that highlighted the unique contributions and benefits that KIU! offered over existing innovations, programs, and services [17, 27]. Tension for change referred to the perceived necessity of change among CBO staff, while compatibility gauged the extent to which KIU! aligned with CBOs' missions, values, workflows, infrastructure, and current programming/services. To summarize and analyze the data, author az employed an inductive process, which involved immersing herself within the specific code's data, thoroughly reviewing and taking notes. Patterns within the coded data were then explored to thematically categorize participant responses [32]. Codes for KIU! value, appeal, and limitations were additionally analyzed by counting the occurrences of those specific themes.

Mixed methods

Based on the average RADHI scores from CBO staff who completed both the survey and interview, the CBOs were categorized into three groups: high, medium, and low readiness. The coded qualitative data was analyzed separately for each RADHI group by authors az and JPZ, who thematically assessed the differences between CBOs in each group using a constant comparison process [33]. The qualitative findings were then utilized to provide context and explanation for the quantitative scores [34]. Three codes (value, appeal, and limitation) were analyzed as undifferentiated counts across groups using the KIU! codes. To assess differences across groups, three separate one-way ANOVAs were conducted for value, appeal, and limitation. No difference was observed across groups in the tension for change code; thus, we did not incorporate this code into the mixed methods portion of the analysis.

Results

RADHI

Table 2 shows the mean RADHI scores by CBO among the full sample of CBO staff (N=85) and among staff who participated in an interview (n=37). Mean RADHI scores for both the full and interview samples were 2.7, suggesting that on average, CBOs were between a “moderate” and “great extent” ready to adopt KIU!. After categorizing the 22 CBOs by quartile, six (27.3%) were considered high readiness (above 3.17), eleven (50%) were medium readiness (between 2.33 and 3.17), and five (22.7%) were low readiness (below 2.33).

Table 2 Mean community-based organization (CBO) Readiness to Adopt Digital Health Interventions (RADHI) scores

Implementation determinants by RADHI score

Innovation compatibility

Across the three RADHI groups, the majority of CBO staff (n=34 out of 37) found KIU! to be aligned with their missions and values. They also noted its seamless integration and complementarity with existing processes, practices, and programming. Staff in the high RADHI group expressed stronger confidence in the long-term compatibility and suitability of KIU! for their needs (see Table 3). One member from the high group shared, “I think it does integrate pretty well. It’s just the beginning stages of us integrating it to what we already currently have…but I think it’s working right now.” Conversely, some staff in the low RADHI group expressed concerns about the potential incompatibility of KIU! with their client population or existing processes. For example, one staff member shared:

Unfortunately, in [state], because we’re a melting pot of like all sorts of cultures, things like that, there may be some areas of the KIU! program, with some of the photos and stuff, that may be a little racy for people around here.

Table 3 Readiness to Adopt Digital Health Interventions (RADHI) and innovation compatibility

Relative priority

The majority of CBO staff in the high and middle RADHI groups (n=28 out of 29) ranked KIU! as a top priority or as equal to currently existing top priorities (i.e., already existing programs and services). For example, one staff member in the middle group explained, “The priority is just as high as all of the rest of them. It is just as valuable to our program as our general testing.” In comparison, those in the low group were mixed about KIU!’s relative priority within their CBO (see Table 4). CBO staff in the low group highlighted numerous higher priorities, including ongoing HIV testing and currently existing programs and services. In part, funding factored into some staff’s responses, with one explaining, “In comparison to other funders, honestly this program, because it’s a research project, is fairly low stakes. I mean if it works out, super. If it doesn’t, we wouldn’t lose one of our large government contracts just so that we could be sure that we met our deliverables for KIU!.”

Table 4 Readiness to Adopt Digital Health Interventions (RADHI) and relative priority of KIU!

Relative advantage

CBOs across groups were in agreement about the advantages of KIU! (see Table 5). Although no CBO had any previous experience with digital health interventions, nearly all CBO staff (n=35 out of 37) emphasized the benefits of KIU! as an eHealth intervention compared to their current face-to-face approaches. The remaining three staff members who did not mention this advantage explained that they were unable to list any potential advantages or disadvantages since they had not yet implemented the intervention. While some staff members mentioned their use of the internet to reach clients, particularly through dating and hookup apps like Grindr, none of the CBOs had previously implemented an eHealth intervention. Therefore, many saw KIU! as a fresh and innovative approach. The relative advantages of KIU! included a non-clinical approach to prevention, which they believed would help clients feel respected, valued, and treated as equals. They found the intervention's presentation of various topics to be more engaging, personable, and realistic compared to other interventions. During an interview, the HIV/STI Prevention Coordinator at one CBO expressed their admiration for KIU!'s approach: They appreciated how KIU! confronts real lived experiences (e.g., seeking sexual partners online, using drugs for pleasure) and acknowledges the risks faced by the target population.

Table 5 Readiness to Adopt Digital Health Interventions (RADHI) and relative advantage of KIU!

An additional advantage of KIU! was its broader range of topics compared to presentations given in public schools, where CBOs often face restrictions on the subjects they can address, the language they can use, and the health interventions they can discuss legally (e.g., condom use and PrEP). Other noted advantages included the ease of implementation for CBO staff, the use of simplified language that makes the intervention accessible to the average client, and the comprehensive coverage of various content areas. They acknowledged that they would not be able to cover the same breadth of content as KIU! due to time and staffing limitations. Additionally, the focus on prevention for HIV-negative clients distinguishes KIU! from other funded programs that solely target HIV-positive clients. CBOs currently face restrictions on engaging in unfunded evidence-based interventions (EBIs), but KIU! offers an opportunity to address this gap. Moreover, the intervention's private nature helps mitigate stigma for clients, allowing them to engage in the intervention discreetly.

Implementation determinants across CBOs

KIU! value, appeal, and limitations

The one-way ANOVAs revealed no statistically significant distinction between CBOs in the low, middle, and high RADHI groups regarding the appeal (p = .080), value (p = .162), and limitations (p = 0.143) of KIU!.

CBO staff identified values of KIU! for their organizations (see Table 6). These included KIU! acting as an extension of their organizations' prevention, testing, education, and PrEP referrals/retention services. KIU! allowed them to expand their services to clients who preferred not to have direct patient-facing interactions. This expansion was made possible because KIU! is an eHealth intervention that enabled CBOs to offer additional services that would otherwise be limited by staffing and time constraints. The staff also emphasized that implementing and delivering KIU! was logistically straightforward: 19 staff members felt that KIU!'s value lay in its compatibility with their existing workflows, programming, services, missions, and values, as well as its seamless integration into their organizational structure. Additionally, 13 staff members explained that KIU! would enable them to provide services to HIV-negative clients for whom they previously had none, while four staff members highlighted the value of KIU!'s proven efficacy in reducing STI rates. During the interview, an HIV/STI Prevention Coordinator highlighted the potential impact of KIU!. They explained that they read published literature identifying a significant decrease of around 20% to 30% in HIV and STI transmission rates from the use of KIU!. While it is important to acknowledge that different environments and communities may yield different outcomes, the coordinator expressed a genuine curiosity and openness to exploring the potential benefits of KIU!. They emphasized the value of giving it a try, recognizing that this situation could hold promising possibilities.

Table 6 KIU! value, appeal, and limitations

The CBO staff identified three key reasons why they found KIU! to be appealing: convenience, relatability, and engaging material. Twenty-seven staff members highly valued the convenience of KIU!, as it allows clients to access the intervention anytime, anywhere, and on any device. Moreover, the staff emphasized that this convenience provides a certain level of privacy, which can help reduce the stigma associated with seeking HIV-focused support from a CBO. Additionally, 23 staff members appreciated how KIU! effectively engages clients through captivating graphics, entertaining skits, interactive games, and humorous videos. This level of engagement is particularly appealing to young individuals, enabling KIU! to effectively reach many YMSM. For example, one interviewee, an HIV Peer Advocate at a prominent CBO, stated, "For me, it [the appeal] was the videos. It felt like I was watching a Netflix special, a complete story in itself."

Twenty-six staff members also found KIU! to be appealing due to its relatability. They highlighted KIU! 's realistic, sex-positive, and non-paternalistic approach when addressing sexual health risks and substance abuse. Staff members also emphasized the casual and simplified language used throughout KIU!. The relatability of KIU! resonated with staff members who watched videos incorporated into the KIU! modules and felt their YMSM participants would see themselves reflected in it as gay, bisexual, and queer men. One interviewee, an HIV/STI Prevention Coordinator at a medium-sized CBO, specifically mentioned the realism of KIU!. They appreciated how the intervention portrayed situations such as being at a club, dancing, and keeping track of the number of drinks consumed when going to the bathroom, as well as the portrayal of drug-related scenarios. The staff member recognized the challenges faced by LGBTQ individuals, especially those of color, in their day-to-day lives. In addition to these reasons, there were other factors that contributed to the appeal of KIU!. These included the intervention's normalization of HIV, discussions about sexual health between partners, the comprehensive information provided, and the representation of diverse races and body sizes in skits, games, and videos. Staff members often mentioned multiple reasons why KIU! was appealing, indicating that these factors were interconnected in their perception.

CBO staff also identified limitations of KIU!, which primarily revolved around the convenience and duration of the intervention. CBO staff perceived that the flexibility of being able to watch KIU! anytime and anywhere may result in clients postponing the intervention, not fully engaging with the content, or delaying HIV/STI testing. Additionally, CBO staff believed the extended duration and self-pace may enable procrastination and be time-consuming. A Clinical Services Coordinator at a medium-sized CBO expressed their concerns about the program's length, questioning the client’s ability to successfully complete it and retain the substantial amount of information.

Seven CBO staff members emphasized that due to the intervention's online nature, there was limited human interaction. This hindered CBOs' ability to establish rapport with clients and prevented clients from building a sense of community with each other. Moreover, disparities in technology access and internet availability meant that CBOs might need to provide devices to clients who are homeless or living in poverty. These clients may not have access to cell phones, laptops, or stable Wi-Fi. Additionally, the staff highlighted challenges faced by clients with learning disabilities, Deaf or Hard-of-Hearing clients, visually impaired clients, and non-English speakers in accessing the intervention. KIU! was designed for a specific underserved and high-risk client group (i.e., YMSM); however, KIU!’s tailored design may limit its applicability to other similar client groups at risk for HIV such as older and transmasculine MSM. Staff members also identified study limitations, such as county restrictions and the need for anonymity. However, these limitations are research artifacts and could be addressed if the intervention were implemented outside the context of a research study. One staff member suggested updating certain aspects of the technology, such as the virtual game, to have a more modern aesthetic. Lastly, one staff member noticed that despite the diverse skits and videos included in the intervention, none explicitly address issues of race and ethnicity.

Tension for change

Thirty CBO staff expressed a strong desire for change within their organizations. Primarily, they emphasized the need for KIU!, an eHealth intervention, as an alternative to their standard in-person prevention services. Some participants highlighted the challenge of providing comprehensive and interactive education to a large number of clients due to limited staffing and time constraints. They also emphasized the importance of extending their services through eHealth interventions to reach potential clients who are reluctant to engage in face-to-face services. Furthermore, half of the survey participants emphasized the importance of addressing and diminishing stigma within their communities and society as a whole. For example, a representative from a CBO elaborated on this issue:

Stigma... is one of the most significant global obstacles in changing people's perceptions regarding HIV, PrEP, PEP, and STIs [sexually transmitted infections]. Reducing stigma is an area where Keep It Up! shows great promise in being highly effective.

CBO staff also emphasized the need for internal change, specifically highlighting the importance of effectively and rapidly reducing HIV and STI rates through EBIs. These CBOs viewed KIU! as a potential eHealth solution based on previous research that demonstrated a 40% decrease in STIs among KIU! participants (Mustanski et al., 2018). During an interview, the Deputy Director of Prevention at one CBO shared their perspective on the changes associated with KIU!: Although their leadership and staff typically resist change, they expressed genuine enthusiasm for the new initiative. They highlighted that when introducing something new, there is often a high likelihood of resistance or frustration. However, once they presented KIU! as a valuable resource for the individuals they serve – those for whom they struggle to make a significant impact or have limited solutions – the response became overwhelmingly positive.

Eight participants from CBOs expressed that their organization's leadership and staff were enthusiastic about a new opportunity but did not explicitly convey a strong need for change or a desire to maintain the status quo. These participants viewed KIU! as an extension of their services, an additional opportunity for their organization and clients, and a "safety net." Two participants showed little inclination for change. One interviewee, a Services Coordinator at a small CBO, explained, "At this moment, KIU! would be one of the activities that we incorporate, because we're compelled to utilize social media." Although they did not express excitement for altering programming or services at the CBO, the organization recognized the necessity to do so in order to attract younger clients from Generation Z and serve clients during the COVID-19 pandemic. Lastly, the other CBO with minimal resistance to change stated that their organization is typically enthusiastic about trying new programs and services, but the COVID-19 pandemic led to staff burnout, as employees had to fulfill their existing job responsibilities in addition to new contact tracing duties.

Discussion

In our study of 22 CBOs that were selected and trained to implement KIU!, a web-based HIV prevention program, CBOs on average reported a moderately high readiness to adopt digital health interventions based on the RADHI. However, we also found variability in their readiness that could be explained by differences in perceptions about the intervention and how it fit with their organization. CBOs in the high RADHI group demonstrated stronger indications that KIU! would be useful for them (KIU! value, relative advantage) and that their clients would like it (KIU! appeal, compatibility). CBOs in the low group expressed concern that KIU! would not be useful or liked by their clients. Although these latter CBOs embraced the idea of KIU!, interview responses reflected their uncertainty about fit and the utility of KIU! for their client population due to the length of the intervention, disparities in access to an online intervention, loss of face-to-face contact, and limited ability of the intervention to adapt to non-English speaking clients and/or disabled clients.

CBOs did not exhibit clear differences in tension for change by RADHI group. This may be because across the board, CBOs shared experiences regarding the decrease in time and resources for non-PrEP HIV prevention activities and that KIU! filled the same niche for all of them. Since the beginning of the HIV epidemic, CBOs have served the populations most affected with a can-do, do-everything-you-can attitude [35]. They have also needed to be nimble to keep up with rapidly changing HIV technologies and policies [36, 37] (e.g., advances in HIV testing, advent of PrEP and U=U, changes in HIV counseling recommendations, shift from behavioral to biomedical prevention). These factors may mean that CBOs are rarely stagnant in their service provision and frequently ready for something new.

In addition to describing the readiness to adopt a digital preventive intervention among a sample of CBOs, our study adds a novel measure to the implementation science and digital health literature. The newly developed RADHI scale demonstrated concurrent and predictive validity with general organizational readiness and the number of cisgender MSM recruited in the first 3 months (see Additional file 1). It also showed convergent validity with qualitative data collected from CBO staff members. Future research may use the RADHI to measure a concept that could potentially moderate implementation effectiveness for digital interventions. The RADHI may also be useful in practice settings to determine which organizations may be more optimally suited to implement digital health versus traditional in-person interventions.

Limitations

We note some limitations of this study. First, the implementation trial of KIU! required CBOs to respond to a request for proposals to implement an eHealth intervention. Therefore, the CBOs in this sample are not representative of all CBOs, as they self-selected to participate in the study and, furthermore, their proposals were selected to support delivery of KIU! and participate in the study. As a result, these findings may not be generalizable to CBOs that are not already interested in adopting digital interventions or that do not already have a minimum level of technological capacity. Future research can explore how the RADHI behaves in a more general sample of CBOs. Second, CBO leadership selected staff members to participate in interviews. This selection may introduce CBO perspective bias since staff were not randomly selected. Third, most data collection occurred during COVID-related closings and reopenings, potentially affecting some respondents’ readiness to adopt a new intervention; CBO staff highlighted in the interviews an increased sense of burnout due to managing a rapidly changing environment for providing HIV prevention services amidst the global pandemic.

Conclusions and next steps

This study was one of the first to create a measure of digital intervention adoption readiness and evaluate the readiness of CBOs to adopt an HIV eHealth intervention. While on average CBOs in this analysis rated themselves between a moderate and great extent ready to adopt KIU!, we found that intervention utility and fit with different client populations gave some CBO staff reservation about adoption. Identifying the unique barriers to adopting digital health interventions, such as technological proficiency and mistrust of data protections, is a critical first step to fulfilling eHealth’s promise of widescale reach with fidelity at low cost [38] and ultimately reducing HIV prevalence among MSM. To build on this work and further demonstrate the utility of the RADHI, researchers should repeat this analysis on larger and more diverse samples of CBOs at more exploratory stages of considering a digital health intervention and evaluate its ability to predict adoption. Researchers should also use these findings to identify and evaluate implementation strategies that can improve readiness to adopt KIU! by addressing concerns listed by CBO staff in the lower RADHI group. To end the HIV epidemic will require a collective focus on implementing our entire armamentarium of EBIs for HIV prevention and treatment, including digital interventions [39]. Our study and the novel RADHI scale help to further implementation research in HIV prevention and provide an adoption readiness tool for any organization that may offer eHealth interventions for a variety of health topics including mental health, care transitions, and disease management.