Skip to main content
Download PDF
Download ePub

Data envelopment analysis to evaluate the efficiency of tobacco treatment programs in the NCI Moonshot Cancer Center Cessation Initiative

Implementation Science Communications volume 4, Article number: 50 (2023) Cite this article

Abstract

Background

The Cancer Center Cessation Initiative (C3I) is a National Cancer Institute (NCI) Cancer Moonshot Program that supports NCI-designated cancer centers developing tobacco treatment programs for oncology patients who smoke. C3I-funded centers implement evidence-based programs that offer various smoking cessation treatment components (e.g., counseling, Quitline referrals, access to medications). While evaluation of implementation outcomes in C3I is guided by evaluation of reach and effectiveness (via RE-AIM), little is known about technical efficiency—i.e., how inputs (e.g., program costs, staff time) influence implementation outcomes (e.g., reach, effectiveness). This study demonstrates the application of data envelopment analysis (DEA) as an implementation science tool to evaluate technical efficiency of C3I programs and advance prioritization of implementation resources.

Methods

DEA is a linear programming technique widely used in economics and engineering for assessing relative performance of production units. Using data from 16 C3I-funded centers reported in 2020, we applied input-oriented DEA to model technical efficiency (i.e., proportion of observed outcomes to benchmarked outcomes for given input levels). The primary models used the constant returns-to-scale specification and featured cost-per-participant, total full-time equivalent (FTE) effort, and tobacco treatment specialist effort as model inputs and reach and effectiveness (quit rates) as outcomes.

Results

In the DEA model featuring cost-per-participant (input) and reach/effectiveness (outcomes), average constant returns-to-scale technical efficiency was 25.66 (SD = 24.56). When stratified by program characteristics, technical efficiency was higher among programs in cohort 1 (M = 29.15, SD = 28.65, n = 11) vs. cohort 2 (M = 17.99, SD = 10.16, n = 5), with point-of-care (M = 33.90, SD = 28.63, n = 9) vs. no point-of-care services (M = 15.59, SD = 14.31, n = 7), larger (M = 33.63, SD = 30.38, n = 8) vs. smaller center size (M = 17.70, SD = 15.00, n = 8), and higher (M = 29.65, SD = 30.99, n = 8) vs. lower smoking prevalence (M = 21.67, SD = 17.21, n = 8).

Conclusion

Most C3I programs assessed were technically inefficient relative to the most efficient center benchmark and may be improved by optimizing the use of inputs (e.g., cost-per-participant) relative to program outcomes (e.g., reach, effectiveness). This study demonstrates the appropriateness and feasibility of using DEA to evaluate the relative performance of evidence-based programs.

Peer Review reports

Background

Tobacco use is a preventable risk factor that can exacerbate adverse health outcomes for individuals with cancer, increasing risk for all-cause mortality, cancer-specific mortality, cancer recurrence, and worsening response to cancer treatment [1, 2]. Failed first-line cancer therapy associated with continued tobacco use adds a major burden to the US healthcare system, amounting to approximately $3.4 billion per year, or $10,678 annual cost per patient [3]. Thus, timely tobacco treatment in patients with cancer is imperative to mitigate the harmful effects of tobacco use on individuals’ health and rising health care costs.

The importance of smoking cessation for individuals with cancer is widely recognized by cancer organizations and the Surgeon General [1, 2]. However, patients with cancer face barriers to accessing tobacco treatment as part of their cancer care [1, 4, 5]. Screening patients for smoking using electronic health records and referring them to smoking cessation programs can effectively facilitate their engagement in these programs [6]. However, tobacco cessation interventions are not routinely offered as a part of standard care in oncology. As such, patients with cancer who smoke are not consistently connected with necessary tobacco treatment resources [7].

In response to this gap, the Cancer Center Cessation Initiative (C3I) was launched in 2017 as part of the National Cancer Institute (NCI) Cancer Moonshot program with the aim of integrating tobacco cessation treatments into routine cancer care. The goal of this multilevel implementation initiative has been to foster and sustain evidence-based tobacco cessation programs for patients with cancer and to encourage system-level change by identifying and referring patients who use tobacco to cessation treatments [8, 9]. C3I includes 52 NCI-designated cancer centers, which have implemented evidence-based tobacco treatments into their standard of care [10]. C3I provided funding to cancer centers across three cohorts: 42 centers received funding for two years over two cycles (i.e., cohort 1: 2017–2019; cohort 2: 2018–2020), and 10 centers received funding for 1 year (i.e., cohort 3: 2020–2021). Eleven cohort 1 and 2 centers were funded for an additional year as enhancement sites. Each funded cancer center offers evidence-based smoking cessation treatments (e.g., in-person/telephone/video-based/point-of-care counseling, cessation mediation, patient education material, Quitline referral), with variability in the provision of type and number of specific treatments across centers.

C3I is guided by the reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework, which facilitates multilevel assessment of the programs’ health-related impact (i.e., individual, organizational, community) [11,12,13]. Identifying factors that contribute to the fidelity and effectiveness of C3I programs is imperative for their sustainability [14, 15]. Cancer centers participating in C3I reported key implementation outcomes biannually for continued program evaluation and to inform future resource allocation needs.

Program reach and effectiveness are two key outcomes of interest that are reported by C3I-funded cancer centers as standardized outcomes and have been used to evaluate the success of the tobacco treatment programs. Reach is defined as the proportion of patients who received at least one component of evidence-based, tobacco treatment. Effectiveness is defined as patient-reported 7-day point prevalence abstinence at 6-month follow-up. C3I centers’ reach and effectiveness rates varied as a function of center characteristics, including cancer center size, implementation strategies used, and types of treatments offered. For instance, Hohl et al. [16] found that cancer center size (i.e., number of patients served) was positively associated with effectiveness and negatively associated with reach. C3I centers that implemented tobacco treatment programs center-wide had similar effectiveness and higher reach than centers that engaged in partial implementation [16]. Further, centers that offered tobacco treatment through interactive voice response (i.e., automated calls) had the highest median reach and lowest effectiveness, whereas centers that implemented in-person face-to-face counseling had the highest effectiveness but the lowest reach. Additionally, offering six or fewer (vs. seven) types of tobacco treatments within a program was associated with higher reach and effectiveness.

In addition to reach and effectiveness, cost is an important factor that can affect program sustainability [17]. An economic evaluation of 15 C3I sites found that monthly operating costs per site ranged from $5129 to $20,751 (median = $11,045), with most costs going towards personnel [18]. Cost per participant ranged from $70 to $2093 (median = $454) and cost per quit was less than $3500 across centers. Overall, C3I centers achieved satisfactory quit rates at reasonable costs [14], and the programs were expected to become more cost-effective as they continued to scale up. Identifying factors associated with high reach and effectiveness while minimizing costs is imperative for maximizing program efficiency and sustainability. However, operating costs of C3I programs in relation to their reach and effectiveness has not been examined. Therefore, the objective of this study was to compare program outcomes relative to expended resources and to identify best practices across cancer centers, including which program components and implementation strategies were associated with optimal efficiency.

To compare C3I program outcomes relative to expended resources, we applied a mathematical optimization method called data envelopment analysis (DEA). DEA is widely used in economics and engineering for measuring the relative performance of production units [19, 20]. One advantage of DEA is that it does not require any parametric assumptions regarding data distributions, and data are not restricted to any functional form [20]. DEA assesses the ratio of outputs to inputs when evaluating performance and produces a “best practice frontier” representing the best performing units. Performance of the remaining units is calculated as a relative score compared to the unit(s) located on the best practice frontier. Thus, DEA can be particularly useful in assessing which C3I programs are operating most efficiently. We assessed efficiency of C3I programs to demonstrate the application of this method in implementation science. In this study, each C3I program was compared against the best practice frontier which consists of the C3I program(s) with the most efficient performance (i.e., proportion of observed outcomes to benchmarked outcomes for given input levels). Multiple inputs and outcomes can be considered simultaneously while using DEA, which allows for evaluation of several factors related to performance, such as cost, reach, and effectiveness. Although these implementation outcomes are commonly assessed in implementation research, efficiency is rarely evaluated as it relates to the implementation of evidence-based practices. Thus, this study also seeks to demonstrate the utility of using DEA as a program evaluation tool within the field of implementation science. By assessing the reach and effectiveness relative to resources expended, decision makers can be better informed regarding which factors contribute to the most effective and sustainable program components in order to maximize impact of their programs.

Methods

Overview

This is a descriptive study using DEA to examine reach and effectiveness of the C3I program relative to resources expended. DEA applications in health services give insights into which organizations are more efficient than others using program outcomes as outputs and resources expended as inputs [21]. This study examined program efficiency in cohorts 1 and 2 of C3I participating cancer centers that had implemented tobacco treatment into oncology care. C3I sites implemented variations in tobacco treatment components and implementation strategies, requiring investments in different types and proportions of resources, including expenditures on staffing, medications, and electronic health record systems. The heterogeneity of components and implementation strategies, as well as the presence of multiple outcomes of interest, pose challenges for evaluating the relative performance of these programs. Cancer centers with existing tobacco treatment programs have historically focused on different outcomes as their primary objectives (e.g., by emphasizing reach vs. effectiveness) [22]. The diversity in the way centers invest in resources and prioritize outcomes is reflective of the differences in implicit valuation that cancer centers assign to various program components and outcomes. DEA allows for multiple inputs and outcomes to be considered simultaneously without any parametric assumptions on data distributions. DEA is appropriate for comparing C3I programs due to its characterization of the implicit valuation placed on program components, which varies by site, and its ability to simultaneously model efficiency for multiple outcomes, such as reach and effectiveness. Sixteen of 42 NCI-designated cancer centers from cohorts 1 and 2 that had complete data for input and outcome measures of interest (i.e., tobacco treatment specialist (TTS), full time equivalent (FTE) of overall staff, cost-per-patient, reach, and effectiveness) were included in this study. We stratified the analysis by cancer center characteristics because identifying factors associated with cancer centers that maintain high reach and effectiveness given budget constraints is important to foster sustainability of C3I programs. Stratifying the analysis by cancer center characteristics clarifies which components are associated with greater efficiency and informs how efficiency can be improved at underperforming centers.

Data collection procedures

Tobacco treatment program evaluation data were reported to the C3I Coordinating Center, based at the University of Wisconsin-Madison Carbone Cancer Center. The Coordinating Center assisted grantees with integrating evidence-based tobacco treatment services into cancer care [10] and created standardized metrics to evaluate the tobacco treatment programs. All C3I cancer centers received an online questionnaire via Qualtrics (Provo, UT) every 6 months from the Coordinating Center, which assessed center characteristics (e.g., size, TTS FTE, treatments offered) and outcomes (e.g., reach, effectiveness). Specific methods regarding C3I measurement are detailed elsewhere [10, 23]. C3I centers were given the option to report implementation costs and other resources expended (e.g., number of tobacco treatment specialists, program staff FTE) using an additional biannual Qualtrics survey [14]. Cost data used in this study were reported during the January to June 2020 reporting cycle. To be included in this study, centers must have reported reach, effectiveness, cost, total program FTE, and tobacco treatment specialist FTE. This study was classified by the University of Wisconsin-Madison and University of Florida Institutional Review Boards as program evaluation and therefore exempt.

Site characteristics

Data reported included: size of the cancer center, smoking prevalence for patients at the center, presence of a point-of-care tobacco cessation intervention (i.e., in-person, telehealth) [24], and whether sites were part of the first or second C3I cohort. Cancer center size was assessed by number of unique adult cancer patients served by the center during the 6-month reporting period. Smoking prevalence was assessed by the proportion of cancer patients within the center who were documented in the electronic health record system as currently smoking cigarettes. Centers that offered point-of-care counseling for tobacco cessation (i.e., in-person or telehealth) included programs with a brief intervention delivered by a health care provider during routine oncology appointments to discuss evidence-based tobacco treatment options and offer tobacco cessation-related advice [10, 16, 23]. C3I programs were also categorized by cohort, whereby cohort 1 sites received funding from 2017 to 2019, and cohort 2 sites received funding between 2018 and 2020. The reporting period was the same for both cohorts, and we did not control for lead time among cohort 1 sites. Therefore, cohort 1 sites had more implementation experience than cohort 2 sites, on average, for each assessment.

Tobacco treatment program components

C3I sites reported the types of evidence-based treatments offered in their programs. These treatments included the following: in-person individual or group counseling, telephone-based counseling, point-of-care counseling, interactive voice response system track and triage services (i.e., TelASK), Quitline referral, SmokefreeTXT text messaging service, online resources (e.g., smokefree.gov), and smoking cessation medications.

Input measures

Measures indicating presence of a TTS on site, total FTE, and cost were collected through the biannual cost surveys and used as input measures. C3I cancer centers reported FTE of tobacco treatment specialists employed in the program. Sites also reported FTE associated with all tobacco treatment program staff by personnel type, which was summed across personnel types to derive the total FTE measure. “Other personnel FTE” was calculated by subtracting TTS FTE from total FTE. We calculated cost-per-patient by dividing total monthly operating costs of each participating C3I center by the number of patients participating in a tobacco treatment program within the 6-month reporting period. Details of how total monthly operating costs were calculated can be found elsewhere [14].

Outcome measures

Reach was assessed as the proportion of unique patients seen during the 6-month reporting period who used tobacco and received at least one type of evidence-based tobacco treatment (e.g., tobacco cessation medications, Quitline referral, point-of-care counseling [24]), relative to the total number of patients who smoked at each C3I center. Effectiveness was assessed as the proportion of patients currently using tobacco who engaged in tobacco treatment and reported abstinence from tobacco use for a minimum of seven days at six months follow-up. The number of total patients using tobacco was assessed using two items on the C3I 6-month survey: (1) for how many patients who received tobacco treatment in the July–December 2020 reporting period do you have follow-up effectiveness data? (2) For how many patients who received tobacco treatment in the July-December 2020 reporting period are 6-month effectiveness data missing? While a small number of programs implemented biochemical verification as part of their assessments, this was not standard across all programs. Therefore, we only used self-reported abstinence for our assessments of program effectiveness. A complete response approach was used wherein each center determined their own denominator for effectiveness based on their center’s reporting practices.

Assessment of program performance

We applied DEA to assess the relative performance of C3I centers [20]. We used the DEA optimization method, which has been applied to estimate the technical or cost efficiency of healthcare programs [25,26,27]. DEA determines how efficiently a program converts inputs into outcomes compared with other programs and produces a best practice frontier comprising the most efficient programs.

Efficiency scores

We used DEA to estimate efficiency scores for each program as the ratio of the weighted sum of outcomes to the weighted sum of inputs, and graphically plotted the efficiency scores according to cost and reach/effectiveness. We applied the input-oriented DEA approach with constant returns-to-scale [19, 28]. Under the input orientation, the efficiency measure is based on the proportion to which the observed input levels can be produced for given outcome levels. Compared to efficiency scores, rankings are robust as they are not based on unstable solutions of linear programming models. We compared efficiency scores across subgroups of sites, by funding cycle, core components, and implementation strategies used. The most efficient program(s) are used as the benchmark for comparison with other programs. The efficiency of any program is relative to the efficiency of other programs in the sample, and the relative efficiency of any given program can change when compared to a different set of programs.

Slacks

Slacks represent excess input utilization or shortages in outcomes within DEA [29]. We assessed the mean amount of slack among inefficient C3I programs, relative to the most efficient program(s), for each input and outcome (i.e., distance between inefficient programs and the most efficient program). We reported the percentage of change needed to eliminate inefficiencies and to achieve performance consistent with the most efficient program(s) on the best practice frontier.

Analyses

Three DEA models assessed the relative efficiency of the sixteen C3I programs with complete data.

  • Model 1 input: cost per participant; outcomes: reach, effectiveness.

  • Model 2 inputs: TTS, other personnel; outcomes: reach.

  • Model 3 inputs: TTS, other personnel; outcomes: effectiveness.

Analyses were also stratified by C3I center characteristics. We conducted all analyses using the PIM-DEA V.3.2 software.

Results

Table 1 includes descriptive statistics summarizing site characteristics, inputs, and outcomes of the included C3I centers. Cancer centers served an average of 24,652 (standard deviation, SD = 21,596, median = 22,075, range = 507–89,485) patients during the 6-month reporting period, and median smoking prevalence was 9.3% (range = 2.2–47.1%) across centers; 44% of cancer centers had implemented point-of-care interventions and 69% were part of cohort 1 (vs. cohort 2). Mean number of tobacco treatment specialist FTE was 0.66 (SD = 0.60), and mean total FTE was 1.39 (SD = 0.74). Average cost-per-patient was $572 (SD = $518, median = $474), and average cost-per-quit was $2981 (SD = $2015, median = 2765). Overall, programs reached 24.4% of patients who smoked (SD = 14.1, range = 2.5–47.8%) and had a 20.4% effectiveness (SD = 10.6, range = 2.6–35.3%), on average.

Table 1 Site characteristics, inputs and outputs of the C3I programs (n = 16)
Full size table

In the first DEA model (Fig. 1), we assessed reach and effectiveness (as outcomes) relative to cost-per-participant (as the input). Only one program was located on the best practice frontier (i.e., benchmark program(s) with the most efficient performance), while the majority of programs clustered near the origin (i.e., away from the best practice frontier). This distribution suggests generally low effectiveness and reach relative to costs, in comparison to the one program on the best practice frontier. Six programs had relatively higher effectiveness (vs. reach), and 10 programs had relatively higher reach (vs. effectiveness).

Fig. 1
figure 1

Efficiency frontier for C3I programs: reach (%) and effectiveness (%) relative to cost-per-participant (n = 16). P1 is on the best practice frontier (i.e., the most efficient unit)

Full size image

Next, we used DEA to assess effectiveness and reach relative to total costs, stratified by program characteristics (Supplementary Materials). C3I centers in cohort 1 were less clustered around the origin than centers in cohort 2, and the best practice frontier was farther from the origin for cohort 1 (vs. cohort 2) (Supplementary Fig. 1). Only one C3I center was located on the best practice frontier for each cohort. Similarly, C3I centers that did not implement point-of-care interventions were clustered closer to the origin, suggesting that point-of-care was associated with higher reach and effectiveness relative to costs (Supplementary Fig. 2). Only one C3I center was located on the best practice frontier for each model assessing point-of-care. Larger cancer centers (i.e., above median size) were more efficient and had generally greater reach than smaller centers (Supplementary Fig. 3). One C3I center was located on the best practice frontier in each model assessing larger and smaller centers. Centers with below-median smoking prevalence were clustered more closely to the origin, suggesting lower reach and effectiveness relative to total cost than centers with above-median smoking prevalence (Supplementary Fig. 4). One C3I center was located on the best practice frontier in the model for greater than median smoking prevalence, and two C3I centers were located on the best practice frontier in the model for lower than median smoking prevalence.

Additionally, we used DEA to assess the reach and effectiveness of C3I centers relative to the personnel involved in tobacco treatment administration (i.e., TTS, other personnel). For the first set of models, the inputs were TTS and other personnel, and the output was reach. C3I centers clustered around the origin, suggesting that most had generally low other personnel and TTS relative to reach. Only one C3I center was located on the best practice frontier. Nine C3I centers had higher FTE for other personnel (vs. TTS) relative to reach, whereas seven C3I centers had higher TTS FTE (vs. other personnel) relative to reach (Fig. 2). Next, we used DEA to assess the effectiveness of C3I centers relative to personnel involved in tobacco treatment administration (i.e., other personnel, TTS). C3I centers were clustered near the origin, suggesting lower use of TTS and other personnel relative to effectiveness. Two C3I centers were located on the best practice frontier. Seven C3I centers had higher use of other personnel relative to effectiveness, whereas six centers had higher use of TTS relative to effectiveness (Fig. 3).

Fig. 2
figure 2

Efficiency frontier for C3I programs: TTS and other personnel relative to reach (n = 16). P9 is on the best practice frontier (i.e., the most efficient unit)

Full size image
Fig. 3
figure 3

Efficiency frontier for C3I programs: TTS and other personnel relative to effectiveness (n = 16). P2 and P5 are on the best practice frontier (i.e., the most efficient units)

Full size image

Table 2 shows the average efficiency scores overall, by cohort, by whether the program had a point-of-care intervention, by cancer center size, and by smoking prevalence. Sites in cohort 1, those with point-of-care interventions, those within larger-than-median cancer centers, and those with higher-than-median smoking prevalence had higher efficiency scores, on average.

Table 2 Technical efficiency scores and returns to the scale of tobacco treatment programs in C3I
Full size table

The performance analysis identified the slacks, representing either excess input utilization or shortages of output production. Table 3 shows the average slack in programs deemed inefficient. These results represent the combined scores of slack for all inefficient programs, for each input and output. Table 3 also shows the percentage of change in the number of inputs or outputs needed to eliminate the inefficiencies and achieve target levels. Based on our preliminary sample, cost per participant should be reduced by an average of 74.34%, TTS FTE should be reduced by an average 10.98%, and other personnel FTE by 52.18% to maximize efficiency.

Table 3 Slacks evaluation for inefficient programs (n = 16)
Full size table

Discussion

This study demonstrated utility of DEA for implementation research by assessing reach and effectiveness of tobacco treatment programs within NCI-designated cancer centers relative to their operating costs. We identified factors associated with the most optimal programs that could be leveraged to increase efficiency of tobacco treatment programs across centers that function less optimally. Programs that were in cohort 1 (i.e., typically more advanced in implementation), programs that had implemented point-of-care interventions, and programs in cancer centers that were larger in size tended to be more efficient.

This information is particularly useful for program evaluation because it directly compares how well C3I programs converted their available resources into measurable outcomes (i.e., reach, effectiveness). Variability in efficiency was high across C3I centers, which is unsurprising given that some centers had existing infrastructure for tobacco treatment programs, whereas others implemented these programs for the first time. Existing program infrastructure may have contributed to program efficiency, given that these sites would have already implemented some tobacco treatment program-related protocols into their workflows. There was also variability in how long tobacco treatment programs had been implemented across centers. Thus, DEA is an important tool for program evaluation because it can identify which programs effectively maximize their resources given budget constraints. Identifying how resources can be allocated to foster sustainability of C3I centers has implications for other tobacco cessation programs in oncology settings. For instance, DEA may be used to characterize treatment efficiencies in others areas of oncology practice, such as value-based care, enrollment in clinical trials, and improving palliative care and cancer survivorship.

Although research using DEA to assess tobacco treatment programs for oncology patients is limited, DEA has been applied in other healthcare settings. For example, DEA has been applied in examining the efficiency of primary healthcare centers, including inputs and outputs such as number of patients and staff, costs, procedures, prescriptions, and referrals [30]. Additionally, DEA has been used to assess public health concerns regarding healthcare systems and the optimal organization of primary care service delivery, using inputs such as primary care governance, workforce development, and economic conditions, and outputs such as comprehensiveness, access, coordination and service delivery indicators of access continuity and comprehensiveness of care [30]. Moreover, application of DEA is not limited to assessing efficiency of programs and systems, and it has been used to support decision-making in clinical settings. For example, DEA was used for real-time benchmarking in radiotherapy treatment planning, where it was associated with improvement of most treatment plans [31]. Thus, DEA has a broad range of applications within healthcare, including within the oncology domain.

DEA can be used as a stand-alone analysis, given its unique ability to assess the relative efficiency of production units. DEA has been widely applied in other fields such as economics [19, 20] as well as in clinical settings as described above. DEA can also be used in combination with other methods, (e.g., qualitative interviews, longitudinal surveys), to glean a more holistic perspective regarding how to improve program efficiency. For example, conducting qualitative interviews or surveys with personnel directly involved with the implementation procedures could elucidate specific recommendations regarding how to improve efficiency, beyond which inputs and outputs are affecting efficiency [16]. As such, DEA can be used either independently to assess program efficiency or complimentarily with other analyses.

Overall, we found that many C3I sites had low efficiency relative to the best practice frontier. In the DEA model assessing reach and effectiveness relative to program costs, only one program was located on the best practice frontier, and this program appeared to achieve substantially higher efficiency compared to other programs. We examined additional DEA models stratified by key organizational characteristics. C3I centers that were part of cohort 1 (vs. cohort 2), had deployed point-of-care tobacco cessation interventions (vs. no point-of-care), were larger (vs. smaller) in size, and had higher (vs. lower) smoking prevalence tended to be more efficient (i.e., greater reach and effectiveness relative to cost). Thus, centers may prioritize implementing point-of-care interventions over other types of tobacco treatment interventions to maximize efficiency. Although point-of-care interventions may be expensive, it is noteworthy that their implementation was associated with greater program efficiency (i.e., ratio of reach and effectiveness relative to costs). Similarly, larger NCI-designated cancer centers generally achieved higher reach and effectiveness while mitigating costs, suggesting that C3I programs are more sustainable when implemented in larger (vs. smaller) cancer centers, and that smaller cancer centers or community oncology practices may require more resources to sustainably implement tobacco treatment programs.

Additionally, centers that had higher smoking prevalence were generally higher in efficiency. It is possible that centers with lower smoking prevalence were less efficient because there were fewer eligible patients to enroll in the program. C3I centers with lower smoking prevalence had lower reach relative to effectiveness, which suggests that these programs may be underutilized and consequently operate less efficiently than programs with higher enrollment rates. This information can be particularly useful for establishing tobacco cessation programs for individuals with cancer. Selecting locations that are more likely to maintain low costs relative to reach and effectiveness may increase the likelihood that cancer centers will sustain these programs long-term. However, this practice may come at the cost of marginalizing patients in settings with limited resources, which may require greater costs to implement and sustain tobacco treatment programs.

Moreover, whether C3I centers had existing infrastructure for tobacco cessation treatments prior to the initiative may have impacted relative program efficiency. For instance, centers that had independently focused on promoting tobacco cessation programs before joining C3I may be more efficient than centers that initiated tobacco cessation programs as part of C3I. Implementation readiness has been shown to be associated with higher chances that a cancer center provides tobacco cessation treatments to its patients [32]. Thus, the stage of program implementation is another important factor that may contribute to a center’s efficiency and sustainability.

Further, programs that employed less TTS FTE and other personnel FTE on average achieved greater efficiency. Previous research regarding the effects of a TTS for tobacco cessation are mixed. Recent studies found that C3I centers with lower TTS-to-patient ratios tended to have higher reach and lower effectiveness [16] and that counseling delivered by TTSs was associated with higher smoking cessation rates [33]. Future research should investigate whether reducing TTS and personnel is associated with greater efficiency among C3I centers in general, or whether centers with particular characteristics may benefit from an increase, in TTS and other personnel (e.g., large centers and/or those with particularly high smoking prevalence and less program staff). More research is needed to identify the most efficient TTS staff-to-patient ratio and explore the contexts in which TTS and other personnel are essential for maximizing program efficiency.

It is important to note that it can be challenging to find a balance between maximizing research and delivering an effective intervention, particularly in the oncology setting. Individuals who continue smoking after their cancer diagnosis can be especially difficult to treat even with a high intensity intervention. Therefore, assessing the external validity and cumulative impact of smoking cessation interventions in oncology settings is of utmost importance. We assessed reach and effectiveness as separate outcomes, however, both must be considered to assess population impact. The cumulative impact of an intervention is a function of every step of dissemination and participation (e.g., proportion of staff that take part, patients that accept participation, patients that benefit from the intervention and continue benefitting 6 months later) [34]. Even interventions that have high effectiveness may yield low population impact after accounting for participation and retention issues at every level of dissemination. As such, consistent and transparent reporting about participation and representativeness at all levels of dissemination are vital for evaluating the cumulative impact of interventions. Future research should evaluate the cumulative impact of C3I programs on tobacco cessation outcomes across various contexts.

Limitations

This study is not without limitations. First, our convenience sample of 16 NCI-designated C3I centers may not be representative of many cancer care programs. This sample consisted of cancer centers that received supplemental funding to improve or expand tobacco cessation resources; thus, results may not be generalizable to other cancer centers and should be interpreted with caution. The programs in this sample reported low proportions of individuals who were American Indian or Alaska Native (≤ 1%), Asian, Native, or Pacific Islander (≤ 1%), or Hispanic (3%); therefore, generalization of findings to these populations may be limited. However, smoking prevalence among C3I centers was similar to estimates of national rates of tobacco use among individuals who have had cancer [35,36,37].

It is possible that the results do not accurately represent the experiences with efficiency across all tobacco treatment programs in C3I. Therefore, a larger sample of C3I centers is needed before factors affecting program efficiency can be reliably assessed and interpreted. Despite this limitation, we achieved the primary goal of this study, which was to demonstrate the benefits of using DEA as a tool for assessing program implementation and performance. Specifically, we showed that DEA can be used to inform program efficiency by assessing readily available practice parameters, such as program reach, effectiveness, and cost.

Another limitation is the reporting of outcomes and program features was voluntary; therefore, data collected from C3I centers may be partially incomplete. Missing data, whether deliberate or coincidental, can skew findings [38]; therefore, more automated data reporting of tobacco treatment program measures would improve future data quality. There was also a lack of uniformity regarding which program personnel reported data to C3I, which may have resulted in between-reporter inconsistencies. Each center also determined their own denominator for effectiveness based on their center’s reporting practices, which exacerbates variability in reporting across centers.

Finally, some of the data for this study were collected during the COVID-19 pandemic, when many non-emergent appointments were postponed or canceled. Other pandemic-related changes, such as limited staff due to illness, staff changes, and program changes with implementation of telehealth, may have impacted program costs and efficiency. Reach and effectiveness may have been affected during this time due to pandemic-related restrictions and barriers. On the other hand, reach may have increased with the pandemic related transformation to telehealth treatment models. Data were reported at the level of the C3I center; therefore, we did not have access to individual-level data. Consequently, we were unable to investigate more granular factors that may be associated with program efficiency, such as which specific tobacco treatments patients were receiving, at what frequency, and whether efficiency was moderated by patient characteristics (e.g., age, cancer site, treatment) [35]. Given these restrictions, we were also unable to undertake a thorough analysis of potential confounding factors that may account for observed relationships, such as differences in the types of patients who received different treatments, and effects of other program features that were not measured or reported.

Conclusion

DEA is a useful tool for assessing the relative efficiency of organizations that implement evidence-based programs in a way that is not possible with other analytic methods. In the case of C3I, identifying factors associated with high reach and effectiveness, while maintaining low operating costs is important for the sustainability of tobacco treatment programs. Decision makers can use findings from DEA to improve efficiency of existing tobacco treatment programs within oncology settings and evaluate how cancer centers could most effectively support implementation of tobacco treatment programs. This study demonstrated that DEA provides valuable information that can foster more sustainable implementation of tobacco treatment programs in oncology settings.

Availability of data and materials

The datasets generated and analyzed during the study are not publicly available due to the sensitive nature of some data. The cost dataset is available from the corresponding author upon reasonable request. All other data are available from the C3I coordinating center on reasonable request.

Abbreviations

C3I:

Cancer Center Cessation Initiative

NCI:

National Cancer Institute

TTS:

Tobacco treatment specialist

FTE:

Full time equivalent

References

  1. U.S. Department of Health and Human Services. Smoking Cessation: A Report of the Surgeon General. Atlanta; 2020.

  2. Warren GW, Alberg AJ, Kraft AS, Cummings KM. The 2014 surgeon general’s report: “The health consequences of smoking-50 years of progress”: a paradigm shift in cancer care. Cancer. 2014;120(13):1914–6.

    Article  PubMed  Google Scholar 

  3. Warren GW, Cartmell KB, Garrett-Mayer E, Salloum RG, Cummings KM. Attributable failure of first-line cancer treatment and incremental costs associated with smoking by patients with cancer. JAMA Netw Open. 2019;2(4):e191703.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Gritz ER, Toll BA, Warren GW. Tobacco use in the oncology setting: advancing clinical practice and research. Cancer Epidemiol Biomark Prev. 2014;23(1):3–9.

    Article  Google Scholar 

  5. Land SR, Toll BA, Moinpour CM, Mitchell SA, Ostroff JS, Hatsukami DK, et al. Research priorities, measures, and recommendations for assessment of tobacco use in clinical cancer research. Clin Cancer Res. 2016;22(8):1907–13.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Baker TB, Berg KM, Adsit RT, Skora AD, Swedlund MP, Zehner ME, et al. Closed-loop electronic referral from primary care clinics to a state tobacco cessation quitline: effects using real-world implementation training. Am J Prev Med. 2021;60(3):S113–22.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Fiore MC, D’Angelo H, Baker T. Effective cessation treatment for patients with cancer who smoke—the fourth pillar of cancer care. JAMA Netw Open. 2019;2(9):e1912264.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Croyle RT, Morgan GD, Fiore MC. Addressing a Core Gap in Cancer Care — The NCI Moonshot Program to help oncology patients stop smoking. N Engl J Med. 2019;380(6):512–5.

    Article  PubMed  PubMed Central  Google Scholar 

  9. National Cancer Institute. Cancer Center Cessation Initiative. 2022.

    Google Scholar 

  10. D’Angelo H, Rolland B, Adsit R, Baker TB, Rosenblum M, Pauk D, et al. Tobacco treatment program implementation at NCI cancer centers: progress of the NCI Cancer Moonshot-funded cancer center cessation initiative. Cancer Prev Res. 2019;12(11):735–40.

    Article  Google Scholar 

  11. D'Angelo H, Ramsey AT, Rolland B, Chen LS, Bernstein SL, Fucito LM, et al. Pragmatic application of the RE-AIM framework to evaluate the implementation of tobacco cessation programs within NCI-designated cancer centers. Front Public Health. 2020:221.

  12. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health. 2013;103(6):e38-46.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Salloum RG, LeLaurin JH, Dallery J, Childs K, Huo J, Shenkman EA, et al. Cost evaluation of tobacco control interventions in clinical settings: a systematic review. Prev Med (Baltim). 2021;146:106469.

    Article  Google Scholar 

  15. The Cancer Center Cessation Initiative Sustainability Working Group. Sustainability of tobacco treatment programs in the cancer center cessation initiative. J Natl Comprehens Cancer Netw. 2021;19(Suppl_1):S16-20.

    Article  Google Scholar 

  16. Hohl SD, Matulewicz RS, Salloum RG, Ostroff JS, Baker TB, Schnoll R, et al. Integrating Tobacco Treatment Into Oncology Care: Reach and Effectiveness of Evidence-Based Tobacco Treatment Across National Cancer Institute–Designated Cancer Centers. J Clin Oncol. 2022:JCO-22.

  17. Malone S, Prewitt K, Hackett R, Lin JC, McKay V, Walsh-Bailey C, et al. The Clinical Sustainability Assessment Tool: measuring organizational capacity to promote sustainability in healthcare. Implement Sci Commun. 2021;2(1):77.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Salloum RG, D’Angelo H, Theis RP, Rolland B, Hohl S, Pauk D, et al. Mixed-methods economic evaluation of the implementation of tobacco treatment programs in National Cancer Institute-designated cancer centers. Implement Sci Commun. 2021;2(1):41.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Charnes A, Cooper WW, Rhodes E. Measuring the efficiency of decision making units. Eur J Oper Res. 1978;2(6):429–44.

    Article  Google Scholar 

  20. Cooper WW, Seiford LM, Tone K. Data envelopment analysis. In: Cooper WW, Seiford LM, Zhu J, editors. Handbook on data envelopment analysis. 1st ed. Springer; 2000. p. 1–40.

    Chapter  Google Scholar 

  21. Sickles RC, Zelenyuk V. Measurement of productivity and efficiency. Cambridge University Press; 2019.

  22. Goldstein AO, Ripley-Moffitt CE, Pathman DE, Patsakham KM. Tobacco use treatment at the U.S. National Cancer Institute’s designated cancer centers. Nicot Tobacco Res. 2013;15(1):52–8.

    Article  Google Scholar 

  23. Craig EJ, Ramsey AT, Baker TB, James AS, Luke DA, Malone S, et al. Point of care tobacco treatment sustains during COVID-19, a global pandemic. Cancer Epidemiol. 2022;78:102005.

    Article  PubMed  Google Scholar 

  24. Ramsey AT, Baker TB, Stoneking F, Smock N, Chen J, Pham G, et al. Increased reach and effectiveness with a low-burden point-of-care tobacco treatment program in cancer clinics. J Natl Compr Canc Netw. 2022;20(5):488-495.e4.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Wang L, Grignon M, Perry S, Chen X, Ytsma A, Allin S, et al. The determinants of the technical efficiency of acute inpatient care in Canada. Health Serv Res. 2018;53(6):4829–47.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Varabyova Y, Schreyögg J. International comparisons of the technical efficiency of the hospital sector: panel data analysis of OECD countries using parametric and non-parametric approaches. Health Policy (New York). 2013;112(1–2):70–9.

    Article  Google Scholar 

  27. Hollingsworth B. The measurement of efficiency and productivity of health care delivery. Health Econ. 2008;17(10):1107–28.

    Article  PubMed  Google Scholar 

  28. Banker RD, Thrall RM. Estimation of returns to scale using data envelopment analysis. Eur J Oper Res. 1992;62(1):74–84.

    Article  Google Scholar 

  29. Alatawi AD, Niessen LW, Khan JAM. Efficiency evaluation of public hospitals in Saudi Arabia: an application of data envelopment analysis. BMJ Open. 2020;10(1):e031924.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Zakowska I, Godycki-Cwirko M. Data envelopment analysis applications in primary health care: a systematic review. Fam Pract. 2020;37(2):147–53.

  31. Raith A, Ehrgott M, Fauzi F, Lin KM, Macann A, Rouse P, et al. Integrating Data Envelopment Analysis into radiotherapy treatment planning for head and neck cancer patients. Eur J Oper Res. 2022;296(1):289–303.

    Article  Google Scholar 

  32. Ostroff JS, Copeland A, Borderud SP, Li Y, Shelley DR, Henschke CI. Readiness of lung cancer screening sites to deliver smoking cessation treatment: current practices, organizational priority, and perceived barriers. Nicotine Tob Res. 2016;18(5):1067–75.

    Article  PubMed  Google Scholar 

  33. Lancaster T, Stead LF. Individual behavioural counselling for smoking cessation. Cochrane Database Syst Rev. 2017;3:CD001292.

    PubMed  Google Scholar 

  34. Glasgow RE, Huebschmann AG, Brownson RC. Expanding the CONSORT figure: increasing transparency in reporting on external validity. Am J Prev Med. 2018;55(3):422–30.

    Article  PubMed  Google Scholar 

  35. Salloum RG, Huo J, Lee JH, Lee J, Dallery J, George T, Warren G. Tobacco and E-cigarette use among cancer survivors in the United States. PloS One. 2019;14(12):e0226110.

  36. Gritz ER, Talluri R, Domgue JF, Tami-Maury I, Shete S. Smoking behaviors in survivors of smoking-related and non–smoking-related cancers. JAMA Netw Open. 2020;3(7):e209072.

  37. Westmaas JL, Alcaraz KI, Berg CJ, Stein KD. Prevalence and correlates of smoking and cessation-related behavior among survivors of ten cancers: findings from a nationwide survey nine years after diagnosis. Cancer Epidemiol Biomarkers Prev. 2014;23(9):1783–92.

  38. Roth PL. Missing data: A conceptual review for applied psychologists. Pers Psychol. 1994;47(3):537–60.

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank the tobacco treatment program staff at participating cancer centers for their support in data collection.

Funding

A contract from the 17GZSK0031) supported the implementation of tobacco treatment programs and data reporting at the NCI-designated cancer centers and coordination efforts at the University of Wisconsin—Madison. Ramzi G. Salloum was supported by NCTATS award UL1TR001427. Jamie S. Ostroff was supported by NCI award P30CA008748-52S1. Li-Shiun Chen was supported by NCI awards P30CA091842-19S5, P50CA24443 and the Siteman Investment Program. Cary A. Presant was supported by NCI grants P30CA033572 and P30CA03572-37S5. Neely Dahl and Kara Wiseman were supported by NCI award P30CA044579. Kara Wiseman was supported by the UVA iTHRIV Scholars Program, NCATS UL1T R003015 and KL2TR003016. Katie Lenhoff was supported by NCI award P30CA023108-43. Kimberly Shoenbill was supported by NCI awards P30CA016086-43S1 and P30CA016086-44S5. Hilary A. Tindle was supported by NCI award 3P30CA068485-24S3. Graham Warren was supported by NCI awards P30CA138313-09S4 and A22-0010–01. Judith J. Prochaska was supported by NCI awards P30CA124435-11S1 and P30CA124435-13S2. Timothy Baker was supported by NCI award P01 CA180945. Justin S. White was supported by award P30CA082103-19S2.

Author information

Authors and Affiliations

  1. Department of Health Outcomes and Biomedical Informatics, University of Florida College of Medicine, 2004 Mowry Rd, Gainesville, FL, 32610, USA

    Kathryn Pluta & Ramzi G. Salloum

  2. University of Wisconsin Carbone Cancer Center, 600 Highland Ave, Madison, WI, 53705, USA

    Sarah D. Hohl, Mara Minion, Betsy Rolland & Michael C. Fiore

  3. School of Medicine and Public Health, University of Wisconsin, 750 Highland Ave, Madison, WI, 53705, USA

    Sarah D. Hohl, Timothy B. Baker & Michael C. Fiore

  4. National Cancer Institute, 9609 Medical Center Dr, Rockville, MD, 20850, USA

    Heather D’Angelo

  5. Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY, 10065, USA

    Jamie S. Ostroff

  6. New York University School of Global Public Health, 708 Broadway, New York, NY, 10003, USA

    Donna Shelley

  7. Rush University Medical Center and Rush Cancer Center, 1725 W Harrison St, Suite 1010, Chicago, IL, 60612, USA

    Yasmin Asvat

  8. Washington University Siteman Cancer Center, 4921 Parkview Pl, St. Louis, MO, 63110, USA

    Li-Shiun Chen

  9. Medical University of South Carolina, 171 Ashley Ave, Charleston, SC, 29425, USA

    K. Michael Cummings & Graham W. Warren

  10. University of Virginia Comprehensive Cancer Center, 1240 Lee St, Charlottesville, VA, 22903, USA

    Neely Dahl & Kara P. Wiseman

  11. University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, TX, 75390, USA

    Andrew T. Day

  12. Fox Chase Cancer Center, 333 Cottman Ave, Philadelphia, PA, 19111, USA

    Linda Fleisher

  13. University of North Carolina Lineberger Cancer Center, 450 West Dr, Chapel Hill, NC, 27599, USA

    Adam O. Goldstein & Kimberly Shoenbill

  14. Virginia Commonwealth University Department of Psychiatry, 501 N 2Nd St, Suite 400B, Richmond, VA, 23219, USA

    Rashelle Hayes

  15. Northwestern University Feinberg School of Medicine and Lurie Comprehensive Cancer Center of Northwestern University, 420 E Superior St, Chicago, IL, 60611, USA

    Brian Hitsman

  16. Indiana University Simon Comprehensive Cancer Center, 535 Barnhill Dr, Indianapolis, IN, USA

    Deborah Hudson Buckles

  17. University of Chicago Medicine Comprehensive Cancer Center, 5758 S Maryland Dr, Chicago, IL, 60637, USA

    Andrea C. King

  18. Huntsman Cancer Institute, University of Utah, 1950 Circle of Hope Dr, Salt Lake City, UT, 84112, USA

    Cho Y. Lam

  19. One Medical Center Drive, Dartmouth-Hitchcock Norris Cotton Cancer Center, Lebanon, NH, 03756, USA

    Katie Lenhoff

  20. University of Colorado Comprehensive Cancer Center, 1665 North Aurora Court, Aurora, 200480045, USA

    Arnold H. Levinson

  21. City of Hope Comprehensive Cancer Center and Beckman Research Institute, 1500 E Duarte Rd, Duarte, CA, 91010, USA

    Cary Presant

  22. Stanford Cancer Institute, Stanford University, 265 Campus Dr, Ste G2103, Stanford, CA, 94305, USA

    Judith J. Prochaska

  23. H. Lee Moffitt Cancer Center, 3011 Holly Dr, Tampa, FL, 33612, USA

    Vani Simmons

  24. Georgetown University Lombardi Comprehensive Cancer Center, 3800 Reservoir Rd NW, Washington, DC, 20007, USA

    Kathryn Taylor

  25. Vanderbilt University Medical Center Vanderbilt-Ingram Cancer Center, 2220 Pierce Ave, Nashville, TN, 37232, USA

    Hilary Tindle

  26. University of California Davis Comprehensive Cancer Center, 2279 45Th St, Sacramento, CA, 95817, USA

    Elisa Tong

  27. Philip R. Lee Institute for Health Policy Studies, University of California San Francisco, 490 Illinois St, Floor 7, San Francisco, CA, 94158, USA

    Justin S. White

  28. University of Wisconsin Institute for Clinical and Translational Research, 750 Highland Ave, Madison, WI, 53705, USA

    Betsy Rolland

Authors
  1. Kathryn Pluta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sarah D. Hohl
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Heather D’Angelo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jamie S. Ostroff
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Donna Shelley
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yasmin Asvat
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Li-Shiun Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. K. Michael Cummings
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Neely Dahl
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Andrew T. Day
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Linda Fleisher
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Adam O. Goldstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Rashelle Hayes
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Brian Hitsman
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Deborah Hudson Buckles
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Andrea C. King
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Cho Y. Lam
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Katie Lenhoff
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Arnold H. Levinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Mara Minion
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. Cary Presant
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Judith J. Prochaska
    View author publications

    You can also search for this author in PubMed Google Scholar

  23. Kimberly Shoenbill
    View author publications

    You can also search for this author in PubMed Google Scholar

  24. Vani Simmons
    View author publications

    You can also search for this author in PubMed Google Scholar

  25. Kathryn Taylor
    View author publications

    You can also search for this author in PubMed Google Scholar

  26. Hilary Tindle
    View author publications

    You can also search for this author in PubMed Google Scholar

  27. Elisa Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

  28. Justin S. White
    View author publications

    You can also search for this author in PubMed Google Scholar

  29. Kara P. Wiseman
    View author publications

    You can also search for this author in PubMed Google Scholar

  30. Graham W. Warren
    View author publications

    You can also search for this author in PubMed Google Scholar

  31. Timothy B. Baker
    View author publications

    You can also search for this author in PubMed Google Scholar

  32. Betsy Rolland
    View author publications

    You can also search for this author in PubMed Google Scholar

  33. Michael C. Fiore
    View author publications

    You can also search for this author in PubMed Google Scholar

  34. Ramzi G. Salloum
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

RGS and GWW conceptualized the study. SH, MM, HD, and BR contributed to the data collection and coordination. RGS analyzed the data. KP and RGS wrote the first draft. YA, L-SC, KMC, ATD, AOG, BH, DHB, ACK, CYL, KL, AHL, JJP, KS, KT, HT, ET, and JSW contributed to the acquisition of data and manuscript revision. JSO, DS, ND, LF, RH, CP, VS, KPW, TBK, and MCF contributed to successive drafts. All authors approved the final manuscript.

Corresponding author

Correspondence to Ramzi G. Salloum.

Ethics declarations

Ethics approval and consent to participate

Data reported in this manuscript were collected for the purpose of program evaluation and quality improvement. The need for informed consent of participants was waived because programs participating in the evaluation provided deidentified, aggregated data. The evaluation was categorized as program evaluation and deemed exempt by the Institutional Review Board of the University of Wisconsin-Madison.

Consent for publication

We consent to publication.

Competing interests

Dr. Baker occupies the Glaxo Wellcome Chair in the Department of Medicine at the University of Wisconsin School of Medicine and Public Health. Dr. Prochaska has provided consultation to pharmaceutical and technology companies that make medications and other treatments for quitting smoking and also has served as an expert witness in lawsuits against tobacco companies. The other authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1:

Supplementary Figure 1. Efficiency frontier for C3I programs in Cohort 1 (left) and Cohort 2 (right): reach and effectiveness relative to cost-per-participant. Supplementary Figure 2. Efficiency frontier for C3I programs with (left) and without (right) a point-of-care intervention: reach and effectiveness relative to cost-per-participant. Supplementary Figure 3. Efficiency frontier for C3I programs at centers larger (left) and smaller (right) than median size: reach and effectiveness relative to cost-per-participant. Supplementary Figure 4. Efficiency frontier for C3I programs at centers with higher (left) and lower (right) than median smoking prevalence: reach and effectiveness relative to cost-per-participant.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pluta, K., Hohl, S.D., D’Angelo, H. et al. Data envelopment analysis to evaluate the efficiency of tobacco treatment programs in the NCI Moonshot Cancer Center Cessation Initiative. Implement Sci Commun 4, 50 (2023). https://doi.org/10.1186/s43058-023-00433-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s43058-023-00433-3

Keywords

Implementation Science Communications

ISSN: 2662-2211