Skip to main content

Framework for identification and measurement of spillover effects in policy implementation: intended non-intended targeted non-targeted spillovers (INTENTS)

Abstract

Background

There is increasing awareness among researchers and policymakers of the potential for healthcare interventions to have consequences beyond those initially intended. These unintended consequences or “spillover effects” result from the complex features of healthcare organisation and delivery and can either increase or decrease overall effectiveness. Their potential influence has important consequences for the design and evaluation of implementation strategies and for decision-making. However, consideration of spillovers remains partial and unsystematic. We develop a comprehensive framework for the identification and measurement of spillover effects resulting from changes to the way in which healthcare services are organised and delivered.

Methods

We conducted a scoping review to map the existing literature on spillover effects in health and healthcare interventions and used the findings of this review to develop a comprehensive framework to identify and measure spillover effects.

Results

The scoping review identified a wide range of different spillover effects, either experienced by agents not intentionally targeted by an intervention or representing unintended effects for targeted agents. Our scoping review revealed that spillover effects tend to be discussed in papers only when they are found to be statistically significant or might account for unexpected findings, rather than as a pre-specified feature of evaluation studies. This hinders the ability to assess all potential implications of a given policy or intervention. We propose a taxonomy of spillover effects, classified based on the outcome and the unit experiencing the effect: within-unit, between-unit, and diagonal spillover effects. We then present the INTENTS framework: Intended Non-intended TargEted Non-Targeted Spillovers. The INTENTS framework considers the units and outcomes which may be affected by an intervention and the mechanisms by which spillover effects are generated.

Conclusions

The INTENTS framework provides a structured guide for researchers and policymakers when considering the potential effects that implementation strategies may generate, and the steps to take when designing and evaluating such interventions. Application of the INTENTS framework will enable spillover effects to be addressed appropriately in future evaluations and decision-making, ensuring that the full range of costs and benefits of interventions are correctly identified.

Peer Review reports

Introduction

Among researchers and policymakers, there is increasing awareness of the potential for the implementation of health interventions and policies to have consequences beyond those initially intended. For example, the Medical Research Council guidance on complex interventions emphasises that these unintended consequences should be considered where possible but provides little guidance on how best to do this systematically [1]. This lack of direction is also evident in the implementation science literature. For example, the recent ImpRes Framework encourages research teams to be “mindful” and to explore the potential unintended consequences of implementation efforts but does not say how [2]. There is also recognition that successful de-implementation at one level may also lead to potentially negative unintended consequences at another level, but how best to mitigate or prevent occurrence is not yet well understood [3].

A growing body of literature now uses the terms “spillover effects” or “spillovers” for such wider impacts [4]. This term is particularly prevalent in the areas of psychology, economics, and programme evaluation, where efforts are often made to quantify spillover effects resulting from interventions. This term is neutral in its connotation, focusing equally on desirable and undesirable consequences not intended at the outset. An obvious spillover occurs because the implementation of interventions and policies requires redeployment of existing resources or use of new resources that could otherwise have been used elsewhere [5]. Other spillovers may arise because of learning or because inputs are involved in multiple activities. The growing literature on spillover effects could provide useful lessons on how to identify and capture these broader effects, informing the development of practical recommendations missing from current guidance.

The identification and measurement of spillovers remain partial and unsystematic. Spillover effects have been examined in relation to pay-for-performance schemes [6, 7] and target waiting times [8, 9] in England. More recently, in relation to the 2010 Affordable Care Act in the USA, studies reported on spillover effects resulting from the expansion of insurance coverage [10, 11] and alternative provider payment models [12, 13].

We develop a framework for the identification and measurement of spillover effects resulting from policy implementation. This framework provides a structured approach to guide consideration of spillover effects and to enable optimal intervention design, maximising the desired outcomes whilst minimising the potential for negative unintended consequences. It is intended for use both when designing interventions and when planning evaluations of their impacts.

Methods

To inform the development of the framework, we conducted a scoping review to map the existing literature on spillover effects in health and healthcare interventions generally.

Scoping review of the literature

We sought to identify and synthesise evidence from studies on both healthcare and broader interventions targeting health and health behaviours (public health, non-pharmaceutical or non-healthcare interventions, health education, etc.). In devising and conducting the search, screening, and synthesis procedures, we adopted the original and updated methodological frameworks for scoping reviews by Arksey and O’Malley [14] and Levac et al. [15]. Scoping reviews offer an appropriate methodology for mapping the field in terms of the existing evidence [15]. We chose a scoping review rather than a systematic literature review because we aimed to gain a conceptual understanding of spillovers [16]. We reported the study according to the new guidelines for scoping reviews [17]. The checklist is available in Additional file 1.

We searched the online databases EMBASE and MEDLINE from January 1991 to September 2020. We focused on these two databases with the goal to strike a reasonable balance between breadth of coverage of the health and healthcare literature and the number of papers to scope for a non-systematic review. We judged this to be sufficiently wide to capture the evolution of the use of the term “spillover” over time and across different journals and branches of the literature. The start date of January 1991 was chosen in order to have a consistent version of the Journal of Economic Literature (JEL) classification system codes available for filtering purposes, as these were updated and expanded after 1990. We excluded papers addressing biological, physiological, and zoonotic spillovers, as the term spillovers is used to refer to a very different and unrelated concept in these literatures. Only full articles in English were included. The full search strategy is available in Additional file 2. After removing duplicates, we conducted the first screening of titles and abstracts for relevance to our inclusion criteria. The second stage involved full-text screening of the remaining articles. At both stages, two reviewers screened the articles independently and resolved any inconsistencies bilaterally.

We focused on articles discussing—primarily or secondarily—at least one spillover effect associated with outcomes of a specific health or healthcare intervention. Therefore, we excluded studies related to spillover effects in other domains (e.g. biology, trade). In the same spirit, we also excluded all editorials, letter to editors, dissertations and review articles, and favouring empirical studies which offered more detail into the analytical approach towards spillovers. We defined further detailed inclusion criteria for our scoping review iteratively. Our initial exploration revealed that a number of papers were using the term spillovers in passing, but did not actually examine the spillover effects in any detail. For example, the possible existence of spillover effects was mentioned only as a potential explanation for unexpected effects measured in the analysis, or the potential for spillover effects was discussed broadly without referring to specific cases. Therefore, we excluded all such articles which mentioned the term “spillover” incidentally or without actually discussing it. Furthermore, we excluded association studies and focused only on articles analysing interventions of some kind. Since our interest lay precisely in how spillover effects have been analysed, we selected intervention studies irrespective of their analytical design (including trials, quasi-experimental, and observational studies).

The selected articles were fully reviewed and details extracted regarding the following set of salient characteristics of the spillover effects: type of intervention generating spillover effects, units involved, whether the spillover effect was measured, whether and how the mechanisms driving the spillover effect were discussed, and the evaluation design used.

Framework development and validation

Based on the evidence synthesised by the scoping review and building on a limited base of literature on the categorisation of spillover effects [18,19,20,21] from different subjects or disciplinary areas, we developed a framework for the identification and measurement of spillover effects. The framework was developed, refined, and validated collectively by all co-authors. During the first joint session, we discussed the validity and relevance of all steps in a draft framework proposed by the lead author and assessed concordance in the classifications of spillover effects based on samples of articles randomly assigned to each co-author. Building on the feedback from the first round, we refined the framework and repeated the exercise, framing the discussion around the categorisation of a new random sample of papers assigned to each co-author participating in the exercise.

Results

Scoping review summary

Figure 1 shows the flow diagram describing all stages of article search and selection. The initial search produced 2441 results. After removing duplicates and conference abstracts, we screened the titles and abstracts for 1247 unique articles. Overall, 1125 articles were excluded, resulting in the inclusion of 122 articles (see Additional file 3 for the full list of selected papers).

Fig. 1
figure 1

Literature search and selection stages

The number of papers matching our inclusion criteria is considerably higher for papers published in the last decade. This finding is consistent with a growth in the volume of published literature addressing spillover effects, although publication volumes in general have also increased during this period.

The selected studies discuss a wide range of interventions in health and healthcare, of various types (including training, regulation, financial incentives, direct provision of healthcare) and affecting subjects at various levels which may not be the same as those targeted by the main intervention (including patients, providers, healthcare organisations). Table 1 groups the interventions featured in the selected literature according to the Cochrane Effective Practice and Organisation of Care (EPOC) taxonomy [22, 23]. Besides synthesising an otherwise wide range of interventions, the grouping is useful to detect potential recurrences between spillover effects generated by different types of interventions.

Table 1 Type of interventions featured in the articles included in the scoping review, grouped according to the EPOC taxonomy

Most selected articles (98 of 122) report a statistically significant spillover effect. Consistent with a generalised bias towards the publication of studies reporting significant effects, this can be interpreted as a broad signal that spillover effects are discussed in papers only when they are found to be significant rather than as a pre-specified feature of evaluation studies. This does not favour a comprehensive evaluation of all implications of policy interventions. If only the studies featuring a statistically significant spillover effect are reported in the literature, policymakers and researchers may make decisions based on a partial overview of the evidence. Null results can still be informative, for example, providing reassurance that interventions do not appear to generate negative spillover effects. Potential impacts may also be missed if studies are underpowered to detect significant effects.

A second important observation emerging from our review is that the spillover effects reported in the identified literature are either experienced by agents or units not originally targeted by the intervention, or concern impacts on outcomes for the targeted agents/units not originally envisaged. This suggests an important distinction about the nature of spillover effects based on the units involved and the extent to which the spillover effect is similar to or different from the intended goal of the intervention. This distinction is summarised in the simple matrix in Table 2.

Table 2 A taxonomy of spillover effects in relation to the main intended intervention effect

On the one hand, interventions can also generate non-intended effects on the units targeted by the intervention, i.e. impacts on outcomes other than those intended by design, which we term within-unit spillovers. For example, a scheme incentivising providers to improve diagnosis of dementia was found to incidentally reduce the time providers allocated to activities related to patients admitted for other health conditions, but also to improve providers’ efforts to raise quality in other care areas [24]. Likewise, some studies found that the Hospital Readmission Reduction Program yielded positive results in terms of readmissions beyond the procedures specifically penalised, therefore improving outcomes for non-targeted patients [25, 26]. Additionally, direct consumer advertising of medicines had the unintended consequence of triggering learning effects and increased willingness to interact with physicians [27]. On this point, it is worth specifying that we define spillovers from the perspective of the unit targeted by the intervention, rather than the unit experiencing the end outcome. For example, if a healthcare organisation is targeted, a change in provider behaviour can lead to a within-unit spillover onto patient outcomes.

On the other hand, the intended effect of a given intervention can affect units that were not initially targeted [21], which we term between-units spillovers in Table 2. Examples of interventions featuring such spillover effects include a cluster-randomised trial promoting the use of insecticide-treated bednets where the protective effects of bednets extended beyond the households using them [28], asset or conditional cash transfer programmes where non-treated households benefitted from the transfers either through friendship/kinship networks or through other market-mediated channels [29,30,31], a body mass reduction programme for overweight individuals [32] and a colorectal cancer screening programme spilling over to non-treated individuals through social networks [33].

Furthermore, non-targeted units can also experience effects of interventions that are different from the intended goals of the programme originating them, which we call diagonal spillovers. At first sight, diagonal spillovers may appear similar to between-units spillovers in that the intervention effect extends to non-targeted units. However, we distinguish them from between-units spillovers based on the outcome affected: diagonal spillovers refer to instances where an intervention on a targeted unit generates a non-intended effect on a non-treated unit. Examples of diagonal spillovers include changes in Medicare provider fees generating non-intended effects on patients covered by different health insurance schemes [34], changes in prescription drug lists reimbursed by Medicare part D affecting younger population through intensified advertising of newly included pharmaceutical products [35], or the introduction of mental health parity laws triggering increases in private insurance premiums and higher enrolment in public health insurance [36]. In the literature reviewed, diagonal spillover effects seemed more pronounced for mediated interventions, i.e. interventions that were implemented at one level of the health system with the intent of producing effects on units at a different level in the system. Examples of mediated interventions are a change in Medicare reimbursement fees affecting quality and quantity of healthcare provision for commercially insured patients, or the extension of the list of prescription drugs covered by Medicare part D causing an increase in pharmaceutical advertisement and consumption among the non-elderly patients (not covered by Medicare). Overall, classifying both the units that could be affected and the domains upon which they may be affected is a crucial first step in mapping out the potential spillover effects that may occur as a result of an intervention.

In light of the above and considering that any given intervention can be associated with multiple types of spillover effects, in Table 3, we map the literature reviewed according to the general taxonomy in Table 2. With the aim of providing a mapping between main and spillover effects observed in the literature, we also report the frequency of different intervention groups within each spillover class.

Table 3 Classes of spillovers and interventions examined in the reviewed literature

Overall, our scoping review suggests that some intervention types are more likely to lead to the reporting of certain types of spillovers in the published literature. For example, interventions related to financial arrangements were more likely to be reported to generate within-unit or diagonal spillovers. In contrast, studies examining policies addressing implementation strategies most often report between-units spillovers.

Spillover mechanisms identified in the reviewed articles

Understanding the mechanisms which generate spillover effects is key if these are to be harnessed to encourage positive spillovers and mitigate negative ones. We identified an interesting recurrence in terms of the spillover mechanisms mentioned in the studies (Table 4). Among the studies addressing spillover effects occurring between units, about half suggested multiplying effects related to the spread of information through social networks. Other recurring mechanisms were spatial dependencies driven by closeness or common social and environmental factors, as well as effects driven by shared resources in healthcare activities involving providers treating multiple groups of patients.

Table 4 Recurrent mechanisms reported to explain the different classes of spillovers in the reviewed literature

In contrast, the articles pointed to various different mechanisms to explain spillovers which we classified as within-unit or diagonal, with none standing out as the predominant explanation. Examples include unanticipated behavioural responses to treatment by both patients and providers, complementarities or substitution in healthcare production (e.g. bundled care), effort diversion, gaming, and complementarity or substitution effects. In Table 4, we also summarise the direction of the spillover effects, highlighting that in the reviewed literature most studies found favourable (or “positive” spillover effects).

Examining the mechanisms hypothesised to cause observed spillover effects is necessary in order to shed light on the nature of spillover effects and hence help in gauging their relevance for the evaluation of the underlying intervention. For example, in studies discussing health insurance coverage expansion, income spillover effects onto non-healthcare consumption patterns may or may not be policy-relevant, depending on the goals of the policy itself (e.g. financial health protection for the poor). On the other hand, income spillover effects associated with conditional cash transfer for health onto consumption of other goods or services may be less interesting for health policy evaluation purposes.

Understanding the mechanisms that have been found to underpin spillover effects in the past is also helpful both for designing interventions and subsequent evaluations going forward. Knowledge and evidence on mechanisms can inform the design of interventions to maximise positive spillover effects whilst minimising negative ones, and guide the groups and outcomes that should be examined in the subsequent evaluation in order to capture an interventions’ full effects using an unaffected comparator group.

This classification of spillover types, along with mechanisms identified as driving spillover effects to date, should serve as a useful basis for researchers wishing to map out the potential spillover effects that may occur as a result of an intervention.

Developing a framework to classify spillover effects: the INTENTS framework

Building on the elements above, we designed a step-wise guide for the identification and measurement of spillovers resulting from health and healthcare interventions, which we named the Intended Non-intended TargEted Non-Targeted Spillovers (INTENTS) framework. This framework is designed to be a useful starting point for researchers, practitioners, and policymakers when selecting a sensible evaluation design for an intervention, prospectively. Potentially, a similar step-wise process may also be informative at the stage of intervention design, to explicitly map potential spillover effects and calibrate an intervention to maximise its benefits, alongside the development of theories of change or theory-based evaluation frameworks. Likewise, the INTENTS framework may be useful to update evaluation plans mid-way through the implementation of an intervention based on early empirical evidence or conceptual understanding. In the next paragraphs, we describe the INTENTS framework step by step, using examples to clarify the meaning of the general activities proposed at each step. In Table 5, we summarise the INTENTS framework, whilst the following sections provide additional details on its different elements.

Table 5 The INTENTS framework for classifying spillover effects

Some preliminary definitions

The use of the INTENTS framework presumes that the intervention in question is related to health or healthcare. The goal is to formulate hypotheses in relation to potential wider effects which may result from the intervention. This can help to determine prospectively whether the intervention is likely to generate spillover effects which should be fostered or mitigated, as well as to decide whether to consider these potential spillovers in subsequent evaluations of the intervention, both in terms of overall effectiveness and evaluation design.

Before describing the framework steps, we define a few key concepts used in the remainder of the paper, based on the taxonomy represented in Table 2.

Intended and non-intended effects

Throughout the description of the framework, we define “intended effects” as the outcomes which are expected to change by whoever designs an intervention, as a result of the implementation of the intervention itself. By extension, we define “non-intended effects” as all other outcomes that the intervention does not explicitly aim to change, but could nonetheless be affected. The type of outcomes classified as non-intended effects depends on the unit targeted by the intervention. For example, in the case of an intervention targeting a primary care practice in relation to a specific group of patients (e.g. patients with chronic conditions), one could classify as non-intended effects both the main performance criterion addressed by the intervention measured on a different (non-targeted) group of patients and different performance criteria measured on the same patients with chronic conditions.

Targeted and non-targeted units

Any intervention typically defines a “targeted” unit, i.e. a unit that the intervention explicitly aims to affect and which should experience a change in outcomes as a result. In healthcare, the treated units are typically individual or groups of patients, healthcare workers, healthcare organisations, and geographic areas. “Non-targeted” units then refer to units that the intervention is not explicitly aiming to affect and that should therefore not experience changes in outcomes associated with the implementation of the intervention.

Overview of framework steps

In this section, we discuss the components of the INTENTS framework, expanding on the questions to ask at each step and their rationale. The section is intended to be read alongside Table 7, which includes an illustrative application of the INTENTS framework to the case of the Quality and Outcomes Framework, a pay-for-performance scheme introduced to improve quality across general practices in the UK [37].

Step #1: what are the expected outcomes of the intervention?

The first crucial step entails understanding the aims of the intervention and the outcomes targeted for impact. This can be achieved through triangulation of different methods, for example:

  • Intervention mapping with key informants [38,39,40,41]

  • Review of related literature [42, 43]

  • Review of relevant policy documents and high-level gap maps [44, 45]

  • Theories of change and negative programme theory [46, 47]

Once a list of outcomes or a logic model resulting from a thorough effort in mapping potential intervention effects is available, one can start to iteratively apply steps 2 to 4 of the INTENTS framework.

Step #2: at what level can spillover effects take place?

Interventions can impact multiple units or actors beyond those explicitly targeted. Some interventions are implemented through units at a given level to affect units at a different level, upwards or downwards in the health system hierarchy (see Table 6 for some examples). Other interventions are implemented to directly affect the targeted units, without intermediary units or without affecting units at different levels. Hence, the characteristics of the intervention demarcate the type of spillover effect that can emerge.

Table 6 Examples of health system hierarchical levels to be considered in the framework

We first consider an intervention addressing the targeted unit directly, without intermediary units at different health system levels. This intervention design ensures that no unit at other levels in the hierarchy is involved in the implementation of the intervention and hence changes behaviour as a result of the intervention. Assuming that the targeted unit also represents the endpoint of the healthcare process addressed by the policy (i.e. there are no cascading relationships), this type of intervention is more likely to generate spillover effects only within the targeted units, or among other units at the same health system level as the targeted units, which have some kind of interaction with the latter. By the characteristics of the intervention, none of these potential spillover effects would be mediated by intermediary units, which is why we define these spillover effects as non-mediated.

On the other hand, interventions targeting a unit at a different hierarchical level (either upwards or downwards) than the level at which the intervention is implemented can additionally generate what we define as mediated spillovers, akin to the mediated pathway of the intervention itself. The units experiencing a mediated spillover effect are typically linked to the targeted units by a specific agency, market or other known transmission mechanisms implicit to the intervention. To properly unpick these details, alongside the step suggested by detailed behaviour specification frameworks (e.g. the AACTT framework [50]), the questions to ask at this stage are as follows:

  1. a)

    Who is targeted by the intervention?

  2. b)

    Who is expected to change behaviour as a result of the intervention?

  3. c)

    Whose behaviour or outcomes may change as a result of the intervention?

In the answers to the questions above, if only one health system level is involved, the intervention can typically only generate non-mediated spillover effects. Instead, if two or more health systems levels are involved, mediated and non-mediated spillover effects may result from implementation. The distinction can be useful to discriminate mechanisms and to better understand the intervention itself and the possible trajectories in terms of spillover effects.

Step #3: which spillover effects could the intervention generate?

In conceiving any healthcare intervention, designers should do their best in considering all possible spillover effects that may be generated. This can be achieved through a comprehensive mapping of all processes and pathways associated to the changes brought by the intervention, as suggested by step #1. Building on the preliminary definitions above and in Tables 2 and 3, we can classify spillover effects generated by healthcare interventions into three main classes:

  • Within-unit spillover effects (non-intended effect on a targeted unit)

  • Between-units spillover effects (intended effect on a non-targeted unit)

  • Diagonal spillover effects (non-intended effect on a non-targeted unit)

The choice is not limited to one—an intervention may generate multiple types of spillover effects. Table 3 provides a useful overview of the type of interventions found to be associated with different classes of spillover effects in the literature to date. Furthermore, step #2 helps in defining the units that could potentially experience spillover effects and thus demarcates the potential search for spillovers. The goal of this step is to obtain a list of potential spillover effects that may result from the intervention as designed. These spillover effects may be positive, and hence worth encouraging, or negative and thus to be discouraged. In listing the potential spillovers emerging from step #3, it is helpful to formulate hypotheses on the direction of the effects. This can be either informed by step #1 or previous literature.

Step #4: what is the nature of the potential spillover effects identified in step #3?

Given a list of spillover effects defined in step #3, the elements proposed below help in distinguishing potential spillover effects that are relevant from a policy evaluation perspective and reflecting on their nature. The questions to ask are as follows:

  1. a)

    Is the spillover effect really different from the intended outcome? For the sake of a comprehensive but coherent evaluation, the spillover effects considered should be sufficiently dissimilar from the intended outcome of the intervention, so that it constitutes a separate but associated effect. To this end, the spillover effect should not be conceptually equivalent nor sequentially related to the target outcome, for example, via a direct clinical pathway or because a process is targeted to affect an outcome downstream.

  2. b)

    Is the spillover relevant and related to the goals of the intervention? Spillover effects, whilst conceptually distinct from the intended effect, should share the same underlying motives, drivers and scope as the intended effect (see Galizzi and Withmarsh [4] for a useful discussion of this argument). Outcomes that do not share the same motives as the intended effect—either randomly or broadly triggered by the intervention of interest—are likely to reveal associations that are not relevant for an analysis of spillovers in health settings.

  3. c)

    Is the spillover effect consistent with the time frame of the intervention? Timing in the effect also helps to disentangle spillover effects from long-run consequences or to identify inconsistencies with the timing of the intervention. In the spirit of programme evaluation, the existence of spillover effects should be evaluated over a time horizon consistent with the intervention. Effects extending far beyond this time horizon may be medium to long-run effects of the intervention. Therefore, it is worth taking into account that mediated spillover effects experienced upwards in the hierarchy (e.g. change in guidelines in few local authorities triggering changes in regulation, patient-level behavioural interventions changing workflows for primary care practitioners) are more likely to develop over longer time periods. The specific time frame of the intervention is not always easy to establish when it is not explicitly defined, for example in the case of new policies or regulations introduced permanently (e.g. the Quality and Outcomes Framework in the UK or different components of the Affordable Care Act), and therefore, some discretion based on experience and case-specific knowledge is needed.

  4. d)

    Is there a credible mechanism for the spillover? Reflecting on the mechanisms driving potential spillovers may aid understanding of the impact pathways of the intervention, potentially improving the design of the intervention itself, for example, knowingly allowing the intervention to extend beyond the target group when this is desirable. The examples listed in Table 4 may offer a useful overview of some potential spillover mechanisms, which may need to be complemented with adequate theoretical concepts depending on the specific intervention area.

A guided example of application of the INTENTS framework

Working through the INTENTS framework should result in a thorough understanding of the potential spillover effects which may be generated by a given healthcare intervention. This includes the units and outcomes which may be affected, and the mechanisms by which spillover effects could be generated. This structured approach should then form the basis for analysis planning, and potential intervention re-design.

In Table 7, we present an illustrative example of the application of these steps, using the UK Quality and Outcomes Framework (QOF), the longest running national primary care pay-for-performance scheme in the world [7, 51] that was first introduced across the UK in 2004. General practices report their performance on a large number of clinical indicators and receive financial bonus payments based on this performance. The clinical indicators refer to patients with specific diagnosis codes, clearly defining the targeted patients whose care contributes to a practice’s performance. A substantial literature has shown that care processes improved for these patients on the incentivised aspects of care, suggesting that the policy achieved the targeted effect [37, 52, 53]. However, the exploration of its unintended consequences or spillover effects has not been approached comprehensively or systematically.

Table 7 Application of the INTENTS framework to the case of the Quality and Outcomes Framework (QOF) in the UK

Discussion

Summary of findings

There is an increasing awareness among both researchers and policymakers of the potential for the implementation of healthcare interventions to have consequences beyond those initially intended. Despite guidance emphasising the importance of evaluating such wider effects, recommendations for how to do so are lacking. We reviewed the literature on spillover effects in order to better understand the use of the term and the ways in which researchers have attempted to identify and measure such effects in practice. We found that spillover effects are frequently cited as explanations for unexpected research findings, but the current approach to their identification and measurement is sporadic and unsystematic.

Depending on their nature, spillover effects can either increase or decrease the overall effectiveness of interventions. The potential existence of spillover effects therefore has important consequences for the design and evaluation of implementation interventions and for decision-making. This is perhaps best highlighted by recent calls for strategies to mitigate adverse impacts of well-intentioned interventions for COVID-19, such as inequalities in access to healthcare or inequitable uptake or treatment for vulnerable/ marginalised populations [61]. Early and systematic assessment of the potential implications of a given intervention could have provided decision makers with earlier warnings and opportunities for faster corrective action. To this end, the INTENTS framework outlined here provides a structured means for researchers and policymakers to think systematically about the potential spillover effects that interventions in healthcare may cause, and crucially the steps they may take to mitigate when attempting to design and evaluate interventions from the outset.

In terms of evaluation design, the potential for spillover effects means that it may be necessary to widen the scope of evaluations and look beyond the targeted outcomes or recipients in order to assess the full impact. Failure to do so could result in over- or under-estimation of the impact of interventions. Secondly, spillovers can invalidate a key assumption for identifying counterfactuals in comparative evaluations because patients, professionals, services, and organisations not explicitly targeted by the programme could nonetheless be affected and hence fail to represent valid “controls”.

An interesting ancillary result of our scoping review is the extent to which we found the term spillover effects to have been applied to describe effects unsuitable to the definition provided by our INTENTS framework. In presenting our results, we make the point that spillover effects are distinct from the main intended effects of the interventions generating them. We also argue that spillover effects need to be gauged in terms of their relevance to the objectives of the original policy.

To this end, we highlight the underlying mechanisms driving spillovers and the time frame over which they may appear as important discriminants. Using this definition, our scoping review of the literature suggests that 24 of the 122 studies could be said to improperly use the spillover label for the effects they studied. Broadly, these studies involved outcomes that we found to be indistinguishable from the main intended goals of the intervention (11 of 24 studies), income effects that can be seen as implicit to the policy evaluated (9 studies, for example, related to the effect of sponsored health insurance coverage), and outcomes that are too far from the objectives of the intervention to appear relevant for the evaluation of the policy itself (4 studies). The application of the INTENTS framework will enable consistency in the assessment of spillover effects going forward, facilitating the appropriate classification of intervention effects.

Strengths and limitations

Our study is the first to present a systematic overview of the use of the term “spillovers” in the healthcare setting and to develop a framework to aid a more systematic consideration of spillover effects in future research and policymaking. With these elements, our contribution offers new insights and a practical framework, adding to a limited body of existing conceptual literature on spillover effects in health and healthcare [18,19,20,21].

There are, however, limitations to our approach. Our study presents an overview of studies explicitly mentioning the term “spillover(s)”, rather than including all possible synonyms that may be used to convey a similar concept. It is exactly for this reason that we chose a scoping review methodology rather than a systematic review. This more flexible approach allowed us to reveal how names for concepts are used interchangeably across disciplines. For example, we found some studies using the terms spillovers and unintended consequences interchangeably. We do not aspire to resolve this semantic dispute, but by conceptualising spillovers as a broader overarching set of effects, we offer researchers a framework that should allow them to use a common language.

Along the same lines, we deliberately focused our scoping review on the healthcare literature. This provides a limited overview compared to the universe of research on health implementation. Nevertheless, given our goal of mapping the use of a specific term in a multidisciplinary research area, we believe our choice strikes the balance between comprehensiveness and conciseness.

Within implementation science there is currently much interest in understanding the mechanisms of implementation strategies, the processes through which they produce desired effects and how they can be measured [62]. There is however little focus on how implementation strategies may deliver unintended effects and whether or how these can be mitigated. We highlight the importance of understanding the mechanisms underpinning spillover effects to define whether the spillover is important and relevant for the evaluation of the underlying intervention or policy, echoing similar earlier calls by Angelucci and Maro [18]. Our paper offers an overview of the mechanisms suggested in the literature to date. This can be a useful starting point for researchers and policymakers interested in designing interventions which maximise positive (and minimise negative) impacts in the domain of healthcare or designing evaluations ensuring that (i) all relevant impacts are captured and (ii) appropriate control groups are defined. Furthermore, the extent to which spillover effects can be modelled by analysts remains of interest for the continuous improvement of intervention design. To this end, future research may explore the regularities between mechanisms and theoretical underpinnings of different spillover classes which we have highlighted.

In the review, we also reported evidence that may indicate a substantial bias towards reporting statistically significant spillover effects. This bias limits our interpretation of the results of the review: our discussion is based only on papers which mostly report statistically significant spillover effects. We are unable to detect the instances where potential spillover effects have been examined but not found. A corollary of our INTENTS framework is the suggestion to always evaluate and report the results for all known or expected spillover effects, in addition to the main outcomes addressed by the intervention/policy. This seems a necessary condition to obtain a comprehensive evaluation of intervention impact.

Conclusions

The complex and interlinked nature of healthcare organisation and delivery means that interventions often have consequences beyond those initially intended. The potential existence of such spillover effects has important consequences for intervention design, evaluation, and decision-making.

The INTENTS framework provides a means for researchers and policymakers to think systematically about the potential spillover effects that interventions in healthcare may cause and the steps they may take when attempting to design and evaluate interventions generating such spillovers. We hope that the application of our framework will enable spillover effects to be planned for a priori in future policymaking and evaluation design and ensure that the full range of costs and benefits of interventions are appropriately identified.

Availability of data and materials

Not applicable.

Abbreviations

EPOC:

Effective Practice and Organisation of Care

JEL:

Journal of Economic Literature

QOF:

Quality and Outcomes Framework

References

  1. Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. Framework for the development and evaluation of complex interventions: gap analysis, workshop and consultation-informed update. Health Technol Assess. 2021;25(57):1–132.

    PubMed  Google Scholar 

  2. Hull L, Goulding L, Khadjesari Z, Davis R, Healey A, Bakolis I, et al. Designing high-quality implementation research: development, application, feasibility and preliminary evaluation of the implementation science research development (ImpRes) tool and guide. Implement Sci. 2019;14(1):80.

    PubMed  PubMed Central  Google Scholar 

  3. Norton WE, Chambers DA. Unpacking the complexities of de-implementing inappropriate health interventions. Implement Sci. 2020;15(1):2.

    PubMed  PubMed Central  Google Scholar 

  4. Galizzi MM, Whitmarsh L. How to measure behavioral spillovers: a methodological review and checklist. Front Psychol. 2019;10 [cited 2020 Oct 1]. Available from: https://www.frontiersin.org/articles/10.3389/fpsyg.2019.00342/full.

  5. Hoomans T, Severens JL. Economic evaluation of implementation strategies in health care. Implement Sci. 2014;9(1):168.

    PubMed  PubMed Central  Google Scholar 

  6. Roland M, Dudley RA. How financial and reputational incentives can be used to improve medical care. Health Serv Res. 2015;50(S2):2090–115.

    PubMed  PubMed Central  Google Scholar 

  7. Sutton M, Elder R, Guthrie B, Watt G. Record rewards: the effects of targeted quality incentives on the recording of risk factors by primary care providers. Health Econ. 2010;19(1):1–13.

    PubMed  Google Scholar 

  8. Bevan G, Hood C. What’s measured is what matters: targets and gaming in the English public health care system. Public Adm. 2006;84(3):517–38.

    Google Scholar 

  9. Kelman S, Friedman JN. Performance improvement and performance dysfunction: an empirical examination of distortionary impacts of the emergency room wait-time target in the English National Health Service. J Public Adm Res Theory. 2009;19(4):917–46.

    Google Scholar 

  10. Kino S, Sato K, Kawachi I. Spillover benefit of improved access to healthcare on reducing worry about housing and meal affordability. Int J Equity Health. 2018;17(174):1:9. https://doi.org/10.1186/s12939-018-0877-y.

  11. Shane DM, Ayyagari P. Spillover effects of the affordable care act? Exploring the impact on young adult dental insurance coverage. Health Serv Res. 2015;50(4):1109–24.

    PubMed  Google Scholar 

  12. Burns LR, Pauly MV. Transformation of the health care industry: curb your enthusiasm? Milbank Q. 2018;96(1):57–109.

    PubMed  PubMed Central  Google Scholar 

  13. Einav L, Finkelstein A, Ji Y, Mahoney N. Randomized trial shows healthcare payment reform has equal-sized spillover effects on patients not targeted by reform. Proc Natl Acad Sci. 2020;117(32):18939–47.

    CAS  PubMed  PubMed Central  Google Scholar 

  14. Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

    Google Scholar 

  15. Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5(1):69.

    PubMed  PubMed Central  Google Scholar 

  16. Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143.

    PubMed  PubMed Central  Google Scholar 

  17. Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

    PubMed  Google Scholar 

  18. Angelucci M, Maro VD. Programme evaluation and spillover effects. J D Effect. 2016;8(1):22–43.

    Google Scholar 

  19. Dolan P, Galizzi MM. Like ripples on a pond: behavioral spillovers and their implications for research and policy. J Econ Psychol. 2015;47:1–16.

    Google Scholar 

  20. Benjamin-Chung J, Abedin J, Berger D, Clark A, Jimenez V, Konagaya E, et al. Spillover effects on health outcomes in low- and middle-income countries: a systematic review. Int J Epidemiol. 2017;46(4):1251–76.

    PubMed  PubMed Central  Google Scholar 

  21. Benjamin-Chung J, Arnold BF, Berger D, Luby SP, Miguel E, Colford JM Jr, et al. Spillover effects in epidemiology: parameters, study designs and methodological considerations. Int J Epidemiol. 2018;47(1):332–47.

    PubMed  Google Scholar 

  22. Effective Practice and Organisation of Care (EPOC). EPOC taxonomy. 2015 [cited 2020 Nov 18]. Available from: epoc.cochrane.org/epoc-taxonomy

  23. Mazza D, Bairstow P, Buchan H, Chakraborty SP, Van Hecke O, Grech C, et al. Refining a taxonomy for guideline implementation: results of an exercise in abstract classification. Implement Sci. 2013;8(1):32.

    PubMed  PubMed Central  Google Scholar 

  24. Liu D, Green E, Kasteridis P, Goddard M, Jacobs R, Wittenberg R, et al. Incentive schemes to increase dementia diagnoses in primary care in England: a retrospective cohort study of unintended consequences. Br J Gen Pract. 2019;69(680):e154–63.

    PubMed  PubMed Central  Google Scholar 

  25. Mehtsun WT, Papanicolas I, Zheng J, Orav EJ, Lillemoe KD, Jha AK. National trends in readmission following inpatient surgery in the hospital readmissions reduction program era. Ann Surg. 2018;267(4):599–605.

    PubMed  Google Scholar 

  26. Demiralp B, He F, Koenig L. Further evidence on the system-wide effects of the hospital readmissions reduction program. Health Serv Res. 2018;53(3):1478–97.

    PubMed  Google Scholar 

  27. Tan ASL. Potential spillover educational effects of cancer-related direct-to-consumer advertising on cancer patients’ increased information seeking behaviors: results from a cohort study. J Cancer Educ. 2014;29(2):258–65.

    PubMed  PubMed Central  Google Scholar 

  28. Jarvis CI, Multerer L, Lewis D, Binka F, Edmunds WJ, Alexander N, et al. Spatial effects of permethrin-impregnated bed nets on child mortality: 26 years on, a spatial reanalysis of a cluster randomized trial. Am J Trop Med Hygiene. 2019;101(6):1434–41.

    Google Scholar 

  29. Raza WA, Van de Poel E, Van Ourti T. Impact and spill-over effects of an asset transfer program on child undernutrition: evidence from a randomized control trial in Bangladesh. J Health Econ. 2018;62:105–20.

    PubMed  Google Scholar 

  30. Shei A, Costa F, Reis MG, Ko AI. The impact of Brazil’s Bolsa Família conditional cash transfer program on children’s health care utilization and health outcomes. BMC Int Health Hum Rights. 2014;14:10.

    PubMed  PubMed Central  Google Scholar 

  31. Egger D, Haushofer J, Miguel E, Niehaus P, Walker MW. General equilibrium effects of cash transfers: experimental evidence from Kenya. NBER; 2019. Report No.: w26600. Available from: https://doi.org/10.3386/w26600

  32. Quinto RA. Estimating the peer effect on youth overweight and inactivity using an intervention study. J Sch Health. 2014;84(10):617–24.

    Google Scholar 

  33. Miettinen J, Malila N, Hakama M, Pitkaniemi J. Spillover improved survival in non-invited patients of the colorectal cancer screening programme. J Med Screen. 2018;25(3):134–40.

    PubMed  Google Scholar 

  34. Henke RM, Karaca Z, Gibson TB, Cutler E, White C, Wong HS. Medicare advantage penetration and hospital costs before and after the affordable care act. Med Care. 2018;56(4):321–8.

    PubMed  Google Scholar 

  35. Lakdawalla D, Sood N, Gu Q. Pharmaceutical advertising and Medicare part D. J Health Econ. 2013;32(6):1356–67.

    PubMed  Google Scholar 

  36. Li X, Ye J. The spillover effects of health insurance benefit mandates on public insurance coverage: evidence from veterans. J Health Econ. 2017;55:45–60.

  37. Mendelson A, Kondo K, Damberg C, Low A, Motúapuaka M, Freeman M, et al. The effects of pay-for-performance programs on health, health care use, and processes of care: a systematic review. Ann Intern Med. 2017;166(5):341.

    PubMed  Google Scholar 

  38. van Mol M, Nijkamp M, Markham C, Ista E. Using an intervention mapping approach to develop a discharge protocol for intensive care patients. BMC Health Serv Res. 2017;17(1):837.

    PubMed  PubMed Central  Google Scholar 

  39. McEachan RR, Lawton RJ, Jackson C, Conner M, Lunt J. Evidence, theory and context: using intervention mapping to develop a worksite physical activity intervention. BMC Public Health. 2008;8(1):326.

    PubMed  PubMed Central  Google Scholar 

  40. Cherrington A, Martin MY, Hayes M, Halanych JH, Andreae SJ, Safford M, et al. Intervention mapping as a guide for the development of a diabetes peer support intervention in rural Alabama. Prev Chronic Dis. 2012;9(110053):E36.

  41. Fassier JB, Sarnin P, Rouat S, Péron J, Kok G, Letrilliart L, et al. Interventions developed with the intervention mapping protocol in work disability prevention: a systematic review of the literature. J Occup Rehabil. 2019;29(1):11–24.

    CAS  PubMed  Google Scholar 

  42. Reeves S, Goldman J, Gilbert J, Tepper J, Silver I, Suter E, et al. A scoping review to improve conceptual clarity of interprofessional interventions. J Interprof Care. 2011;25(3):167–74.

    PubMed  Google Scholar 

  43. Brett J, Staniszewska S, Newburn M, Jones N, Taylor L. A systematic mapping review of effective interventions for communicating with, supporting and providing information to parents of preterm infants. BMJ Open. 2011;1(1):e000023.

    PubMed  PubMed Central  Google Scholar 

  44. Petticrew M, Anderson L, Elder R, Grimshaw J, Hopkins D, Hahn R, et al. Complex interventions and their implications for systematic reviews: a pragmatic approach. J Clin Epidemiol. 2013;66(11):1209–14.

    PubMed  Google Scholar 

  45. Snilstveit B, Vojtkova M, Bhavsar A, Gaarder M. Evidence gap maps ? A tool for promoting evidence-informed policy and prioritizing future research. World Bank. 2013:34 [cited 2020 Nov 19]. (Policy Research Working Papers). Available from: https://elibrary.worldbank.org/doi/abs/10.1596/1813-9450-6725.

  46. Funnell SC, Rogers PJ. Purposeful program theory: effective use of theories of change and logic models. Hoboken: Wiley; 2011. p. 471.

  47. Birckmayer JD, Weiss CH. Theory-based evaluation in practice: what do we learn? Eval Rev. 2000;24(4):407–31.

    CAS  PubMed  Google Scholar 

  48. Gilson L. Alliance for Health Policy and Systems Research, World Health Organization. Health policy and systems research: a methodology reader; 2012.

  49. Savedoff WD, Hussman K. The causes of corruption in the health sector: a focus on health care systems. Berlin: Transparency International; 2006. p. 4–16.

    Google Scholar 

  50. Presseau J, McCleary N, Lorencatto F, Patey AM, Grimshaw JM, Francis JJ. Action, actor, context, target, time (AACTT): a framework for specifying behaviour. Implement Sci. 2019;14(1):102.

    PubMed  PubMed Central  Google Scholar 

  51. Roland M, Guthrie B. Quality and outcomes framework: what have we learnt? BMJ. 2016;354:i4060.

    PubMed  PubMed Central  Google Scholar 

  52. Gillam SJ, Siriwardena AN, Steel N. Pay-for-performance in the United Kingdom: impact of the quality and outcomes framework: a systematic review. Ann Fam Med. 2012;10(5):461–8.

    PubMed  PubMed Central  Google Scholar 

  53. Flodgren G, Eccles MP, Shepperd S, Scott A, Parmelli E, Beyer FR. An overview of reviews evaluating the effectiveness of financial incentives in changing healthcare professional behaviours and patient outcomes. Cochrane Database Syst Rev. 2011;(7) [cited 2021 Oct 1]. Available from: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD009255/full.

  54. Doran T, Kontopantelis E, Valderas JM, Campbell S, Roland M, Salisbury C, et al. Effect of financial incentives on incentivised and non-incentivised clinical activities: longitudinal analysis of data from the UK quality and outcomes framework. BMJ. 2011;342:d3590.

    PubMed  PubMed Central  Google Scholar 

  55. Forbes LJ, Marchand C, Doran T, Peckham S. The role of the quality and outcomes framework in the care of long-term conditions: a systematic review. Br J Gen Pract. 2017;67(664):e775–84.

    PubMed  PubMed Central  Google Scholar 

  56. Allen T, Whittaker W, Sutton M. Does the proportion of pay linked to performance affect the job satisfaction of general practitioners? Soc Sci Med. 2017;173:9–17.

    PubMed  Google Scholar 

  57. McDonald R, Harrison S, Checkland K, Campbell SM, Roland M. Impact of financial incentives on clinical autonomy and internal motivation in primary care: ethnographic study. BMJ. 2007;334(7608):1357.

    PubMed  PubMed Central  Google Scholar 

  58. Khan N, Rudoler D, McDiarmid M, Peckham S. A pay for performance scheme in primary care: meta-synthesis of qualitative studies on the provider experiences of the quality and outcomes framework in the UK. BMC Fam Pract. 2020;21(1):142.

    PubMed  PubMed Central  Google Scholar 

  59. Minchin M, Roland M, Richardson J, Rowark S, Guthrie B. Quality of care in the United Kingdom after removal of financial incentives. N Engl J Med. 2018;379(10):948–57.

    PubMed  Google Scholar 

  60. Kaarboe O, Siciliani L. Multi-tasking, quality and pay for performance. Health Econ. 2011;20(2):225–38.

    PubMed  Google Scholar 

  61. Wensing M, Sales A, Armstrong R, Wilson P. Implementation science in times of COVID-19. Implement Sci. 2020;15(1):42.

    PubMed  PubMed Central  Google Scholar 

  62. Lewis CC, Boyd MR, Walsh-Bailey C, Lyon AR, Beidas R, Mittman B, et al. A systematic review of empirical studies examining mechanisms of implementation in health. Implement Sci. 2020;15(1):21.

    PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors are grateful to HOPE seminar participants at the University of Manchester, discussant (Dr. Becky Pennington) and participants in the Winter 2021 meeting of the UK Health Economists’ Study Group, Prof. Naomi Fulop, Prof. Louise Locock, Prof. Graham Martin, and Prof. Ruth McDonald for their insightful comments on earlier versions of the manuscript.

Funding

This work was supported by the Medical Research Council (MRC, grant number MR/R013314/1, for the project titled “Identifying and Measuring Spillover Effects of Changes to Healthcare Organisation and Delivery”, co-applicants are MS, RM, SRK).

Author information

Affiliations

Authors

Contributions

IF and JE took the lead in conducting the scoping review. IF also took the lead in drafting the manuscript. IF, RM, JE, SRK, PB, DLP, and MS all made substantial contributions to the conception or design of the work. IF, RM, JE, SRK, PB, DLP, PW, and MS made substantial contributions to the drafting, provided critical revisions to the work, approved the final version of the manuscript, and agreed to be accountable for all aspects of the work.

Corresponding author

Correspondence to Igor Francetic.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist.

Additional file 2.

Search strategy.

Additional file 3.

Detailed list of papers included in the scoping review.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Francetic, I., Meacock, R., Elliott, J. et al. Framework for identification and measurement of spillover effects in policy implementation: intended non-intended targeted non-targeted spillovers (INTENTS). Implement Sci Commun 3, 30 (2022). https://doi.org/10.1186/s43058-022-00280-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s43058-022-00280-8

Keywords

  • Health policy
  • Spillover effects
  • Evaluation
  • Healthcare economics and organisations
  • Programme evaluation
  • Unintended effects