Table 1 Summary of sustainment measures

RE-AIM Framework Dimension Items Checklist - Maintenance Measure [18, 35, 47]

(http://www.re-aim.org/resources-and-tools/measures-and-checklists/)

To plan for and evaluate maintenance of a health program or policy at the individual and setting level. Maintenance is described as the extent to which a program is institutionalized or becomes a routine part of practice (setting level), and extent to which individuals see long-term effects of the program 6+ months after most recent contact (individual level).

Maintenance = sustainment

Public health

Developed by The Implementation Science Team at NCI Division of Cancer Control and Population Sciences with key RE-AIM leaders and authors as part of project to review grant proposals using RE-AIM

Public health and health services researchers. The website also states that RE-AIM is intended to be considered by “program planners, evaluators, readers of journal articles, funders and policy-makers.”

N/A. However, there has been a systematic review of the RE-AIM Framework (Gaglio, Shoup and Glasgow, 2013).

Individual: 5

Setting: 4

Qualitative (open-text responses) in the RE-AIM Planning Tool and quantitative (dichotomous, yes/no) in the Measuring the Use of the RE-AIM Model Dimension Items Checklist.

Checklist developed as a guide rather than a specific measure. Not a direct\replicable outcome measure of sustainment. However, the checklist or planning tool could be used as an outcome measure that is tracked over time by researchers or implementers.

Level of Institutionalization Scale (LoIn) [12]

(Goodman, McLeroy, Steckler, and Hoyle, 1993)

Measures institutionalization or the extent to which an innovative program has gone through the stages of passages, routines, and niche saturation to become an integral part of organization.

Institutionalization = sustainment

Health programs

Initial questionnaire reviewed by 5 multidisciplinary experts, revised and pilot tested in local health department

Administrators in health programs

8 subscales. 4 routine subscales (average interfactor correlations = .85) 4 niche saturation subscales (average interfactor correlations = .76). Chronbach’s alpha moderate to high (not listed). Internal consistency and factor structure.

45 items (15 3-part items)

4-point Likert-style response scale

Complex measure to complete, respondent burden. Can only be completed by higher-level administrator

Evidence-based Practice (EBP) Sustaining Telephone Survey [48]

(Swain, Whitley, McHugo, and Drake, 2010)

Mixed methods telephone survey to assess sustainability of EBPs in mental health agencies. Has sections on sustainment of practice, adaptation of practice, and facilitators of sustainment. Administered two years after implementation.

Sustainability = sustainment and sustainability

Mental health agencies

Not reported

Program leaders, administrator, and trainers

Not reported

47 items, quantitative and qualitative

Qualitative answers transcribed and coded, quantitative answers analyzed with descriptive statistics. Sustainment factors measured with 5-point Likert-style response scale.

Long, difficult to score, telephone survey (retrospective self-report).

The Team Check-Up Tool (TCT) [49]

(Lubomski et al., 2008)

Designed to be completed over the course of a team-based quality improvement (QI) intervention to assess dynamic context and progress.

Dynamic context and progress = sustainability

Healthcare QI teams

Developed by a statewide QI collaborative. Items were developed by clinicians and health service researchers, and later modified and refined in response to feedback and implementation experiences. Experts rated items on 4-point Likert scale, and team members participated in focus groups to provide item feedback.

QI team members

Evidence supporting the temporal stability, construct validity, and responsiveness of TCT

Content Validity Index = .87. Positive qualitative user feedback.

15 topics with subscales, 36 total items assessed for validity

Experts rated on 4-point Likert-style response scale

Looks at individual components of the specific intervention in the study, how many people are doing them, and what the barriers are. Appears to be closer to a measure of fidelity and barriers to rather than sustainment.

Stages of Implementation Completion (SIC) [50]

(Chamberlain, Brown, and Saldana, 2011)

Purpose: To document implementation progress of an EBP within an implementation project. The SIC measures 31 activities across 8 stages of implementation that span three phases: pre-implementation, implementation, and sustainability. Sustainability: defined as competency and credentialing as a sustainable program.

Sustainability = sustainment

Randomized control trials testing EBP implementation in health settings

SIC developed by members of study team and California Institute of Mental Health with 12 stages. Applied SIC to sites; iterative process using initial observations lead to 8 stages. Used in the context of a study comparing effectiveness of 2 implementation strategies for MTFC. Constructed to reflect the same stages for both strategies: identical requirements for stage completion, equivalent activities within the stages, etc.

Information is collected from individuals at multiple levels (providers, agency leaders, etc.). As an example, the SIC was completed by 53 sites from 51 counties in California and Ohio that implemented Multidimensional treatment foster care [REF].

Difficult to use traditional psychometric models because many SIC proportion items are dichotomous and SIC duration items are time-distributed. Reliability and validity were estimated using IRT-based Rasch models; adequate reliability.

8 stages, 31 activities across all stages

Three scores are calculated: 1) speed of implementation (duration of each stage), [2] proportion of implementation activities completed within each stage and [3] the number of stages (out of 8) completed.

Does not include all implementation activities or provide information on why activities were not completed. It also requires fee-for-service consultation/contracting with the measure developers to track and complete the SIC.

NHS Sustainability Model and Guide [51]

(Maher, Gustafson, and Evans, 2010; https://improvement.nhs.uk/resources/Sustainability-model-and-guide/)

Diagnostic tool to predict the likelihood of sustainability of an improvement initiative across three factors, process, organization and staff.

Sustainability= sustainment and sustainability

Health organizations

Co-produced for the NHS by front line teams, improvement experts, senior administrators, clinical leaders, people with content expertise from academia and other industries, as well as the literature on change and sustainability.

Individuals or teams working in NHS/healthcare

Not reported

10 subscales, 4 questions each (40 items)

Each factor has four detailed response scenarios. Participants mark the response that corresponds to their project. Each response is given a specific score. These are added to give a final sustainability score.

Developed as a formative self-assessment and guide for implementers rather than a research tool. No psychometric properties. Blends predictors and sustainment indicators.

A framework and a measurement instrument for sustainability of work practices in long-term care [15]

(Slaghuis, Strating, Bal, and Nieboer, 2011)

Measure of sustainability of new work practices in long-term health care practices where sustainability is the dynamic process through which actors develop/adapt organizational routines to a new work method.

Two domains, routinization: new work method becomes part of everyday activities; subscales of principles, practices, and feedback; and institutionalization: new work method is embedded in the organizational context, structures, and processes; subscales of skills, documentation materials, practical materials, and team reflection

Sustainability (routinization and institutionalization) = sustainment and sustainability

Healthcare organizations

Literature review conducted by authors on sustainability and related themes in health care. Created statements to be evaluated on five point Likert scale; Content validity assessed by authors and 11 experts, majority had worked in long-term care organizations as care professionals, quality staff or management. About half of the experts also had practical professional experience in organizing quality improvement projects.

Staff at long-term care organizations (e.g., nursing homes, elderly homes, care for disabled) and members of improvement teams

For 52 items tested: Hierarchical 2 factor model w/ routinization and institutionalization as separate constructs = best fit. Average factor loadings of each item = .54. Structure coefficients ranged from .68-1.

Short version: bivariate correlations between routinization and institutionalization = .79. Scores within subscales of these ranged from .49-.80.

Tested: 52

Long version: 40

Short version: 30

Translation: N/A

Each sub-dimension had a scale of 5-10 statements describing several aspects. Evaluated on a five point Likert scale, ranging from '1: I don't agree at all' to '5: I agree very much', including the option 'I don't know'.

Low response rate (33%) and small sample size (n = 112, across 63 teams, with missing data); need to assess test-retest reliability. Some subscales not generalizable across contexts and innovations.

Normalization Process Theory-based Measurement (NoMAD) [14, 72]

(Finch et al., 2013)

Measures intervention normalization using Normalization Process Theory (NPT) constructs. NPT states that practices become implemented, embedded, and integrated when people work individually and collectively to enact them. Four generative processes: coherence, cognitive participation, collective action, and reflexive monitoring.

Normalization = sustainment and sustainability

Healthcare organizations

Developed items to reflect each of 4 NPT constructs.

Process: lit review, identifying concepts, generating and appraising items, iterative item testing

6 sites implementing health-related interventions; respondents had a variety of roles (clinical, managerial, etc.)

Good face validity, construct validity, and internal consistency. Construct validity for 4 constructs was supported. Chronbach’s alpha for 4 NPT construct groupings ranged from .65-.81. Overall normalization scale high reliability (20 items, alpha = 0.89)

Bivariate correlations between NPT construct measures ranged from .49 to .68. Correlation between 43 NPT and 3 normalization items was low.

Tested: 46

Final: 23

5-point Likert-style response scale of agreement (strongly agree - strongly disagree) for NPT construct items

11-point Likert-style response scale for normalization assessment items

Needs to be validated across implementation settings; need to assess test-retest reliability

Program Sustainability Assessment Tool (PSAT) [52]

(Luke, Calhoun, Robichaux, Ellito, and Moreland-Russell, 2014)

Assesses the capacity for sustainability of public health-related programs.

Sustainability = sustainment and sustainability

Public health programs

Developed pilot based on lit review and concept mapping

Tested in large number of community and state-level programs. 592 program managers and staff in 252 public health programs aimed at addressing tobacco control, diabetes, obesity prevention, or oral health

Good fit of the 8 domain model. Sig diff in item-factor loadings across state and community levels (p < .001). Average Chronbach’s alpha of subscales = .88 (range 0.79-0.92). CFI for final 40 items = .89.

Pilot: 63 items, 9 sustainability domains

Final: 40 items, 8 sustainability domains (5 items/domain)

Items rated on 7-point Likert-style response scale of agreement

Relatively long scale. Can only be completed by higher-level administrators.

Clinical Sustainability Assessment Tool (CSAT) [53] (http://www.sustaintool.org/csat/)

Assesses the capacity for sustainability of clinical programs

Sustainability

Clinical programs

Developed based on lit review and concept mapping

Pilot tested with 120 individuals (physicians, pharmacists, nurses, administrative staff, leadership, and others) assessing clinical practice from a range of clinical settings and environments.

Developers are in the process of validating the tool.

Pilot: 35 items, 7 domains (5 items/domain)

Items rated on 7-point Likert-style response scale ranging from no extent to a full extent.

Relatively long scale.

Program Sustainability Index (PSI) [54]

(Mancini and Marek, 2004)

Purpose: measure sustainability of community-based programs using the following conceptual, model of 3 linked domains:

1. Sustainability: “capacity of programs to continuously respond to community issues… maintains a focus consonant with original goals and objectives… key element… is providing continued benefits” focus on sustained benefits to families vs. sustained activities.

2. Middle-range program results. Impacted by sustainability.

3. Ultimate results. Impacted by first two domains; pertains to program sustainability.

Sustainability= sustainment and sustainability

Community-based programs

Sustainability elements determined in previous mixed methods studies w/ 100+ interviews of community program personnel and 4000 responses to open-ended questions. Pilot survey developed with qualitative results then administered across 153 community-based programs ➔ 7 elements consistently influencing sustainment.

243 human development and family life professionals at varying levels of program development/evaluation voluntarily completed survey. Data collected at annual Children, Youth and Families at Risk initiative meeting

Initial analyses: unsatisfactory fit. Factor analysis suggested 6 domains. Acceptable internal consistencies (0.67-0.88), acceptable performance validity for the 29 items.

Tested: 53 items, 7 domains

Final: 29 items, 6 domains

3-point Likert-style response scale of extent. Items within each of the six PSI subscales were averaged to create scores.

Respondents fairly homogenous, should be tested in other settings. Frequently adapted or certain subscales selected, indicating lack of utility or fit in current form