Audit feedback interventions to address high-risk prescriptions in long-term care homes: a costing study and return on investment analysis
Implementation Science Communications volume 2, Article number: 125 (2021)
Audit and feedback is a common implementation strategy, but few studies describe its costs. ‘MyPractice’ is a province-wide audit and feedback initiative to improve prescribing in nursing homes. This study sought to estimate the costs of ‘MyPractice’ and assess whether the financial benefit of ‘MyPractice’ offsets those costs.
We conducted a costing study from the perspective of the Ontario government. Total cost of ‘MyPractice’ was calculated as the sum of the costs of producing and disseminating the reports (covering three report releases) which were obtained from Ontario Health staff interviews and document reviews. Return on investment (ROI) was calculated as the ratio of net cost-savings and the intervention cost. Cost savings were based on the effectiveness of ‘MyPractice’ derived from a published cohort study. Cost-savings attributable to ‘MyPractice’ were estimated from the changes in the rates of antipsychotics over time between physicians who signed up and viewed the reports and those who did not sign up to the reports.
Total intervention costs were C$223,691 (C$838 per physician and C$74,564 per release). Costs incurred during the development phase accounted for 74% of the total cost (C$166,117), while implementation costs for three report releases were responsible for 26% of the total costs (C$57,575). The ROI for every C$1 spent on the ‘MyPractice’ intervention was 1.02 (95% CI 0.51, 1.93) for three report releases.
‘MyPractice’ report offers a good return on investment and the value for money could improve with greater number of report releases.
Approximately 20.7% of long-term care home residents are prescribed antipsychotics without concurrent psychiatric diagnosis in Canada . Inappropriate use of these medications could increase risk of stroke, heart disease, and kidney failures which increase risk of premature death [1, 2]. To reduce inappropriate prescribing of antipsychotics among long-term care home residents, quality improvement initiatives are increasingly being utilized . Audit and feedback (A&F) has emerged as a common method to improve the quality of health care practices. A&F interventions highlight discrepancies between desired and actual performance and encourage prescribers to address these discrepancies .
Previous studies have shown the effectiveness of A&F on prescriber behaviors [5,6,7]. An Australian study, for example, examining the impact of A&F on antipsychotics prescribing for schizophrenia showed that A&F was effective in changing prescribing rates towards recommended levels . Another randomized control trial reported that the multi-strategic intervention consisting of A&F, including staff education and interdisciplinary reviews, resulted in a significant reduction in the proportion of long-term care home residents taking antipsychotics .
Despite growing evidence on effectiveness of A&F, little is known about the resources required to develop and implement such intervention. Given the constraint on health care budgets, credible information about A&F intervention costs could help inform the decision on whether an A&F intervention should be adopted or expanded. In this study, we estimated the costs of the ‘MyPractice’ reports, the A&F intervention that targets prescribing of antipsychotic medications in long-term care homes in Ontario and assessed whether the added costs could offset the benefits gained from the intervention.
Setting and intervention
Prescription drug costs for long-term care home residents in Ontario are covered by the Ontario Drug Benefit Program. Ontario Health, formerly Health Quality Ontario (HQO), is the agency that advises government and health care providers in Ontario on evidence to support high-quality care‚ to support improvements in quality, and to monitor and report on quality of health care provided in Ontario. The agency produces several ‘MyPractice’ clinician-focused reports with input from relevant stakeholders, including clinicians, epidemiologists, quality improvement experts, front-line clinicians, health services researchers, sector organizations and associations, and policy makers (4). The report aimed to address high-risk prescriptions for long-term care home residents in the province of Ontario, Canada, with an initial focus on antipsychotic medications. The reports were promoted to all family physicians working in long-term care homes across the province via communication materials distributed by the agency and external partners. The reports provided physicians the opportunity to voluntarily sign up to receive quarterly confidential feedback reports which compare the recipient’s prescribing rate with the provincial average. Physicians were asked to provide consent to receive the ‘MyPractice’ reports on the HQO website and verify their email address and identity. Physicians who signed up received an email notification when the report became available for download. They could log into their account through HQO’s secure web portal, download and view the report. Several audit and feedback initiatives are available to family physicians in Ontario; however, most physicians, especially those working in nursing homes, do not engage in these initiatives [8, 9].
The study period started from April 2016 to June 2017, covering development and implementation phases of ‘MyPractice’ reports focusing on antipsychotics. The ‘MyPractice’ initiative was designed and developed over 8 months (from April to November 2016). Three ‘MyPractice’ reports were released over a period of 7 months (from December 2016 to June 2017). At the time of the study, 267 physicians (28%) of 944 eligible physicians working in long-term care homes across Ontario signed up to receive the reports .
We conducted a costing study of the ‘MyPractice’ intervention from the perspective of government of Ontario (i.e., the insurer). We also calculated the return on investment (ROI) for ‘MyPractice’ by comparing the intervention cost to savings from the reduction in antipsychotic medication prescriptions, as a result of the intervention. The ROI analysis is a method to estimate net financial gains (or losses) from an intervention, considering resources invested to implement the program and the amount gained though increase in revenue, reduced costs, or both . The target population for ROI analysis included all physicians working in long-term care homes in Ontario who were provided the opportunity to sign up to receive ‘MyPractice’ reports.
Resource use data collection and unit cost
We estimated the intervention cost using a gross costing technique, where costs were calculated as a product of resource use and unit costs. We obtained resource use data for developing and implementing the intervention and their unit costs from program financial records, service level agreements, and the program budget, through close consultation with Ontario Health staff. Costs were categorized into development and implementation costs.
Development costs comprised of the costs of data acquisition and personnel time for planning, report content development, developing technical infrastructure and administrative, and managerial support for the development phase. Implementation costs included the costs of personnel time for analysis, quality assurance, outreach, support for participant queries, administrative and managerial support. Cost data were presented in 2019 Canadian dollars.
We estimated the total intervention cost as the sum of development and implementation costs for three report releases. We calculated the cost per physician by dividing the total intervention costs by the number of physicians who worked in long-term care homes and signed up to receive ‘MyPractice’ reports. We also projected the cost per physician if the ‘MyPractice’ initiative was adopted as a mandatory program, i.e., all physicians working in long-term care homes in Ontario received ‘MyPractice’ reports. Furthermore, we calculated the intervention cost per release by dividing the total intervention costs by the total number of report releases. Key drivers of development and implementation costs were also described.
In addition, we forecasted the annual cost of the ‘MyPractice’ intervention for the first year and subsequent years. The annual cost of ‘MyPractice’ for the first year was calculated as the sum of development costs and implementation costs for four quarterly report releases, while the upfront development costs were excluded for each subsequent year.
The ROI of the ‘MyPractice’ intervention was calculated as :
A ROI greater than zero indicates that the savings generated from the intervention are greater than the costs of the developing and implementing the intervention.
Cost savings due to ‘MyPractice’ were based on the attributable effectiveness of ‘MyPractice’, which was measured as the change in antipsychotic prescription rates over time between the intervention and reference groups. For our base-case analysis, the intervention group comprised of physicians who signed-up and viewed the reports, while physicians who did not sign-up were considered the reference group. In a scenario analysis, physicians who signed up and did not view the report were considered the reference group.
Cost-savings were estimated from a retrospective cohort study that compared the mean proportion of long-term care home days on antipsychotics over the quarter before ‘MyPractice’ reports were released, the quarter immediately following the first report and the quarter that followed . The cohort study reported changes in prescription rates over 6 months between the three exposure groups after adjustment for nursing home, physician, and resident characteristics . Baseline prescription rates for antipsychotics were similar across the three exposure groups (signed up and viewed: 25.0%; signed up and not viewed: 26.3%; not signed up: 25.8%). For physicians who signed up and viewed the report, there was a 0.94% decrease in the mean proportion of long-term care home days on antipsychotics (95% CI 0.35%, 1.54%) compared to those who did not the sign up , and a 0.47% decrease (95% CI − 0.15%, 1.09%) compared to those who signed up and did not view the report.
Based on the reduction in the mean proportion of days on antipsychotics reported in the cohort study, we estimated cost-savings attributable to ‘MyPractice’ by multiplying the antipsychotic medication cost saved per resident and the total number of long-term care home residents in the province of Ontario . Antipsychotic medication cost saved per resident was estimated by multiplying the reduction in the number of days on antipsychotics attributable to ‘MyPractice’  and the daily costs of antipsychotic medications. Daily costs of antipsychotic medications were calculated by multiplying the prevalence of atypical and typical antipsychotic drugs commonly prescribed for nursing home residents with their daily costs [12, 13].
We performed scenario analyses projecting the ROI for ‘MyPractice’ covering four and eight report releases, with the assumptions that reports were released every quarter and that the effectiveness for ‘MyPractice’ reports observed over the study period was sustained for subsequent report releases.
We performed a probabilistic sensitivity analysis using the Monte Carlo simulation technique. We varied all parameters over ± 25% of their base values and repeated the ROI analysis over 1,000 iterations. Of these 1000 iterations, we estimated the probability that that ‘MyPractice’ would provide a good return on investment (i.e., ROI ≥ 0).
The total intervention costs were estimated to be C$223,691. The cost per physician who signed up for ‘MyPractice’ reports was C$838. The cost per release was C$74,564. The cost of developing the intervention was C$166,117 (74% of total cost) and the cost of implementation for three report releases was C$57,575 (26% of total costs). Data acquisition and analysis accounted for the largest share (35%) of development costs, followed by report production (29%) and management (29%) costs (Table 1). Quality assurance of reports accounted for majority (55%) of implementation costs. If the ‘MyPractice’ initiative was mandatory, the cost per physician was projected to be C$237.
Since most costs incurred upfront, the average cost per release decreased significantly with increasing number of report release (Fig. 1). We forecasted that if ‘MyPractice’ were to be released quarterly, its annual cost was estimated to be C$241,370 for the first year (December 2016 to December 2017) and C$70,716 for each subsequent year.
The ROI for ‘MyPractice’ for three report releases was 0.02 (95% CI − 0.49, 0.93) for the base-case. Furthermore, the ROI increased with greater number of report releases (Fig. 2). The projected ROI was 0.60 (95% CI − 0.19, 2.03) for four releases and 1.47 (95% CI 0.25, 3.54) for eight releases.
Results from the scenario analysis showed that the ROI was highly sensitive to the effectiveness of ‘MyPractice’. The larger impact of the intervention on the mean proportion of long-term care home days on antipsychotics, the greater the ROI values. The ROI for ‘MyPractice’ was − 0.51 (95% CI − 0.93, 1.14) in a scenario analysis where the attributable effectiveness of ‘MyPractice’ was calculated using the physicians who signed up and did not view the reports as a reference group.
The probability of ‘MyPractice’ being a good return on investment (ROI ≥ 0) was 48.5% for the base-case; this probability reached 89.3% if at least four reports were released (Fig. 3).
Our study highlights that the majority of ‘MyPractice’ costs (74%) were fixed and incurred upfront; the average cost per release‚ therefore‚ reduced significantly with increasing number of releases. This would suggest that releasing more reports or scaling the intervention to larger number of long-term care homes would not significantly increase the total costs of the intervention. We observed a positive ROI (ROI = 0.02) for the ‘MyPractice’ initiative, suggesting that for every C$1 spent on ‘MyPractice’, C$1.02 will be gained through the reduction in costs associated with antipsychotic prescriptions. The ROI was found to increase with the greater number of report releases if the same reduction in antipsychotic prescribing attributable to ‘MyPractice’ was sustained over subsequent report releases. The ROI was highly sensitive to the attributable reduction in antipsychotic prescription rates as a result of ‘MyPractice’ reports.
We identified only one existing study that assessed the costs of implementing an A&F intervention. Fretheim et al. determined the costs of implementing a multifaceted intervention consisting of audit and feedback, outreach visits and computerized reminders to improve adherence to clinical practice guidelines for prescribing of antihypertensive and cholesterol-lowering drugs in primary practices in Norway . The intervention was conducted across 70 primary practices over 1 year. The total intervention cost for 257 physicians was C$122,584Footnote 1 (C$476.98 per physician). The total intervention cost included the costs of training, software development, printing, salary of pharmacists doing outreach visits, personnel time for technical and administrative support, travel, and physician opportunity costs. The cost per physician reported by Fretheim et al. was lower than our study possibly because they excluded costs of designing the intervention, which was the key driver of the total costs of ‘MyPractice’.
Our study has certain limitations that must be acknowledged. Data on resources required for development and implementation of the ‘MyPractice’ initiative was collected retrospectively. Personnel time used for each activity may not be accurately reported. Moreover, reduction in prescribing of high-risk antipsychotics could reduce health system cost due to fewer readmissions and outpatient visits. However, we did not account for this potential downstream cost savings due to limited information on long-term effectiveness of A&F interventions. Furthermore, ‘MyPractice’ is a simple audit feedback strategy; therefore, the results from our study may not be generalizable to other audit feedback strategies that may vary in terms of resources required for implementation.
Despite these limitations, our study is one of the few comprehensive economic analyses of an audit and feedback initiative. Findings could help inform the decision to adopt or expand audit and feedback initiatives such as ‘MyPractice’ to address prescribing of high-risk medications in long-term care homes. More studies on the cost of A&F interventions are required as cost data from diverse contexts could highlight A&F design and delivery approaches that can make the intervention more affordable.
In summary, ‘MyPractice’ is a good return on investment intervention to address prescribing of high-risk medication in long-term care homes. The financial benefits of this and other A&F interventions would depend on how they are designed and delivered. If most intervention costs are either incurred upfront or spread over several years, the A&F interventions may be economically attractive even with a small effect size.
Availability of data and materials
Data for cost analysis: The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.
Cost converted from US$ to C$ for comparison purposes.
Audit and feedback
Frequently asked questions
Return on investment
Canadian Institute for Health Information. Potentially Inappropriate Use of Antipsychotics in Long-Term Care. Available from: http://indicatorlibrary.cihi.ca/display/HSPIL/Potentially Inappropriate Use of Antipsychotics in Long-Term Care
Fazio PD, Manfredi V, Gareri P, Bruni A, Ciambrone P, Cerminara G, et al. Use of atypical antipsychotics in the elderly: a clinical review. Clin Interv Aging. 2014;9:1363.
Coon JT, Abbott R, Rogers M, Whear R, Pearson S, Lang I, et al. Interventions to reduce inappropriate prescribing of antipsychotic medications in people with dementia resident in care homes: a systematic review. J Am Med Direct Assoc. 2014;15(10):706–18.
Ivers N, Desveaux L, Presseau J, Reis C, Witteman HO, Taljaard MK, et al. Testing feedback message framing and comparators to address prescribing of high-risk medications in nursing homes: protocol for a pragmatic, factorial, cluster-randomized trial. Implement Sci. 2017;12(1):86.
Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;13(6):CD000259.
Wheeler A, Humberstone V, Robinson E, Sheridan J, Joyce P. Impact of audit and feedback on antipsychotic prescribing in schizophrenia. Journal of Evaluation in Clinical Practice. 2009;15(3):441–50. https://doi.org/10.1111/j.1365-2753.2008.01032.x.
Westbury J, Jackson S, Gee P, Peterson G. An effective approach to decrease antipsychotic and benzodiazepine use in nursing homes: the RedUSe project. Int Psychogeriatr. 2009;22(1):26–36.
Vaisson G, Witteman HO, Chipenda-Dansokho S, Saragosa M, Bouck Z, Bravo CA, et al. Testing behavior change techniques to encourage primary care physicians to access cancer screening audit and feedback reports: protocol for a factorial randomized experiment of email content. Curr Oncol. 2019;26(3):205–16. https://doi.org/10.3747/co.26.4829.
Wagner DJ, Durbin J, Barnsley J, Ivers NM. Beyond quality improvement: exploring why primary care teams engage in a voluntary audit and feedback program. BMC Health Serv Res. 2017;17:803.
Agency for Healthcare Research and Quality. The toolkit for using the AHRQ quality indicators: how to improve hospital quality and safety. Available from: https://www.ahrq.gov/patient-safety/settings/hospital/resource/qitool/index.html
Ivers N, Taljaard M, Giannakeas V, Reis C, Mulhall C, JMC L, et al. Effectiveness of confidential reports describing prescriptions for antipsychotic medications in long-term care homes. Implement Sci Commun. 2020;1(1):1–4.
Canadian Agency for Drugs and Technologies in Health. Pharmacoeconomic review report: brexpiprazole (Rexulti): (Lundbeck Canada Inc. and Otsuka Canada Pharmaceutical Inc.). 2017Dec. Available from: https://www.ncbi.nlm.nih.gov/books/NBK535508/pdf/Bookshelf_NBK535508.pdf
Kamble P, Chen H, Sherer J, Aparasu RR. Antipsychotic drug use among elderly nursing home residents in the United States. Am J Geriatr Pharmacother. 2008;6(4):187–97. https://doi.org/10.1016/j.amjopharm.2008.10.002.
Fretheim A, Aaserud M, Oxman AD. Rational Prescribing in Primary Care (RaPP): economic evaluation of an intervention to improve professional practice. PLoS Med. 2006;3(6):e216.
This study was supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). This study also received funding from The Ontario SPOR SUPPORT Unit. Parts of this material are based on data and information compiled and provided by: MOHLTC and CIHI. The analyses, conclusions, opinions, and statements expressed herein are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred.
Ontario SPOR SUPPORT Unit.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Thavorn, K., Kumar, S., Reis, C. et al. Audit feedback interventions to address high-risk prescriptions in long-term care homes: a costing study and return on investment analysis. Implement Sci Commun 2, 125 (2021). https://doi.org/10.1186/s43058-021-00225-7