Keywords
Artificial intelligence
Health economics and outcomes research
HEOR
Search filters
How to Cite
Abstract
Background: Search strategies used to identify evidence on automation in Health Economics and Outcomes Research (HEOR) often lack sensitivity and specificity, resulting in information overload or missed studies. This umbrella review evaluated and compared the performance and overlap of search filters commonly used to retrieve automation-related HEOR evidence.
Methods: Systematic literature reviews (SLRs) and search filters focusing on any form of automation in HEOR were included. Searches (January 01, 2023–July 03, 2024) were conducted in EMBASE, ISSG Resource, and Google Scholar. Subject headings, search terms, and performance metrics were extracted. Reference lists were cross-checked. Screening was performed by one reviewer, with 20% verified by a second reviewer. The PRESS checklist was used to assess search strategy quality. The protocol was registered with the Open Science Framework (OSF).
Results: Seven SLRs and one standalone filter, reporting 11 search strategies, met the inclusion criteria. HEOR relevance was defined by studies applying search filters in contexts of SLRs, indirect treatment comparisons, and economic modeling. Included SLRs retrieved between 5-273 studies. PubMed was the most frequently searched database. Commonly used subject headings included “artificial intelligence,” “deep learning,” “machine learning,” and “natural language processing,” with “artificial intelligence” the most frequent free-text term. Inclusion rates varied: title/abstract (1%–8%), full-text (27%–86%), and final inclusion (0.13%–2.31%). Time to include one study ranged from 0.6-8 hours.
Conclusions: Considerable variability in search filter performance was observed, causing lower specificity and inefficient evidence retrieval. Standardized, high-performing search strategies are needed to enhance efficiency and reliability in identifying automation-related HEOR evidence.
References
1. van Nooten F, Holmstrom S, Green J, Wiklund I, Odeyemi IA, Wilcox TK. Health economics and outcomes research within drug development: challenges and opportunities for reimbursement and market access within biopharma research. Drug Discov Today. 2012: 615 [PMID: 22366662, https://doi.org/10.1016/j.drudis.2012.01.021]
2. Reason T, Rawlinson W, Langham J, Gimblett A, Malcolm B, Klijn S. Artificial Intelligence to Automate Health Economic Modelling: A Case Study to Evaluate the Potential Application of Large Language Models. Pharmacoecon Open. 2024: 191 [PMID: 38340276, https://doi.org/10.1007/s41669-024-00477-8]
3. Nice. Use of AI in evidence generation: NICE position statement. 2024.
4. Mason J, Morrison A, Visintini S. An overview of clinical applications of artificial intelligence. Ottawa: CADTH. 2018:
5. Health NIoEi, Services S. Development and evaluation of a software tool based on GPT-4 to assist in sorting documents in the context of literature reviews: Literature, a proof of concept. 2024.
6. Escobar Liquitay CM, Garegnani L, Garrote V, Sola I, Franco JV. Search strategies (filters) to identify systematic reviews in MEDLINE and Embase. Cochrane Database Syst Rev. 2023: MR000054 [PMID: 37681507, https://doi.org/10.1002/14651858.MR000054.pub2]
7. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hrobjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021: n71 [PMID: 33782057, https://doi.org/10.1136/bmj.n71]
8. Pollock M, Fernandes RM, Becker LA, Pieper D, Hartling L. Chapter V: overviews of reviews. Cochrane handbook for systematic reviews of interventions version. 2020:
9. OpenAi. GPT-4 Technical Report 2023. 2023.
10. McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016: 40 [PMID: 27005575, https://doi.org/10.1016/j.jclinepi.2016.01.021]
11. J G, C L, P M, S R, I B, Woods L e. ISSG Search Filter Resource. 2006.
12. Toth B, Berek L, Gulacsi L, Pentek M, Zrubka Z. Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed. Syst Rev. 2024: 174 [PMID: 38978132, https://doi.org/10.1186/s13643-024-02592-3]
13. de la Torre-López J, Ramírez A, Romero JR. Artificial intelligence to automate the systematic review of scientific literature. Computing. 2023: 2171 https://doi.org/10.1007/s00607-023-01181-x]
14. Schmidt L, Mohamed S, Meader N, Bacardit J, Craig D. Automated data analysis of unstructured grey literature in health research: A mapping review. Res Synth Methods. 2024: 178 [PMID: 38115736, https://doi.org/10.1002/jrsm.1692]
15. Schmidt L, Finnerty Mutlu AN, Elmore R, Olorisade BK, Thomas J, Higgins JPT. Data extraction methods for systematic review (semi)automation: Update of a living systematic review. F1000Res. 2021: 401 [PMID: 34408850, https://doi.org/10.12688/f1000research.51117.3]
16. Yao X, Kumar MV, Su E, Flores Miranda A, Saha A, Sussman J. Evaluating the efficacy of artificial intelligence tools for the automation of systematic reviews in cancer research: A systematic review. Cancer Epidemiol. 2024: 102511 [PMID: 38071872, https://doi.org/10.1016/j.canep.2023.102511]
17. Doneva SE, Qin S, Sick B, Ellendorff T, Goldman J-P, Schneider G, Ineichen BV. Large language models to process, analyze, and synthesize biomedical texts: a scoping review. Discover Artificial Intelligence. 2024: https://doi.org/10.1007/s44163-024-00197-2]
18. Santos AOD, da Silva ES, Couto LM, Reis GVL, Belo VS. The use of artificial intelligence for automating or semi-automating biomedical literature analyses: A scoping review. J Biomed Inform. 2023: 104389 [PMID: 37187321, https://doi.org/10.1016/j.jbi.2023.104389]
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright (c) 2025 ASIDE Health Sciences