AI Tools for Evidence Synthesis

How do we define Artificial Intelligence (AI)?

It is the theory and development of computer system able to perform tasks that normally require human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages. 

Examples of AI: 

• Automated synthesis of information
• Natural language processing
• Speech and text analysis 
• Facial recognition

AI has become an increasingly valuable asset in the realm of systematic reviews and evidence synthesis. These advanced tools can be leveraged at various stages of the process:

- Search strategy development
- Identification of relevant articles and resources
- Data screening and extraction
- Synthesis of information
- Creation of plain language summaries

While AI tools offer significant advantages, experts emphasize the importance of:

1. Understanding potential biases and limitations
2. Using new AI tools alongside established, validated methods
3. Considering ethical implications, copyright issues, and intellectual property concerns

The National Institute for Health and Care Excellence (NICE) has issued a position statement regarding the use of AI in evidence generation and reporting. This statement aims to:

- Outline expectations for AI usage in evidence processes
- Provide guidance on regulations, best practices, and standards
- Support committee members and assessment groups in evaluating AI applications

A collaborative effort led by international organizations, including:

- International Collaboration for Automation in Systematic Reviews
- Cochrane and Campbell
- JBI (formerly Joanna Briggs Institute)

This initiative is developing guidance and recommendations for the responsible use of AI in evidence synthesis, currently in draft form for consultation and revision.

As AI continues to evolve, its role in systematic reviews and evidence synthesis is likely to expand, necessitating ongoing evaluation and adaptation of best practices.

• Syllabus Guidelines
• See the academic integrity policy in the current Academic Catalog. 

Purpose

• Using AI as a mediating step in between sections of the systematic review process

• Creates efficient operations and reduces the amount of time spent on more time-heavy portions

• Using AI as an aid to make faster decisions

• Increasing transparency and clarity in review questions

Strategies

• Determine the strengths and weaknesses of different sections of the systematic review process

• Identify the areas that take the most amount of time

• Assess the risk in automation 

• Talk to the research and library team about where automated processes would benefit the process

Human Review Primary (in between first and second step):

• AI can synthesize information to form a protocol

• Checking to make sure elements of DEI are included in the protocol and all components are present

Human Review Secondary (in between the second and third step):

• Autogenerated search strings

• Automated literature selections; Conducting the quality check after return results 

Human Review Tertiary (in between the third and fourth step):

• Automated selection of studies; review selection criteria and process

• Automated data extraction; review the type of data and what is included and excluded

• Automated synthesis of data; review for any biases and exclusions

Machine Bias

• Overestimations of research data input

• Inaccurate or unfair predictions

• Information exclusion

• Over specification 

• Discrimination against specific groups

Research Bias

• Lack of representation for marginalized groups in medical research 

• Grey literature may not always be considered

Equity Considerations

• May not consider equitable practices

• With the presence of machine discrimination, equity may go out the window

• Equity can be highlighted from the human lens

To strengthen the processes that use AI, it is important to provide feedback and speak up about any inconsistencies or biases noticed in the intermediate reviews. Also, always remember to assess the role of AI in your project and document when it was used in your methods section.

For more information on writing a Systematic or Scoping Review go to the Library Research Guide. 

• Systematic Review Toolbox

  The Systematic Review Toolbox is an online catalog of tools that support various tasks within the systematic review and wider evidence synthesis process.

• Rayyan

  Free web tool designed to speed up the process of screening and selecting studies

• Abstrackr

  Aids in citation screening. Please note you will need to create a free account before accessing the tool.

• DistillerSR

  An online application designed to automate all stages of the systematic literature reviews. "Another powerful tool for screening, DistillerSR uses AI to assist in the initial screening of titles and abstracts. It can prioritize studies based on relevance, apply predefined criteria, and flag potential duplicates. DistillerSR also supports collaborative work, allowing teams to share screening tasks and discuss decisions in real-time" (Verma, 2024). In addition, it is a great tool for the full-text screening phase. Priced packages are available (please note we cannot offer support on using this system).

• Elicit 
  Use in keyword suggestions and data extraction. This AI tool specializes in enhancing the research question by suggesting keywords that are relevant to your PICO elements. By providing initial terms, Elicit can generate a list of related keywords that might be used in literature searches, ensuring comprehensive coverage. Full data extraction including adding additional fields that may not be standard. It can be prompted to extract aspects of interest from the results such as population, outcomes measured, main findings, etc. Information can be downloaded via RIS or CSV with a subscription. This YouTube video has a great example of data extraction from an uploaded PDF in comparison with SciSpace.

• SciSpace
  Full data extraction including adding additional fields that may not be standard. It can be prompted to extract aspects of interest from the results such as population, outcomes measured, main findings, etc. Information can be downloaded via RIS or CSV with a subscription. This YouTube video has a great example of data extraction from an uploaded PDF in comparison with Elicit.

• ExaCT (not currently available)

  Information Extraction system. Please note you will need to request a free account. The system is trained to find key information from scientific clinical trial publications, namely the descriptions of the trial's interventions, population, outcome measures, funding sources, and other critical characteristics. Please note you will need to request a free account. Research article on this product. 

• RobotReviewer

  RobotReviewer is a machine learning system that aims to support evidence synthesis. The demonstration website allows users to upload RCT articles and see automatically determined information concerning the trial conduct (the 'PICO', study design, and whether there is a risk of bias).

• EPPI-Reviewer 
  An application for all types of literature review, including systematic reviews, meta-analyses, 'narrative' reviews, and meta-ethnographies. It is suitable for small or large-scale reviews (with some of our existing reviews containing over a million items). Connects with Zotero to help you locate full text. Visualizer, mapping, and multiple other awesome tools in the product to create reviews. Supports the development of systematic reviews from study screening through data collection, analysis, and synthesis. It supports authors and editors in writing all types of reviews, particularly in complex areas including meta-analysis, framework synthesis, and thematic synthesis. Used with Cochrane reviews and Campbell Reviews and is free for authors that write for those publishers. 
• RevMan (ReviewManager)

  RevMan is Cochrane's bespoke software for writing Cochrane reviews. RevMan has been designed to integrate with other systematic review software and new features and updates are added regularly. Cochrane review authors can log in to RevMan to view the dashboard (all reviews) and edit reviews online. Watch a 6-minute YouTube tutorial for authors using RevMan. RevMan is now available for non-Cochrane reviews. Click here to find out more.

• Covidence
  Covidence allows you to collaborate online with your co-authors on tasks such as uploading and sharing search results, screening and selecting studies, data collection, and risk of bias assessment.

• ASReview

  Free and open source tool that uses active learning as you screen to resort the remaining items by most to least likely to be included in your review. Like Rayyan, it doesn't do the screening for you. It sorts items to screen based on your previous inclusion/exclusion decisions.

• Litsearchr

  Web interface for the R package litsearchr, helps identify search terms and generate reproducible search strategies. The web version and the R package are both completely free.

• Khalil, H., Ameen, D., & Zarnegar, A. (2022). Tools to support the automation of systematic reviews: A scoping review. Journal of Clinical Epidemiology, 144, 22–42. https://doi.org/10.1016/j.jclinepi.2021.12.005
• Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010). ExaCT: automatic extraction of clinical trial characteristics from journal publications. BMC medical informatics and decision making, 10, 56. https://doi.org/10.1186/1472-6947-10-56
• Verma, P. (2024, September 1). How to write systematic reviews step by step with the help of ai tools. LinkedIn. https://www.linkedin.com/pulse/how-write-systematic-reviews-step-help-ai-tools-dr-preeti-rrs4c
• Hill, G. (2024, October 15). Libguides: Systematic and literature reviews: using ai’s in reviews. Royal Malborne Hospital Library. https://libguides.mh.org.au/systematic_and_literature_reviews/_AI

 

Recording of 1 hour webinar exploring Artificial Intelligence (AI) and its potential impact on the process of systematic reviews (August 15th, 2023). Note PICO Portal is a systematic review platform that leverages artificial intelligence to accelerate research and innovation.

Moderator Dr Greg Martin. Presenters: Eitan Agai - PICO Portal Founder & AI Expert; Riaz Qureshi - U. of Colorado Anschutz Medical Campus; Kevin Kallmes - Chief Executive Officer, Cofounder; Jeff Johnson - Chef Design Officer.

An update on machine learning AI in systematic reviews

June 2023 webinar including a panel discussion exploring the use of machine learning AI in Covidence (screening & data extraction tool).

Web Clinic: Artificial intelligence (AI) technologies in Cochrane

The session was delivered in May 2024 and you will find the videos from the webinar, together with the accompanying slides to download [PDF]. Recordings from other Methods Support Unit web clinics are available here.

Part 1: How Cochrane currently uses machine learning: implementing innovative technology
Part 2: What generative AI is, the opportunities it brings and the challenges regarding its safe use
Part 3: Cochrane's focus on the responsible use of AI in systematic reviews
Part 4: Questions and answers

The CLEAR path: A framework for enhancing information literacy through prompt engineering.

This article introduces the CLEAR Framework for Prompt Engineering, designed to optimize interactions with AI language models like ChatGPT. The framework encompasses five core principles—Concise, Logical, Explicit, Adaptive, and Reflective—that facilitate more effective AI-generated content evaluation and creation.

Lo, L. S. (2023). The CLEAR path: A framework for enhancing information literacy through prompt engineering. The Journal of Academic Librarianship, 49(4), 102720.

Various AI tools are invaluable throughout the systematic review or evidence synthesis process. While the consensus acknowledges the significant utility of AI tools across different review stages, it's imperative to grasp their inherent biases and weaknesses. Moreover, ethical considerations such as copyright and intellectual property must be at the forefront.

• Application ChatGPT in conducting systematic reviews and meta-analyses

• Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation?

• Artificial intelligence in systematic reviews: promising when appropriately used

• Harnessing the power of ChatGPT for automating systematic review process: methodology, case study, limitations, and future directions

• In-depth evaluation of machine learning methods for semi-automating article screening in a systematic review of mechanistic

• Tools to support the automation of systematic reviews: a scoping review

• The use of a large language model to create plain language summaries of evidence reviews in healthcare: A feasibility study

• Using artificial intelligence methods for systematic review in health sciences: A systematic review

Publishers may have different policies on whether or not generative AI is allowed and how to cite it. Check your publisher's information for authors' webpage, or contact their editorial staff, for details.

If using a chatbot or other generative AI-created content, here are ways to acknowledge that usage:

• Cite it in the text and references of your work
• Describe your use as part of your methodology
• Include an appendix with screenshots or transcripts of prompts and AI-generated responses

Examples of different citation styles:

APA 7 reference	OpenAI. (Year). ChatGPT (Month Day version) [Large language model]. https://chat.openai.com
MLA 9 works cited entry	“Tell me about confirmation bias” prompt. ChatGPT, Day Month. version, OpenAI, Day Month Year, chat.openai.com.
Chicago footnote	ChatGPT, response to “Tell me about confirmation bias,” Month Day, Year, https://chat.openai.com.