Skip to Main Content

Scoping & Systematic Reviews

In this guide you will find information about how to conduct a scoping and systematic review plus information on how librarians can support your in the process.

About Step 1: Pre-Review Tasks.

Average time (hours) to complete
       160                                                                                                                                        

This step will help you prepare to conduct your review. You will:

  1. End Goals
  2. Develop your research question.
  3. Look at the literature to decide if you need to do a systematic review or scoping review.
  4. Build your research team.
  5. Decide which citation manager and software you will use.
  6. Identify your Eligibility Criteria

This page has information about research questions and review teams. Librarians can help you edit your research question based on the literature.

Click on the other tabs to see how they apply to Step 1: Pre-Review Tasks.

The PRISMA (Preferred Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines detail what you should report about your review methods.

SCOPING REVIEW SYSTEMATIC REVIEW
  • Checklist - your guideline for essential reporting in the final manuscript.
  • Flow Diagram - your visualization of the exclusion/inclusion process.
  • Statement - PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation.

For this step of the process, you can review the PRISMA checklist and flow diagram and download the PRISMA flow diagram template that matches your review type and sources, then list out the databases and other sources you plan to search. Common questions about the Prisma diagram have been written up by Rethlefsen & Page (2021). We suggest that you read over this so you are not missing pieces to your diagram.

Please see Systematic & Scoping Review Service for more
detailed information and to submit a request form. 

Before you begin conducting a review, your librarian can help you 

  • Develop and refine your research question framework 
  • Determine if any prior reviews have been published on the same or similar topics
  • Determine how much literature might be available on your topic

If the APU Librarian is listed as a co-author, they will work with your team to design and report thorough search strategies based on guidelines and standards set forth by Cochrane, PRISMA, JBI and IOM.  Our librarians provide:

  • Comprehensive, replicable search strategies for multiple databases and grey literature resources.
  • Mendeley or Zotero library of de-duplicated results or RIS file of de-duplicated results.
  • Excel workbook of search results.
  • Write the complete search methodology that can be included in the final systematic review/scoping review manuscript
  • A completed PRISMA flow diagram in png format.
  • For more info see how your librarian can help

Please allow for a minimum of 4 weeks for the scoping/systematic review literature search to be completed. Your librarian can give you a more precise estimate after you submit the review request form.

Conducting a Review

Publication - End Goals to think about

The publication component is where librarians can become useful again. A librarian can help you identify high-impact journals and help you consider the differences between open access and traditional publishing. In addition, the library has partnerships with publishers to allow authors from APU to publish open access with free or limited APC fees. 

Do your research on the journal(s) you intend to submit to. Make sure you understand their audience, their scope and tone, and any standards or expectations they may have around systematic & scoping reviews.

Locate materials on how to write for publication in your discipline. For example here are a number of materials about writing in the healthcare field.

Additional Publications

Think through how to maximize your publications with your work

  1. Published Systematic or Scoping Review
  2. Register the Protocol
  3. Published data & protocol
  4. Policy brief or white papers for an organization or popular or news rather than scholarly publication

To see some examples of publications co-authored by APU Librarians go to: https://apu.libguides.com/reviews/librarian-co-author

Automation, AI, and other upcoming review technologies

Purpose and Strategies

Purpose

  • Using AI as a mediating step in between sections of the systematic review process

  • Creates efficient operations and reduces the amount of time spent on more time-heavy portions

  • Using AI as an aid to make faster decisions

  • Increasing transparency and clarity in review questions

Strategies

  • Determine the strengths and weaknesses of different sections of the systematic review process

  • Identify the areas that take the most amount of time

  • Assess the risk in automation 

  • Talk to the research and library team about where automated processes would benefit the process

AI in Systematic Review Process

Steps of planning phase

Human Review Primary (in between first and second step):

  • AI can synthesize information to form a protocol

  • Checking to make sure elements of DEI are included in the protocol and all components are present

Steps of searching phase (literature)

Human Review Secondary (in between the second and third step):

  • Autogenerated search strings

  • Automated literature selections; Conducting the quality check after return results 

Study review, selection, and information synthesis phase

Human Review Tertiary (in between the third and fourth step):

  • Automated selection of studies; review selection criteria and process

  • Automated data extraction; review the type of data and what is included and excluded

  • Automated synthesis of data; review for any biases and exclusions

Review composition phase

Benefits and Challenges

Machine Bias

  • Overestimations of research data input

  • Inaccurate or unfair predictions

  • Information exclusion

  • Over specification 

  • Discrimination against specific groups

Research Bias

  • Lack of representation for marginalized groups in medical research 

  • Grey literature may not always be considered

Equity Considerations

  • May not consider equitable practices

  • With the presence of machine discrimination, equity may go out the window

  • Equity can be highlighted from the human lens

To strengthen the processes that use AI, it is important to provide feedback and speak up about any inconsistencies or biases noticed in the intermediate reviews. Also, always remember to assess the role of AI in your project and document when it was used in your methods section.

For more information on writing a Systematic or Scoping Review go to the Library Research Guide. 

Develop and refine research question

Comprehensive Literature Search Planning Form (coming soon): A planning document such as this one is highly useful to fill out as you are beginning your project.

Before you start your research, it's important you have a clear objective and one or more questions aligned to this.

  • The objective is the rationale behind why the review should be conducted. It should be clear, succinct, and convey to the reader what the study will add to the research field.
  • To define an objective, think about what you are trying to understand about the topic you are reviewing.
  • Once you have defined your objective, you can start putting together inclusion criteria for the review.
  • Inclusion criteria are the elements or factors that must be present in each source of evidence for it to be eligible for inclusion in the review.
  • To create inclusion criteria, simply break down objectives and questions into their component parts.
  • You may also need to consider primary and secondary questions.
    • Primary questions directly relate to the topic and must be addressed by all sources.
    • Secondary questions provide additional or contextual information that doesn’t need to be addressed by all sources.
  • You can focus your objectives by adding more elements to your inclusion criteria, e.g. adding contextual factors such as health care setting or geographic location, or a population group.

--------------------------

'Clarity of the review question assists in developing the protocol, facilitates effectiveness in the literature search, and provides a clear structure for the development of the scoping review'

- JBI Manual for Evidence Synthesis 11.2.2 - Developing the title and question

'Provide an explicit statement of the questions and objectives being addressed with reference to their key elements (e.g., population or participants, concepts, and context) or other relevant key elements used to conceptualize the review questions and/or objectives.'

- PRISMA-ScR Checklist and Explanation, p. 471

'Objectives give the review focus and must be clear before appropriate eligibility criteria can be developed. If the review will address multiple interventions, clarity is required on how these will be addressed (e.g. summarized separately, combined or explicitly compared).'

 - Cochrane Handbook, Chapter 2, 2.3

An example of the process
  Definition Example

Topic

 

Question

A broad statement outlining the focus of the review.

Sometimes this is posed as a broad question instead. This primary question relates directly to the topic, and determines the content that must be addressed by all sources of evidence to be considered relevant. 

This is the question that the review must pose in order to meet your objective.

Nurse-led models of care in the field of chronic disease management

What nurse-led models of care are used to manage chronic disease?
Objective

An objective is a clear, succinct statement that conveys:

  • Why the review should be conducted
  • What the review will add to the reader's knowledge in the field
  • What specifically is being investigated about the topic under investigation
 Nurse-led models of care are an emerging approach across a number of areas of health care. The objective of this review is to report on nurse-led models of care in chronic disease. The review seeks to examine whether there are different types of nurse-led models, the kinds of chronic disease care for which nurse-led models have been used and what facilitators and/or barriers have been reported relating to the success and/or failures of the model.
Secondary questions/ sub-questions Not required for every review. Provide contextual or additional information about the topic. Not all of your included articles need to address these questions.

What has been reported to be a facilitator/barrier for the success of these models?

The PCC framework/mnemonic (for scoping reviews) and PICO (for systematic reviews) is recommended as an aid in developing your review title, question and inclusion criteria. A clearly formulated question is also helpful when designing your search strategy.

A sequence of а funnel, a circle full of questions and a single big question mark. A sign above this sequence says, "Narrow topic, Think of questions, and Focus question."The process for developing a research question

There are many ways of framing questions depending on the topic, discipline, or type of questions.

Try Elicit to generate a few options for your initial research topic and narrow it down to a specific population, geographical location, disease, etc. You may explore a similar tool, LitSense to identify additional search terms.

Because the aim of a scoping review differs from that of a systematic review, question development may not fit into the PICO (Patient/Intervention/Comparison/Outcome) framework. Therefore, PCC (Population/Concept/Context) may be a more useful framework.

Per JBI's Scoping Review Manual: "The 'PCC' mnemonic is recommended as a guide to construct a clear and meaningful title for a scoping review. The PCC mnemonic stands for the Population, Concept, and Context. There is no need for explicit outcomes, interventions or phenomena of interest to be stated for a scoping review; however, elements of each of these may be implicit in the concept under examination."

The PCC Question outline can help frame your scoping review question and highlight important concepts for the literature search.

P (Population)

"Important characteristics of participants, including age and other qualifying criteria...

You may not need to include this element unless your question focuses on a specific condition or cohort. For example, for a scoping review that is focused upon mapping the types and details of research designs that have been used in a particular field, it may not be useful or within scope to detail the types of participants involved in that research" (10.2.4)
C (Concept) "The core concept examined by the scoping review should be clearly articulated to guide the scope and breadth of the inquiry. This may include details that pertain to elements that would be detailed in a standard systematic review, such as the “interventions”, and/ or “phenomena of interest”, and/or “outcomes” (11.2.4). (as relevant for the particular scoping review)
C (Context) "The “Context” element of a scoping review will vary depending on the objective/s and question/s of the review. The context should be clearly defined and may include, but is not limited to, consideration of cultural factors, such as geographic location and/or specific racial, social, cultural, or gender-based interests. In some cases, context may also encompass details about the specific setting (such as acute care, primary health care or the community). Reviewers may choose to limit the context of their review to a particular country or health system or healthcare setting, depending on the topic and objectives" (10.2.4).
Example 1:

Topic: Non-pharmaceutical treatments for ADHD

Research Question: What research is available about non-pharmaceutical treatments to treat ADHD?

  • P = ADHD
  • C = Non-pharmaceutical treatments
  • C = N/A
Example 2: 

Topic: Models of nurse-led care in chronic disease in high-income countries

Research Question: What nurse-led models of care are used to manage chronic disease in high-income countries?

  • P = N/A
  • C = nurse-led models of care used to manage chronic disease
  • C = high-income countries

Source: JBI Manual for Evidence Synthesis: Chapter 10.2.4

The PICO framework is useful for developing an answerable clinical question. On the Systematic Review Request form you will be asked to outline your research question in PICO format. This allows us to easily understand the main concepts of your research question. Here is what PICO stands for:

P = Problem/Population

I = Intervention (or the experimental variable)

C = Comparison (or the control variable) [Optional]

O = Outcome

If your research question does not fit neatly into PICO that is okay. Just try to include the elements of your question as closely as possible into the format. Your collaborating librarian will discuss any questions or concerns about your research topic before putting together your systematic review search strategy.

For example: Is a paleo diet more effective than a Mediterranean diet for weight reduction in obese adults?

P Population (patient)
What are the characteristics of the patient or population? OR
What is the condition or disease you are interested in?		 obese adults
I/E    Intervention or Exposure
What do you want to do with the patient (e.g. treat, diagnose, observe etc.)?		 paleo diet
C Comparison
What is the alternative to the treatment (e.g.placebo, different drug, surgery)?		 mediterranean diet
O Outcome
What are the relevant outcomes (e.g. morbidity, complications)?		 weight reduction

Use the PICO worksheet and template to get started with your question. If you are using another framework, adapt it.

Watch the 4 min. video on how to frame a research question with PICO.

Did you know there are at least 25 other question frameworks besides variations of PICO? 

Find Existing Reviews

Why?

Once you have a clearly defined research question, you will want to search other reviews on your topic. Finding existing reviews

  1. tells you if your planned systematic/scoping review has already been done before, in which case you can either amend your research question or find a way to materially improve on or update the existing systematic/scoping review
  2. shows you where your review fits into the scholarly conversation and enables you to acknowledge the existence of related reviews in your introduction
  3. gives you a means to mine relevant sources from related reviews
  4. provides examples for how to conduct your own review (note what questions and criteria are included in these reviews, pay attention to search terms and databases used to search for studies)
How to find reviews?

You can easily search in the library databases like Cinahl, Medline, PsycInfo, SocIndex, Cochrane, Pubmed, and any of the number of health science databases on the Proquest platform for the keyword search "scoping review" or "systematic review" to see what is out there on a topic.

Search Tips for Existing Reviews

1) Use broader search terms than you will include in your own search protocol. This will help you find related reviews that may not explicitly match your research question but still be useful. 

2) The term "systematic review" or "scoping review" will not always appear in the title or abstract on an article. In some databases, you cannot limit your searches to a systematic/scoping review methodology. So, using a search string like the one below helps you catch more potential reviews of interest...

“systematic review*” OR "research synthesis" OR "synthesis of research" OR "meta analysis" OR "meta-analysis" or "scoping review*"

Recruit Team Members

Review teams consist of three people at a minimum (2 screeners 1 librarian) but can be much larger. It's important to consider who will fill these roles. Define your roles and expectations early in the review process. Keep these guidelines in mind when establishing your review team:

 

  • Project manager/team lead - This person may assist in multiple parts of the  review, but they are responsible for seeing the project from initiation to completion. Tasks include corresponding with the librarian, ensuring team members stay on the project schedule, and managing the manuscript submission.
  • Screeners - Reviews involve two rounds of screening, title/abstract and full-text. For both rounds, it is recommended that each article is screened by two individuals. Screeners need to have sufficient subject matter expertise to determine if articles should be included or excluded in the review. Conflicts between screeners may be resolved between the screeners or by a third party. At a minimum, you will require two screeners. This is also true for data extraction and critical appraisal. Depending on the size of the literature, you may want to add additional team members. A team of up to ten or twelve people is not unusual for a large systematic review. Conscientiously create a diverse team to ensure different voices and perspectives are represented.
  • Search specialist (the librarian) - Working with a librarian or information specialist to develop and implement your search is recommended by the major systematic/scoping review handbooks. The librarian will handle all aspects of the review related to search development and translation, importing the articles into screening software, and providing general methodological advice. 
  • Statistician - Depending on the nature of your review, particularly if you plan to conduct a meta-analysis, you may require significant statistical expertise. 

Choose Review Tools

Prepare for Screening (deduplicating search if necessary)
  • Zotero is a free, open-source, web-based tool for managing citations, organizing pdfs, and creating bibliographies. Self-guided instruction in the form of tutorials, FAQs, and help are available through Zotero. The University of Otowwa has a detailed Guide on Zotero. To increase success with Zotero's in getting full text, go to Edit, Preferences, then Advanced. Then under "Open URL" select North America, then Azusa Pacific University.
  • Endnote is citation management software produced by Clarivate. A free 30-day trial is available through EndNote. Self-guided instruction in the form of tutorials, FAQs, & videos are available through Endnote APU has a guide on how to use Endnote. To increase success with Endnote’s “Find Full Text” feature, add APU’s OpenURL in preferences: https://apu.on.worldcat.org/atoztitles/link
  • Mendeley iis a free, web-based tool from Elsevier for managing references, creating bibliographies, and pdf organizer. Self-guided instruction in the form of tutorials, FAQs, & videos are available through Mendeley help guides. Fulltext can be added through the libraries subscription to Elsevier and Open Access articles are are added automatically. Check out our tutorial on Managing Citations in Mendeley for more help with this tool.
  • Other Deduplication techniques can be found here: https://apu.libguides.com/deduplicate
Screening Tools
  • The Systematic Review Accelerator (SRA)  is a suite of automation tools, purpose-built to speed up multiple steps in the systematic review (SR) process. Includes screening, deduplication, searching, and methods wizard. This is a free resource.
  • Covidence is an online systematic review program developed by, and for, systematic reviewers. It can import citations from reference managers like EndNote, facilitate the screening of abstracts and full-text, populate risk of bias tables, assist with data extraction, and export to all common formats. Offers free trials. We recommend this service.
  • CADIMA is a free, online, open-access review management tool developed to facilitate research synthesis and structure documentation of the outcomes. The software supports and guides users through the entire systematic review process, including protocol development, literature searching, study selection, critical appraisal, and documentation of the outcomes. The flexibility in choosing the steps also makes CADIMA suitable for conducting systematic mapping and rapid reviews. Does not handle duplicate removal (deduplicate in Zotero then, upload into CADIMA). We recommend this service.
  • Rayyan is a web based collaborative application that facilitates team screening, including the upload of citations and recording of the decisions behind the screening process. A mobile app allows you to screen articles any place. The basic account is free. Training material, including how-to videos on Rayyan can be found here. 
  • SysRev is a platform for the collaborative extraction of data from academic articles and abstracts, PDF documents, and other entities. The basic account is free.
  • Abstrackr is a software for semi-automated abstract screening for systematic reviews. At present, Abstrackr is a free, open-source tool for facilitating the citation screening process. Upload your abstracts, invite reviewers, and get to screening!
  • DistillerSR: DistillerSR is web-based software that primarily focuses on screening and data extraction. It is developed by Evidence Partners in Canada. Pricing is based on a subscription model.
  • More AI Tools for Evidence Synthesis

Here is a comparison chart of some of the Review Tools

The Systematic Review Toolbox (also includes information on Scoping Reviews) is a community-driven, searchable, web-based catalog of tools that support the systematic review process across multiple domains. The resource aims to help reviewers find appropriate tools based on how they provide support for the systematic review process. Users can perform a simple keyword search (i.e. Quick Search) to locate tools, a more detailed search (i.e. Advanced Search) allowing users to select various criteria to find specific types of tools and submit new tools to the database.

  • In searching for research articles, we often want to obtain lists of references from across studies, and also obtain lists of articles that cite a particular study. In systematic/scoping reviews, this supplementary search technique is known as "citation chasing": forward citation chasing looks for all records citing one or more articles of known relevance; backward citation chasing looks for all records referenced in one or more articles.
  • Traditionally, this process would be done manually, and the resulting records would need to be checked one-by-one against included studies in a review to identify potentially relevant records that should be included in a review.
  • This tool automates this process by making use of the Lens.org API. An input article list can be used to return a list of all referenced records, and/or all citing records in the Lens.org database (consisting of PubMed, PubMed Central, CrossRef, Microsoft Academic Graph and CORE); read more here.
More information on Citation tools including tutorials on how use them can be found here: https://apu.libguides.com/citations

Types of Searches

There are four kinds of searching that go into a systematic and scoping reviews.

Preliminary Search – This is the kind of searching most people do when they start thinking about conducting a systematic review. The results from a preliminary search are not exhaustive, and should not be used as the sole source of data for your systematic review. The goals of the preliminary search include identifying existing reviews, assessing number of potentially relevant studies (assume an exhaustive search will identify about 2-3 times the number located in a preliminary search), and locating at least 2-5 example articles that meet your review criteria. Use Cinahl & Medline to begin your preliminary search.

Exhaustive Database and Grey Literature Search – This is the search designed by a librarian trained on how to design searches for systematic reviews. One goal of an exhaustive search is to Identify all publications and as much grey literature as possible that meet study requirements. Another goal is to document and report the exhaustive search in such a way that it can be replicated for updates and reproduced by others after publication. For a librarian designed systematic review search fill out the Systematic Review Search Request form. 

Hand Search - Identify grey literature like conference proceedings, abstracts for posters, and presented papers not indexed in online databases. Sources to hand search include subject-specific professional association websites, major relevant journals, bibliographies of all included studies, and bibliographies of on-topic reviews. It is best for the subject experts to do the hand-searching since they are most likely to have access to conference archives on professional society websites. It is important to document all sources searched by hand and what was located for reporting and creating the PRISMA Flow Diagram.

Contact Experts – The goal of contacting experts is to determine if more than one paper has been published on the same study and to identify unregistered studies with unpublished results or potential results. 

Eligibility Criteria

In order to reduce bias, eligibility criteria (also known as inclusion and exclusion criteria) refer to what you plan to include and exclude from your scoping/systematic review. These criteria are decided after the research question is developed but before searches are completed. 

To identify the inclusion and exclusion criteria: 

  • Consider the core components of the primary review questions.
  • Elements of the secondary questions relate to content that is 'nice to have' but not essential, so these do not typically form part of the inclusion criteria.

Below are examples of criteria that may be used to determine inclusion or exclusion.

Type of criteria Example
Type of participants May be limited to specific groups of people or age ranges
Study design May include specific study designs and exclude others based on which best answer the research question
Intervention of interest Includes interventions of interest and excludes any others
Outcomes of interest Includes outcomes of interest and may exclude studies reporting outcomes not of interest
Setting May be limited to a specific setting like inpatient, ambulatory, classroom, etc.
Type of publication Reviews, editorials, commentaries, and letters are often excluded
Date of publication Date ranges may be applied when updating a systematic review or when specific to an intervention or therapy
Language of publication Non-English articles are often excluded from reviews; however, this may allow language bias to affect the quality of your review
Note: your inclusion criteria may necessitate changes to the review objective and questions!

 

For example:

The field of nurse-led care in chronic disease is too broad for the allocated time, so the reviewer decides to limit the search to higher-income countries only. She rationalizes health care in low to middle-income countries can be affected by contextual factors, and these might act as confounders. This:

  • adds an additional inclusion criterion of high income countries, and consequently
  • changes the review topic and objective to: models of care in chronic disease in high-income countries.