Skip to Main Content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Systematic Reviews: Developing the review protocol

Additional considerations

Developing a well considered review protocol provides a sound foundation for the development of a well-defined and rigorous search strategy. The protocol provides the parameters of what, where and how literature will be searched contributing greatly to the review's reliability and accuracy. Decisions around the type of literature to include, the parameters to apply to the review as well as how bias will be assessed all need to be considered.

Inclusion and exclusion criteria

Eligibility criteria

You are required to set out your review's parameters/boundaries to ensure the focus of your research question is clear. The aspects of your research you wish to focus on will become your inclusion criteria, and the aspects of your research topic that you decide not to focus on will become your exclusion criteria. The selection of these criteria are based firmly on the research question as well as the framework used to define the question. Once you have decided on your criteria, you will then apply them to decide which studies you are going to include and which studies you are not going to include.  

How to determine eligibility criteria using the PICO framework

It is important to ensure you detail the criteria you have selected, to ensure the reader of your review clearly understands what the focus of your review is and what its limitations might be. Information of the inclusion and exclusion criteria are normally detailed in the Methods section of the review.

There are many factors that, depending on the research question, can be used as inclusion and exclusion criteria. To effectively determine the criteria you are going to use, you need to fully understand each part of your question - this is where using a framework, such as PICO can help.

More information can be found at: 

Cochrane Library https://community.cochrane.org/mecir-manual/standards-conduct-new-cochrane-intervention-reviews-c1-c75/developing-protocol-review-c1-c23/setting-eligibility-criteria-including-studies-review-c5-c13
University of Melbourne  https://unimelb.libguides.com/c.php?g=492361&p=3368110
City, University of London  https://libguides.city.ac.uk/postgraduate_research/criteria

PICO question 

So, what do you need to consider when deciding on the inclusion and exclusion criteria? In the section below, we will take an example research question, based on the PICO framework, and consider the sort of questions you might wish to ask yourself when setting out the study's parameters. 

Example research PICO question

In treating adults with a sinus infection, what is the effect of steam inhalation on reducing sinus pain compared with the effect of saline rinsing?  

PICO framework Research question
Population adults with a sinus infection
Intervention steam inhalation
Comparator saline rinsing
Outcome reduction in sinus pain

Example questions/criteria to be considered:

  • How will the condition be defined?
  • How will the condition have been diagnosed?
  • Will all adults be included or will only adults of a specific date range be focused on?
  • Will there be groups of people excluded from the review (for example, those with other conditions, such as those with asthma)?
  • Will there be any other demographic factors to be considered? 
  • Within the studies, will the features of the interventions be sufficiently similar to be comparable? 
  • How will pain levels be measured?
  • What study designs will be prioritised?
  • Will other factors, such as language or date range need to be considered? 

Adapted from: https://libguides.ucmerced.edu/systematic-reviews/question

Study types

Hierarchy of evidence

A system of grading different types of studies has been devised based their methodological design and on their usefulness to clinical decision-making. The hierarchy is often displayed as a pyramid. The highest quality study type is seen to be systematic reviews and meta-analyses as the studies included  within have been subjected to a rigorous search and selection processes defined by eligibility criteria and critical appraisal. 

Source:

Evidence-based medicine (EBM) pyramid

                                   

Source: "File:Evidence-based Medicine (EBM).png" by Tingjoh is licensed under CC BY-SA 4.0

The evidence-based medicine (EBM) pyramid categories evidence-based studies on the quality of the evidence they produce. The different types are then positioned in ascending order with the highest level of study types, systematic reviews and meta-analyses, at the top. Systematic reviews, meta-analyses and critically-appraised topics/articles are considered high quality evidence types as they have all been subjected to an evaluation process. The information they contain is sometimes known as filtered information. Any study types positioned below the 'filtered' studies, have not been through an evaluation/critical appraisal process and so are deemed to produce less quality, 'unfiltered' information. 

As the shape of the pyramid suggests, there are likely to be less filtered studies available for your research topic. If you find there are very few studies, or possibly no studies of the higher level study types, you will need to consider moving down the pyramid to include unfiltered information.

 

Adapted from:  https://libguides.winona.edu/c.phpg=11614&p=61584#:~:text=Levels%20of%20

evidence%20(sometimes%20called,or%20strength)%20of%20recommendation.%22

Types of evidence

Within the EBM pyramid, there are a number of different types of research studies identified within it. The table below gives a a brief overview of the different types.

Filtered sources  
Systematic Reviews and Meta-Analyses

Systematic Review:
It seeks to review, in a transparent and repeatable way, all literature that is relevant to answering a specific research question. It uses established methods to identify, select, critically appraise and synthesise the data from the selected studies.

Meta-analysis:
Reviews quantitative studies that are focused on the same topic area and have the same outcome measures, using an established statistical methodology.

Critically Appraised Topics and Articles

Critically Appraised Topic (CAT):
This is a summary of the best available research evidence on a specific topic that has gone through a critical appraisal process. Usually, a CAT is focused on answering a specificclinical question.  

Critically Appraised Article:
whereas a CAT appraises and synthesises the evidence from multiple studies, a critically appraised article focusses on the analysis, and appraisal, of one individual research study.

Unfiltered sources  
Randomised Control Trials (RCTs) RCTs study the effect of a treatment on a group of patients. The methods employed by a RCT look to reduce any potential bias within the trial. This may result in the inclusion of a control group as a comparator for the patient group receiving the treatment.
Cohort Studies These studies focus on large groups (cohorts). They identify two cohorts with the members of one cohort one sharing something in common, such as a particular condition or the same treatment. They are compared with another group who do not have that common aspect shared by the first group. These studies are then studied for a period of time. 
Case-Controlled Studies and Case Series/Reports

Case-Controlled Study:
This is a study where one group of people (the case group) with the outcome of interest, such as a 
particular disease/condition, are compared with a group of people (the control group) that do have that outcome of interest.  It is retrospective in design as it starts with the case group already having the outcome being studied.  It then traces back in time to identify exposures, and frequency of those exposures, experienced by the case group when compared to the control group. 

Case Series/Report:
A report focuses on the treatment of a patient that for some reason is identified as having a particular condition, or set of characteristics, that are of particular interest. When there are a few case reports, this becomes a case series. 

Background Information/Expert Opinion

Background information:

Background information can be found in sources, such as textbooks and encyclopedias, and provide a general summary (although it may have taken several years to have been published).

Expert opinion:

An expert opinion is based on a person’s own knowledge and experience. It can be influenced by the expert’s opinions, or viewpoints, and can be based on anecdotal rather than research-based evidence.

Adapted from: https://libguides.winona.edu/c.php?g=11614&p=61584

Considering bias

Why does bias need to be considered?
When developing the protocol, it is important to consider how the review will assess the risk of bias.. Bias can occur within studies where certain factors affect its results and conclusions. This can result in the study being inaccurate which, in turn, can lead to the publication of inaccurate recommendations/guidelines. 


The Cochrane Handbook defines bias as:
“…a systematic error, or deviation from the truth, in results. Biases can lead to under-estimation or over-estimation of the true intervention effect…” (section 7.1 Introduction)
A study that has been able to effectively minimise bias is said to have internal validity.
*Internal validity is not to be confused with external validity which is said of a study whose results are able to be applied to other situations outwith the parameters of the original study*

Types of bias

There are different types of bias, for example:
Publication bias
Selection bias  
Reporting bias


An explanation, and discussion, of these and other types of bias can be found at:
Chapter 7 of the Cochrane Handbook
Catalogue of Bias
NHMRC’s Assessing risk of bias guidelines
TedTalk video found on this web page within the section, Video on bias in medical research.

Assessment tools

An important component of the protocol is to include an assessment of the risk of bias for all the included studies.

To make sure the assessment is as objective, consistent and as thorough as possible, it is advised that an established assessment tool is used. There are many published and established tools available. Depending on the types of studies included, more than one assessment might need to be used. 
For more information about these crtitical appraisal assessment tools, please refer to the Critical Appraisal libguide.  

Video on bias in medical research

Methodological search filters

Search filters (sometimes known as hedges) are pre-set (and tested) search strategies that have been designed to uncover sources based by study type or by a particular focus. Search filters are commonly used in systematic reviews to include only randomised control trials as they are considered the best type of research for evidence-based medicine.

There are filters available for other methodological designs, such as for observational studies or diagnostic studies. Search filters can also be designed to improve sensitivity (increasing the comprehensiveness of the search but also increasing the number of irrelevant results) or to improve precision (more focussed thus reducing the number of irrelevant records but increasing the risk of missing relevant results).

In addition, the search filters have been adapted to suit specific databases with CINAHL, EMBASE, MEDLINE and PsycINFO being common inclusions.  As with your own search strategy, the search filters consist of free text and thesaurus terms, with functions included to instruct where, and how, the terms should be applied in the search. 

Some examples of search filters, designed for specific study types and databases can be found at:

Cochrane Handbook - section 4.4.7 and the Technical Supplement - section 3.6.1

SIGN (Scottish Intercollegiate Guidelines Network) Search filters

McMaster University Hedges (search filter) Project

InterTASC Information Specialists' Sub-Group (ISSG) Search Filter Resource
ISSG is a collaborative group that looks to identify and test search filters

PubMed Clinical Queries
Designed specifically for retrieving clinical and diseased-based subject areas. 

You can copy and paste search filters into your own search and save them along with your own searches. By saving them, you can re-run them for another search. As with your own searches, you will need to combine them using the AND operator (please see this guide's Search strategy tab for more information about database searching).