Developing a well considered review protocol provides a sound foundation for the development of a well-defined and rigorous search strategy. The protocol provides the parameters of what, where and how literature will be searched contributing greatly to the review's reliability and accuracy. Decisions around the type of literature to include, the parameters to apply to the review as well as how bias will be assessed all need to be considered.
EBOOK: How to Do a Systematic Literature Review in Nursing: a Step-By-step Guide by Eligibility criteria
You are required to set out your review's parameters/boundaries to ensure the focus of your research question is clear. The aspects of your research you wish to focus on will become your inclusion criteria, and the aspects of your research topic that you decide not to focus on will become your exclusion criteria. The selection of these criteria are based firmly on the research question as well as the framework used to define the question. Once you have decided on your criteria, you will then apply them to decide which studies you are going to include and which studies you are not going to include.
How to determine eligibility criteria using the PICO framework
It is important to ensure you detail the criteria you have selected, to ensure the reader of your review clearly understands what the focus of your review is and what its limitations might be. Information of the inclusion and exclusion criteria are normally detailed in the Methods section of the review.
There are many factors that, depending on the research question, can be used as inclusion and exclusion criteria. To effectively determine the criteria you are going to use, you need to fully understand each part of your question - this is where using a framework, such as PICO can help.
More information can be found at:
PICO question
So, what do you need to consider when deciding on the inclusion and exclusion criteria? In the section below, we will take an example research question, based on the PICO framework, and consider the sort of questions you might wish to ask yourself when setting out the study's parameters.
Example research PICO question
In treating adults with a sinus infection, what is the effect of steam inhalation on reducing sinus pain compared with the effect of saline rinsing?
| PICO framework | Research question |
| Population | adults with a sinus infection |
| Intervention | steam inhalation |
| Comparator | saline rinsing |
| Outcome | reduction in sinus pain |
Example questions/criteria to be considered:
Adapted from: https://libguides.ucmerced.edu/systematic-reviews/question
Hierarchy of evidence
A system of grading different types of studies has been devised based their methodological design and on their usefulness to clinical decision-making. The hierarchy is often displayed as a pyramid. The highest quality study type is seen to be systematic reviews and meta-analyses as the studies included within have been subjected to a rigorous search and selection processes defined by eligibility criteria and critical appraisal.
Source:
Evidence-based medicine (EBM) pyramid
Source: "File:Evidence-based Medicine (EBM).png" by Tingjoh is licensed under CC BY-SA 4.0
The evidence-based medicine (EBM) pyramid categories evidence-based studies on the quality of the evidence they produce. The different types are then positioned in ascending order with the highest level of study types, systematic reviews and meta-analyses, at the top. Systematic reviews, meta-analyses and critically-appraised topics/articles are considered high quality evidence types as they have all been subjected to an evaluation process. The information they contain is sometimes known as filtered information. Any study types positioned below the 'filtered' studies, have not been through an evaluation/critical appraisal process and so are deemed to produce less quality, 'unfiltered' information.
As the shape of the pyramid suggests, there are likely to be less filtered studies available for your research topic. If you find there are very few studies, or possibly no studies of the higher level study types, you will need to consider moving down the pyramid to include unfiltered information.
Adapted from: https://libguides.winona.edu/c.phpg=11614&p=61584#:~:text=Levels%20of%20
evidence%20(sometimes%20called,or%20strength)%20of%20recommendation.%22
Types of evidence
Within the EBM pyramid, there are a number of different types of research studies identified within it. The table below gives a a brief overview of the different types.
| Filtered sources | |
| Systematic Reviews and Meta-Analyses |
Systematic Review: Meta-analysis: |
| Critically Appraised Topics and Articles |
Critically Appraised Topic (CAT): Critically Appraised Article: |
| Unfiltered sources | |
| Randomised Control Trials (RCTs) | RCTs study the effect of a treatment on a group of patients. The methods employed by a RCT look to reduce any potential bias within the trial. This may result in the inclusion of a control group as a comparator for the patient group receiving the treatment. |
| Cohort Studies | These studies focus on large groups (cohorts). They identify two cohorts with the members of one cohort one sharing something in common, such as a particular condition or the same treatment. They are compared with another group who do not have that common aspect shared by the first group. These studies are then studied for a period of time. |
| Case-Controlled Studies and Case Series/Reports |
Case-Controlled Study: Case Series/Report: |
| Background Information/Expert Opinion |
Background information: Background information can be found in sources, such as textbooks and encyclopedias, and provide a general summary (although it may have taken several years to have been published). Expert opinion: An expert opinion is based on a person’s own knowledge and experience. It can be influenced by the expert’s opinions, or viewpoints, and can be based on anecdotal rather than research-based evidence. |
Adapted from: https://libguides.winona.edu/c.php?g=11614&p=61584
Why does bias need to be considered?
When developing the protocol, it is important to consider how the review will assess the risk of bias.. Bias can occur within studies where certain factors affect its results and conclusions. This can result in the study being inaccurate which, in turn, can lead to the publication of inaccurate recommendations/guidelines.
The Cochrane Handbook defines bias as:
“…a systematic error, or deviation from the truth, in results. Biases can lead to under-estimation or over-estimation of the true intervention effect…” (section 7.1 Introduction)
A study that has been able to effectively minimise bias is said to have internal validity.
*Internal validity is not to be confused with external validity which is said of a study whose results are able to be applied to other situations outwith the parameters of the original study*
Types of bias
There are different types of bias, for example:
Publication bias
Selection bias
Reporting bias
An explanation, and discussion, of these and other types of bias can be found at:
Chapter 7 of the Cochrane Handbook
Catalogue of Bias
NHMRC’s Assessing risk of bias guidelines
TedTalk video found on this web page within the section, Video on bias in medical research.
Assessment tools
An important component of the protocol is to include an assessment of the risk of bias for all the included studies.
To make sure the assessment is as objective, consistent and as thorough as possible, it is advised that an established assessment tool is used. There are many published and established tools available. Depending on the types of studies included, more than one assessment might need to be used.
For more information about these crtitical appraisal assessment tools, please refer to the Critical Appraisal libguide.
Search filters (sometimes known as hedges) are pre-set (and tested) search strategies that have been designed to uncover sources based by study type or by a particular focus. Search filters are commonly used in systematic reviews to include only randomised control trials as they are considered the best type of research for evidence-based medicine.
There are filters available for other methodological designs, such as for observational studies or diagnostic studies. Search filters can also be designed to improve sensitivity (increasing the comprehensiveness of the search but also increasing the number of irrelevant results) or to improve precision (more focussed thus reducing the number of irrelevant records but increasing the risk of missing relevant results).
In addition, the search filters have been adapted to suit specific databases with CINAHL, EMBASE, MEDLINE and PsycINFO being common inclusions. As with your own search strategy, the search filters consist of free text and thesaurus terms, with functions included to instruct where, and how, the terms should be applied in the search.
Some examples of search filters, designed for specific study types and databases can be found at:
CADTH Search Filters Database Canadian Agency for Drugs and Technologies in Health Filters include: systematic reviews/meta-analysis/health technology assessment; randomised controlled trials; guidelines; economic evaluations/cost/economic models; health utilities/quality of life; narrow economic filter
Cochrane Handbook - section 4.4.7 and the Technical Supplement - section 3.6.1
Flinders Filters (Flinders University) (including CareSearch filters) Includes: Integrated care; Bereavement, Grief and Loss; Contraception; Dementia; Opthalmology; Heart Failure; Aboriginal and Torres Strait Islander Health; Lung Cancer; Palliative Care; Primary Health Care; Residential Aged Care; Stroke
InterTASC Information Specialists' Sub-Group (ISSG) Search Filter Resource ISSG is a collaborative group that looks to identify and test search filters
McMaster University Hedges (search filter) Project
Methodological Search Filters - CEBMa (Center for Evidence-Based Management) Filters include: meta-analyses and systematic reviews; controlled and/or longitudinal studies
PubMed Clinical Queries Designed specifically for retrieving clinical and diseased-based subject areas. You can copy and paste search filters into your own search and save them along with your own searches. By saving them, you can re-run them for another search. As with your own searches, you will need to combine them using the AND operator (please see this guide's Search strategy tab for more information about database searching).
SIGN (Scottish Intercollegiate Guidelines Network) Search filters
Topic search filters: a systematic scoping review (R. Damarell et al.) This scoping review identifies 58 filters across a range of topics.
