Implementing a Systematic Review Process

TL;DR

A systematic review is a structured way to answer a research question by finding, appraising, and synthesizing all relevant evidence. It's crucial for avoiding bias and ensuring your conclusions are based on solid, comprehensive data. Following a clear, repeatable process ensures your findings are credible and transparent.

1. The Mental Model

Think of a systematic review like building a very sturdy, well-documented house. You need a clear blueprint (protocol), carefully select every brick (studies), inspect them for flaws (appraisal), and then assemble them logically (synthesis) so anyone can see how you built it.

2. The Core Material

Systematic reviews aren't just literature reviews; they're much more rigorous and structured. They aim to minimize bias by following a predefined protocol to identify, select, appraise, and synthesize all relevant evidence on a specific research question. This systematic approach makes findings more reliable and reproducible.

Here's the typical flow:

Defining Your Research Question (PICO)

Photo by Leeloo The First on Pexels

Before you do anything else, you need a crystal-clear research question. A common framework for clinical and intervention-focused questions is PICO:

• Population/Problem: Who are you interested in? (e.g., adults with type 2 diabetes)
• Intervention: What are you looking at? (e.g., mindfulness-based stress reduction)
• Comparison: What are you comparing it to? (e.g., standard care, no intervention)
• Outcome: What effect are you measuring? (e.g., reduction in HbA1c, improved quality of life)

Your question should be specific enough to guide your search strategy, but broad enough to capture all relevant evidence.

Developing a Protocol

Photo by Kampus Production on Pexels

This is your blueprint. A protocol details every step of your systematic review before you start. It outlines your PICO question, inclusion/exclusion criteria, search strategy, data extraction plan, and how you'll assess bias and synthesize data. Registering your protocol (e.g., with PROSPERO) helps prevent selective reporting and duplicate reviews.

Comprehensive Literature Search

Photo by Tima Miroshnichenko on Pexels

You'll search multiple databases (e.g., PubMed, Scopus, Web of Science, Embase) using a combination of keywords, MeSH terms, and Boolean operators (AND, OR, NOT). The goal is to find all potentially relevant studies, not just a few convenient ones. You'll also often check reference lists of included studies ("snowballing") and look for grey literature (e.g., dissertations, technical reports) to ensure thoroughness.

Study Selection

Photo by RDNE Stock project on Pexels

This usually involves two phases, ideally with two independent reviewers to reduce bias:

1. Title/Abstract Screening: Review the titles and abstracts of all studies found in your search against your inclusion/exclusion criteria. Discard clearly irrelevant studies.
2. Full-Text Screening: Retrieve the full text for studies that passed the first screen. Read them thoroughly and apply your criteria again. Document reasons for exclusion.

Data Extraction

For each included study, you'll systematically extract relevant information using a standardized form. This might include: study design, participant characteristics, intervention details, outcome measures, and results. Again, two reviewers often do this independently, then compare and resolve discrepancies.

Quality Appraisal / Risk of Bias Assessment

Not all studies are created equal. You need to assess the methodological quality and risk of bias for each included study. Tools like the Cochrane Risk of Bias tool for randomized controlled trials (RCTs) or Joanna Briggs Institute (JBI) critical appraisal tools help you do this systematically. This assessment informs how much you trust a study's findings and helps explain heterogeneity.

Data Synthesis

Depending on the homogeneity of your studies (how similar they are in terms of population, intervention, outcome, and design), you'll synthesize the data in one of two ways:

• Qualitative Synthesis (Narrative Synthesis): If studies are too diverse for statistical pooling, you describe and summarize the findings thematically or by outcome. You look for patterns, consistencies, and inconsistencies across studies.
• Quantitative Synthesis (Meta-Analysis): If studies are sufficiently similar, you can statistically combine their results to produce a single, precise estimate of an effect. This involves statistical software and expertise.

Here's how the process generally flows:

graph TD
    A["Formulate Research Question (PICO)"] --> B["Develop & Register Protocol"];
    B --> C["Systematic Literature Search"];
    C --> D["Screen Titles & Abstracts (Two Reviewers)"];
    D --> E{Potential Full Texts};
    E --> F["Retrieve Full Texts"];
    F --> G["Screen Full Texts (Two Reviewers)"];
    G --> H{Included Studies};
    H --> I["Data Extraction (Two Reviewers)"];
    I --> J["Quality Appraisal / Risk of Bias"];
    J --> K["Synthesize Data (Narrative or Meta-Analysis)"];
    K --> L["Write & Disseminate Report"];

3. Worked Example

Let's say your PICO question is: "In adults undergoing elective surgery (P), does preoperative carbohydrate loading (I), compared to fasting (C), reduce postoperative length of hospital stay (O)?"

1. Protocol: You'd detail that you'll search PubMed, Embase, and Web of Science for RCTs published in English. You'll include studies with adult surgical patients receiving carbohydrate loading versus standard fasting, measuring hospital stay. Your data extraction form will capture patient demographics, type of surgery, carb loading regimen, and average length of stay with standard deviations. You'll use the Cochrane Risk of Bias tool.
2. Search: You run searches like (carbohydrate loading OR "carb loading") AND (surgery OR surgical OR perioperative) AND ("hospital stay" OR "length of stay").
3. Screening: You get 1500 hits. Two reviewers independently screen titles/abstracts. 1400 are excluded. For the remaining 100, you retrieve full texts. After full-text screening, you exclude 80 more (e.g., not RCTs, wrong population, did not report length of stay). You're left with 20 studies.
4. Data Extraction & Quality Appraisal: For each of the 20 studies, you (and another reviewer) extract the data into your pre-defined form. You also independently assess the risk of bias for each study (e.g., one study might have high risk due to inadequate blinding, another low risk).
5. Synthesis: You find that the 20 studies are mostly RCTs with similar patient groups and outcomes. You decide to perform a meta-analysis. Using statistical software, you combine the results of the 20 studies to calculate an overall effect estimate and its confidence interval, showing that carbohydrate loading significantly reduces hospital stay by 0.5 days (95% CI: 0.3 to 0.7 days).

4. Key Takeaways

• A systematic review starts with a focused, answerable research question, often using the PICO framework.
• A detailed, pre-registered protocol is essential for transparency and reducing bias.
• The literature search must be comprehensive and well-documented across multiple databases.
• Study selection, data extraction, and quality appraisal should ideally involve two independent reviewers to enhance rigor.
• Data synthesis can be qualitative (narrative) or quantitative (meta-analysis), depending on study homogeneity.
• Systematic reviews provide robust evidence to inform practice and policy.
• Thorough documentation of every step is critical for reproducibility.

Common mistakes to avoid:
* No Protocol: Jumping straight into searching without a clear plan leads to disorganized, biased results.
* Incomplete Search: Missing key studies by only searching one database or using limited search terms.
* Single Reviewer: Relying on one person for screening and data extraction introduces significant bias.
* Ignoring Quality Appraisal: Treating all studies as equally reliable, regardless of their methodological flaws.
* Poor Documentation: Not clearly recording decisions, exclusions, or methods, making your review unreproducible.

5. Now Try It

Take a common research question you're interested in, perhaps from your field of study, and try to formulate it using the PICO framework. Then, draft a very brief (2-3 paragraphs) "mini-protocol" for how you'd conduct a systematic review to answer it. Focus on what databases you'd search, what your key inclusion/exclusion criteria would be, and what type of data you'd extract.

What success looks like: Your PICO question is clear and specific. Your mini-protocol outlines a logical, systematic approach for finding and evaluating evidence, showing you understand the key steps involved.