Systematic Reviews, Meta-Analysis & Forest Plots
How pooled evidence is built, drawn, and distorted
Systematic Reviews and Meta-Analysis
Systematic review. A systematic review answers one focused question, usually framed as PICO (population, intervention, comparator, outcome), with methods fixed in advance and often registered (for example, in PROSPERO). Its defining steps are an exhaustive search of several databases and unpublished sources, explicit eligibility criteria, screening and data extraction by two independent reviewers, and a formal risk-of-bias appraisal of every included study (such as the Cochrane tool for randomized trials or the Newcastle-Ottawa scale for observational studies). Reporting follows PRISMA, whose flow diagram tracks records identified, screened, excluded with reasons, and included. A traditional narrative review has none of these safeguards, so its author can cite only the studies that fit a preferred conclusion.
Meta-analysis. The optional statistical step of a systematic review. Study effect sizes are combined as a weighted average, usually with inverse-variance weights, so large and precise studies contribute most. Pooling raises power and precision: several underpowered, nonsignificant trials can together yield a significant estimate with a narrow confidence interval. When studies are too different to combine, the review stops at a narrative synthesis.
A meta-analysis is only as valid as the studies inside it. Pooling shrinks random error but never systematic error, so biased trials or confounded observational studies produce a narrow CI around the wrong answer. Observational data suggesting that postmenopausal hormone therapy protected the heart, later not confirmed by a large randomized trial, is the classic cautionary example.
Reading a Forest Plot
| Element | What it shows | Reading tip |
|---|---|---|
| Square | A study's point estimate; area ∝ its weight | Big squares (large, precise studies) dominate the pooled result |
| Horizontal whiskers | That study's 95% CI | Long whiskers = small or imprecise study |
| Vertical line | No effect: 1 for RR, OR, HR; 0 for mean or risk difference | A CI touching or crossing it is not significant |
| Diamond | Pooled estimate; center = point estimate, width = pooled CI | Significant only if the whole diamond clears the line |
| Axis labels | "Favors treatment" vs "favors control" | Show which side represents benefit for this outcome |
Scale. Ratio measures are plotted on a logarithmic axis, so a halving (0.5) and a doubling (2.0) lie the same distance from 1. Difference measures use a linear axis centered on 0.
Direction. For an adverse outcome (death, MI, stroke), a ratio below 1 favors treatment; for a desirable outcome (remission, wound healing), a ratio above 1 does. Read the axis labels before deciding which side is "good."
Numbers. A pooled HR of 0.75 (95% CI 0.64–0.88) is a 25% relative reduction in the hazard and is significant because the interval excludes 1; absolute benefit still depends on baseline risk. Individual rows may cross the line while the diamond does not, which is exactly the power that pooling adds.
- Identify the measureRatio (null = 1) or difference (null = 0)
- Read the axis labelsWhich side favors treatment for this outcome
- Scan the study rowsSquare sizes, spread, and overlap of the CIs
- Read the diamondPooled estimate and whether its full width clears the line
- Check heterogeneityCochran Q p-value and I² printed beneath the plot
Heterogeneity and Choice of Model
Definition. Heterogeneity means true effects differ among studies rather than varying only by chance, usually because of different populations, doses, durations, outcome definitions, or study quality. On the plot, estimates scatter to both sides of the line and CIs overlap poorly.
Cochran Q. A chi-square test of whether between-study variation exceeds chance. It has low power with few studies, so p < 0.10 is often used as the cutoff, and a nonsignificant Q does not prove homogeneity.
I². The percentage of total variability across studies due to true heterogeneity rather than chance: roughly 25% low, 50% moderate, 75% or more high; above 50% is usually called substantial.
| Feature | Fixed-effect | Random-effects |
|---|---|---|
| Assumption | One true effect shared by all studies | True effects vary around an average |
| Variance included | Within-study (sampling) error only | Within-study plus between-study variance (τ²) |
| Weighting | Large trials dominate | More even; small trials gain relative weight |
| Pooled CI | Narrower | Wider, more conservative |
| Best suited to | Low heterogeneity | Moderate to substantial heterogeneity |
Responding to heterogeneity. Besides using a random-effects model, investigators probe its sources with prespecified subgroup analyses (dose, age, baseline risk, risk of bias) or meta-regression, and run sensitivity analyses that drop high-risk-of-bias or outlying trials, or swap models, to test whether the conclusion holds. Subgroup differences are hypothesis-generating. If heterogeneity is extreme and unexplained, results are better presented without a single pooled number.
A random-effects model widens the CI but does not explain heterogeneity. Because it gives small trials relatively more weight, it can amplify bias when the small trials are the flawed ones.
Publication Bias and Funnel Plots
Mechanism. Positive, statistically significant studies are more likely to be published, published quickly, and published in English; small null or negative studies stay in the "file drawer." A meta-analysis of published trials therefore overestimates benefit. Selective outcome reporting, in which a trial publishes only its favorable endpoints, distorts results the same way.
Funnel plot. Each study's effect size (x-axis) is plotted against its precision, usually the standard error (y-axis, most precise studies at the top). Large studies cluster near the true effect and small ones scatter widely at the base, forming a symmetric inverted funnel. An empty lower corner, where small studies showing no benefit or harm belong, makes the plot asymmetric and suggests publication bias; weaker methods in small trials or genuine heterogeneity can also cause asymmetry. Egger's regression test quantifies it, and trim-and-fill estimates the pooled effect after imputing the missing studies.
Prevention. Prospective trial registration (for example, on ClinicalTrials.gov) makes unpublished trials and switched endpoints traceable; ICMJE member journals consider a trial only if it was registered before the first participant enrolled. Reviewers also search gray literature (conference abstracts, dissertations, regulatory documents such as FDA reviews, registry results) and request unpublished data from investigators.
Strength of Evidence
Evidence hierarchy. Systematic reviews and meta-analyses of well-conducted RCTs sit at the top, above single RCTs, cohort and case-control studies, case series, and expert opinion. A meta-analysis of observational studies inherits their confounding.
Individual-patient-data (IPD) meta-analysis. Investigators reanalyze the raw participant-level records from every trial. This permits uniform outcome definitions, consistent adjustment, time-to-event analysis, and credible subgroup analyses (for example, by age or sex) that published aggregate data cannot support, making it the most rigorous form of evidence synthesis.
Rating certainty. Frameworks such as GRADE downgrade a body of evidence for risk of bias, inconsistency (unexplained heterogeneity), indirectness, imprecision (wide CIs), and publication bias, the same weaknesses described above.