Risk of bias & quality appraisal
Finding all the studies isn’t enough — you have to judge how much to trust each one. Risk of bias is the degree to which flaws in a study’s design or conduct could have distorted its result. Assessing it for every included study is what stops a weak study from carrying the same weight as a strong one in your synthesis. It’s a required, two-reviewer step in any systematic review.
Risk of bias vs “quality”
These aren’t the same thing. “Quality” is a broad notion that can include how completely a study was reported or how applicable it is. Risk of bias is narrower and sharper: did the design and conduct threaten the validity of the effect estimate? A beautifully written paper can still be at high risk of bias. Modern tools deliberately assess risk of bias rather than awarding generic quality scores, which were easy to game and hard to interpret.
Match the tool to the design
| Tool | For |
|---|---|
| RoB 2 | Randomised controlled trials |
| ROBINS-I | Non-randomised studies of interventions |
| Newcastle–Ottawa Scale | Cohort & case-control studies |
| CASP checklists | A range of designs, including qualitative |
Use the tool your protocol specifies, applied independently by two reviewers. Most tools assess bias across domains — e.g. RoB 2 covers the randomisation process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported result — and you reach a domain-level and overall judgement (low / some concerns / high).
From single studies to the whole body: GRADE
Risk of bias is judged per study. GRADE goes one level up and rates the certainty of the body of evidence for each outcome — high, moderate, low, or very low. It starts from the study design and can downgrade for risk of bias, inconsistency, indirectness, imprecision, and publication bias (or upgrade for a large effect). This is how your review tells a reader how much confidence to place in each conclusion — and it feeds directly into the synthesis.
Get the free Systematic Review toolkit
A risk-of-bias summary template, a data-extraction form, and a protocol starter from The Dissertation Literature Review Sprint. We’ll email you the download link.
Frequently asked questions
What is risk of bias?
The degree to which a study’s design or conduct could have distorted its result — assessed per included study to weight trust.
Risk of bias vs quality?
Quality is broad (incl. reporting); risk of bias is specifically about threats to the validity of the effect estimate.
Which tool?
RoB 2 for RCTs, ROBINS-I for non-randomised interventions, Newcastle–Ottawa for cohort/case-control, CASP for mixed/qualitative.
What is GRADE?
A system rating the certainty of the whole body of evidence per outcome — high to very low — across five downgrading domains.