A critical appraisal template for Notion that you can defend

See inside the Critical Appraisal Studio

The problem it solves

"How did you judge study quality?" "I read each paper carefully" is not an answer a supervisor or examiner will accept. Neither is a score out of ten.

Summed quality scores are a known trap. When Jüni et al. (1999) applied 25 different quality scales to the same set of trials, the conclusion about whether quality mattered changed with the scale. A total hides which flaw matters: a paper can score 8 out of 10 and still have unadjusted confounding that undermines its main result.

The other common problems are drift and choice. Students appraise the first ten papers one way and the last ten another, and they are unsure which checklist fits which design: CASP, JBI, RoB 2 or ROBINS-I. And once a paper has problems, they still have to decide whether to rely on it, use it with caveats or set it aside.

This studio gives you one consistent way to record those judgements, with the evidence behind each one.

How it works: Judge → Decide → Profile

1. Judge

   Rate a paper one domain at a time, for example confounding, missing data or reflexivity, on a single scale: Low concerns, Some concerns, High concerns or Unclear. Quote the sentence, with page or section, that supports each judgement.

2. Decide

   A transparent worst-domain rule, the logic behind RoB 2 and ROBINS-I, suggests an overall judgement. You make the final call, explain any departure from the rule, and write what it means for how you will use the paper.

3. Profile

   See which domains are weak across your whole literature. That is the limitations paragraph your review chapter needs, written from evidence rather than impression.

What the studio checks for you

Every paper, every judgement and every domain carries a status worked out from what you enter. These are the main labels.

Suggested overall, per paper

• ⚪ Not appraised yet: no domain judged.
• ⚠️ Set the design first: judgements exist, but the paper's design is blank.
• ⏳ Incomplete: 3 of 6: some domains are still to judge, and none is High so far.
• 🔴 High concerns: any domain is High. If domains are still open, it says how many.
• 🟠 Some concerns: find the missing information: no High, but at least one domain is Unclear. Missing information is not reassurance.
• 🟡 Some concerns: no High or Unclear, at least one Some concerns.
• 🟢 Low concerns: every domain Low.

Checks on your decisions

• ⚠️ Quote the paper (page or section): a domain judgement has no real supporting text.
• ⚠️ Domain does not fit this design: for example a qualitative domain on a cohort study.
• ⚠️ Explain why you departed from the rule: your final judgement differs from the rule with no reason given.
• ⚠️ Relying on a high-concern paper: justify it in the note: a High concerns paper is marked Rely on it without a written justification.
• Consider: several concerns together may justify High: three or more Some or Unclear domains. Your call; record why.

Profile across review, per domain

• 🔴 Weak spot in this literature: High in at least half the papers judged on it.
• 🟠 Recurring concern: not Low in at least half.
• 🟡 Occasional concern: not Low in at least one.
• 🟢 Sound across papers: Low everywhere it was judged.

The thresholds (half the papers, three concerns) and the use of the worst-domain rule for qualitative studies and reviews are GradSummit conventions, labelled as such in the studio. The studio never shows a total or a percentage.

The worked example

Fictional study

The worked example uses the fictional Brookfield study: 8 papers on feedback timing and course completion, of six designs (a randomised trial, cohorts, a case-control study, a cross-sectional survey, two qualitative studies and a systematic review), with 42 domain judgements. Paper H is deliberately unfinished, so you can see the prompts. The profile shows one weak spot: how the exposure, feedback timing, was measured. That is the sentence a Brookfield review chapter would need. Clearing the example takes a few minutes.

Method and sources

Domain-based judgement follows Sterne et al. (2019) for RoB 2 and Sterne et al. (2016) for ROBINS-I: any High domain makes the overall judgement High, and several concerns together may justify High at the reviewer's discretion. The case against summed scores follows Jüni et al. (1999). AMSTAR 2 (Shea et al., 2017) is noted as rating confidence from critical weaknesses, not a score.

The domain bank holds 18 domains for randomised, non-randomised, qualitative and review designs, written in GradSummit's own words with questions to ask and what a High looks like. Each paper shows the official tool for its design: RoB 2, ROBINS-I or ROBINS-E, JBI, CASP or AMSTAR 2. The method page links to each.

What the studio does not do

• It is not the official RoB 2, ROBINS-I, CASP, JBI or AMSTAR 2 tool, and it reproduces none of their checklists. It is not affiliated with Cochrane, CASP, JBI or AMSTAR.
• For a formal systematic review, use the official tool, follow its guidance, and have two reviewers appraise each paper independently.
• It does not judge your papers for you. The rule suggests; you decide and record why.
• It does not rate the certainty of a body of evidence, as GRADE does.

What you get

• A one-page PDF with your private link, sent by Gumroad straight after checkout. Click it, then Duplicate, and the studio is in your own Notion workspace, yours to keep.
• Simple mode: three steps, one small table each, with a plain instruction above every table. You can start in Notion for the first time today.
• A papers table, domain judgements grouped by paper, a quality profile of your review, a board of papers by final judgement, and the 18-domain bank.
• Export views that turn your appraisal table, judgements and quality profile into clean CSV tables for an appendix.
• A Start here page, a method guide with full references, and a worked fictional example you can clear in a few minutes.
• Works on Notion's Free plan. No charts, no paid features and no Notion AI needed.

Related guides and templates

• How to critically evaluate a research article

  a reading checklist

• Risk of bias and quality appraisal

  a plain-English guide to the tools

• How to do a systematic review

  where formal appraisal fits

• Systematic Review & Meta-Analysis Protocol Workbook

  plan how you will assess risk of bias

Other research studios

• Literature Synthesis Matrix

  Turn paper summaries into a real synthesis, with a gap meter. $12.

• Scoping & Systematic Review Manager

  A PRISMA flow diagram that adds itself up. $15.

• Grounded Theory Studio

  Saturation by properties, theoretical sampling and a core category check. $15.