A scale development template for Notion you can defend
Write each item against one domain, let the studio work out the content validity indices from your expert ratings, screen the pilot statistics, and keep or drop every item with a written reason.
For master's and doctoral researchers who need a new questionnaire or scale for their thesis, in nursing, psychology, education, public health and related fields.
- Write → Rate → Decide
- Simple mode
- Works on Notion's Free plan
- Worked example
$15
One-time price. Checkout on Gumroad; an instant PDF with your link to duplicate the studio.
See inside the Scale & Instrument Development Studio
The problem it solves
"Which S-CVI did you report?" "Why did you drop item 14?" "Is an alpha of .89 your validity evidence?" These are the questions examiners ask about a new scale.
Content validity is the first trap. There are two ways to compute a scale-level CVI, and papers often do not say which they used (Polit & Beck, 2006). The rule for an item also changes with the size of the panel: with three to five experts, every expert must rate it relevant (Lynn, 1986).
The second trap is the decision trail. Items get reworded after the panel, dropped after the pilot or split after a cognitive interview, and by the write-up nobody remembers why. A scale you cannot account for item by item is hard to defend.
This studio keeps the arithmetic, the flags and the reasons in one place, item by item.
How it works: Write → Rate → Decide
Write
Draft each item and tie it to one domain of your construct. Wording flags prompt you to check items that may ask two things at once, are negatively worded, or run long.
Rate
Enter how many of your experts rated each item relevant. The studio works out the I-CVI, the chance-adjusted modified kappa and a plain check that applies the rule for your panel size.
Decide
Keep, revise or drop each item with a written reason, then watch the scale-level scores and the pilot screen update.
What the studio checks for you
Each item and each scale version carries a status worked out from what you enter. These are the main labels.
CVI check, per item
- ⚪ Not yet rated: no panel ratings entered.
- ⚠️ Fewer than 3 experts (Lynn, 1986, recommends at least 3): the panel is too small.
- ✅ Full agreement (panel of 3–5): every expert rated it relevant.
- ✏️ Panel of 3–5: needs full agreement (Lynn, 1986): revise or drop: a small panel without full agreement.
- ✅ Meets .78: six or more experts and an I-CVI of .78 or higher.
- ✏️ Below .78: revise or drop (read the comments): a larger panel below the cut-off.
- ☑️ Below the rule: item dropped / 🔁 Below the rule: reworded, re-rate the new wording: you have already acted on a low item.
Wording flags and pilot flags, per item
- ⚠️ and/or: check double-barrelled: the item contains and or or. ⛔ Double-barrelled once you confirm it.
- ↩️ Negative wording · 📖 Reading grade above 8 · 📏 Over 20 words: prompts to check, not verdicts.
- Consider: weak item-total r · Consider: low loading · Consider: cross-loads: pilot values below .30, below .40, or a cross-loading of .32 or more. Otherwise No flag.
- ✅ Reason recorded: a keep, revise or drop decision with a real reason. ⚠️ Give the reason for dropping until you write one.
Per scale version
- ✅ Meets .90 (items not dropped) / ⚠️ Below .90 (items not dropped): the S-CVI/Ave of the items you kept.
- 4 / 5 sources supported · Missing: Consequences of testing: validity evidence counted against the five sources in the testing Standards.
The .78 and full-agreement rules follow Lynn (1986) and Polit & Beck (2006); the .32 cross-loading follows Tabachnick & Fidell (2019). The .30 and .40 pilot cut-offs, grade 8 and 20 words are GradSummit conventions, labelled as such in the studio.
The worked example
Fictional study
The worked example is fictional: a doctoral student developing the Academic Help-Seeking Reluctance Scale for graduate students. It has 4 domains, 16 items, one review round with a six-expert panel, 7 cognitive interview notes, a pilot screen, 7 pieces of validity evidence and 5 decision log entries. Thirteen items meet .78, one dropped item shows ☑️ and two reworded items show 🔁. The kept items reach an S-CVI/Ave of .924, and the evidence count reads 4 / 5 sources supported, with consequences of testing still missing. Clearing the example takes a few minutes.
Method and sources
The studio follows Boateng et al. (2018) for the overall process, Lynn (1986), Polit & Beck (2006) and Polit, Beck & Owen (2007) for content validity and the modified kappa, Willis (2005) and Tourangeau, Rips & Rasinski (2000) for cognitive interviewing, and the AERA, APA & NCME (2014) Standards for the five sources of validity evidence.
A method page explains each index and where its threshold comes from. Reliability is logged beside the validity evidence but not counted as validity, because a high alpha does not show that a scale measures what it claims to (McNeish, 2018).
What the studio does not do
- It does not run factor analysis or reliability analysis. Run them in your statistics software and paste the values in.
- It does not certify that a scale is valid. Validity is an argument you build from evidence.
- The flags are prompts. You decide what to keep, revise or drop, and the studio checks that a reason is written, not that it is good.
- Reading grade has to be pasted in from a readability checker; Notion cannot count syllables.
What you get
- A one-page PDF with your private link, sent by Gumroad straight after checkout. Click it, then Duplicate, and the studio is in your own Notion workspace, yours to keep.
- Simple mode: three steps, one small table each, with a plain instruction above every table. You can start in Notion for the first time today.
- Construct and domains with coverage against target numbers, an expert-panel engine (I-CVI, modified kappa, S-CVI/Ave and S-CVI/UA), a cognitive interviewing log, a pilot item screen, validity evidence by source and a decision log.
- Export views that turn your item bank, interviews, evidence and decisions into clean CSV tables for an appendix.
- A Start here page, a method guide with full references, and a worked fictional example you can clear in a few minutes.
- Works on Notion's Free plan. No charts, no paid features and no Notion AI needed.
Questions
Which S-CVI does it report?
Both. S-CVI/Ave, the average of the item I-CVIs, is shown for all rated items and for the items you kept; S-CVI/UA, the share of items every expert rated relevant, is shown as well. Polit & Beck (2006) recommend saying which one you report, and the method page explains the difference.
Does it work for adapting or translating an existing scale?
Partly. You can mark an item's source as Adapted; adapted items need permission and a citation to the original measure. The panel, interview log, pilot screen and decision log all apply, but translation steps have no table of their own.
Will I get updates?
The link in your PDF points to the published studio, so if it is updated you can duplicate the current version again. Your own copy never changes underneath you, which matters halfway through a thesis.
Does it work on Notion's Free plan?
Yes. The studio uses ordinary databases, relations, rollups and formulas. It needs no charts, no paid Notion features and no Notion AI, so the Free plan is enough.
How do I get the studio after I pay?
Checkout is on Gumroad. Straight after you pay $15, you get a one-page PDF with your private link. Open the link, click Duplicate, and the studio is copied into your own Notion workspace. It is yours to keep.
What if the link does not work or something is wrong?
Digital products cannot be returned once delivered, but if the link does not work or something is wrong with the studio, email rafiq@gradsummit.com and we will fix it or refund you.
Can my research group or class use it?
Each purchase is for one person. For a class or research group, email rafiq@gradsummit.com about a group licence.
Related guides and templates
Other research studios
Scale & Instrument Development Studio
Write → Rate → Decide, with a worked example and a method guide. $15, one-time.
Scale & Instrument Development Studio by Rafiq Muhammad, MD, MIHMEP, PhD (GradSummit). The example study and all its data are fictional and created for teaching.
Notion is a trademark of Notion Labs, Inc. GradSummit is not affiliated with or endorsed by Notion.