Calibration activity
90 minutes, before first observations, then twice per semester · For site coordinators and anyone who scores tutors with the rubric
Why calibrate
The measurement plan tracks rubric averages and growth (+0.5 between observations 1 and 3) — numbers that only mean something if the people producing them score the same session the same way. Inter-rater drift is the quiet killer of observation systems: one generous scorer and one severe scorer turn a program average into noise. Calibration is the fix, and it is cheap: score the same fictional transcript, compare, argue the gaps, converge.
The activity (90 minutes)
Materials: the observation rubric, printed; three calibration transcripts — T1, T3, T5 — chosen because they anchor different ends and mixtures (T1 ≈ strong across domains; T3 = mixed, split scoring; T5 = the weak end); independent scoring sheets.
Round 1 · Independent scoring (30 min). Everyone scores T1 silently, alone: five domain scores + one evidence line each. No talking. Collect the sheets before any discussion — comparison needs independent judgments, not agreed ones.
Round 2 · Compare (20 min). Post all scores for T1. For each domain: range? If Domain 3 ranges 2–4, the discussion is the activity:
Scorer A: “I gave a 4 — every problem opened with a question.” Scorer B: “I gave a 3 — she explained first on the second problem.” Facilitator: “Read the level-4 anchor out loud. ‘Every problem opens with a probing question.’ Is ‘every’ met?” Scorer A: “…no. Three of five. That’s a 3.”
The transcript annotations (numbered after each exchange) are the tiebreaker: point at the tick that settles it. Anchors beat seniority — the most experienced voice doesn’t win; the anchor text does.
Round 3 · The hard one (20 min). Score T5 (weak end). The drift that matters most is severity: new scorers round up for warmth (“she was so prepared!”). Anchor discipline: the rubric scores student production, not tutor effort. If Domain 3 comes out ≥2 for T5, re-read annotation 1: ninety seconds, zero questions, students produce nothing. The correct score is 1 — feel the difference; that’s the calibration.
Round 4 · Split scoring (20 min). T3 exists to prove the rubric scores domains separately: questioning at 4, data use at 1, in the same session. Anyone who “averages T3 to a 3” misses the design — and would coach T3’s tutor wrong (the T3 debrief leads with the strength precisely because the weakness is in a different domain). Note: the transcripts’ own “Rubric read” sections were re-scored against the anchors on 2026-10-26 (T3’s Domain 2 = 1 per the exit-ticket cap, Domain 1 = 3; T6’s Domain 2 = 1; T4’s Domain 5 = 2), so the published scores and this protocol now agree — if you and a page ever disagree again, the anchor text wins.
Close (a few minutes): record the agreed scores in the calibration log and set the year’s rules of the house (below).
Rules of the house (established in round 2, kept for the year)
- Lower level wins when between levels — unless the higher behavior is clearly present (rubric rule; calibration makes it habit).
- Count, don’t vibe. “Wait time” is 3–5 counted seconds; “specific feedback” names the work. If you can’t point at the tick, you can’t score it.
- Warmth is not a score. Prepared, kind tutors can score 1–2; sessions are scored on student production ([T5] is the standing reminder).
- Score domains separately, always. No overall averages in the debrief.
- The annotations settle disputes — in calibration and in real debriefs, point at the exchange, not at your experience.
Calibration log (per site)
| Date | Scorers | Transcript | Domains agreed first pass | Domains converged after discussion | Rules clarifications made |
|---|---|---|---|---|---|
| (template row) | T1 / T3 / T5 | /5 | /5 |
The log travels with the measurement plan: “calibration evidence” is part of what makes the rubric-growth metric honest. A program that can’t show scorer agreement can’t claim rubric growth — and this page is where that claim starts.
Fictional program, real routine. Bridge Math Tutoring is fictional; the transcripts are fictional; the inter-rater problem is not. Any program adopting the rubric should adopt the calibration first — the evaluation methods page notes that these materials’ own scoring evidence has the same dependency.