Pilot report
Status: PENDING — the pilot has not been run. This page is a prepared template with the protocol, tasks and scoring design already fixed (protocol). When the pilot runs, the report is written into this page and the status label changes. No results, quotes, or numbers appear on this page, and none are claimed anywhere in this repo.
Planned design (fixed in advance — registered before data)
Participants: 3–5 adults who have tutored or intend to (per the protocol: consent, no personal data collected beyond optional first name, no minors).
Materials under test: Module 3, Module 4, and the spot-the-move practice — the smallest slice that exercises the program’s hardest skill.
Flow (60–75 min per participant):
- Pre-task — respond in writing to three scripted student errors (error set A).
- Study — Modules 3 and 4 at their own pace, then the practice tool.
- Post-task — three new scripted errors (set B, parallel difficulty).
- Survey — clarity, usefulness, the “one thing I’d change” question.
Scoring: the written pre/post responses are scored with the rubric’s Domain 3 anchors (probing question · expose-don’t-correct · wait-time respect in writing · no takeover), blind to pre/post ordering (sheets are shuffled and de-identified before scoring — the blind scoring sheet implements this).
What the results will be allowed to claim — and not
| Will say | Will NOT say |
|---|---|
| “Descriptive comparison of written responses, n=3–5, per the rubric anchors” | Any learning-gain or significance claim |
| “Participants found X confusing; revised Y” | “The training works” or “validated with users” |
| Quoted feedback (with consent), labeled as pilot-participant quotes | Anything generalizable to real tutors or students |
| ≥3 revisions made in response, logged in the changelog | That the revisions were ‘proven’ by the pilot |
Small n, no control, self-selected adults: the pilot’s job is to find clarity and usability failures in the materials — ambiguous instructions, unrealistic scenarios, missing feedback — before real tutors meet them.
Report skeleton (to be filled after the pilot)
- Dates, participants (n, background), consent status — table
- Pre/post rubric scores — per participant per anchor, descriptive; no averages presented as gains
- Participant quotes — with consent, labeled
- What confused people — task-level findings
- Revisions made — linked diffs / changelog entries (minimum 3 to claim the pilot was used)
- Limitations — n, self-selection, writing-vs-live gap (written responses can’t show wait time), no student data, authors scored with the AI-judge method labeled as such
Interim note (while pending)
Until the pilot runs: this page stays empty of results, the README evaluation-status notice says “no real learners have tested this yet,” and the evaluation methods table lists the pilot as “not run — protocol prepared.”