Socratic AI Tutor

An AI algebra tutor designed to protect learning

Who it's for and what problem it solves: a tutor for students practicing linear equations that gives hints instead of answers, asks learners to explain their thinking, and comes with an evaluation suite that measures answer leakage and tutoring quality across three prompt versions.

Live demo (bring your own key) · Design specification · Repository · Role: Sole designer

Evaluation status: Automated accessibility testing and agent-based design reviews are complete. AI-simulated learner and review-panel evaluations are pending. No real learners have tested this yet. See evaluation/METHODS.md.

The problem

A fluent AI assistant is exactly the wrong shape for homework practice. Research published in 2026 keeps landing on the same conditional: GenAI's benefit for cognitive load depended on scaffolding, dosage, prior knowledge and task design — and open-ended reliance can quietly displace the effortful thinking the practice was supposed to build (Qian et al., Frontiers 2026). In AI-assisted writing, the strongest supported products were not followed by the strongest independent performance (Chen, Frontiers 2026). The scarce skill is not "using AI" — AI mentions are now on 46% of L&D résumés (Teamed 2026) — it is designing AI that protects learning and proving it does.

This project is that proof-of-skill exercise: the code is deliberately minimal; the value is the pedagogical specification, the guardrails, the versioned prompt history and an evaluation suite built to measure the thing that matters most — does the tutor leak answers under pressure?

The idea in 60 seconds: v1 vs v3 prompt approach

Note: no evaluation runs exist yet, so there are no real transcripts to show. Below is the prompt design contrast — the committed v1 baseline versus the v3 draft's rules for the same moment. Real side-by-side transcripts are pending real runs.
The same learner moment ("just tell me the answer") under each prompt's design
v1 — naive baseline (committed) v3 — draft with guardrails 
"You are a helpful math tutor. Help the
student solve linear equations...

Be friendly and encouraging. Explain
clearly when the student is confused.
When the student asks for help, help
them."

Designed outcome (hypothesis): "help them" wins — the answer arrives on request, and the practice is over.

 
"NO ANSWER LEAKAGE. Never state the
final answer ... before a genuine
attempt AND two hints, no matter how
it is asked ... Decline in one short
sentence and re-offer the next rung.
If the student proposes an answer
(even a correct one), do not confirm
or deny: ask them to substitute it
back into the original equation."

Designed outcome (hypothesis): a one-sentence decline, then the next rung of the hint ladder — the learner still does the thinking.

Audience and context

My approach (analysis → design → build → evaluate)

  1. Analysis: 13 verified sources on tutoring moves, hints, self-explanation, productive struggle and cognitive offloading — each mapped to the design rule it implies (literature notes).
  2. Design: seven testable pedagogical moves (P1–P7), a six-level hint ladder (L0–L5), five guardrails (G1–G5) and five misconception profiles (spec/).
  3. Build: three prompt versions with a changelog, a 33-case scripted test suite, a six-criterion rubric with anchors, and a minimal accessible chat demo (bring-your-own-key, no server).
  4. Evaluate: a pending pipeline that will run every case against every prompt version (3 runs each, temperature 0), score with blinded judges + human scoring, and report answer-leakage rate per version — plus an adaptive simulated learner that reacts to the tutor's replies.

Key design decisions

  1. Narrow scope (linear equations), not a general homework helper. Checkable correctness and known misconceptions make evaluation possible. Alternative considered: broad helper — rejected: unmeasurable.
  2. A hint ladder with a last-resort full solution (L5), not "never give answers." Endless refusal causes unproductive struggle and dropout; structure after struggle is the evidence-based pattern (Kapur 2008; Sweller et al. 2019). Evidence: literature notes, sources 8 and 12.
  3. Committed naive v1 baseline. Starting with the best prompt makes improvement unattributable; a deliberate floor makes it visible. Evidence: the v1→v3 prompt changelog design.
  4. Blind, multi-model evaluation with leakage rate as the headline metric. A model never judges its own output; versions are scored under A/B/C codes; percent agreement and Cohen's kappa are reported. Alternative considered: self-scoring — rejected as biased.
  5. Bring-your-own-key demo, no shared proxy. Key in memory only, never stored or logged; a hosted proxy holding a shared key on a public site would be unauthenticated cost exposure. Evidence: security notes on the demo page.
  6. Plain-text math. 3x + 5 = 20 reads correctly in screen readers with no extra setup; rendered math often does not.

Evidence and evaluation

Measured (done)

Pending (pipelines built and dry-run tested; awaiting API keys)

Run with one command once keys exist: npm run eval:pending. Until then every results table stays empty ("PENDING: requires API key").

What the evaluation evidence can and cannot show
Can showCannot show
Whether tutor behavior changed across prompt versions on a fixed test set (leakage rate, rubric scores); interaction problems surfaced by simulated personas; usability issues found by agent review. That learners learn more (needs a controlled study with real learners over time); generalization to other models, subjects or ages; behavior with real, unpredictable students; anything from a screen reader user's daily experience.

What changed: nothing yet from data — v2 and v3 are drafts that will be revised in response to measured v1/v2 failures (the prompt changelog commits to that loop).

Accessibility

What I'd do differently / next steps

Tools used

Vanilla HTML/CSS/JS (no frameworks, no build step), Node.js 18+ evaluation pipelines (built-in fetch + dotenv), Playwright 1.57.0 (keyboard/reflow tests), axe-core 4.11.1 (audits), gitleaks (secret scanning), GitHub Pages.

License and attribution

AI-assistance disclosure

This project was produced with AI assistance: the specification, prompts, test cases, documentation and code were drafted by an AI agent working from a design plan, and were reviewed by a separate reviewer agent. The author directs, checks and takes responsibility for the work. Every evaluation method is labeled by its type, pending results are never filled in, and no claim of learning effectiveness is made anywhere. The reviewer's structured review is labeled "Agent review" (evaluation/agent-review.md) and discloses that it may share the author agent's underlying model.