TraceBook: an evidence ledger for curriculum claims
TraceBook is a CLI + GitHub Action that turns course content into a versioned claims ledger. It extracts factual claims from curriculum material, checks each one against its cited primary source with deterministic quote-back matching, rewrites corrections in place with a note, and re-checks every source weekly so citations can't rot silently. It is built for curriculum developers, instructional designers, and education researchers.
What a green ledger does not mean: it is not a certification that content is "fact-checked". TraceBook surfaces claims and evidence for human judgment; a human reviewer decides. Every verdict in the ledger is a prompt to look, not a verdict to trust.
The problem
Every curriculum artifact makes factual claims: an effect size, a legal effective date, a platform version number. Verification usually happens by hand, once — and then silently rots. Sources move, versions bump, and a claim drifts away from its citation with nobody noticing. This exact process — claim → source → URL → access date → verdict → correction note — was done manually across a seven-repo portfolio before it was productized as this tool.
How it works
The ledger stores the exact sentence each claim relies on
(stored_quote). That is what makes drift detectable: if the
live source no longer contains the stored quote, the weekly check flags
it — even though the link itself still works.
Verdict vocabulary
| Verdict | Meaning |
|---|---|
| VERIFIED | The stored source quote appears verbatim in the live source, and every number in the claim appears in the quote. |
| VERIFIED-SNIPPET | Partial match — possible source drift or paraphrase. Flagged for human judgment. |
| UNVERIFIED | Source unreachable, access-restricted (e.g. paywalled — never circumvented), quote not found, or the claim's numbers contradict the quote. |
| CORRECTED | Content was fixed in place against the source, with a correction note left in the file. |
| NOT-APPLICABLE | A human marked the claim as not checkable this way. |
Try it
git clone https://github.com/iwyusdaz/tracebook.git
cd tracebook
npm test # negative test: all seeded fixture errors caught
npm run extract -- --prepare # extraction request for your AI agent
npm run verify && npm run report
See the live demo of the fixture run's ledger, or read the full documentation and honest limits in the repository README.
Design decisions
Quote-back matching instead of trust-the-link
A working link says nothing about whether the claim is still backed. Storing the exact source sentence makes verification a string comparison: deterministic, repeatable, and cheap enough to run weekly.
Corrections rewrite content, not just reports
A report nobody acts on rots too. With a human's approval recorded in
the ledger (apply_correction: true), TraceBook rewrites the
source file in place and leaves a dated correction note next to the fixed
sentence. It never corrects anything automatically.
Honest labeling of restricted sources
Paywalled and 403 sources are labeled ACCESS-RESTRICTED.
TraceBook never retries with different identities, strips paywalls, or
uses workarounds.
Extraction is agent-run and checked on import
Claim extraction is done by an AI agent you already use. The script
writes a request file, your agent answers in JSON, and the script imports
only claims that appear verbatim in your content. It needs no API key and
makes no network calls. --dry-run produces stubs labeled
"DRY RUN – NOT REAL RESULTS" for testing the plumbing only.
Evidence: the negative test
The repository ships a fixture content set with five seeded errors (a
wrong statistic, a stale version string, a dead link, a paywalled source,
drifted wording) plus one correct control claim. The mandatory test
npm test proves the pipeline catches all of them.
It runs 47 assertions and exits 0 only when every seeded error is caught:
1 verified, 1 snippet-verified, 4 unverified. See the
fixture ledger and the
methods note.
Honest limits
- Verification quality depends on source accessibility; paywalled sources stay honestly labeled UNVERIFIED.
- Quote-back matching can false-positive on heavily paraphrased sources — those are labeled VERIFIED-SNIPPET and require human review.
- Agent-run extraction has been tried once, by the build agent, on the fixture content only. Its quality on real curricula has not been measured.
- Supported content formats today: Markdown, HTML, plain text. DOCX must be converted first.
- Claim extraction is done by an AI agent and is imperfect: it can miss claims or propose wrong sources. The human gate exists for this reason.
AI-disclosure
This repository, including its code and documentation, was authored with AI assistance and reviewed by the repository owner. The fixture content and source pages are fictional and labeled as such. No real learners, educators, or publications were involved in the fixture run.