TraceBook · course content in, evidence ledger out

TraceBook: an evidence ledger for curriculum claims

TraceBook is a CLI + GitHub Action that turns course content into a versioned claims ledger. It extracts factual claims from curriculum material, checks each one against its cited primary source with deterministic quote-back matching, rewrites corrections in place with a note, and re-checks every source weekly so citations can't rot silently. It is built for curriculum developers, instructional designers, and education researchers.

What a green ledger does not mean: it is not a certification that content is "fact-checked". TraceBook surfaces claims and evidence for human judgment; a human reviewer decides. Every verdict in the ledger is a prompt to look, not a verdict to trust.

The problem

Every curriculum artifact makes factual claims: an effect size, a legal effective date, a platform version number. Verification usually happens by hand, once — and then silently rots. Sources move, versions bump, and a claim drifts away from its citation with nobody noticing. This exact process — claim → source → URL → access date → verdict → correction note — was done manually across a seven-repo portfolio before it was productized as this tool.

How it works

TraceBook pipeline: extract, verify, report, rot-check A five-step flow. Curriculum content feeds EXTRACT (done by your AI agent, no API key) into a claims ledger, which a human reviews. VERIFY fetches each source and does quote-back matching. REPORT emits CLAIMS.md and applies human-approved corrections in place. A weekly ROT CHECK re-fetches sources and opens an issue per new finding. curriculum content 1 EXTRACT (agent-run) 2 LEDGER + human gate 3 VERIFY quote-back 4 REPORT + 5 ROT CHECK weekly re-fetch
Steps 3 to 5 are fully deterministic scripts. Only EXTRACT involves AI: your own agent reads a request file and returns claims, and the script imports only claims found verbatim in your content. No API key is needed.

The ledger stores the exact sentence each claim relies on (stored_quote). That is what makes drift detectable: if the live source no longer contains the stored quote, the weekly check flags it — even though the link itself still works.

Verdict vocabulary

Five verdicts. Partial matches are labeled, never silently upgraded.
VerdictMeaning
VERIFIEDThe stored source quote appears verbatim in the live source, and every number in the claim appears in the quote.
VERIFIED-SNIPPETPartial match — possible source drift or paraphrase. Flagged for human judgment.
UNVERIFIEDSource unreachable, access-restricted (e.g. paywalled — never circumvented), quote not found, or the claim's numbers contradict the quote.
CORRECTEDContent was fixed in place against the source, with a correction note left in the file.
NOT-APPLICABLEA human marked the claim as not checkable this way.

Try it

git clone https://github.com/iwyusdaz/tracebook.git
cd tracebook
npm test                 # negative test: all seeded fixture errors caught
npm run extract -- --prepare   # extraction request for your AI agent
npm run verify && npm run report

See the live demo of the fixture run's ledger, or read the full documentation and honest limits in the repository README.

Design decisions

Quote-back matching instead of trust-the-link

A working link says nothing about whether the claim is still backed. Storing the exact source sentence makes verification a string comparison: deterministic, repeatable, and cheap enough to run weekly.

Corrections rewrite content, not just reports

A report nobody acts on rots too. With a human's approval recorded in the ledger (apply_correction: true), TraceBook rewrites the source file in place and leaves a dated correction note next to the fixed sentence. It never corrects anything automatically.

Honest labeling of restricted sources

Paywalled and 403 sources are labeled ACCESS-RESTRICTED. TraceBook never retries with different identities, strips paywalls, or uses workarounds.

Extraction is agent-run and checked on import

Claim extraction is done by an AI agent you already use. The script writes a request file, your agent answers in JSON, and the script imports only claims that appear verbatim in your content. It needs no API key and makes no network calls. --dry-run produces stubs labeled "DRY RUN – NOT REAL RESULTS" for testing the plumbing only.

Evidence: the negative test

The repository ships a fixture content set with five seeded errors (a wrong statistic, a stale version string, a dead link, a paywalled source, drifted wording) plus one correct control claim. The mandatory test npm test proves the pipeline catches all of them. It runs 47 assertions and exits 0 only when every seeded error is caught: 1 verified, 1 snippet-verified, 4 unverified. See the fixture ledger and the methods note.

Honest limits

AI-disclosure

This repository, including its code and documentation, was authored with AI assistance and reviewed by the repository owner. The fixture content and source pages are fictional and labeled as such. No real learners, educators, or publications were involved in the fixture run.