Live demo: the fixture run's claims ledger
This is the actual output of
npm run verify and npm run report on the
repository's test fixtures — fictional content with five seeded
errors and one correct control claim. The fixture ledger's
claims and stored quotes were hand-authored to isolate
the deterministic steps; only the verdicts and verification notes below
come from the pipeline. Fixture source URLs point
at a local server (127.0.0.1:8641) that only runs during
npm test; that is why the links here are local addresses.
The result: 1 verified · 1 snippet-verified · 4 unverified. The pipeline
refused to green-light every seeded error.
Reminder: a green ledger surfaces claims for human judgment. It does not certify content as "fact-checked."
Summary
6 claim(s) · 1 verified · 1 snippet-verified · 4 unverified · 0 corrected · 0 not-applicable
The ledger
| ID | Claim | Seeded error / purpose | Verdict | Verification note (from verify.mjs) |
|---|---|---|---|---|
| C001 | "Large-scale randomized studies show that high-impact tutoring lifts achievement by 0.41 standard deviations on average." | Wrong statistic. The source says 0.10–0.20 SD. | UNVERIFIED | NUMERIC-MISMATCH — the stored quote was found in the source, but the claim states number(s) [0.41] that do not appear in the quote. |
| C002 | "Spaced practice typically improves long-term retention by 10 to 30 percent compared with massed practice in classroom studies." | Control: a correct claim. | VERIFIED | QUOTE-FOUND — stored quote found verbatim in the live source; numbers consistent. |
| C003 | "One 2024 randomized trial found that structured tutoring reduced course dropout by 18 percent." | Paywalled source (HTTP 403). | UNVERIFIED | ACCESS-RESTRICTED — source returned HTTP 403 (paywalled or blocks automated access). Labeled honestly; the source was not circumvented. |
| C004 | "As of this writing, the CourseKit platform runs on version 3.8.5." | Stale version string. The live changelog says 4.2.1. | UNVERIFIED | QUOTE-NOT-FOUND — similarity 0.67. Either the source changed (content drift), the quote was mis-transcribed, or the source never contained it. |
| C005 | "Retrieval practice strengthens memory more than rereading in most classroom experiments." | Drifted wording (source now says "restudying"). | VERIFIED-SNIPPET | PARTIAL-MATCH — similarity 0.86. Possible content drift or paraphrase. Human judgment required; never treated as fully verified. |
| C006 | "The 2019 accreditation guideline requires programs to publish their evidence base annually." | Dead link (HTTP 404). | UNVERIFIED | SOURCE-UNREACHABLE — source unreachable (HTTP 404). |
What the negative test adds
npm test runs this exact pipeline and asserts every verdict
above, plus: the agent-run extractor imports only claims found verbatim
in the content and makes no network calls; --dry-run writes stubs labeled "DRY RUN – NOT REAL
RESULTS"; report --correct rewrites the wrong statistic in
place with a dated correction note; and the weekly rot-check flags the
dead link (LINK-ROT), the paywall
(ACCESS-RESTRICTED), and both changed sources
(CONTENT-DRIFT) — then opens no duplicate findings on the
next run. 47 assertions, all passing.
Machine-readable ledger: ledger/claims-verified.json · Markdown version: CLAIMS.md