Emitall
Emitall checks that the numbers quoted in a report or paper still agree with the stored result files they came from. In one specification file you list each quoted number, its file and an expression that recomputes it, and Emitall reports every quote that disagrees with the recomputed value at the quote's printed precision. Three companion programs check the Python scripts that write summary sentences and a paper's LaTeX sources.
What it does
Quoted numbers go wrong in ordinary ways. A value is copied by hand and a digit slips, or the result file is regenerated after the sentence was written. A count depends on an unstated choice of subset, or a range is rounded to the nearest digit when it should have been rounded outward. None of these errors looks wrong on the page.
The input is a claims spec (the specification file) in JSON, or in YAML if PyYAML is installed. Its receipts block (the package's word for stored result files) gives an alias to each JSON or CSV file, optionally with newer_than and max_age_days freshness conditions. Its claims list has one entry per quoted number: a section and label, the quoted string exactly as printed, and an expr.
Expressions use Python syntax but are evaluated by a restricted parser, never eval. They reach the files only through accessors (functions that read a result file) such as f('s','/n_sites_rated'), the value at a slash-separated path. A longer derivation belongs in your own script: write its result into the result file and quote that.
Each claim is compared in one of three modes:
printed: the default. The recomputed value is formatted with the quote's number of decimals and compared as text, ignoring thousands separators and a trailing percent sign. A value of 10.4166... matches a quoted "10.4%", while -0.6845 does not match "-0.69" because it prints as -0.68.exact: the two strings are compared with no normalization.outward-band: the mode for a quoted interval. At granularity $g$ (the rounding step) the printed edges must equal $g\lfloor \mathrm{lo}/g \rfloor$ and $g\lceil \mathrm{hi}/g \rceil$, so the printed range always contains the computed one. The test runs in exact rational arithmetic. A failure is markedinward-edgewhen a printed edge cuts into the computed interval andoutward-loosewhen the band is wider than the outward rounding.
The output is a table with one line per claim, marked MATCH, FINDING, EMITTED (a claim with no quoted string is recorded, never compared) or MISSING-RECEIPT. A missing result file is a finding in its own right. So is a stale one, older by modification time than a file named under newer_than or than max_age_days. An expression that cannot be evaluated becomes an eval-error finding, and the rest of the spec still runs.
The first companion program, lint.py, reads the scripts that write summary sentences without running them. It examines every string that feeds a variable, key, argument or function whose name contains emit, headline or summar, and reports three kinds of finding:
digit-literal: a digit typed into the literal text of such a string. Interpolated{...}fields and%placeholders are exempt.unordered-iteration: a set or dictionary view iterated into such a string withoutsorted().unseeded-sampling:random.sample,choice,choicesorshufflewith no seed set earlier in the file.
A comment # emitall-lint: allow-literal <reason> on a line of the string permits its typed digits. The reason is required, and a comment without one is itself reported.
The second program, battery.py, checks a paper's .tex files and the text of its built PDF against a JSON spec of assertions in four classes:
FROZEN: a number or phrase appears verbatim in its section file or the PDF text, optionally an exact number of times.NEVER: a phrasing appears nowhere, apart from contexts the spec allows.NEG: a string occurs exactly zero times.ABSTRACT: a phrase is in the abstract, or is gone from it.
Comments are stripped before the comparison, and the PDF text is rejoined across hyphen wraps and page breaks. With pdf_fresh_vs set, battery.py refuses a PDF that is older than its source.
The third program, paper_seams.py, takes a master .tex file, follows its \input and \include lines, and reports every % comment block dropped inside a sentence. After such a slip the opening words of the sentence sit on a comment line, and the built text reads "...end. lowercase fragment". Table rows and macro lines are skipped. When the built PDF sits beside the master, fragments found in its text are cross-checked against the source.
Emitall has these limits:
- It checks only the claims in the spec. Whether the spec covers every number in the document is up to you.
printedmode confirms agreement at the printed precision, nothing more.lint.pydetects by name pattern, so it does not examine a list of summary sentences that is namedlines.paper_seams.pyfollows\inputand\includethrough one level of nesting and no deeper.
Examples
Run the included example. The package comes with a complete spec for an invented star-rating program, specs/toy_stars.json, with synthetic result files in specs/toy_receipts/. With Python 3 installed (see Requirements), run it from the package directory tools/emitall/:
python3 run.py specs/toy_stars.json
Two of its twelve claims are a percentage in printed mode and a dollar range in outward-band mode. The file stores dollars, and scale converts them to millions before the rounding at 0.1.
{"section": "RATINGS", "label": "pct changed", "quoted": "10.4%",
"expr": "100 * f('s','/n_star_changes') / f('s','/n_sites_rated')"},
{"section": "BANDS", "label": "payout band, 0.1 $M", "mode": "outward-band",
"quoted": "[1.2-2.8]", "quoted_lo": "1.2", "quoted_hi": "2.8",
"granularity": "0.1", "scale": "1e-6",
"lo_expr": "f('s','/payout_band_usd/lo')", "hi_expr": "f('s','/payout_band_usd/hi')"}
The run prints a line for each of the twelve claims, eleven MATCH and one EMITTED, then 12 headlines emitted; 11 MATCH; 0 FINDINGS, and exits 0. In the self-tests, quoting "6" star changes where the file says 5 gives one mismatch finding and exit code 1. Copy this spec as the starting point for your own.
Call the comparison functions from Python. Put the directory that contains emitall/ on sys.path, as the tests do:
from emitall import compare_printed, compare_outward_band compare_printed(6.652513268810491, "6.7%") # (True, "6.7") compare_printed(-0.6845146732002659, "-0.69") # (False, "-0.68") compare_outward_band(154867600, 232301400, "154.9", "232.3", "0.1", "1e-6") compare_outward_band(154867600, 232301400, "154.8", "232.4", "0.1", "1e-6")
compare_outward_band takes the computed low and high (here in dollars), the quoted edges, the granularity and a scale. The computed interval is [154.8676, 232.3014] million. The first call therefore fails with an inward-edge note naming both edges, and the second, the correct outward rounding, passes.
Check the scripts that write summary sentences. Give lint.py one or more files or directories, from tools/emitall/:
python3 lint.py <script-or-dir> [more paths...] [--json report.json] [--quiet]
Directories are scanned recursively for .py files. The findings are printed by file, one line each with the line number, kind and a short detail, then a summary count. In the README's motivating case every number in a headline string is interpolated except a typed "(max 5 cents)", while the stored maximum was 0.09 dollars. lint.py reports that 5 as a digit-literal, and it reports the year labels typed into the same string too.
Routines
| Command | What it does |
|---|---|
python3 run.py SPEC.json [--out report.json] [--quiet] | Evaluates a claims spec, prints the table and, with --out, writes the JSON report |
python3 lint.py <script-or-dir> [more paths...] [--json report.json] [--quiet] | Checks the scripts that write summary sentences; --json writes the findings to a file |
python3 battery.py SPEC.json [--quiet] | Runs the assertions of a spec over a paper's .tex files and PDF text |
python3 paper_seams.py <master.tex> [--snapshot <dir>] [--marker <label> ...] [--allow <phrase> ...] [--quiet] | Reports % comment blocks dropped inside a sentence, in the master and the files it inputs |
Every command exits with 0 when nothing is found and 1 on any finding or failed assertion. Exit code 2 means a malformed spec (run.py, battery.py), a PDF older than its source (battery.py) or a usage error (lint.py, paper_seams.py).
For paper_seams.py, --allow names a legitimate lowercase sentence opening and --snapshot compares against an earlier copy of the tree. --marker gives the label of the comment blocks that comparison examines (default FOLD). The format of a battery.py spec is in the docstring of battery.py.
The Python functions are imported with from emitall import .... The lower-level names (fmt_like, evaluate, resolve, resolve_pattern, Env and the exceptions) are listed in __init__.py.
| Function | What it returns |
|---|---|
run_spec(spec_path, out_path, quiet) | (report, exit_code) from running a spec |
load_spec(path) | The parsed JSON or YAML spec; raises SpecError when malformed |
compare_printed(emitted, quoted, decimals) | (ok, printed_string), the printed comparison |
compare_outward_band(lo, hi, quoted_lo, quoted_hi, granularity, scale) | (ok, display, note_or_None), the outward-rounding test in exact fractions |
lint_source(source, path), lint_file(path) | (findings, allowed); each finding is a dict with file, line, kind and detail |
lint_paths(targets) | (findings, allowed, files_scanned) |
Inside a spec expression the accessors are f, vals, rows, cw and joinrows. The helpers are where, split, join, jsondump, zipjoin and frac, with len sum min max abs int float str round bool as in Python. The docstring of expr.py gives their arguments and the full grammar.
Requirements
- Runs on: Python 3 with the standard library only. The report timestamp is read from the system
date -ucommand, so a Unix-like system is assumed. - Install: nothing to build. Run the commands from the package directory
tools/emitall/. - Self-test:
python3 -m pytest tools/emitall -qfrom the repository root, orpython3 tests/test_all.pyandpython3 tests/test_paper_seams.pyfromtools/emitall/. The tests also pass underpython3 -O.python3 battery.py --selftestneeds nopdftotextand ends withselftest: 8/8 gates green. - Optional: PyYAML, for YAML specs. The
pdftotextprogram is needed only forbattery.pyspecs that read a PDF (a pre-renderedtxtfile avoids it) and for the PDF cross-check inpaper_seams.py. That cross-check is skipped with a notice when the program or the PDF is absent.
Source and references
The code is in tools/emitall/ in the BootLoops repository, released under the MIT license. README.md there describes the spec format, the expressions and the four programs.
| Reference | What it supplies |
|---|---|
pdftotext, from Poppler | The text of a built PDF, for battery.py and paper_seams.py |