Emitall

Emitall checks that the numbers quoted in a report or paper still agree with the stored result files they came from. In one specification file you list each quoted number, its file and an expression that recomputes it, and Emitall reports every quote that disagrees with the recomputed value at the quote's printed precision. Three companion programs check the Python scripts that write summary sentences and a paper's LaTeX sources.

What it does

Quoted numbers go wrong in ordinary ways. A value is copied by hand and a digit slips, or the result file is regenerated after the sentence was written. A count depends on an unstated choice of subset, or a range is rounded to the nearest digit when it should have been rounded outward. None of these errors looks wrong on the page.

The input is a claims spec (the specification file) in JSON, or in YAML if PyYAML is installed. Its receipts block (the package's word for stored result files) gives an alias to each JSON or CSV file, optionally with newer_than and max_age_days freshness conditions. Its claims list has one entry per quoted number: a section and label, the quoted string exactly as printed, and an expr.

Expressions use Python syntax but are evaluated by a restricted parser, never eval. They reach the files only through accessors (functions that read a result file) such as f('s','/n_sites_rated'), the value at a slash-separated path. A longer derivation belongs in your own script: write its result into the result file and quote that.

Each claim is compared in one of three modes:

The output is a table with one line per claim, marked MATCH, FINDING, EMITTED (a claim with no quoted string is recorded, never compared) or MISSING-RECEIPT. A missing result file is a finding in its own right. So is a stale one, older by modification time than a file named under newer_than or than max_age_days. An expression that cannot be evaluated becomes an eval-error finding, and the rest of the spec still runs.

The first companion program, lint.py, reads the scripts that write summary sentences without running them. It examines every string that feeds a variable, key, argument or function whose name contains emit, headline or summar, and reports three kinds of finding:

A comment # emitall-lint: allow-literal <reason> on a line of the string permits its typed digits. The reason is required, and a comment without one is itself reported.

The second program, battery.py, checks a paper's .tex files and the text of its built PDF against a JSON spec of assertions in four classes:

Comments are stripped before the comparison, and the PDF text is rejoined across hyphen wraps and page breaks. With pdf_fresh_vs set, battery.py refuses a PDF that is older than its source.

The third program, paper_seams.py, takes a master .tex file, follows its \input and \include lines, and reports every % comment block dropped inside a sentence. After such a slip the opening words of the sentence sit on a comment line, and the built text reads "...end. lowercase fragment". Table rows and macro lines are skipped. When the built PDF sits beside the master, fragments found in its text are cross-checked against the source.

Emitall has these limits:

Examples

Run the included example. The package comes with a complete spec for an invented star-rating program, specs/toy_stars.json, with synthetic result files in specs/toy_receipts/. With Python 3 installed (see Requirements), run it from the package directory tools/emitall/:

python3 run.py specs/toy_stars.json

Two of its twelve claims are a percentage in printed mode and a dollar range in outward-band mode. The file stores dollars, and scale converts them to millions before the rounding at 0.1.

{"section": "RATINGS", "label": "pct changed", "quoted": "10.4%",
 "expr": "100 * f('s','/n_star_changes') / f('s','/n_sites_rated')"},
{"section": "BANDS", "label": "payout band, 0.1 $M", "mode": "outward-band",
 "quoted": "[1.2-2.8]", "quoted_lo": "1.2", "quoted_hi": "2.8",
 "granularity": "0.1", "scale": "1e-6",
 "lo_expr": "f('s','/payout_band_usd/lo')", "hi_expr": "f('s','/payout_band_usd/hi')"}

The run prints a line for each of the twelve claims, eleven MATCH and one EMITTED, then 12 headlines emitted; 11 MATCH; 0 FINDINGS, and exits 0. In the self-tests, quoting "6" star changes where the file says 5 gives one mismatch finding and exit code 1. Copy this spec as the starting point for your own.

Call the comparison functions from Python. Put the directory that contains emitall/ on sys.path, as the tests do:

from emitall import compare_printed, compare_outward_band

compare_printed(6.652513268810491, "6.7%")       # (True, "6.7")
compare_printed(-0.6845146732002659, "-0.69")    # (False, "-0.68")

compare_outward_band(154867600, 232301400, "154.9", "232.3", "0.1", "1e-6")
compare_outward_band(154867600, 232301400, "154.8", "232.4", "0.1", "1e-6")

compare_outward_band takes the computed low and high (here in dollars), the quoted edges, the granularity and a scale. The computed interval is [154.8676, 232.3014] million. The first call therefore fails with an inward-edge note naming both edges, and the second, the correct outward rounding, passes.

Check the scripts that write summary sentences. Give lint.py one or more files or directories, from tools/emitall/:

python3 lint.py <script-or-dir> [more paths...] [--json report.json] [--quiet]

Directories are scanned recursively for .py files. The findings are printed by file, one line each with the line number, kind and a short detail, then a summary count. In the README's motivating case every number in a headline string is interpolated except a typed "(max 5 cents)", while the stored maximum was 0.09 dollars. lint.py reports that 5 as a digit-literal, and it reports the year labels typed into the same string too.

Routines

CommandWhat it does
python3 run.py SPEC.json [--out report.json] [--quiet]Evaluates a claims spec, prints the table and, with --out, writes the JSON report
python3 lint.py <script-or-dir> [more paths...] [--json report.json] [--quiet]Checks the scripts that write summary sentences; --json writes the findings to a file
python3 battery.py SPEC.json [--quiet]Runs the assertions of a spec over a paper's .tex files and PDF text
python3 paper_seams.py <master.tex> [--snapshot <dir>] [--marker <label> ...] [--allow <phrase> ...] [--quiet]Reports % comment blocks dropped inside a sentence, in the master and the files it inputs

Every command exits with 0 when nothing is found and 1 on any finding or failed assertion. Exit code 2 means a malformed spec (run.py, battery.py), a PDF older than its source (battery.py) or a usage error (lint.py, paper_seams.py).

For paper_seams.py, --allow names a legitimate lowercase sentence opening and --snapshot compares against an earlier copy of the tree. --marker gives the label of the comment blocks that comparison examines (default FOLD). The format of a battery.py spec is in the docstring of battery.py.

The Python functions are imported with from emitall import .... The lower-level names (fmt_like, evaluate, resolve, resolve_pattern, Env and the exceptions) are listed in __init__.py.

FunctionWhat it returns
run_spec(spec_path, out_path, quiet)(report, exit_code) from running a spec
load_spec(path)The parsed JSON or YAML spec; raises SpecError when malformed
compare_printed(emitted, quoted, decimals)(ok, printed_string), the printed comparison
compare_outward_band(lo, hi, quoted_lo, quoted_hi, granularity, scale)(ok, display, note_or_None), the outward-rounding test in exact fractions
lint_source(source, path), lint_file(path)(findings, allowed); each finding is a dict with file, line, kind and detail
lint_paths(targets)(findings, allowed, files_scanned)

Inside a spec expression the accessors are f, vals, rows, cw and joinrows. The helpers are where, split, join, jsondump, zipjoin and frac, with len sum min max abs int float str round bool as in Python. The docstring of expr.py gives their arguments and the full grammar.

Requirements

Source and references

The code is in tools/emitall/ in the BootLoops repository, released under the MIT license. README.md there describes the spec format, the expressions and the four programs.

ReferenceWhat it supplies
pdftotext, from PopplerThe text of a built PDF, for battery.py and paper_seams.py

← back to the tools index