Gatekeeper
Gatekeeper is a Python package that checks the numerical values of Feynman integrals that go into a fit for an exact formula, and then checks the fitted formula. It scans the value files for duplicated, empty or unreadable data, and accepts a fit only if each value, left out in turn, is reproduced from the others to a set number of digits. The fit it checks is the method of Barrera, Dersy, Husain, Schwartz and Zhang (arXiv:2507.17815).
What it does
A family of Feynman integrals reduces to a finite set of master integralsthe basis integrals that every integral in the family can be written in terms of; find them and the family is solved. One way to find a master exactly is a value fit. At each order in the dimensional-regularization parameter $\varepsilon$, write it as a combination of known functions with unknown coefficients and solve for them from numerical values at several kinematic points. Then recognize the coefficients as rationals with PSLQan integer-relation algorithm: given high-precision numbers it finds small integers combining them to zero, which turns a floating-point coefficient into an exact fraction. The master values come from a reference evaluator such as AMFlow, one JSON file per master per point.
Two failures look like success. The same numbers stored under two master labels, or all-zero coefficients from an evaluator that returned nothing, make the fit report masters that were never computed. Separately, a small residual at the fitting points says nothing about other points.
The integrity scan, gatekeeper.oracle_integrity, reads every *.json under the directories you name.
- Hash. It strips the identity fields (
tag,k,indices,master,name) and hashes the sorted remainder with SHA-256. Different labels sharing a hash form a duplicate group. - Zero test. It lists the files whose coefficients are empty or all below
--zero-tol(default $10^{-80}$), and the files that do not parse. - File forms. It reads the tagged one-record-per-file form and AMFlow's own
solve_integralsoutput, where everyresultentry counts as one record tagged<file tag>#<indices>. - Arb balls. Values printed as Arb balls (a midpoint with an error radius), such as
[0.25 +/- 1e-115], parse, and the zero test is applied to the midpoint. A ball whose radius exceeds its midpoint is listed underconsistent_with_zeroand does not change the verdict unless--strict-zero-ballsis given. - Verdict. It writes
integrity_report.jsonwith averdictofCLEANorQUARANTINE_NEEDEDand exits with status 0 or 2 to match. With--quarantineit also moves the offenders (all but the first file of each duplicate group, plus the zero and unreadable ones) into a sibling<dir>_QUARANTINE/folder.
The leave-one-out certifier, gatekeeper.heldout_cv, takes each master at one order and drops the flagged files.
- Fit. For every point $P_k$ it fits the coefficients by least squares on the other $N-1$ points, predicts the value at $P_k$, and records $-\log_{10}\lvert\text{predicted}-\text{actual}\rvert$.
- Verdict. The master is marked
certifiedonly if the smallest of these numbers reaches the threshold (30 digits by default), so the verdict is set by the worst left-out point rather than the average. Only then are the coefficients converted to rationals. - Conditioning. For an ill-conditioned basis matrix,
auto_conditionapplies LLLLenstra–Lenstra–Lovász lattice reduction, which finds short integer combinations of a set of vectors; here it exposes exact integer relations among the basis functions and a better-conditioned integer change of basis to the columns and drops exact dependencies among the basis functions (typically shuffle relations among iterated integrals).pslq_with_lllthen finds the integer relations where plain PSLQ stalls.
The probes, gatekeeper.probes, test a quadrature rule (the nodes and weights of a numerical integration) before a long high-precision run.
- Convergence probe. In a product quadrature each one-dimensional rule has its own level (node count).
quad_proberaises one level with the others fixed, so that their errors cancel in consecutive differences, and reports the digits gained per level by that rule alone. Below five it printsMETHOD-MISMATCHED, which means the rule is wrong for the integrand and more nodes will not help. - Rule choice.
ell_substbuilds the matching rule for $\int_a^b h(K)\prod_i P_i(K)^{-1/2}\,dK$ with $a$, $b$ simple roots of the polynomial product. The rule depends on the other real roots.
| Other real roots of the product | Rule |
|---|---|
| None near the interval | Gauss–Chebyshev |
| One just outside each end | A Jacobi $\mathrm{sn}$ substitution |
| A single one just outside one end only | A Jacobi $\mathrm{sn}^2$ substitution |
| An interior root, a repeated endpoint root, or two nearby roots on the same side | None: kind='unsupported' with a reason |
The parser, gatekeeper.amf_result, reads AMFlow result files. amf_laurent parses one entry of a solve_integrals result into a dictionary from $\varepsilon$-order to value. It reads each coefficient's order as the absolute power, because adding leading_order on top double-counts, and it warns when the two disagree. amf_ball and amf_arb read the Arb ball strings.
Gatekeeper has these limits:
- A
certifiedverdict is numerical evidence and is not a proof. - The verdict is only as strong as the independence of the reference values from the fitted functions, and on unscanned files it means nothing.
- Gatekeeper computes no values itself.
- The loaders expect files with invariants
sandt, an identity field, and acoeffsmap from order to{"re": ..., "im": ...}strings.
Nestor's nestor.probe is this quad_probe, and the PSLQ and LLL page covers the integer-relation step. Two related tools live in other packages: the singularity-subtracted dispersion quadrature is Nestor's nestor.dispersion module, and the $\varepsilon\to0$ extrapolator is Wayfinder's wayfinder.epslimit.
Examples
Run the self-tests. With Python, mpmath and sympy installed (see Requirements), run from the repository's tools/ directory:
python3 gatekeeper/test_gatekeeper.py
It prints (T4's diagnostic lines omitted; tests T5b–T12 abridged to ...):
T1 integrity-dedup: PASS (dup tags=['m00', 'm02'], zero=m04) T2 heldout-CV: PASS (min=89.7d mean=90.0d rats=['1', '3']) T3 Lyndon/Witt: PASS (w3 over 2 letters: 2 Lyndon words; full shuffle basis dim=8, |det T|=144) T4 ill-cond LLL (before/after via heldout_certify): PASS T5 pslq_with_lll: PASS (method=lll, rel=[1, -7, 2, -11, 5, -3]) ... ALL TESTS PASS
T1 writes five small value files, one a copy of another under a different label and one all zero, and requires the scan to flag exactly those two. T2 writes six files from $\zeta_2\log(-s)+3\log^2(-t)$, fits them in the basis $[\zeta_2\log(-s),\,\log^2(-t)]$ with a 40-digit threshold, and requires certification with rationals ['1', '3']. The other tests cover the conditioning step, the AMFlow order convention, the Arb ball forms and the solve_integrals file form.
Scan a set of value files, then certify a fit. With value files under mdata/ and mdata_nt/ and the basis functions evaluated at the same points in basis_values.json:
python -m gatekeeper.oracle_integrity mdata/ mdata_nt/ \
--report integrity_report.json [--quarantine]
python -m gatekeeper.heldout_cv \
--oracle mdata/ mdata_nt/ \
--basis basis_values.json --weight 2 --gate 30 \
--integrity integrity_report.json --report cv_report.json
The first command prints the counts of good, duplicate, suspicious-zero, consistent-with-zero and unreadable files with the verdict. Resolve or quarantine what it lists before going on. The second excludes the flagged files, runs the leave-one-out fit at order 2, and prints one line per master: CERT or blank, points used, minimum and mean digits, condition number, rationals. It writes cv_report.json with a summary block and the per-master records.
basis_values.json is {"names": [...], "points": {"-3|-5": {"B0": "<digits>", ...}, ...}} keyed by "s|t", or the same per order under "weights": {"2": {...}}.
The same from Python.
from gatekeeper import scan_integrity, heldout_certify, load_oracle_values
rep = scan_integrity(["mdata/", "mdata_nt/"])
excl = rep.quarantine_set() # files to drop from a fit
vals, kin, _ = load_oracle_values(["mdata/"], exclude_files=excl)
res = heldout_certify(vals, kin, weight=2,
basis_fn=lambda pk,s,t: [...], basis_names=[...],
gate=30, dps=80,
n_duplicates_excluded=len(excl))
res maps each master tag to a CVResult. Read .certified, .min, .heldout_digits and .pslq_rationals.
Choose a quadrature rule. Two integrands on $[-1,1]$ from the acceptance tests:
import mpmath as mp, sympy as sp
from gatekeeper.probes import select
x = sp.symbols('x'); mp.mp.dps = 80
plan = select([1 - x**2, 4 - x**2], -1, 1, N=64, dps=70) # plan.kind == 'jacobi-sn'
plan3 = select([1 - x**2, x**2 + 4], -1, 1, N=96, dps=70) # plan3.kind == 'chebyshev'
$1/\sqrt{(1-x^2)(4-x^2)}$ has a root at each end and one just outside each end. select therefore returns the Jacobi-sn rule with modulus $1/4$, and plan.integrate() equals the complete elliptic integral $K(1/4)$. With $x^2+4$ in place of $4-x^2$ only the endpoint roots remain, the rule is Gauss–Chebyshev, and plan3.integrate(mp.exp) matches adaptive quadrature to more than 50 digits at 96 nodes.
Each plan has nodes, weights and a report. For a configuration the selector does not cover, plan.kind is 'unsupported' and plan.report['reason'] says why.
Routines
The python -m commands run from the repository's tools/ directory. Paths that begin with tools/ are relative to the repository root.
| Command | What it does |
|---|---|
python -m gatekeeper.oracle_integrity DIR ... [--report FILE] [--quarantine] [--zero-tol X] [--strict-zero-balls] | The integrity scan; exit status 0 if clean, 2 otherwise |
python -m gatekeeper.heldout_cv --oracle DIR ... --basis FILE --weight W [--gate D] [--integrity FILE] | Leave-one-out certification at one order; scans first if no integrity report is given |
python -m gatekeeper.probes.quad_probe --workdir DIR --module MOD --func FN --base DPS=26,LEVEL_K=4 --chains LEVEL_K [--span 2] | Convergence probe, one rule at a time; exit status 1 on METHOD-MISMATCHED |
python -m gatekeeper.registry with --rebuild, --point s,t --masters lo..hi or --dupes | Indexes the stored value files, lists the masters that have values at a point, or lists duplicate files |
python3 tools/gatekeeper/ctrl_per_order.py --towers FILE --oracle FILE [--masters lo:hi] | Digits of agreement per master and order between differential-equation boundary values and reference values |
tools/quad_probe.py, tools/ell_subst.py, tools/oracle_registry.py | Flat-path scripts that forward to the package |
quad_probe passes the levels to your module as environment variables, and the registry takes its roots from ORACLE_REG_ROOTS. ctrl_per_order.py reports ALIAS-LIMITED with the highest usable order when the digits lost per order match $\lvert\log_{10}\varepsilon_0\rvert$ (the eps0_log10 value stored in the towers file), the pattern of series truncation rather than insufficient precision.
These names import from gatekeeper:
| Function | What it returns |
|---|---|
scan_integrity(dirs, zero_tol, strict_zero_balls) | An IntegrityReport: the lists of flagged files and the verdict; .quarantine_set() lists the files to exclude |
load_oracle_values(dirs, exclude_files) | values[tag][point][order] and kinematics[point], with each point keyed by s and t joined by a vertical bar |
heldout_certify(values, kinematics, *, weight, basis_fn, basis_names, gate, dps) | {tag: CVResult}, each with certified, min, mean, heldout_digits, pslq_rationals, cond_before, cond_after, pslq_method and note |
lyndon_basis(alphabet, weight) | The well-conditioned Lyndon-word basis of the shuffle algebra at that weight, with its rational transition matrix |
auto_condition(basis_matrix, target_cond) | The LLL column reduction of an ill-conditioned basis matrix |
pslq_with_lll(values, prec, max_coeff_bits) | (relation, method): an integer relation by LLL, with PSLQ as fallback |
amf_laurent(result_entry) | {absolute_order: mpc} from one solve_integrals entry |
amf_arb(v), amf_ball(v) | The midpoint of an Arb ball string at the caller's precision; the midpoint and radius as decimal strings |
These import from gatekeeper.probes:
| Function | What it returns |
|---|---|
probe(evaluator, base_levels, chains, span) | Per rule: levels, differences, digits per level, and the at_noise_floor and mismatched flags |
EnvEvaluator(workdir, module, func) | An evaluator that runs module.func in parallel subprocesses with the levels as environment variables |
select(factors, a, b, N, dps) | A SubstPlan with kind, nodes, weights, m, report, .integrate(h) and .log_frame() |
cubic_under_sqrt_K(a3, a2, a1, a0) | $\int dz/\sqrt{\text{cubic}}$ between its two lowest real roots, in closed form through $K(m)$ |
.log_frame() splits off endpoint logarithms. It is defined for the Chebyshev and sn kinds only and raises for jacobi-sn2. The exported names are listed in __init__.py and probes/__init__.py, and the full argument lists are in the docstrings.
Requirements
- Runs on: Python 3 with the standard library and
mpmath.sympyis needed forprobes/ell_subst.pyand the tests. - Self-test:
python3 tools/gatekeeper/test_gatekeeper.pyends withALL TESTS PASS. - Probe tests:
python3 tools/gatekeeper/probes/ell_subst_test.pyends withELL_SUBST ACCEPTANCE: PASS. Its T4 uses a reference script that is not included and is skipped unlessELL_KLINE_DIRis set. - Tests on your own data:
ctrl_per_order.py --selftestandregistry --selftestneed your own data (CTRL_TOWERS,CTRL_ORACLE,ORACLE_REG_ROOTS) and skip or fail without it. - Optional:
fpylllfor the LLL step. Without it the reduction is pure Python.
Source and references
The code is tools/gatekeeper/ in the BootLoops repository, released under the MIT license.
| Reference | What it supplies |
|---|---|
| O. Barrera, A. Dersy, R. Husain, M. D. Schwartz and X. Zhang, "Analytic regression of Feynman integrals from high-precision numerical sampling", arXiv:2507.17815 | The value fit that the certifier checks |
| H. R. P. Ferguson, D. H. Bailey and S. Arno, "Analysis of PSLQ", Math. Comp. 68 (1999) 351 | The PSLQ algorithm |
| F. Johansson and the mpmath contributors, mpmath | PSLQ as called here, and the arbitrary-precision arithmetic |
| A. K. Lenstra, H. W. Lenstra Jr. and L. Lovász, Math. Ann. 261 (1982) 515; the FPLLL development team, fplll | LLL lattice reduction, run through fpylll when it is installed |
| X. Liu and Y.-Q. Ma, AMFlow, arXiv:2201.11669; the AMFlow.cpp contributors, AMFlow.cpp | The result files that the scan and the parser read |
| Duval's algorithm; Radford's theorem | The Lyndon words and the Lyndon basis of the shuffle algebra in lyndon_basis |
| Gradshteyn and Ryzhik, formula 3.131 | The closed form in cubic_under_sqrt_K and the substitution of the $\mathrm{sn}^2$ rule |