FFCapital
FFCapital recovers the exact coefficients that had already finished when a FireFly rational-function reconstructionfinding an exact rational function from its values modulo large primes stopped early, for example because the job ran out of memory or the machine went down. Its input is the ff_save/ directory that Kira with FireFly writes while reducing a family of Feynman integrals, and each recovered function is checked against a value stored there. FireFly is the program of Jonas Klappert, Sven Yannick Klein and Fabian Lange; FFCapital reads the files FireFly leaves on disk and never resumes or advances the reduction.
Where it comes from
FireFly and the format of its saved-state files are by Jonas Klappert, Sven Yannick Klein and Fabian Lange. FFCapital is a new BootLoops program that reads those files. The file layout was taken from FireFly's source code, and so were the first 40 entries of its table of 63-bit primes. No FireFly routine is included or run. recon_symbolic_d.py reads the table from the user's copy of ReconstHelper.cpp.
| Part of FFCapital | Source | Authors |
|---|---|---|
File readers that follow the layout their code writes: section names, line formats, the ZERO marker and validation.gz | RatReconst.cpp and Reconstructor.hpp in FireFly | Klappert, Klein and Lange |
| Constants copied from their code: the first 40 primes of the table, in the two harvest scripts | ReconstHelper.cpp in FireFly | Klappert, Klein and Lange |
| New code: the check against the stored values, the prime detection, the slot assignment, the fit in $d$ and its checks | Written for BootLoops | Code by Claude |
Please cite the authors' papers when you use it. FFCapital is not affiliated with or endorsed by the FireFly authors.
What it does
In an IBP reductionintegration-by-parts identities relate the many integrals of a Feynman family to a small basis; solving that large linear system is the reduction each coefficient is a rational function of the spacetime dimension $d$ and the kinematic variables. Kira with FireFly finds it by sampling the linear system modulo large primes and reconstructing the exact function from the samples.
FireFly saves the state of each function in ff_save/states/<n>_<k>.gz, where <n> is the function index and <k> is the counter of the last prime at which the state was saved. A state file with is_done equal to 1 already holds the complete exact numerator and denominator coefficients. FireFly also writes ff_save/validation.gz: one probe point and the value of every function there, computed from the linear system directly and not from the reconstruction. Both files remain on disk after a crash.
FFCapital rebuilds the function $N/D$ (numerator over denominator) for each finished function. It accepts the function only if $N/D$ at the probe point reproduces the stored value modulo the FireFly prime. The prime is detected from FireFly's table of 63-bit primes and must come out the same for a sample of up to eight finished functions; otherwise the run stops with an error instead of guessing.
A function that fails is marked in the output and named on standard error, the others are unaffected, and the exit code is nonzero. All arithmetic on the value path is exact, with no floating-point numbers anywhere.
The package has three scripts. Which one to use depends on the shape of the saved run.
- One variable.
harvest_ffsave_eta.pyhandles a save in which $d$ was fixed to a rational number and one kinematic variable,eta, remains. - Two variables.
harvest_ffsave_2var.pyhandles a save in $(d,\eta)$, optionally with earlier snapshot copies. It also works out which (target integral, master integralone of the small basis of integrals to which every integral of the family reduces) coefficient each FireFly function index is. It certifies that assignment by exact polynomial cross-multiplication against at least two finished reductions of the same system at fixed rational $d$. - Many fixed-$d$ runs.
recon_symbolic_d.pygoes the other way: from one-variable extractions of many runs at different rational $d$ ("slices") it fits the $d$-dependence of every coefficient exactly. It uses four more slices than interpolation needs, checks every slice left out of the fit and refits without each fit slice in turn. It also compares against a separate multi-prime FireFly state that never enters a fit.
Equality is always tested as equality of rational functions, $P\,D_s = \lambda\,Q\,N_s$ with one overall scale $\lambda$. Here $P/Q$ is the fitted function at the $d$ of slice $s$ and $N_s/D_s$ is the function stored for that slice. The test has this form because a slice whose numerator and denominator share a factor stores a reduced but correct function.
FFCapital has these limits:
- The per-function check compares a function with its own run's probe value, so a run whose linear system was set up wrongly can still pass. Only the two-variable and cross-slice scripts test agreement across runs.
- Do not read a save that a solver process is still writing: FireFly rotates the state files.
- Do not use the scripts on states with more than two variables.
- Resuming an unfinished run is FireFly's job; the AMFlow wrapper covers resumable runs.
ffsave_degree_census.py, on the Kira, FireFly and Fermat page, reads the same states for degree statistics. Numkin sets up the fixed-$d$ slice runs that recon_symbolic_d.py combines.
Examples
With Python set up (see Requirements), run the scripts with python3 from the package directory tools/ffcapital/ on your own ff_save/.
Inventory a stopped one-variable run, then extract what finished. From the usage text of harvest_ffsave_eta.py:
harvest_ffsave_eta.py census --ffsave <dir> --out census.jsonl [--workers 16]
harvest_ffsave_eta.py extract --ffsave <dir> --out funcs.jsonl
(--fns 0,17,26683 | --all-done) [--workers 16]
[--no-validate]
census writes one JSON line per state file: function index fn, FireFly tag, the done flag, np (how many primes had been combined) and the degrees degN and degD. extract writes one line per requested function, with N and D as maps from the power of eta to an exact string "p/q" and a validated field naming the prime index used. A function that fails gets "validated": false with the reason, standard error lists the failing indices, and the exit code is 2.
Recover and assign a two-variable save. From the usage text of harvest_ffsave_2var.py:
harvest_ffsave_2var.py salvage \ --primary <ff_save dir> [--extra label=<ff_save dir> ...] \ --slice label=<kira_target.m>=<d_num>/<d_den> ... (>=2) \ [--census <ffsave_degree_census json>] (degree stats for RESIDUAL_TODO) \ [--residual-target label=<target file> ...] (staged slices to shrink) \ --bank <outdir> [--family myfam] [--workers 24] \ [--min-slice-pass 2] [--kill-rate 0.001]
Each --slice is a finished Kira reduction of the same system at $d =$ d_num/d_den, exported with kira2math; at least two are required. The files written to the output directory are listed under Routines. If hard failures exceed --kill-rate (default 0.1%) of the finished functions, the save is treated as contaminated. The recovered reduction is then not written, EXTRACTION_RECEIPT.json records the verdict CONTAMINATED_STOP, and the exit code is 3.
Reconstruct the $d$-dependence from many fixed-$d$ runs. From the package guide:
recon_symbolic_d.py control|index|probe3|run --outdir <dir> [--coverage ...] [--wave1 ...] [--bank-states ...] [--ff-helper <ReconstHelper.cpp>] [--p2-file ...] [--config <json>] [--base <root>] [--skip-banked <prior>]
Run control first and again after any code change. It needs only --coverage, --ff-helper and --outdir. Then index writes slice_index.json, and run writes shard_NNN.jsonl with one line per function. The inputs, the control check and the output lines are described under Routines.
Routines
| Command | What it does |
|---|---|
harvest_ffsave_eta.py census | Inventory of every state file of a one-variable save: done flag, primes combined, degrees |
harvest_ffsave_eta.py extract | Exact coefficients of the listed (--fns) or all finished (--all-done) functions, each checked against validation.gz; exit code 2 on any mismatch |
harvest_ffsave_eta.py extract --no-validate | The same without the check; only for a quick look |
harvest_ffsave_2var.py salvage | Extracts and checks a $(d,\eta)$ save, assigns each function to its (target, master) slot and certifies it; exit code 3 on CONTAMINATED_STOP |
recon_symbolic_d.py control | Self-check of the cross-slice reconstruction on the run's own slice grid; needs no ff_save/ data; must pass before run |
recon_symbolic_d.py index | Builds slice_index.json, the byte-offset index of the per-slice files |
recon_symbolic_d.py run | Reconstructs the exact $(d,\eta)$ functions in parallel; options --workers, --fns, --limit, --no-loo, --skip-banked |
recon_symbolic_d.py probe3 | Diagnostic: for a few functions, reports which coefficient FireFly normalized to 1 on each slice, then reconstructs them |
harvest_ffsave_2var.py salvage writes these files to the --bank directory:
| File | What it holds |
|---|---|
funcs_2var.jsonl.gz | Exact numerator and denominator, assigned slot and checks passed, one line per function |
extracted_<family>.m | The recovered reduction in kira2math format |
RESIDUAL_TODO.json, residual_targets/<label>.target | What still has to be reduced |
QUARANTINE.jsonl | One record per rejected function, naming the failed check and the error |
EXTRACTION_RECEIPT.json | Summary of the run, with its verdict |
recon_symbolic_d.py takes these inputs. Each path may come from a flag, the --config JSON, or --base; a needed path left unset stops the run with CONFIG: <NAME> unset.
| Option | What it holds |
|---|---|
--wave1 | Directory of per-slice extract outputs node_eps_<a>_<b>_harvest.jsonl, with $d = 4 - 2a/b$; a _HELDOUT suffix marks a slice used only for checking |
--coverage | Each function's $d$-degrees and slice list |
--bank-states | The ff_save/states directory of the reference multi-prime run |
--ff-helper | FireFly's ReconstHelper.cpp, read for the prime table |
control slices a known synthetic function at the run's own $d$-grid (the slice names in the coverage file) and pushes it through the same code. Exact recovery is required, and each of several deliberately corrupted inputs must fail. The result goes to control_result.json, and the exit code is 0 only on PASS. A grid with fewer than 16 slices is too small for the synthetic function, and the run stops with a CONTROL: message.
Each line of shard_NNN.jsonl holds integer $d$-coefficient lists per power of eta in N and D, the slices used, and a status. The status is BANKED when every check passed. Otherwise it is the name of the failed check (FIT_FAIL, HELDOUT_FAIL, ...).
Requirements
- Runs on: Python 3. The two harvest scripts use only the standard library;
recon_symbolic_d.pyalso needspython-flint. - Input: every script needs a user-supplied
ff_save/directory from a Kira + FireFly run. Only thecontrolself-check runs without one. No sample save is included. - Self-test:
python3 selftest.pyin the package directory. It printssmoke okfor the two--helpchecks and for theCONFIGerror ofcontrolrun without inputs. It then runscontrolon three generated slice grids and printscontrol okper grid, or a named skip ifpython-flintis missing. - Exit code of the self-test: nonzero on purpose (code 4). The last line begins
DATA GATE:and says that the harvest checks need your ownff_save/data.
Source and references
The code is tools/ffcapital/ in the BootLoops repository. GUIDE.md there lists the checks each script applies.
| Reference | What it supplies |
|---|---|
| J. Klappert and F. Lange, "Reconstructing rational functions with FireFly", arXiv:1904.00009; code: FireFly | FireFly, the program that writes the saved-state files FFCapital reads |
| J. Klappert, S. Y. Klein and F. Lange, "Interpolation of dense and sparse rational functions and other improvements in FireFly", arXiv:2004.01463 | The same program, second paper |
| J. Klappert, F. Lange, P. Maierhöfer and J. Usovitsch, "Integral reduction with Kira 2.0 and finite field methods", arXiv:2008.06494 | Kira, the program whose reductions are recovered |