FFCapital

FFCapital recovers the exact coefficients that had already finished when a FireFly rational-function reconstructionfinding an exact rational function from its values modulo large primes stopped early, for example because the job ran out of memory or the machine went down. Its input is the ff_save/ directory that Kira with FireFly writes while reducing a family of Feynman integrals, and each recovered function is checked against a value stored there. FireFly is the program of Jonas Klappert, Sven Yannick Klein and Fabian Lange; FFCapital reads the files FireFly leaves on disk and never resumes or advances the reduction.

Where it comes from

FireFly and the format of its saved-state files are by Jonas Klappert, Sven Yannick Klein and Fabian Lange. FFCapital is a new BootLoops program that reads those files. The file layout was taken from FireFly's source code, and so were the first 40 entries of its table of 63-bit primes. No FireFly routine is included or run. recon_symbolic_d.py reads the table from the user's copy of ReconstHelper.cpp.

Part of FFCapitalSourceAuthors
File readers that follow the layout their code writes: section names, line formats, the ZERO marker and validation.gzRatReconst.cpp and Reconstructor.hpp in FireFlyKlappert, Klein and Lange
Constants copied from their code: the first 40 primes of the table, in the two harvest scriptsReconstHelper.cpp in FireFlyKlappert, Klein and Lange
New code: the check against the stored values, the prime detection, the slot assignment, the fit in $d$ and its checksWritten for BootLoopsCode by Claude

Please cite the authors' papers when you use it. FFCapital is not affiliated with or endorsed by the FireFly authors.

What it does

In an IBP reductionintegration-by-parts identities relate the many integrals of a Feynman family to a small basis; solving that large linear system is the reduction each coefficient is a rational function of the spacetime dimension $d$ and the kinematic variables. Kira with FireFly finds it by sampling the linear system modulo large primes and reconstructing the exact function from the samples.

FireFly saves the state of each function in ff_save/states/<n>_<k>.gz, where <n> is the function index and <k> is the counter of the last prime at which the state was saved. A state file with is_done equal to 1 already holds the complete exact numerator and denominator coefficients. FireFly also writes ff_save/validation.gz: one probe point and the value of every function there, computed from the linear system directly and not from the reconstruction. Both files remain on disk after a crash.

FFCapital rebuilds the function $N/D$ (numerator over denominator) for each finished function. It accepts the function only if $N/D$ at the probe point reproduces the stored value modulo the FireFly prime. The prime is detected from FireFly's table of 63-bit primes and must come out the same for a sample of up to eight finished functions; otherwise the run stops with an error instead of guessing.

A function that fails is marked in the output and named on standard error, the others are unaffected, and the exit code is nonzero. All arithmetic on the value path is exact, with no floating-point numbers anywhere.

The package has three scripts. Which one to use depends on the shape of the saved run.

Equality is always tested as equality of rational functions, $P\,D_s = \lambda\,Q\,N_s$ with one overall scale $\lambda$. Here $P/Q$ is the fitted function at the $d$ of slice $s$ and $N_s/D_s$ is the function stored for that slice. The test has this form because a slice whose numerator and denominator share a factor stores a reduced but correct function.

FFCapital has these limits:

ffsave_degree_census.py, on the Kira, FireFly and Fermat page, reads the same states for degree statistics. Numkin sets up the fixed-$d$ slice runs that recon_symbolic_d.py combines.

Examples

With Python set up (see Requirements), run the scripts with python3 from the package directory tools/ffcapital/ on your own ff_save/.

Inventory a stopped one-variable run, then extract what finished. From the usage text of harvest_ffsave_eta.py:

harvest_ffsave_eta.py census  --ffsave <dir> --out census.jsonl [--workers 16]
harvest_ffsave_eta.py extract --ffsave <dir> --out funcs.jsonl
                              (--fns 0,17,26683 | --all-done) [--workers 16]
                              [--no-validate]

census writes one JSON line per state file: function index fn, FireFly tag, the done flag, np (how many primes had been combined) and the degrees degN and degD. extract writes one line per requested function, with N and D as maps from the power of eta to an exact string "p/q" and a validated field naming the prime index used. A function that fails gets "validated": false with the reason, standard error lists the failing indices, and the exit code is 2.

Recover and assign a two-variable save. From the usage text of harvest_ffsave_2var.py:

harvest_ffsave_2var.py salvage \
  --primary <ff_save dir> [--extra label=<ff_save dir> ...] \
  --slice label=<kira_target.m>=<d_num>/<d_den> ... (>=2) \
  [--census <ffsave_degree_census json>]   (degree stats for RESIDUAL_TODO) \
  [--residual-target label=<target file> ...] (staged slices to shrink) \
  --bank <outdir> [--family myfam] [--workers 24] \
  [--min-slice-pass 2] [--kill-rate 0.001]

Each --slice is a finished Kira reduction of the same system at $d =$ d_num/d_den, exported with kira2math; at least two are required. The files written to the output directory are listed under Routines. If hard failures exceed --kill-rate (default 0.1%) of the finished functions, the save is treated as contaminated. The recovered reduction is then not written, EXTRACTION_RECEIPT.json records the verdict CONTAMINATED_STOP, and the exit code is 3.

Reconstruct the $d$-dependence from many fixed-$d$ runs. From the package guide:

recon_symbolic_d.py control|index|probe3|run --outdir <dir> [--coverage ...]
  [--wave1 ...] [--bank-states ...] [--ff-helper <ReconstHelper.cpp>] [--p2-file ...]
  [--config <json>] [--base <root>] [--skip-banked <prior>]

Run control first and again after any code change. It needs only --coverage, --ff-helper and --outdir. Then index writes slice_index.json, and run writes shard_NNN.jsonl with one line per function. The inputs, the control check and the output lines are described under Routines.

Routines

CommandWhat it does
harvest_ffsave_eta.py censusInventory of every state file of a one-variable save: done flag, primes combined, degrees
harvest_ffsave_eta.py extractExact coefficients of the listed (--fns) or all finished (--all-done) functions, each checked against validation.gz; exit code 2 on any mismatch
harvest_ffsave_eta.py extract --no-validateThe same without the check; only for a quick look
harvest_ffsave_2var.py salvageExtracts and checks a $(d,\eta)$ save, assigns each function to its (target, master) slot and certifies it; exit code 3 on CONTAMINATED_STOP
recon_symbolic_d.py controlSelf-check of the cross-slice reconstruction on the run's own slice grid; needs no ff_save/ data; must pass before run
recon_symbolic_d.py indexBuilds slice_index.json, the byte-offset index of the per-slice files
recon_symbolic_d.py runReconstructs the exact $(d,\eta)$ functions in parallel; options --workers, --fns, --limit, --no-loo, --skip-banked
recon_symbolic_d.py probe3Diagnostic: for a few functions, reports which coefficient FireFly normalized to 1 on each slice, then reconstructs them

harvest_ffsave_2var.py salvage writes these files to the --bank directory:

FileWhat it holds
funcs_2var.jsonl.gzExact numerator and denominator, assigned slot and checks passed, one line per function
extracted_<family>.mThe recovered reduction in kira2math format
RESIDUAL_TODO.json, residual_targets/<label>.targetWhat still has to be reduced
QUARANTINE.jsonlOne record per rejected function, naming the failed check and the error
EXTRACTION_RECEIPT.jsonSummary of the run, with its verdict

recon_symbolic_d.py takes these inputs. Each path may come from a flag, the --config JSON, or --base; a needed path left unset stops the run with CONFIG: <NAME> unset.

OptionWhat it holds
--wave1Directory of per-slice extract outputs node_eps_<a>_<b>_harvest.jsonl, with $d = 4 - 2a/b$; a _HELDOUT suffix marks a slice used only for checking
--coverageEach function's $d$-degrees and slice list
--bank-statesThe ff_save/states directory of the reference multi-prime run
--ff-helperFireFly's ReconstHelper.cpp, read for the prime table

control slices a known synthetic function at the run's own $d$-grid (the slice names in the coverage file) and pushes it through the same code. Exact recovery is required, and each of several deliberately corrupted inputs must fail. The result goes to control_result.json, and the exit code is 0 only on PASS. A grid with fewer than 16 slices is too small for the synthetic function, and the run stops with a CONTROL: message.

Each line of shard_NNN.jsonl holds integer $d$-coefficient lists per power of eta in N and D, the slices used, and a status. The status is BANKED when every check passed. Otherwise it is the name of the failed check (FIT_FAIL, HELDOUT_FAIL, ...).

Requirements

Source and references

The code is tools/ffcapital/ in the BootLoops repository. GUIDE.md there lists the checks each script applies.

ReferenceWhat it supplies
J. Klappert and F. Lange, "Reconstructing rational functions with FireFly", arXiv:1904.00009; code: FireFlyFireFly, the program that writes the saved-state files FFCapital reads
J. Klappert, S. Y. Klein and F. Lange, "Interpolation of dense and sparse rational functions and other improvements in FireFly", arXiv:2004.01463The same program, second paper
J. Klappert, F. Lange, P. Maierhöfer and J. Usovitsch, "Integral reduction with Kira 2.0 and finite field methods", arXiv:2008.06494Kira, the program whose reductions are recovered

← back to the tools index