Qinvert

The content on this page was written by AI under human supervision.

Qinvert works backward from published summary statistics to the data rows that must have produced them. It takes statistics an agency published exactly (quartiles, outlier fences, minima, maxima, means) together with the part of the dataset the agency released, and returns the smallest sets of added or removed rows that make every published statistic recompute exactly. All arithmetic is done in Python Fractions, so a match means equality, never agreement within a tolerance.

What it does

Agencies often publish statistics computed on a full dataset while releasing only part of it: rows for closed entities, small cells or private records are withheld. When the released rows do not reproduce the published numbers, the discrepancy says something about the withheld rows, and Qinvert turns that into an exact search. A target is a pair (statistic, published value). The statistic is a quantilethe value below which a given fraction p of the data lies; statistical packages define it in several slightly different ways under a named definition, a Tukey fencean outlier cutoff placed a multiple m of the interquartile range below the first quartile or above the third, an order statistic (the j-th smallest value), a mean, or a linear combination of these. The published value is an exact number, or an Interval when the publication is capped or only bounds it. Given the released values and the targets for one dataset, solve finds delta families: added values plus removed released values, as few in total as possible, after which every target recomputes exactly.

Statistical software disagrees on how to compute a quantile, so Qinvert implements the five SAS definitions ("sas1" to "sas5", SAS's QNTLDEF 1–5) and the nine R types of Hyndman and Fan ("r1" to "r9"). Each target is stated under the convention the publisher used. A Tukey fence pair with multiplier $m$ is $\mathrm{lo} = (1+m)Q_1 - mQ_3$, $\mathrm{hi} = (1+m)Q_3 - mQ_1$, and when neither published fence sits at its cap the pair fixes both quartiles:

$$Q_1 = \frac{(1+m)\,\mathrm{lo} + m\,\mathrm{hi}}{1+2m}, \qquad Q_3 = \frac{m\,\mathrm{lo} + (1+m)\,\mathrm{hi}}{1+2m}.$$

A fence published at its cap value gives only an inequality, and the solver then scans candidate values for the quartile left free.

Every family is verified before it is returned, by rebuilding the dataset and re-evaluating every target, so a reported family is always a genuine solution. The converse is limited by search caps (total delta size k_cap, removals per family rem_cap, families kept max_witnesses, and the budgets in DEFAULT_CAPS), and every cap the search reaches is named in the result's caps_hit list. An empty result whose only note is k_cap-exhausted means no family exists up to that depth, within the structural limits listed in the README; an empty result with further notes means only that a truncated search found nothing.

stack covers one withheld entity appearing in several datasets, such as one closed contract entering every measure published for its year. It solves each dataset separately, then assembles the families into shared entities carrying one value per dataset they take part in. A dataset whose targets already recompute exactly must stay exact, and stack raises StackerError instead of returning a delta that breaks one. The assembly is greedy: it shows the returned entity set works, not that no smaller set exists.

Limits. A family shows the publication is consistent with some delta of that size and shape; it does not identify the withheld rows, and several families usually fit the same publication. Published values must be the exact rationals the publisher's convention produces; if the publisher rounds for display, enter the display range as an Interval. Floats are refused with a TypeError; pass integers, Fractions or decimal strings such as "1.41". Qinvert does not address the forward question, whether a printed number can be reproduced from its stated inputs at all.

Examples

One dataset with a capped fence pair (the README quickstart). The released scores have two decimals; the publisher printed a low outer fence of 0, which is this measure's cap, and a high outer fence of 1.41, under SAS's default quantile definition. values is your list of released scores.

from fractions import Fraction as F
from qinvert import (Quantile, TukeyFence, Mean, Interval,
                     solve, Dataset, stack)

# one dataset, published Q1/Q3 (SAS default definition) + capped fences
targets = [
    (TukeyFence("lo", 3, "sas5", cap=0), F(0)),        # published "0" = cap
    (TukeyFence("hi", 3, "sas5"),        F("1.41")),   # exact published
]
res = solve(values, targets, register=F(1, 100),  # data are 2dp
            value_lo=0, k_cap=8, rem_cap=2)
res["k"]          # minimal delta size, or None if unreachable within caps
res["families"]   # [{k_add, k_rem, removed, adds:[{value, window, role}]}]
res["caps_hit"]   # every truncation leaves a note

register=F(1, 100) puts added values on the 0.01 grid and value_lo=0 forbids negative scores. res["k"] is the smallest number of added plus removed rows that reproduces both fences, searching up to 8 rows with at most 2 removals per family; if the released rows already match, it is 0 and res["exact_already"] is True. Each family lists removed values and adds. An add's window (lo, hi, lo_strict, hi_strict), when present, means any grid value inside it also keeps the order-statistic targets exact. An answer therefore often reads "one extra row anywhere above some value." Check res["caps_hit"] before reading anything into an empty result.

Invert a fence pair for the quartiles. From the package tests; the last element of the returned tuple is the mode, which says which quartiles the pair determines.

from fractions import Fraction as F
from qinvert import invert_fence_pair

invert_fence_pair("0", "1.41", 3, F(0), None)[2]     # "hi_only"
invert_fence_pair("57", "100", 3, F(0), F(100))[2]   # "lo_only"
invert_fence_pair("0", "100", 3, F(0), F(100))[2]    # "none"

In the first call the low fence equals its cap of 0, so only the high-fence equation constrains the quartiles; in the second the high fence is at its cap; in the third both are, and only inequalities remain. With two uncapped fences the mode is "both" and the first two elements are the exact $Q_1$ and $Q_3$ from the formula above.

Several datasets sharing withheld entities (also from the README). Each Dataset holds released rows as (entity_id, value) pairs plus its own targets. Two entity classes are declared, the second never participates in D2, and an entity present in both datasets must carry the same value in each.

d1 = Dataset("D1", rows1, targets1, register=F(1, 100))   # rows: (eid, value)
d2 = Dataset("D2", rows2, targets2, register=F(1, 100))
out = stack([d1, d2], classes=("MA-PD", "MA-only"),
            participation=lambda cls, ds: not (cls == "MA-only" and ds == "D2"),
            couplings=[("D1", "D2")],   # entity in D2 carries equal value in D1
            k_cap=8, rem_cap=1)
out["entities"], out["removed_entities"], out["battery"], out["regressions"]

out["entities"] lists the added entities, each with an id, a class cls and a values map from dataset name to the value it carries there; out["removed_entities"] lists released entity ids taken out. out["battery"] holds one record per dataset with its before_exact and after_exact flags, and out["unsolved"] and out["notes"] name any dataset not solved within the caps and why. regressions is empty on any normal return, since a regressing delta raises StackerError instead.

Routines

Statistics and targets (targets.py)

Solving one dataset (solver.py)

Stacking across datasets (stacker.py)

Tests

Used on this site

Requirements and source

Python 3, standard library only. Run the tests with python3 tests/test_all.py from the package directory (or python -m pytest tests/test_all.py -q). Two tests cross-check against the original Medicare audit's own code and data files, which are not part of the repository; they look for that tree at the path in the QINVERT_ORIGIN_ROOT environment variable and are skipped when it is unset or the files are absent. The code is in tools/qinvert/ in BootLoops' bootloops-dev repository (GitHub organization BootLoops-ai), released under the MIT license; its README.md has the algebra and the full list of structural limits. Quantile definitions follow Hyndman and Fan, "Sample Quantiles in Statistical Packages," The American Statistician 50(4), 361–365 (1996), and the Base SAS 9.4 Procedures Guide, "The UNIVARIATE Procedure: Calculating Percentiles."

← back to the tools index