Mixture models (statistics)
Mixture models describe data drawn from several overlapping populations, and deciding how many populations there are is notoriously hard. The Bayesian evidence for that number can be computed exactly, from two components to infinitely many.
Mixtures have a bad reputation among statisticians, “inherently evil” in one much-quoted verdict, and the Bayesian way to count the populations rests on an integral long considered intractable. For mixtures of one discrete variable that integral can now be computed exactly. One consequence, worked out for a simple clinical-trial design: ruling out a spurious second population takes millions of participants, where the textbook criterion says twenty-two would do.

The running example. Each of $N$ trial participants answers yes or no four times, and the bars tally how many said yes 0 to 4 times in a simulated trial where everyone answers yes half the time; violet marks show what one population predicts, dashed orange an equal blend of two hidden cohorts answering yes 40 and 60 percent of the time. The dial is the Bayes factor for one population against any two-cohort blend, on the conventional scale where 3 counts as positive evidence and 20 as strong. It grows only as the fourth root of $N$ and passes 20 near 3.3 million participants. (Illustration; the sample is simulated, the law is the paper’s.)
The twilight zone
In 1894 Karl Pearson fit a sum of two normal curves to Raphael Weldon’s crab measurements and asked whether the animals formed one population or two. His model is the prototype of a mixture model: each observation comes from one of $g$ hidden components, chosen with fixed probabilities called the weights, and each component has parameters of its own.
In August 2012 Larry Wasserman, a statistician at Carnegie Mellon, put the case against mixtures on his blog Normal Deviate under the title “Mixture Models: The Twilight Zone of Statistics.” Granting that mixtures are useful, he listed what goes wrong in the simplest case, a mixture of bell curves, and closed: “I have decided that mixtures, like tequila, are inherently evil and should be avoided at all costs.” The post has been quoted and argued with ever since, from Andrew Gelman’s reply eleven days later to the 2019 Handbook of Mixture Analysis, whose chapter on choosing the number of components closes by quoting the tequila verdict and declining to go that far.
Wasserman’s seven items amount to five complaints. First, the likelihoodthe probability of the observed data, read as a function of the model’s parameters; fitting usually means finding the parameters that make it largest misbehaves. For a mixture of bell curves it grows without limit as one component shrinks onto a single data point. Even where it stays finite it has many peaks, and the EM algorithm, the standard iterative fitting method, finds only one of them. Second, the components cannot always be told apart. Swapping two labels changes nothing, and wherever two components coincide or a weight is zero a parameter drops out of the model. Third, as a result, standard large-sample theory fails: likelihood-ratio tests and the Bayesian information criterion (BIC) assume a single well-behaved best fit. Fourth, a flat priorthe probability distribution placed on a model’s parameters before seeing the data; a flat (improper) prior spreads evenly over an infinite range on a component’s location gives an improper posterior, one that can never be normalized, at any sample size, so a Bayesian must commit to a proper prior. Fifth, fitted components need not mean what one hopes. A mixture of $g$ bell curves can have fewer or more than $g$ peaks, and assigning each point to its most probable component can carve up the data in unintuitive ways.
Two more difficulties, which the post leaves aside, concern the Bayesian route. A Bayesian compares one component against two through each model’s evidencethe marginal likelihood: the probability of the observed data under a model with its parameters averaged over their prior instead of set to best-fit values, and the ratio of two evidences is the Bayes factorthe factor by which the data multiply the prior odds between two models; by convention 3 counts as positive evidence and 20 as strong. For a mixture the evidence is an integral with $g^N$ terms at $N$ observations, one for each way of allocating observations to components, too many to sum directly. In practice it is estimated by Monte Carlocomputational methods that estimate a quantity by averaging over many random draws; the answer comes with a statistical error bar of its own methods, and on mixtures these have troubles of their own: Chib’s method, for one, falls short by $\log g!$ unless the simulation visits every relabeling of the components, the behavior called label switching.
Exact evidence for discrete data
Everything from here to the last section concerns mixtures of one discrete variable; the continuous case, Pearson’s bell curves, is taken up at the end. The exact computation applies to discrete data. Each observation falls in one of $k$ cells, and a dataset is summarized by the cell counts $U=(U_1,\ldots,U_k)$ of its $N$ observations. Each component is a vector of cell probabilities with a uniform or, more generally, a Dirichlet prior, the standard family of priors on probability vectors. With whole-number Dirichlet parameters the two-component evidence is an explicit formula in the cell counts. Under uniform component priors the evidence is a finite sum at any number of components and in the infinite limit. These sums are evaluated exactly at sample sizes up to a hundred thousand for two components, and at smaller sizes as $g$ and $k$ grow. Nothing is simulated, so the values carry no Monte Carlo error bar.
For discrete data the complaints about the likelihood disappear: it is bounded, and because the evidence integrates over every parameter value no peak has to be located, no EM iteration is needed and the label symmetry is integrated over with everything else. The non-identifiability does not disappear, since it is a property of the models themselves, and for a single discrete variable it is complete: a blend of $k$-cell distributions is again a $k$-cell distribution, and Teicher proved in 1963 that the components can then never be recovered from the data. Models with one, two or infinitely many components describe the same set of distributions, so their evidences differ only through the priors that the component structure places on that set, and the Bayes factor compares priors rather than fits. The exact values below show what that comparison comes to. Standard large-sample theory does fail, as Wasserman said; its replacement for models of this kind, Watanabe’s singular learning theory, fixes the rate at which the log evidence grows but leaves a constant undetermined, and that constant is computed here for the trial design below. Every result uses proper priors and holds for a stated range of them.
The collapse of the sum
The computation rests on a cancellation. Multiplying out the two-component likelihood turns the evidence into a sum over all $2^N$ allocations of the observations to the components, each term a textbook integral in factorials. Evans, Gilula and Guttman wrote such sums down in 1989 and judged them too complicated for practical use. In 2009 Lin, Sturmfels and Xu reorganized them, proved that the evidence is a rational number, and evaluated it exactly for datasets of a few hundred observations. The new observation is that for one discrete variable the factorials from the likelihood cancel those from the component integrals. A term then depends only on how many observations went to the first component, whichever ones they are. The $2^N$ terms collapse to $N+1$, and partial fractions evaluate what remains in harmonic numbers, $H_j = 1+\tfrac12+\cdots+\tfrac1j$, which grow like $\log j$. For binary data with $n$ observations in each cell the two-component evidence under uniform priors is (Theorem 1)
$$Z(n,n) \;=\; \frac{(n!)^2}{(2n+1)!}\left[\frac{1}{n+1} + 2\big(H_{2n+1}-H_{n+1}\big)\right]$$
Here $Z(n,n)$ is the probability that the two-component model assigns to the observed data in the order recorded; Theorems 2 and 3 give the closed form for any counts, any number of cells and any whole-number Dirichlet priors. The one-component evidence at the same counts is the factor in front, $(n!)^2/(2n+1)!$, so the Bayes factor for one population against two is the reciprocal of the bracket. Since $H_{2n+1}-H_{n+1}$ tends to $\log 2$, the Bayes factor tends to $1/(2\log 2)=0.72134\ldots$ (Corollary 1), and it is already $0.7265$ at fifty observations per cell. A single yes-or-no variable split fifty-fifty thus never yields strong evidence either way, at any sample size, and the mild preference that remains favors the mixture. Away from fifty-fifty the limit is another constant, which favors one population only for splits more lopsided than about 20:80.
Components, readings and participants
Three whole numbers set the size of a mixture problem: the number of components $g$, of cells $k$ and of observations $N$. The applications range widely in all three. The trial design below has two components, five cells and millions of observations. At the other extreme, mutational signatures in cancer gene panels involve ten components and ninety-six cells but as few as five observations. In record linkage, which asks how many distinct individuals a file of noisy records describes, the components are those individuals, tens of thousands of them in the largest files analyzed.
The trial design that serves as the running example adds a fourth number, the readings $m$ per participant. A trial records a yes-or-no outcome $m$ times for each of $N$ participants, who thus fall into one of $k=m+1$ cells by their number of yes answers. If everyone responds with the same probability $\theta_0$ the expected counts are binomialthe distribution of the number of successes in a fixed number of independent yes-or-no trials that share one success probability; if the participants secretly form two cohorts, responders and non-responders, the counts follow a blend of two binomials, a little flatter (figure below).

A mixture question in the four-reading design ($m=4$), drawn as coin flips (a head stands for a yes answer). Bars: the fraction of participants with 0 to 4 yes answers out of four when everyone responds with probability one half. Orange line: an equal blend of two cohorts responding with probabilities 0.4 and 0.6, which differs from the bars by three percentage points at two out of four. Dotted line: a blend of 0.3 and 0.7, visibly flatter. A mild blend is nearly indistinguishable from one population. (Illustration drawn for this page from elementary binomial probabilities; not a figure of the paper.)
With two or more readings the two-cohort model falls outside the family covered by the closed forms, since one response probability fixes all of a cohort’s $m+1$ cell probabilities. It is handled instead through a large-sample expansion of its allocation sum, and the exact Bayes factors quoted below come from direct numerical integration over its three parameters. To design such a trial one needs to know how large $N$ must be, at given $m$, before the Bayes factor can favor one cohort ($g=1$) over a spurious second one ($g=2$) when the data really come from one cohort.
The fourth-root law
With a single reading per participant ($m=1$), as shown above, no number of participants suffices. At $m=4$ the Bayes factor does grow with $N$, though slowly. In Watanabe’s theory the log evidence of each model falls below the best attainable log likelihood by $\lambda\log N$ plus a constant. The learning coefficienthow fast a model’s log evidence falls behind its best fit as data accumulate; half the parameter count in regular models, often less for mixtures $\lambda$ is $1/2$ for one cohort and, by a 2004 result of Yamazaki and Watanabe, $3/4$ for the two-cohort binomial mixture. The missing constants can be obtained for data with every count at its expected value. For such data the terms of the allocation sum peak along a whole ridge of ways to split the observations between the cohorts, and summing around the ridge gives the constant in closed form (Theorems 6 and 7). The Bayes factor for one cohort against two is then
$$\mathrm{BF}_{\,\rm ind:mix} \;=\; \frac{2\cdot 3^{1/4}}{\pi\,\Gamma(1/4)}\;\cdot\;\frac{N^{1/4}}{\sqrt{\theta_0(1-\theta_0)}}\;\big(1+o(1)\big)\,, \qquad \frac{2\cdot 3^{1/4}}{\pi\,\Gamma(1/4)} \;=\; 0.231089\ldots$$
where ind:mix means one population (the “independence model”) against the mixture, $\Gamma$ is Euler’s gamma function and $o(1)$ a correction that vanishes as $N$ grows (Corollary 2). The power $1/4$ is the difference of the two learning coefficients, and the number in front comes from the ridge. Fourth-root growth is very slow: at $\theta_0=\tfrac12$ the exact Bayes factor first reaches 3, conventionally positive evidence, at $N=1{,}281$, and reaches 20, strong evidence, near $N=3.3$ million (figure below). BIC, the criterion most used in practice, divides each model’s best-fit likelihood by $\sqrt{N}$ for every free parameter. The mixture has two parameters more than a single cohort, so the BIC version of the Bayes factor grows in proportion to $N$. On the median dataset generated by one cohort, BIC gives twenty-to-one odds against a second cohort already at $N=22$, where the true Bayes factor is about $1.2$, so BIC understates the required sample size by five orders of magnitude.

How slowly the evidence against a spurious second cohort accumulates. The curve is the Bayes factor for one population against a two-cohort mixture, with four readings per participant at a fifty-fifty response rate, on data with every count at its expected value; both axes are logarithmic. Slate line: the large-sample law above, $0.462\,N^{1/4}$. Open circles: exact values by numerical integration. Filled dots: the exact crossings of 3 at $N=1{,}281$ and of 20 near $3.3$ million. Orange dashes: the growth in proportion to $N$ implied by the parameter-counting BIC, which passes 20 at about twenty participants. Violet: the single-reading design, whose Bayes factor settles at $1/(2\log 2)\approx0.72$ and never resolves. Strong evidence takes millions of participants, five orders of magnitude more than BIC suggests. (Redrawn for this page after Figure 1 of the paper, from the same law and exact values.)
The derivation goes through for any number of readings $m\ge 2$ and makes the trade-off between readings and participants explicit (equation (6.2)):
$$c_m(\theta_0) \;=\; \frac{\sqrt2\,\big(m(m-1)\big)^{1/4}}{\pi\,\Gamma(\tfrac14)\,\sqrt{\theta_0(1-\theta_0)}}\,, \qquad N^*(B) \;=\; \frac{B^4\,\pi^4\,\Gamma(\tfrac14)^4\,\big(\theta_0(1-\theta_0)\big)^{2}}{4\,m(m-1)}$$
Here $c_m(\theta_0)$ is the coefficient of $N^{1/4}$ in the Bayes factor, reducing at $m=4$ to the formula above, and $N^*(B)$ is the number of participants at which this law reaches a target value $B$. The exact Bayes factor crosses a little sooner than the law, at $1{,}281$ rather than $1{,}775$ for $B=3$. The target enters as $B^4$, so asking for odds of 20 instead of 3 multiplies the enrollment by $(20/3)^4$, nearly two thousand. The readings enter through the factor $1/(m(m-1))$, which is infinite at $m=1$ (the never-resolving case) and equals $1/2$, $1/12$ and $1/90$ at two, four and ten readings; ten readings instead of four therefore divide the required enrollment by $7.5$. The factor $m(m-1)$ counts pairs of readings on one participant: hidden cohorts change how often the same person answers yes twice, and a single reading has no pairs to count. BIC, by contrast, reaches twenty-to-one at about twenty participants however many times each is read.
These sample sizes are design values: they hold for idealized data with every count at its expected value and for the stated priors. Sampling noise or a different prior moves the constant by a factor of order one without changing the fourth-root rate.
Any number of components, and infinitely many
The cancellation behind the two-component closed forms persists at any number of components $g$, provided the component priors are uniform (Theorem 4). A term then depends only on the $g$ component sizes, and the allocations with given sizes are counted by a classical generating function for contingency tables, whole-number arrays with fixed row and column totals.
The cost depends on all three of $g$, $k$ and $N$. After the collapse the two-component evidence takes time polynomial in the number of cells and observations, so a thousand cells at a hundred thousand observations is a routine computation on one processor core, with the answer an exact fraction. With more components the work grows as the smaller of $N^{g-1}$ and $N^k$: a few components on dozens of cells reach about ten thousand observations, while on two cells even a million components at five thousand observations is quick (in floating point, checked against exact values at smaller sizes).
The modern way to avoid choosing $g$ is to make it infinite. Give the weights a symmetric Dirichlet prior with parameter $\alpha/g$ in each coordinate and let $g\to\infty$ at a fixed concentration $\alpha$; the limit is the Dirichlet processFerguson’s random probability distribution; as a prior over mixtures it lets the data fill as many clusters as they support, more readily at larger concentration mixture, built on the random distribution that Ferguson introduced in 1973. Its evidence is a sum over every partition of the observations into clusters, some $10^{115}$ terms at $N=100$, and earlier exact computations had reached about twenty observations. The infinite limit can nevertheless be taken exactly (Theorem 5). With the $\alpha/g$ weights the $g$-component evidence is a polynomial in $1/g$ whose constant term is the Dirichlet-process evidence, a finite sum over cluster sizes weighted by the same table counts. Passing to infinitely many components is the substitution $1/g=0$. On two cells the infinite case runs at five thousand observations about as easily as the finite ones.
A standard test case for the Dirichlet-process evidence is Escobar and West’s counts of eye-tracking anomalies in 101 patients with schizophrenia, where it had been computed before only by Monte Carlo. The counts are binned into two or three cells; unbinned, on 35 cells, they are beyond the method’s reach. At seven concentrations from $1/3$ to $22/7\approx3.14$ the Dirichlet process is favored over its own truncations at one, two and three components every time, though never strongly: the $\log_{10}$ Bayes factors run from $0.04$ to $0.61$ (Table 5). The posterior on the number of occupied clusters, on the other hand, is driven by the concentration $\alpha$, a number the analyst chooses as part of the prior. The most probable cluster count moves from 2 to 12 across the grid, and the distribution is broad everywhere (figure below).

Exact posterior for the number of occupied clusters under the Dirichlet process mixture on the eye-tracking data (101 patients, binned into two cells), one curve per concentration from $\alpha=1/3$ (lightest) to $\alpha=22/7$ (orange). As the concentration rises the most probable number of clusters moves from 2 to 12 and the distribution stays broad throughout: the data alone do little to fix the number of clusters. (Redrawn from the exact values behind Figure 2 of the paper.)
Checking the estimators
Exact values also make it possible to measure each Monte Carlo estimator’s error directly instead of trusting its own error bar. Five standard estimators, four in our own implementations and nested sampling through the dynesty package of Speagle, were run on the one-reading and four-reading families at sample sizes from a hundred to a million. The harmonic mean is biased upward at every size. Chib’s method diverges on the four-reading family, and averaging over label permutations leaves the divergence in place, so label switching does not appear to be its cause. Bridge sampling agrees with the exact values up to a few thousand observations and drifts low beyond, far short of the millions at which the trial question above is settled. Stepping-stone and nested sampling agree with the exact values throughout, stepping-stone with a drift of about half a natural-log unit at a million.
Continuous data: two bell curves
Pearson’s original question was about bell curves, and there no exact evidence is available: with continuous data every one of the $2^N$ allocations gives a different term and nothing collapses. The large-sample route of the fourth-root law still goes through, because the ridge computation behind it needs only the shape of the likelihood near the truth, not a finite sum. Take one normal component of known width against a mixture of two such components with a uniform prior on the weight and a normal prior on each mean, and data that really come from one component. The Bayes factor for one component again grows as the fourth root of the sample size, with an explicit constant set by the prior density of the means at the true value (Proposition 1):
$$\mathrm{BF}_{\,\rm one:two} \;=\; \frac{\sqrt2}{\pi\,\Gamma(1/4)\,\varphi(\mu_0)}\;N^{1/4}\,\big(1+o(1)\big)\,, \qquad N^*(B) \;=\; \frac{B^4\,\pi^4\,\Gamma(\tfrac14)^4\,\varphi(\mu_0)^4}{4}\,,$$where $\varphi(\mu_0)$ is the prior density of a component mean evaluated at the true mean $\mu_0$, in units of the known width, and $N^*(B)$ is the sample size at which the law reaches a target Bayes factor $B$. This is the trial-design formula above with $\varphi(\mu_0)^4$ in place of the readings-and-response factor. For standard normal priors centered on the truth the Bayes factor reaches 3 near $8{,}600$ observations and 20 near $17$ million, so the slow accumulation of evidence against a spurious second component is not a peculiarity of discrete data. A prior that puts less density at the true mean favors one component sooner. Reduced-rank regression, treated the same way, gives a second continuous family in which the constants come out in closed form. The bell-curve result covers two components of known, equal width in the large-sample idealization; it is a statement about how fast evidence can accumulate, not an exact evidence for a given dataset.
The paper
- Exact Bayesian evidence for mixture models: from two components to infinity (PDF) — Matthew D. Schwartz and Subhabrata Sen. A preliminary version of the paper.
Supplementary material
- Mixalot — the Python package that evaluates the exact evidences on this page, with usage, a capability map and worked examples; the code is in the BootLoops repository under
tools/mixalot.
References
| K. Pearson, III. Contributions to the mathematical theory of evolution, Phil. Trans. R. Soc. Lond. A 185 (1894) 71 | two normal curves fit to Weldon’s crab measurements; the first “one population or two” |
| H. Teicher, Identifiability of finite mixtures, Ann. Math. Statist. 34 (1963) 1265 | a mixture of one binary variable cannot be identified at any sample size |
| T. S. Ferguson, A Bayesian analysis of some nonparametric problems, Ann. Statist. 1 (1973) 209 | the Dirichlet process |
| G. Schwarz, Estimating the dimension of a model, Ann. Statist. 6 (1978) 461 | the BIC, a parameter-counting approximation whose assumptions mixtures violate |
| M. Evans, Z. Gilula and I. Guttman, Latent class analysis of two-way contingency tables by Bayesian methods, Biometrika 76 (1989) 557 | the mixture evidence as a finite sum, judged too complicated for practical use |
| R. E. Kass and A. E. Raftery, Bayes factors, J. Amer. Statist. Assoc. 90 (1995) 773 | the conventional scale on which a Bayes factor of 3 is positive evidence and 20 strong |
| S. Chib, Marginal likelihood from the Gibbs output, J. Amer. Statist. Assoc. 90 (1995) 1313 | Chib’s estimator of the evidence, one of the five tested against the exact values |
| S. Richardson and P. J. Green, On Bayesian analysis of mixtures with an unknown number of components, J. R. Stat. Soc. B 59 (1997) 731 | reversible-jump sampling of mixture posteriors with the number of components unknown |
| J. Pitman and M. Yor, The two-parameter Poisson–Dirichlet distribution derived from a stable subordinator, Ann. Probab. 25 (1997) 855 | the Pitman–Yor process, the two-parameter generalization of the Dirichlet process; one of the partition priors compared in the supplement’s record-linkage analysis |
| M. D. Escobar and M. West, Computing nonparametric hierarchical models, in Practical Nonparametric and Semiparametric Bayesian Statistics, Lecture Notes in Statistics 133, Springer (1998) 1 | the eyetracking data and their Monte Carlo analysis under a Dirichlet process mixture |
| K. Yamazaki and S. Watanabe, Newton diagram and stochastic complexity in mixture of binomial distributions, in Algorithmic Learning Theory, Lecture Notes in Computer Science 3244 (2004) 350 | the exponent $\lambda=3/4$ for the binomial mixture, without its constant |
| S. Watanabe, Algebraic Geometry and Statistical Learning Theory, Cambridge University Press (2009) | singular learning theory: the $\lambda\log N$ form of the log evidence for singular models |
| S. Lin, B. Sturmfels and Z. Xu, Marginal likelihood integrals for mixtures of independence models, J. Mach. Learn. Res. 10 (2009) 1611 | exact rational evidence for datasets of a few hundred observations; the family this paper builds on |
| L. Wasserman, Mixture models: the twilight zone of statistics, Normal Deviate (blog), August 4, 2012 | the list of mixture pathologies and the tequila verdict |
| Further reading for this page (not cited in the paper) | |
| A. Gelman, How I think about mixture models, Statistical Modeling, Causal Inference, and Social Science (blog), August 15, 2012 | the reply to Wasserman’s post, eleven days later |
| A. Feller, E. Greif, L. Miratrix and N. Pillai, Principal stratification in the Twilight Zone: weakly separated components in finite mixture models, arXiv:1602.06595 (2016) | weakly separated mixture components in causal inference; revised in 2019, with N. Ho, as Weak separation in mixture models and implications for principal stratification |
| G. Celeux, S. Frühwirth-Schnatter and C. P. Robert, Model selection for mixture models: perspectives and strategies, in Handbook of Mixture Analysis, S. Frühwirth-Schnatter, G. Celeux and C. P. Robert (eds.), Chapman and Hall/CRC (2019) 117 | chapter 7, on choosing the number of components, which closes by quoting the tequila verdict without going that far |