Disputed authorship (textual analysis)

Hamilton and Madison claimed some of the same Federalist essays. Several plays printed as Shakespeare's have long been suspected of containing a second dramatist's lines, and critics have proposed anything from one writer to many for Isaiah and the books of Moses. The paper summarized on this page treats a disputed text as a statistical mixture and computes, in closed form, how probable its counts of small grammatical words are under each proposed division. Most answers agree with the scholars; a few do not.

The method in one picture. The function words of a disputed text, tinted here by the hand whose word-rate profile each is drawn from, are modeled as a blend of two authors’ profiles, and their counts alone give a posterior distribution for the second hand’s share; the content words, in gray, never enter. (Site illustration; not a result of the paper.)

Deciding authorship by counting

Who wrote a text is ordinarily settled by evidence outside it: a title page, a printer's register, a letter. When that evidence is missing or contradictory, scholars turn to the language of the text itself. The idea that counting could decide such arguments is Victorian. T. C. Mendenhall compared authors by curves of word length in 1887, the word "stylometry" comes from Wincenty Lutosławski's 1897 book on Plato, and G. Udny Yule studied sentence lengths in 1939. Neither Mendenhall nor Yule attached a probability to a hypothesis.

Frederick Mosteller and David Wallace did, in 1963 and 1964, with the Federalist essays of 1787–88. Hamilton left a list of his essays the day before his duel with Burr; Madison's appeared in 1818. The lists disagree on twelve essays, three more (Nos. 18, 19 and 20) are acknowledged joint work, and five are John Jay's. The two men proved "practically twins" in sentence length, so Mosteller and Wallace separated them by the rates of function wordsClosed-class words such as articles, prepositions, conjunctions, pronouns and particles. Their rates vary more from writer to writer than from subject to subject, which is why authorship tests count them and leave content words out.: upon occurs at three per thousand words in Hamilton and one sixth per thousand in Madison. They computed posterior odds for each disputed essay and gave all twelve to Madison; Nos. 18 and 19 they judged mainly his.

Most later work frames attribution as classification; where a text has more than one hand, existing methods locate the change of style or divide the text among a given number of authors. The composite hypothesis itself, that an essay is one-third Hamilton's or that a book had four sources, seldom receives a probability of its own.

Texts as mixtures

In the paper a text is reduced to the counts $U = (U_1, \ldots, U_k)$ of $k$ listed function words, treated as independent draws. A hypothesis about who wrote it constrains the probability $p_v$ that a token (any one running word of the text) is word $v$, and its support is its evidence: the probability of the observed counts once every unknown rate, share or boundary has been integrated over its prior. For one homogeneous text with a symmetric Dirichlet prior of parameter $\alpha$ on its rates, the evidence is

Here $N$ is the total count of list-word tokens in the text, $\Gamma$ is the gamma function (the factorial extended beyond whole numbers), and $\mathrm{Dir}(p \mid \alpha)$ is the prior on the rate profile $p = (p_1, \ldots, p_k)$, which treats all $k$ words alike; the source-counting analyses below are run at three settings of $\alpha$. For rational $\alpha$ each factor is a ratio of whole numbers, and the evidence integrals in the paper are evaluated directly, many in exact rational arithmetic, with no sampling error. For a text that two known authors may share, the profiles $p_A$ and $p_B$ are the word rates of each author's undisputed writings, held fixed. Under the blend hypothesis each token comes from $A$'s profile with probability $w$, an unknown share with a uniform prior between 0 and 1:

The pure hypotheses are the endpoints $w = 1$ and $w = 0$. The integrand is a polynomial in $w$, so the integral is exact, and the same polynomial, normalized, is the posterior density of $w$; for the Federalist $A$ is Hamilton and the posterior mean of $w$ is the Hamilton share. In real texts, though, an author's rate for a word varies from text to text more than independent draws allow, which is why Mosteller and Wallace rejected that idealization. Every two-author analysis is therefore repeated under a second, hierarchical model, in which each text's rates scatter about its author's profile by an amount fitted to that author's undisputed texts. The evidences under this model are again finite sums in closed form.

For a book whose sources may lie end to end, a hypothesis divides its chapters into $g$ contiguous segments, each with its own rates integrated out by the first formula. Summing over every placement of the boundaries gives a posterior over $g$ and over where the cuts fall. The source of each segment is a "contiguous generating process," never an author: a seam can mark genre or language as easily as a hand. Two hypotheses are compared by their Bayes factor,

positive values favoring $H_1$, each unit a factor of ten in the odds. Two cautions apply throughout. First, the magnitudes are conditional on the count model. The paper therefore treats as robust only which author the posterior of $w$ favors (not where it sits or how wide it is), and which boundaries stay put across prior settings. Second, a negative result counts only where detection power, the chance of catching a division that is really there, has been measured, here by inserting a second author's text into single-author works. A play is long enough to test only if verse by any comparison dramatist of its period, planted to make up a fifth of the play's verse, changes the classification at least eight times in ten. A biblical book needs roughly 4,800 function-word tokens to qualify.

The Federalist and Hamilton's share

For the Federalist, three word lists are used, each of which assigns 64 of the 65 papers of known authorship correctly when a paper is left out and scored: the seventy function words Mosteller and Wallace screened, a 318-word stop-word list from a software library, and the 300 most frequent words. Under all three the joint papers are Madison-dominant (first figure). The posterior mean Hamilton share runs from 0.037 to 0.071 for No. 18, 0.058 to 0.230 for No. 19 and 0.118 to 0.258 for No. 20. For scale, papers Madison wrote alone, left out of his profile in turn, receive Hamilton shares with medians 0.13, 0.19 and 0.26. The joint papers' Hamilton weight is thus comparable to, or below, a typical solo Madison paper's, and no Hamilton contribution above that level is resolved in them. Under the second model both sets of numbers fall (the largest joint-paper mean to 0.094) and the comparison is unchanged.

The Hamilton share $w$ in the three acknowledged joint papers and the twelve disputed ones under the three word lists used in the paper. Dots are posterior means and bars run from the 5th to the 95th percentile; the dashed line is an even split, and the dotted lines mark the median share the same analysis gives a paper Madison wrote alone under each list (0.13, 0.19, 0.26). Every joint paper is Madison-dominant, with a share comparable to or below that level, and No. 55 is the only disputed paper whose mean passes one half under any list, and then only under the inherited seventy-word list. (Drawn from Tables 3 and 4 of the paper; its Figure 1 shows the full posterior curves for Nos. 18–20.)

Of the twelve disputed papers only No. 55 has a Hamilton share above one half under any list, and only under the inherited seventy words (0.5146). Under the two lists that did best on the known papers its shares are 0.3198 and 0.3879 and the pure-author comparison favors Madison ($-9.313$ and $-6.764$): a Madison majority with a Hamilton share near a third. That share may reflect a Hamilton contribution or paper-to-paper variation that the independent-draws model omits, and these data do not separate the two. Under the second model, which builds that variation in, the shares under those two lists fall to 0.075 and 0.132. That model, however, also misses a constructed even mixture of two papers about a third of the time, so the drop does not settle whether No. 55 is joint work either.

Shakespeare's collaborators

Doubts about sole authorship in the canon go back at least to Edmond Malone in 1787; in 2016 the New Oxford Shakespeare printed Marlowe's name beside Shakespeare's on the Henry VI title pages. All 41 plays are scored alike: a blend of Shakespeare's profile with another dramatist's (eight in all, matched by period) is tested against the better single profile, on 365 function words. A play counts as having had help only when its blend gain beats, by a set margin, the largest gain that any securely solo play of the period earns when scored the same way (the ceiling); at least one act must also be decisive on its own. Six plays pass: 2 and 3 Henry VI, Edward III, Henry VIII, The Two Noble Kinsmen and Titus Andronicus. Under the second model the blend gains of all six fall at least threefold and four still show a second hand; in 2 and 3 Henry VI a gain survives but no single act stays decisive. Four show none at lengths where a fifth of the verse in another hand would have been caught: King John, A Midsummer Night's Dream, Richard III and The Tempest. In each, such a fifth planted from any comparison dramatist of the period moves the play out of the “none” class at least 84 times in a hundred. The other thirty-one cannot be resolved. In eight, Hamlet among them, a planted fifth changes the classification only 9 to 52 times in a hundred. In sixteen a simpler power calculation already falls short of eight in ten or is marginal; nine of these are late plays, such as Macbeth and Pericles, that face a high Middleton-side ceiling. In the remaining seven the gain clears a ceiling but fits no single pair of profiles. That leaves the shares scholars find in those plays untested, not doubted. For eight of the ten resolved plays the result, help or none, matches the published position; for 2 and 3 Henry VI, where scholars debate revision or collaboration, it is stronger.

On two finer points the results depart from the scholarly attributions. In Henry VIII the second hand, given to Fletcher since 1850, is closer on function words to Middleton's profile (log Bayes factor 62.55) than to Fletcher's, which barely clears its own ceiling (3.34 against 2.36). Under the second model Fletcher's gain falls below its ceiling while Middleton's stays decisive. This need not mean a different collaborator, though: two dramatists writing in the same late idiom may each fit the other's profile. In Titus Andronicus, under the independent-draws model, Peele's profile is decisive in act 3, whereas Brian Vickers's division gives him chiefly act 1. Marlowe's profile also clears its ceiling for Titus and is decisive in act 1, and under the second model the classification rests on that act alone.

The three parts of Henry VI are then analyzed more closely, against profiles of Shakespeare, Marlowe and Kyd, on a 142-word list chosen by testing nine reference plays. In all three a Shakespeare–Marlowe blend beats the best single profile, by $10^{9.9}$, $10^{12.7}$ and $10^{30.9}$, with Shakespeare shares of 0.70, 0.64 and 0.48; Kyd is never preferred. Left out of its own profile and scored the same way, Edward II, accepted as Marlowe's unaided work, receives a blend factor of $10^{38.8}$ and a share of 0.49. A blend preference as strong as any part's can therefore arise without a second hand, and by itself does not show collaboration. Under the second model the three factors fall to $10^{4.0}$, $10^{4.9}$ and $10^{11.8}$ and Edward II's to $10^{4.5}$, so only Part Three's preference stands clear of what solo plays produce. Edward II is also the one reference play the 142-word list misclassifies, so every Marlowe-side number here is conditional on that one play.

The posterior Shakespeare share $w$ in the Shakespeare–Marlowe blend for every scene of the three parts of Henry VI, act by act: dots are means, bars the 5th to 95th percentiles, blue Shakespeare-side, orange Marlowe-side, gray uncertain, with each act’s class written above it and the play-level numbers above each panel. Marlowe-side acts occur only in Part Three (acts 1 and 4) and in act 4 of Part Two, where the New Oxford editors and Craig and Kinney put Marlowe; most individual scenes are too short to class at all. (Redrawn from Figure 2 of the paper.)

Classed act by act (second figure), Marlowe-side acts occur only in Part Three, acts 1 and 4, and in act 4 of Part Two, the Jack Cade scenes that Hugh Craig and Arthur Kinney associated with Marlowe in 2009. The Part Two act clears its threshold by only 0.11, and without Edward II in Marlowe's profile all three acts favor Shakespeare, so the paper claims no more of the agreement than those conditions support.

Isaiah and the sources of the Pentateuch

Isaiah was divided in the eighteenth century, when Döderlein and Eichhorn assigned chapters 40 onward to a later author writing in the Babylonian exile; in 1892 Bernhard Duhm cut the book in three, chapters 1–39, 40–55 and 56–66. Instead of fixing the number of sources, the paper gives a posterior over that number and over the boundaries, computed from the 66 chapters of the Leningrad Codex of 1008 CE.

How many contiguous processes, and where the boundaries fall, for Isaiah and two control books on the medieval codex. Left, the posterior on the number of segments $g$ at the three prior settings (blue $\alpha = 1/k$, orange $1/2$, green $1$); right, the posterior probability of a cut after each chapter at $\alpha = 1/2$, dark bands marking cuts. Isaiah divides at 39|40, and with three processes also after chapter 35 at the Hezekiah narrative, with no mass at 55|56; Zechariah divides at its accepted boundary after chapter 8, though how many processes it has depends on the prior setting; Esther stays whole, its strip showing only where a cut would go if one were forced. (Redrawn from Figure 3 of the paper.)

For Isaiah a single process is rejected at every prior setting, by log Bayes factors of 41.5 to 98.2. Given two processes the boundary falls after chapter 39 with posterior 0.979 to 0.994; given three, the extra boundary falls at or inside the prose Hezekiah narrative of chapters 36–39 (third figure). A boundary after chapter 55 has posterior 0.000, given three or four processes, at every chapter-level setting. That zero concerns function-word rates and sharp boundaries only. The case for a third Isaiah rests substantially on content, theology and historical setting, which the word list excludes on purpose, and gradual drift would not show as a sharp cut. Chapters 40–66 tested alone give the same zero, but they are short. A change at 55|56 as large as the one measured between chapters 1–39 and 40–66 would be caught only about six times in ten, and a smaller one would usually be missed.

The Great Isaiah Scroll from Qumran, copied around 125 BCE and a millennium older than the codex, is a second witness. On the scroll a single process is again rejected, at 44.0 to 85.3. Given two processes the boundary lies within chapters 37–40 with posterior 1.0000 at every setting, most probably after chapter 39, though more weakly than on the codex. At one setting the scroll's count spreads over three to six processes, so the shared finding is the boundary, not the count. The scroll also provides a control for scribal hand: handwriting analysis indicates two scribes probably copied one half each, with the seam after chapter 33, and the boundary posterior there is 0.000. After chapter 55 it is 0.000 on the scroll as well, so neither manuscript supports a third Isaiah at the power the later chapters allow.

The Pentateuch is a different case: on the documentary hypothesis, to which Julius Wellhausen gave its classical form in 1878, the five books interleave, verse by verse, four sources now called J, E, D and P. Interleaved sources cannot be contiguous segments, so the verse-by-verse assignments tabulated by Carpenter and Harford-Battersby in 1900 are scored as fixed hypotheses, by how well each source map predicts blocks of text left out of it. Separating the Priestly P verses improves prediction by about 1,600 nats (natural-log units; 2.303 nats make a factor of ten), 13.2 standard deviations above the mean of random relabelings of the blocks. Adding the Deuteronomic D gains a further 943 to 966. Splitting the rest into J and E does not help: the gain is $-18.1$ to $-27.2$ nats, inside what fake partitions of single-source Esther produce by chance (up to $+33.5$).

On function words, then, P is a distinct generating process. It stays one when the comparison is confined within single books, where it keeps about three-fifths of its gain, or within narrative alone. D separates too, but nearly all D material is the book of Deuteronomy, and with labels shuffled only within books its gain is no longer exceptional, so on these words D coincides with that book. J and E cannot be told apart: the tables' J/E labeling predicts no better than a random one. The test had power, though: a J/E difference two-thirds to five-sixths the size of P's contrast with the rest would have been caught at least eight times in ten. The divine names, doublets and theology on which the J/E case rests are excluded by construction, so the null does not refute the documentary hypothesis any more than the separations prove documents.

Every book of the Bible

The same boundary-finding model (a changepoint model, in statistical terms) is then applied unchanged to every book of the Bible. Twenty-one are classed as composite, nine show no division at their length, and thirty-six, including twenty-one of the twenty-seven New Testament books, are too short for any result. Each of these non-results is reported as a limit of the test's power, never as evidence that a book is a unity. Many composite results fall where source critics put a division, as in Leviticus at the Holiness Code and Deuteronomy at its law code. Ezekiel, though, usually read as the most unified major prophet, is the strongest single-language composite in the survey (log Bayes factor 363.5). It divides after chapter 39, where the prophecies give way to the temple vision, and the paper notes that this seam may be one of subject rather than hand.

When whole books within each section of the canon are grouped by their word rates, the synoptic gospels (Matthew, Mark and Luke) share one process, as Matthew's and Luke's use of Mark predicts. Acts, however, separates from Luke, although most commentators assign both to one author; the paper concludes only that their function-word processes differ. Each section of the canon thus has its own count of processes, and summed over sections the counts come to 24 to 30. That sum, however, assumes no profile is shared across sections, and comparing all sixty-six books at once shows that several are. At one prior setting Genesis groups with Judges, Samuel, Ruth and two of the minor prophets, Isaiah with four minor prophets, and the synoptic gospels with James. Merged wherever the evidence favors a shared profile, the books fall into roughly a dozen to thirty distinguishable function-word profiles. The exact count, 13 to 32, depends on the prior and on whether each comparison uses only the word categories that the books compared share. These numbers are counts of profiles at this resolution, not of authors or sources, and not a bound on either. When the analysis is repeated on the oldest witnesses, Codex Sinaiticus and the Septuagint among them, 19 of 26 checkable books keep their classification on their earliest witness, and 25 on at least one line of transmission.

The limits of counting

The Bayes-factor magnitudes depend on the count model. Under the second model the main results survive: the joint papers stay Madison-dominant, four of the six Shakespeare detections remain and Henry VIII still prefers Middleton. Finer ones do not: no single act of 2 or 3 Henry VI stays decisive, three of the four plays with no second hand become unresolved, and one of fifteen solo plays, Henry V, falls on Marlowe's side. In both models a ninety-percent interval for the blend weight misses a known true share well over one time in ten, so the weights order texts without measuring fractions of authorship. A collaborator with no undisputed work has no profile and cannot be detected; in only one play would a tenth of the verse in another hand change the classification; and help written in prose is outside the test. Over half the biblical books and most scenes are too short to test; a detected division may mark genre, period or language rather than a second writer; and content, theology and meter, on which much of the traditional argument rests, never enter the counts. A count model whose weight intervals were calibrated would turn the weights into measured fractions of authorship, and a second securely attributed solo play for Kyd, Nashe or Wilkins (none is known to survive) would make their profiles usable.

The paper

• Measuring shared authorship in the Federalist Papers, Shakespeare and the Bible (PDF) — Matthew D. Schwartz; the version hosted here is marked preliminary. Reviews the three disputes and the history of deciding authorship by counting, sets out the models and evidence calculations for two-author blends and for sources in sequence, checks them on four books of known structure, and then treats the Federalist, the Shakespeare canon with the three parts of Henry VI, and the Bible (Isaiah, the Pentateuch and all sixty-six books) in turn; the supporting checks, reference sets, word lists and computational details are in five appendices. The closed-form evidence integrals used throughout are proved in a companion paper on exact Bayesian evidence for mixture models (site page, web summary).

Supplementary material

The code is hosted on this site; all texts analyzed are public editions.

• Mixalot — tool page for the Python package that evaluates the evidence integrals, with a manual and worked examples.

References

← back to the web summaries