The strange phenomenon of the cult of facts: three case studies

by Massimo Pigliucci

I am a scientist, I appreciate the importance of verifiable facts. Moreover, my empirical research was in quantitative genetics, so I have a salutary respect for quantification and statistical analysis.

But I’m also a philosopher, which means I recognize that there simply isn’t such a thing as facts without a given theoretical framework. [1] So did Darwin, by the way. He famously wrote to his friend Henry Fawcett: “How odd it is that anyone should not see that all observation must be for or against some view if it is to be of any service!”

I’m telling you all this because I think we are currently suffering from what Leon Wieseltier has recently called “the cult of facts” [2]. Wieseltier was complaining about famed data cruncher Nate Silver, who has been referring to opinion journalism as, and I quote, “bullshit.”

Case 1: Nate Silver, bullshit and opinion journalism

Silver is justly famous for his Bayesian number crunching and meta-polling which, among other things, has led him to formulate increasingly accurate predictions about Presidential and Congressional elections (and which has now gotten him a gig at ESPN, where he is turning his talent to what he loves most: baseball analysis) [3].

Silver’s complaint about opinion journalism is based on his perception that op-ed columnists at outlets like The New York Times, The Washington Post and so forth are predictable and repetitive, which in turn is because they have “very strong ideological priors [which prevent them from] evaluating the data as it comes in [and] doing a lot of [original] thinking.”

Like Wieseltier, I am aware that the state of public intellectualism and opinion making isn’t exactly without problems. But, with Wieseltier, I find it oddly naive of Silver to talk as if “ideological priors” (otherwise known as beliefs about the world) weren’t inevitable in anyone (including Silver), and — within limits — were not actually a good thing.

Moreover, as a fellow Bayesian, Silver ought to know that his own analogy is ironically flawed: in Bayesian analysis you always begin with priors, and the whole point is to revise those priors as new data comes in. That is, embedded in the very fabric of the Bayesian approach [4] is that you start with beliefs, you add data (collected on the basis of your beliefs!), and end up with (likely modified) beliefs. You just can’t take the belief components out of the analysis, it’s integral to it, and it’s both affected by the data one gathers and determines which bits of information “out there” actually get to count as data.

As Wieseltier astutely observes, “Silver wishes to impugn not only the quality of opinion journalism, he wishes to impugn also its legitimacy. The new technology, which produces numbers the way plants produce oxygen, has inspired a new positivism, and he is one of its princes. He dignifies only facts … He does not recognize the calling of, or grasp the need for, public reason.”

And that is the crucial issue. It is fine — indeed, a good idea — to criticize individual opinionators when they get their facts wrong, or when their reasoning is faulty. That is the essence of democratic discourse in an open society. It is downright reactionary, however (regardless of whether Silver himself intends it that way) to delegitimize the whole idea that smart and well read people — we used to call them intellectuals — have become irrelevant because all we need to grasp the truth is tables and graphs.

Wieseltier accuses Silver of attempting to impose an auctoritas ex numero as the final arbiter for our judgments. The number of crucial issues that simply do not lend themselves to this sort of neo-positivism is staggering, with Wieseltier himself citing whether gays should have the right to marry, the scope of the social net, and the question of whether we have a moral duty to intervene in a case of genocide as obvious examples. Note that he is not saying that facts are irrelevant to these questions: social nets and military interventions in other countries are costly affairs, both in terms of financial and human resources (though it’s harder to imagine what sort of fact would be relevant to the issue of gay marriage). The point is — in perfect Humean fashion [5] — that facts will help us arrive at judgments, but will not uniquely determine those judgments. Our values and our critical reasoning are the additional ingredients entirely left out of Silver’s narrow view.

Wieseltier points out another bit of revealing naiveté on the part of Silver: his complaint that commentators like Paul Krugman or George Will are “repetitive,” which Silver again attributes to the rigidity of their (ideological) priors. But as Wieseltier immediately notes, Krugman, Will and others are in the business of public reasoning and persuasion, and the latter requires repetition. Indeed, for someone who is so much into evidence-based assertions, Silver would do well to check the cognitive science literature on how people change their minds: they rarely do it on the spot, as soon as they hear a beautifully reasoned argument (or pore over a cleverly put together infographic). People change their minds — when they do — because of multiple exposures to a given idea, from multiple sources and in various fashions. So it isn’t enough to make sure one gets his facts straight and his reasons well articulated. One also has to write elegantly and convincingly. And one has to do it over and over, if one wishes to accomplish anything at all.

Wieseltier ends his piece by asking whether numeracy is truly the American public’s most pressing problem. Seems to me that a vibrant democratic discourse could use more numeracy among its participants, and Nate Silver has certainly contributed his share in acquainting people with the power of data crunching. But that’s peanuts compared to the hurdle of fostering critical thinking abilities without which no amount of data crunching will help move society forward.

Case 2: the Ngram viewer

A second instructive case comes courtesy of a book review that appeared recently in The New Yorker, penned by Mark O’Connell, who was commenting on Uncharted: Big Data as a Lens on Human Culture, by Erez Aiden and Jean-Baptiste Michel [6].

Aiden and Michel have apparently single-handedly founded a new field, which goes by the unwieldy name of “culturomics,” i.e. the quantitative study of culture. The promise is the usual one: out with the old-fashioned humanistic approach to culture; in with the scientific and quantitative method.

As in the case of Wieseltier’s criticism of Silver, O’Connell too is (rightly) not dismissive of the new approach. After all, it is only reasonable to welcome a better handle on the facts of whatever it is one wishes to study or understand. The issue, rather, is one of emphasis, and of what exactly is being promised vs what can be delivered.

The centerpiece of Aiden and Michel’s book is the tool they invented for Google back in 2010: the Ngram viewer, a piece of software that allows you to graph the recurrence of a given word or phrase in Google’s gigantic library of scanned books. With so much data and computing power at their disposal, what did Aiden and Michel discover about the inner workings of human culture?

Well, while investigating the infamous Nazi campaign against “degenerate [i.e., modern] art,” Aiden and Michel looked up Marc Chagall’s trajectory in Ngram and concluded that “between 1936 and 1943, Marc Chagall’s full name appears only once in our German book records.” Which, of course, has been well known (qualitatively) to historians for a while. They also discovered that “Marcel Proust became famous for writing good books” (really?). They further quantitatively documented that Hitler is the most famous person born in the past two centuries, which led them to the apparently novel conclusion that “darkness, too, lurks among the n-grams, and no secret darker than this: Nothing creates fame more efficiently than acts of extreme evil. We live in a world in which the surest route to fame is killing people, and we owe it to one another to think about what that means.” Just one question: for whom, exactly, was this a secret?

And therein lies the problem. It’s not that Ngram isn’t a fun and potentially even somewhat useful tool. Nor is the point that humanity scholars cannot benefit from more scientific literacy and, when appropriate, some statistical training. But none of this amounts to the hyperbolic, TED-like statements of Aiden and Michel reported by O’Connell: “this big data revolution is about how humans create and preserve a historical record of their activities. Its consequences will transform how we look at ourselves. It will enable the creation of new scopes that make it possible for our society to more effectively probe its own nature. Big data is going to change the humanities, transform the social sciences, and renegotiate the relationship between the world of commerce and the ivory tower.” I seriously doubt it, but at any rate I’d like to see the data backing up this overly grandiose statement before accepting the claim at face value.

Case 3: Who’s assessing the assessors?

If you are a faculty at a state university anywhere in the United States you will recognize the term “assessment,” and you will likely have strong feelings about it. It refers to the latest legislative and administrative fad to engineer the impression that the powers that be actually give a damn about public education — at the same time as legislators keep slashing funds for it with gusto, and administrators keep hiring people like themselves and granting them handsome salaries the benefits of which to students is far from clear. [7]

The whole idea behind assessment exercises is that we need quantitative (of course) ways to figure out if our students are learning what we think we are teaching them. As one of the administrators at my own university keeps repeating, “I am data driven.” Well, so am I, truly, though I prefer the broader expression “evidence driven.” Nonetheless, I thought that giving students assignments in the form of tests and papers, and then grading said students on such assignments, was precisely the way we do check the degree to which students are learning at least a certain percentage of what we teach them.

Apparently not. Instead, we need to spend precious faculty time, and of course invest in expensive, custom made (and usually awfully designed) software, to “assess.” But as Steven Hales has pointed out in an entirely data-free editorial in The Chronicle of Higher Education [8], the whole approach quickly degenerates into epistemic skepticism and eventually into downright epistemic suicide.

Here is Hales’ satirical analysis: “the outcomes-assessment tool faces the same dilemma that grades [do]: Either (1) we know that it accurately measures the degree to which a student has mastered the course material and achieved the objectives of the course, or (2) we do not know. … Obviously we can’t use the outcomes-assessment tool itself to prove its own veracity, since that, again, is circular. … the demand that we prove the reliability of every method of gaining beliefs leads directly to a vicious regress. Ultimately we are left with skepticism: We have no knowledge at all.”

If only state legislators and administrators had bothered to take a course in epistemology, or introductory logic!

There is more: whenever I inquire with administrators about the ultimate purpose of assessment exercises, I eventually get them to admit that what they really want is to show state legislators that the university is improving by the only two measures that seem to have any traction with politicians: graduation rates and time to degree completion.

Now, of course nobody wants students to drop out of college, if it can be at all avoided. And nobody wants students to spend an extra minute beyond what is necessary in college, because — given the outrageous cost of tuition — they’d be sinking further and further into perilous debt before they even get their first job.

But surely those can’t be the only measures that count! To show this, my favorite retort to administrators is to use a reductio argument (again, those pesky intro philosophy courses!): if we really care only about graduation rates and time to completion, then there is a sure way to guarantee that we have one hundred percent graduation and that all students finish their degree in exactly four years. All we need to do is — ironically — to drop any assessment, including grades, and pass every student in every course, regardless of how s/he has mastered the material.

“Surely you’re joking, Massimo!” immediately responds the somewhat flabbergasted administrator. But I’m not, or at the least only in part. (Good philosophical points can often be made with jokes anyway. [9]) The point is that clearly better graduation rates and shorter time to completion are not what we are after. What we are after is a thoughtful education that allows our young to both reflect on the kind of life they want and develop the skills necessary to become thoughtful citizens of a vibrant democracy. Oh, and yes, to be able to find a job too.

But these latter goals are very difficult to quantify, contra the ease of estimating graduation rates and time to completion. So here is a perfect example of a situation where quantitative data not only are not helpful, but are positively harmful. That’s because the data is being gathered in response to a very poorly thought out question. It is simply astounding that we even need to have this discussion, and moreover that faculty all over the country are — at the least at the moment — surely on the loosing side of said discussion (which means, of course, that so are the students).

As I made clear at the beginning of this essay, I am not data-phobic, anti-science, or a luddite. I am simply trying to resist the latest quantification fad whenever it doesn’t help, or in fact hampers what we are trying to do. (And I haven’t even mentioned the app-based obsession with quantifying selves! [10]) Bayesian statistics, Ngram, and even (some) assessment exercises are surely tools we want to keep handy in our conceptual and technological toolbox. But the tools by themselves are no panacea, and indeed in some cases are simply irrelevant to the task, or downright harmful to it. David Hume once said that a wise person proportions his beliefs to the evidence. True, but a wise person is also capable of formulating good questions and then choose the best approaches to answer them, instead of the other way around.

_____

Massimo Pigliucci is a biologist and philosopher at the City University of New York. His main interests are in the philosophy of science and pseudoscience. He is the editor-in-chief of Scientia Salon, and his latest book (co-edited with Maarten Boudry) is Philosophy of Pseudoscience: Reconsidering the Demarcation Problem (Chicago Press).

[1] See section 4 of Jim Bogen’s “Theory and observation in science,” Stanford Encyclopedia of Philosophy.

[2] “The emptiness of data journalism,” by Leon Wieseltier, The New Republic, 19 March 2014.

[3] See Silver’s FiveThirthyEight site.

[4] See The Theory That Would Not Die: How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy, by Sharon Bertsch McGrayne (2011).

[5] Here is how David Hume framed the problem in his A Treatise of Human Nature (1739): “In every system of morality, which I have hitherto met with, I have always remarked, that the author proceeds for some time in the ordinary ways of reasoning, and establishes the being of a God, or makes observations concerning human affairs; when all of a sudden I am surprised to find, that instead of the usual copulations of propositions, is, and is not, I meet with no proposition that is not connected with an ought, or an ought not. This change is imperceptible; but is however, of the last consequence. For as this ought, or ought not, expresses some new relation or affirmation, ’tis necessary that it should be observed and explained; and at the same time that a reason should be given; for what seems altogether inconceivable, how this new relation can be a deduction from others, which are entirely different from it. But as authors do not commonly use this precaution, I shall presume to recommend it to the readers; and am persuaded, that this small attention would subvert all the vulgar systems of morality, and let us see, that the distinction of vice and virtue is not founded merely on the relations of objects, nor is perceived by reason.”

[6] “Bright Lights, Big Data,” by Mark O’Connell, The New Yorker 20 March 2014.

[7] It is easy to find both data and opinion pieces backing up these statements. For instance: “State of American Higher Education: More Adjuncts, More Administration, More Tuition, Fewer Full-time Faculty, and Less State Support!,” by Anthony Picciano, CUNY Academic Commons, 17 February 2014; The Fall of the Faculty, by Benjamin Ginsberg, Oxford University Press, 2011; “Administrator Hiring Drove 28% Boom in Higher-Ed Work Force, Report Says,” by Scott Carlson, The Chronicle of Higher Education, 5 February 2014; and many, many more.

[8] “Who’s Assessing the Assessors’ Assessors?” by Steven Hales, Chronicle of Higher Education, 11 March 2013.

[9] See Plato and a Platypus Walk into a Bar . . .: Understanding Philosophy Through Jokes, by Thomas Cathcart and Daniel Klein, Penguin, 2007.

[10] For an insightful commentary see: “Quantified Self: The algorithm of life,” by Josh Cohen, Prospect Magazine, 5 February 2014.