Weekly Philo of Economics: More Critical Reflections on the Cognitive Reflection Test measure

Last week Catarina did a critical post on a widely cited 2005 article by Shane Frederick that was  published in the Journal of Economic Perspectives. (In what follow I assume that readers are familiar either with Catarina's post or with The Cognitive Reflection Test (CRT).) Now it is worth mentioning that "Articles appearing in the journal are normally solicited by the editors and associate editors." [Full disclosure: once I tried to have this article "solicited" there!] I mention this because when I shared my interest in the piece with some of my usual 'informants' from economics, one of them (who shall remain anonymous) immediately shut back and reminded me that (a) Frederick is not a trained economist (he is a decision-scientist trained at CMU) and (b) that the refereeing at JEP is (shall we say) non-standard. The special provenance of this paper is reflected in its stated purpose: to explore "the possibility that general intelligence or various more specific cognitive abilities are important causal determinants of decision making. To provoke interest in this neglected topic," (25-26). It is, thus, a good proxy for the kind of ideology that excites certain incrowd social scientists. (I return to this at the end of the post.)

Now, Frederick argues that "CRT scores are predictive of the types of choices that feature prominently in tests of decision-making theories, like expected utility theory and prospect theory." (26) But as Nobel Laureate (Economics), the experimental economist, Vernon Smith, pointed out to me the "decision-making theories" that Frederick is exploring are all about isolated individuals. They are not about decision making understood as social psychology (of the sort that he and I associate with Adam Smith's The Theory of Moral Sentiments).  Smith's philosophically sophisticated fellow experimentalist at Chapman, Bart Wilson remarked, that Frederick's approach simply ignores how "distributed knowledge [is] built into our institutions, both formal and informal, which are smarter than any single mind;" If such a focus on institutions were included "then exploring human cognitive difference in economics could prove fruitful."

Moreover, the decision-making theories that CRT tends to be predictive of are very bad at exploring learning effects, background history (and filters) of subjects of the sort that Vernon Smith and Wilson have been exploring in their  recent experimental work (here and here). (It's interesting that excessive methodological individualism is under scrutiny within Classical Liberal-friendly experimental political economy.) Frederick seems remarkably blind to subject's previous (educational/social) history. Catarina was also critical about this in her post. Vernon Smith puts the criticism very nicely:

In the paper the MIT students look like they are from a different planet than the other university students; at Caltech [where Vernon Smith did his BA--ES] the program was four years of the kind of problem solving that constitute these cognitive measurement exercises. You were trained to be wary of quick judgments, and structure problems in steps.  

Another important economist, Steve Durlauf, argued that "All [Frederick's article] does is find some correlations between measures of cognitive ability and some forms of task performance, but establishes nothing about the causal role of genes in cognition, let alone establish a causal link between cognitive ability, however, measured, and the performance." In this context, Durlauf reminded me of (another Nobel Laureate in economics) James Heckman's research. (I have taught Heckman's work in the context of philosophy of social science courses that focused on IQ testing.) Heckman shows that  "cognitive measures such as IQ are an outcome of complicated gene/environment interactions." For a fantastic introduction to Heckman's line of research see here;  Heckman cites this piece which shows that "Frederick's "measure of “cognition” is substantially influenced by personality traits and is not a measure of pure cognition as measured by Raven’s progressive matrices." (Quoted from here.)

Finally, I want to turn to the ideological orientation of Frederick's approach. Frederick is attracted to the idea that more weight  "should be placed on the opinions of those with higher cognitive abilities," even though Frederick modifies this by adding this "clearly depends on the type of decision in question." (40; but the piece implies that when it comes to financial decisions the brightest should rule.) Frederick's piece flirts with providing the intellectual scaffolding for the kind of elite rule that tempts a lot of highly educated people. (Recall how Jason Brennan wants to disenfranchise voters who lack his knowledge in economics.) Now, I have spelled out some of my rules by which we should think about experts and the way they relate to their own research (here and here). In particular, I claim that (i) experts can't keep themselves (their incentives/their roles, etc) out of the model/proposal. (In practice this means that we can't simply assume that policy-scientists are disinterested truth-seekers in the context of policy-sensitive science.)  (ii) Experts shouldn't promote policies where the down-side risks of implementation are (primarily) shifted onto less fortunate others. (iii) Experts should make an effort to educate policy and opinion-makers to counter-arguments to the policies they advocate.

Amusingly enough, the main bit of advice that one might infer from Frederick' argument never gets entertained in the piece. According to Frederick the "high CRT group was more willing to gamble—particularly when the gamble had higher expected value (top panel),"(33) and "women were considerably more risk averse than men, and this remains true even after controlling for CRT score." (38) If Frederick really accepts his own research, he should advocate banning high CRT males from the trading desks of banks with deposit insurance! I bet we won't hear this very soon from a professor at Yale's School of Management.