Social science and policy

One of the important reasons that we value scientific knowledge is the possibility that it will allow us to intervene in the world to solve problems that we care about. Good climate science allows us to have high confidence in the causes of global climate change; and it also provides a sound basis for policy interventions to help to mitigate the pace of climate change. Good cellular biology permits a better understanding of autoimmune disease; and it also suggests avenues for prevention and treatment. There is thus an important component of pragmatism in our esteem for scientific knowledge.

In the social sciences we would like to assume that something similar is possible. If we have good sociological understanding of the causes of teen pregnancy or gang violence, perhaps that understanding will also provide a basis for designing effective interventions that reduce the incidence of the social problems we study. In other words, perhaps we can count on social science to provide a valuable and effective basis for the design of social policy.

The philosophy of social science that I’ve developed in this blog and in New Directions in the Philosophy of Social Science raises some challenges to that hope. It is argued here that the social world is contingent, heterogeneous, plastic, and conjunctural. In the words of Roy Bhaskar, social causation takes place in an open system in which we cannot arrive at confident predictions of particular social outcomes. In place of general theories and comprehensive social laws, it is argued here that we are best advised to seek out particular causal mechanisms that underlie various social outcomes of interest. And it is emphasized that it is difficult to make predictions in particular circumstances even when we have an idea of some of the operative social mechanisms, because of the perennial possibility of contingent interventions by additional factors.

So the hard question is this: to what extent is it at all possible for social science research to provide a confident basis for the design and implementation of social policies to address important social problems?

One approach that does not seem promising is the methodology of random controlled trials (RCT). The logical shortcomings of this approach when applied to social phenomena have been highlighted by Nancy Cartwright and Jeremy Hardie in Evidence-Based Policy: A Practical Guide to Doing It Better, and I discuss these problems In an earlier post (link). So it does not seem promising to expect that we will be able to isolate causal mechanisms (for example, “provide after-school tutoring”) and use the method of RCT to demonstrate the efficacy of this mechanism in reducing a given social harm (say, “high school absenteeism”).

The problem of establishing a strong relationship between theory and policy has been considered in several areas of social research. One such study is in the field of international relations. Stephen Walt’s 2005 article, “The relationship between theory and policy in international relations”, is an extended treatment of the topic (link). Here is the abstract to Walt’s paper:

Policy makers pay relatively little attention to the vast theoretical literature in IR, and many scholars seem uninterested in doing policy-relevant work. These tendencies are unfortunate because theory is an essential tool of statecraft. Many policy debates ultimately rest on competing theoretical visions, and relying on a false or flawed theory can lead to major foreign policy disasters. Theory remains essential for diagnosing events, explaining their causes, prescribing responses, and evaluating the impact of different policies. Unfortunately, the norms and incentives that currently dominate academia discourage many scholars from doing useful theoretical work in IR. The gap between theory and policy can be narrowed only if the academic community begins to place greater value on policy-relevant theoretical work.

Fundamentally the article raises the question of whether there is a useful relationship between international relations theories and the practice of diplomacy and foreign policy. Can IR theory guide the construction of a successful foreign policy?

Here are some of the ways that Walt believes theory can be used to support policy analysis. Walt believes that theory can assist policy analysis in four important ways, including diagnosis, prediction, prescription, and evaluation. Unfortunately, none of the examples that he offers provide much confidence in any of these capabilities in a significant way. Diagnosis comes down to classification; but given that the idea of a social kind is suspect, we do not add much to our knowledge by classifying a given regime as “fascist”, because we know that there is substantial variation across the group of fascist states. Prediction (as Gandhi said about Western civilization) would be nice; but it is almost never attainable in real social situations. Prescription requires a sound knowledge of the likely causal dynamics of a situation; but the open nature of social reality implies that we cannot have such knowledge in any comprehensive way. And evaluation is subject to similar issues. Walt assumes we can evaluate the success of a policy in a quasi-experimental way — observe the cases where the intervention took place and measure the frequency of the desired outcome. But unfortunately this quasi-experimental method is also suspect.

An important drawback of Walt’s treatment is the fairly traditional view that Walt takes with regard to the content of scientific knowledge. There is an underlying preposition of a fairly Humean view of cause and effect.

Policy makers can also rely on empirical laws. An empirical law is an observed correspondence between two or more phenomena that systematic inquiry has shown to be reliable. (25)

But in fact, there are very few useful “empirical laws” in the social realm that might serve as a basis for simple cause-and-effect policy design.

At present, then, there is a still a significant gap between an empirically supported social theory and a well designed social intervention. Unfortunately social causation is rarely as simple and regular as the empiricist framework presupposes. This is indeed disappointing, because it is certainly true that we most urgently need guidance in designing strategies for solving important social problems. (Here is an earlier post that offers a somewhat more positive assessment of the relevance of theory to policy; link.)

Advertisements

Sustaining a philosophy research community

 

The European Network for Philosophy of Social Science (ENPOSS) completed its annual conference in Krakow last week. It was a stimulating and productive success, with scholars from many countries and at every level of seniority. ENPOSS is one of the most dynamic networks where genuinely excellent work in philosophy of social science is taking place (link). Philosophers from Germany, Poland, Norway, Spain, France, the Netherlands, the UK, and other countries came together for three intensive days of panels and discussions. The discussions made it clear that this is an integrated research community with a common understanding of a number of research problems and a common vocabulary. There is a sense of continuing progress on key issues — micro-macro ontology, social mechanisms, naturalism, intentionality, institutional imperatives, fact-value issues, computational social science, and intersections of disciplinary perspectives, to name several.

Particular highlights were keynote addresses by Dan Hausman (“Social scientific naturalism revisited”), Anne Alexandrova (“Are social scientists experts on values?”), and Bartosz Brozek (“The architecture of the legal mind”). There were also lively book discussions on several current books in the philosophy of social science — Chris Mantzavinos’s Explanatory Pluralism, Lukasz Hardt’s Economics Without Laws: Towards a New Philosophy of Economics, and my own New Directions in the Philosophy of Social Science. Thanks to Eleonora Montuschi, Gianluca Manzo, and Federica Russo for excellent and stimulating discussion of my book.

It is interesting to observe that the supposed divide between analytic and Continental philosophy is not in evidence in this network of scholars. These are philosophers whose Ph.D. training took place all over Europe — Italy, Belgium, Finland, Germany, France, Spain, the UK … They are European philosophers. But their philosophical ideas do not fall within the stereotyped boundaries of “Continental philosophy.” The philosophical vocabulary in evidence is familiar from analytic philosophy. At the same time, this is not simply an extension of Anglo-American philosophy. The style of reasoning and analysis is not narrowly restricted to the paradigms reflected by Russell, Dummett, or Parfit. It is, perhaps, a new style of European philosophy. There is a broad commitment to engaging with the logic and content of particular social sciences at a level that would also make sense to the practitioners of sociology, political science, or economics. And there is a striking breadth to the substantive problems of social life that these philosophers are attempting to better understand. The overall impression is of a research community that has the features of what Imre Lakatos referred to as a “progressive research programme” in Criticism and the Growth of Knowledge — one in which problems are being addressed and treated in ways that sheds genuinely new light on the problem. Progress is taking place.

There were two large topic areas that perhaps surprisingly did not find expression in the ENPOSS program. One is the field of critical realism and the ideas about social explanation advanced by Roy Bhaskar and Margaret Archer. And the second is the theory of assemblages put forward by Deleuze and subsequently elaborated by DeLanda and Latour. These topic areas have drawn a fair amount of attention by social theorists and philosophers in other parts of the philosophy of social science research community. So it is interesting to realize that they were largely invisible in Krakow. This leads one to think that this particular network of scholars is simply not very much influenced by these ideas.

Part of the dynamism of the ENPOSS conference, both in Krakow and in prior years, is the broad sense that these issues matter a great deal. There was a sense of the underlying importance of the philosophy of social science. Participants seem to share the idea that the processes of social change and periodic crisis that we face in the contemporary world are both novel and potentially harmful to human flourishing, and that the social sciences need to develop better methods, ontologies, and theories if they are to help us to understand and improve the social world in which we live. So the philosophy of social science is not just a contribution to a minor area within the grand discipline of philosophy; more importantly, it is a substantial and valuable contribution to our collective ability to bring a scientific perspective to social problems and social progress.

Next year’s meeting will take place in early September at the University of Hannover and will be a joint meeting with the US-based Philosophy of Social Science Roundtable. The call for papers will be posted on the ENPOSS website.

Time for a critical-realist epistemology

The critical realism network in North America is currently convened in Montreal in a three-day intensive workshop (link). In attendance are many of the sociologists and philosophers who have an active interest in critical realism, and the talks are of genuine interest. A session this morning on pragmatist threads of potential interest to critical realists, including Mead, Abbott, and Elias, was highly stimulating. And there are 29 sessions altogether — roughly 85 papers. This is an amazing wealth of sociological research.

Perhaps a third of the papers are presentations of original sociological research from a CR point of view. This is very encouraging because it demonstrates that CR is moving beyond the philosophy of social science to the concrete practice of social science. Researchers are working hard to develop research methods in the context of CR that permit concrete investigation of particular social and historical phenomena. And this implies as well that there is a growing body of thinking about methodology within the field of CR.

CR theorists began with ontology, and a great deal of the existing literature takes the form of theoretical expositions of various ontological theses. And this was deliberate; following Bhaskar, theorists have argued that we need better ontology before science can progress. (This seems particularly true in the social realm; link.) So ontology needs to come first, then epistemology.

I believe the time has come when CR needs to give more explicit and extended attention to epistemology.

What is epistemology? It is an organized effort to answer the question, what is (scientific) knowledge? It attempts to provide a justified theory of empirical justification. Epistemology is an attempt to articulate the desired relationship between evidence and assertion; more specifically, it is an attempt to uncover the nuances of the domain of “evidence” across the realm of social research. Most fundamentally, it is an attempt to articulate how the practices of science are “truth-enhancing”: a given set of epistemic practices (methodologies) are hoped to result in a higher level of veridicality over time.

Like a left handed quarterback, CR has a disadvantage in formulating an epistemology because of its blind side. In the case of CR, the blind side is the movement’s visceral rejection of positivism. CR theorists are so strongly motivated to reject all elements of positivism that they are disposed to avoid positions they actually need to take. For example, The two following statements sound very similar:

A “Sociological claims must be evaluated on the basis of objective empirical evidence”

B “Sociological claims need to be confirmed or falsified”

And so the CR theorist is inclined to reject A as well as B. But this is a philosophical misstep caused by fear of the blind side. A is actually a perfectly valid requirement of epistemological rationality.

So what do we need from a developed epistemology for CR? Essentially we need three things.

First, we need an explicit commitment to empirical evaluation.

Second, we need a nuanced discussion of the complications involved in identifying “empirical evidence” in social research; for example, the impossibility of theory-independent or perspective-independent social data, the constructive nature of most historical and social observation, and the problem of selectivity in the collection of evidence.

Third, we need a discussion of the modes of inference — deductive, inductive, statistical, causal, and Bayesian — on the basis of which social scientists can arrive at an estimate of likelihood for a statement given a set of evidence statements.

Finally, our CR epistemology needs to give an appropriate discussion of the fallibility of all scientific research.

The epistemological frame that I currently favor is the coherence methodology described by philosophers like Quine and Goodman. The social sciences constitute a web of belief, and provisional conclusions in one area may serve to establish a method or valuation for findings in another area of the web. both ontological positions and epistemological maxims may require adjustment in light of future empirical and theoretical findings. Rawls’s conception of reflective equilibrium illustrates this epistemology in the moral field. This approach has an unexpected affinity for CR, because there is an emerging interest in the pragmatist philosophy from which this approach derives.

Epistemology allows us to place various specific methodological approaches into context. So we can locate the method of process tracing into the context of justification, and therefore into epistemology. It also validates the idea of methodological pluralism: there are multiple avenues through which researchers can create evidence through which to prove and evaluate a variety of sociological claims.

Critical realism seeks to significantly influence the practice and content of social science theory and research. In order to do this it will need to be able to state with confidence the commitments made by CR researchers to empirical standards and evidence-based findings. This will help CR to fulfill the promise of discovering some of the real structures and processes of the social world based on publicly accessible standards of theory discovery and acceptance.

First generation anti-positivism: Wellmer

In Critical Theory Of Society (1969) Albrecht Wellmer announced a critique of positivist assumptions in the study of society. Proceeding from the perspective of critical theory and especially Horkheimer and Adorno, Wellmer denounced the embrace of positivism by “bourgeois” social science. But perhaps more surprisingly, he addresses this critique to Marx’s system as well.

Probably Horkheimer himself offered the most impressive statement of the Frankfurt school’s estimate of its own function and importance when, in his article on traditional and “critical” theory, he joined issue with bourgeois science and its objectivist misconception of its own nature. The essay shows clearly that the confrontation between critical, Marxist and traditional “bourgeois” science had hardly moved by then into the vague realm of methodological abstractions; to the extent that the debate was concerned with methodology, critical theory was more inclined to view it as the mere reflection of actual social conflicts. (10)

The main thrusts against positivism consist of the claim that positivists look at social arrangements as purely objective and factual; whereas they require interpretation.

Apart from strict behaviorists, social scientists would in general no longer dispute the fact that access to the measured or observed

data

of their field of study is obtained through the medium of communication. but, they opine, the role of interpretations finishes with its provision of a means of access to the data, and perhaps also of a heuristic value for the discovery of explanations; in addition, they would claim for their science the methodological status of a natural science, and therefor of a science entitled, with the aid of universal laws, to explain and predict unusual phenomena. (35)

And positivism assumes that value perspectives can be filtered out in the perception of facts; whereas critical theorists maintain that perspective is inseparable from perception. (Proletarians see the social world differently from the bourgeois.)

So what is “critical social science”? To start, it is hermeneutic, according to Wellmer: it has to do with the interpretation of meanings in social action.

It has already been shown that explanatory sociology is always interpretative sociology as well; and that an interpretative sociology cannot be merely a subjective sociology of the interpretation of meaning. It is also clear now that the empirical content of social scientific theories is peculiarly proportional to the historical concretion to which they attain. (38)

Further, critical social science is fundamentally responsive to historical context. So hermeneutic interpretation cannot be extracted from its historical context.

So much of the specific content of a certain historical period enters into the basic theoretical assumptions and the framework of reference used for categorization, that its hypotheses cannot be transferred without violence to more distant socio-historical situations. (36)

This point parallels the post-positivist view that observation is theory-laden; but it goes beyond that by postulating that social observations are framed by conceptual systems that are themselves historically specific.

Third, critical social science rests on a recognition (even more explicit in Habermas) that knowledge and interest are interwoven:

Already apparent is the changed relation of theory and practice that exists for a critical social theory derived form a practical interest in cognition. Critical theory is derivable from a notion of the “good life” already available to it as part of the socio-historical situation it subjects to analysis; which, as the notion of an acknowledgement of each individual as a person by each other individual, and as the idea of a non-coercive communal human life of dialogue, is a draft meaning of history already fragmentarily embodied in a society’s traditions and institutions. (41)

The tension between the Frankfurt School and orthodox Marxist theory is evident here, because Wellmer’s critique of “bourgeois social science” is extended to Marx himself.

The critique of the objectivism of Marx’s philosophy of history was directed at a latently positivistic misconception, which, according to Habermas’s thesis, arises from the part played by the concept of labor. (67)

“Objectivism” here means the stance of the social researcher to regard the social as “given”, not subject to interpretation. And Wellmer argues that there is a strand of Marx’s thought that does precisely that. Marx’s materialist theory of history — “history consists of a dialectic of change driven by conflict between the forces and relations of production” — leaves no room for the radical interpretivism that Wellmer favors. Crudely, Marx in the German Ideology and Capital is not at all interested in the ideas that men and women have, but rather the objective system of social relations that underlies their actions and interactions. Even arguments that Marx makes about ideology, false consciousness, and fetishism of commodities takes the form of demystification — dissolution of the system of false consciousness rather than interpretation of how these representations relate to the workings of the social order. Consciousness plays no role in the dynamics of history. And in fact Wellmer believes that this stance plays a self-defeating role in Marx’s system:

The union of historical materialism and the criticism of political economy in Marx’s social theory is inherently contradictory. (74)

What Wellmer favors for social theory is expressed here:

This means that critical theory does not wish to replace an ideological consciousness with a scientific consciousness, but — of course by means of empirical and historical analyses — to assist the practical reason existing in the for of ideological consciousness to “call to mind” its distorted form, and at the same time to get control of its practical-utopian contents. Ultimately, therefore, critical theory can prove itself only by initiating a reflective dissolution of false consciousness resulting in liberating praxis: the successful dissolution of false consciousness as an integrative aspect of emancipatory practice is the proper touchstone for its truth. (72)

So Wellmer’s “critical theory of society” is criticism all the way down: critique of the assumptions about knowledge, action, and social relations that underlie both bourgeois social science and large swathes of Marx’s own theoretical framework. In place of orthodox “empiricist” ideas about empirical confirmation and “hypothetico-deductive method”, he advocates the “hypothetico-practical model of validation. Those theories which unfold into a basis for human liberation are for that reason rationally preferable to those that do not. Rather than theory and observation, Wellmer’s philosophy of science rests on a view of theory and praxis, or theory and liberation.

However, it is difficult today to interpret this series of observations as a serious and credible approach to social science epistemology. It offers suggestive ideas about what is involved in making sense of a given historical-social reality. But it gives little guidance about how to evaluate various theories and interpretations.

(The choice of Millet’s painting “Peasants planting potatoes” is apt for a discussion of Wellmer’s philosophy of social science. The painting represents a set of “facts”; but we cannot say what facts these are without substantial interpretation, and various interpretations are possible. The painting might be regarded as a small piece of critical social theory all by itself, with a gesture towards social reality, a depiction that can be understood as a system of domination, and a call for liberation.)

Explanation and critical realism

 

To explain something is to provide a true account of the causes and circumstances that brought it about. There is of course more to say on the subject, but this is the essential part of the story. And this normative account of explanation should work as well for investigations created within the framework of critical realism as any other scientific framework. 

Moreover, CR is well equipped with intellectual resources to produce explanations of social outcomes based on this understanding. In particular, CR emphasizes the reality of causal mechanisms in the social world. To explain a social outcome, then — perhaps the rise of Trumpism — we are instructed to identify the causal mechanisms and conditions that were in play such that a novice from reality television would gain the support of millions of voters and win the presidency. So far, so good. 

But a good explanation of an outcome is not just a story about mechanisms that might have produced the outcome; instead, we need a true story: these mechanisms existed and occurred, they brought about the outcome, and the outcome would not have occurred in the absence of this combination of mechanisms. Therefore we need to have empirical methods to allow us to evaluate the truth of these hypotheses.

There is also the important and interesting point that Bhaskar makes to the effect that the social world involves open causal configurations, not closed causal configurations. This appears to me to be an important insight into the social world; but it makes the problem of validating causal explanations even more challenging. 

This brings us to a point of contact with the theme of much current work in critical realism: a firm opposition to positivism and an allegiance to post-positivism. Because a central thrust of positivism was the demand for substantive empirical confirmation or verification of substantive claims; and that is precisely where we have arrived in this rapid analysis of explanation as well. In fact, it is quite obvious that CR theories and explanations require empirical validation no less than positivistic theories. We cannot dispense with empirical validation and continue to believe we are involved in science. 

Put the point another way: there is no possible avenue of validation of substantive explanatory hypotheses that proceeds through purely intuitive or theoretical avenues. At some point a good explanation requires empirical assessment. 

For example, it is appealing in the case of Trumpism to attribute Trump’s rise to the latent xenophobia of the disaffected lower working class. But is this true? And if true, is it critical as a causal factor in his rise? How would we confirm or disconfirm this hypothetical mechanism? Once again, this brings us into proximity to a few core commitments of empiricism and positivism — confirmation theory and falsifiability. And yet, a rational adherence to the importance of empirical validation takes us in this direction ineluctably. 

It is worth pointing out that the social and historical sciences have indeed developed empirical methods that are both rigorous and distinctive to the domain of the social: process tracing, single-case and small-N studies, comparative analysis, paired comparisons, and the like. So the demand for empirical methods does not imply standard (and simplistic) models of confirmation like the H-D model. What it does imply is that it is imperative to use careful reasoning, detailed observation, and discovery of obscure historical facts to validate one’s hypotheses and claims. 

Bhaskar addresses these issues in his appendix on the philosophy of science in RTS. He clearly presupposes two things: that rigorous evidence must be used in assessment of explanatory hypotheses in social science; and flat-footed positivism fails in providing an appropriate account of what that empirical reasoning ought to look like. And, as indicated above, the open character of social causation presents the greatest barrier to the positivist approach. Positivism assets that the task of confirmation and refutation concerns only the empirical correspondence between hypothesis and observation. 

Elsewhere I have argued for the piecemeal validation of social theories and hypotheses (link). This is possible because we are not forced to adopt the assumption of holism that generally guides philosophy in the consideration of physical theory. Instead, hypotheses about mechanisms and processes can be evaluated and confirmed through numerous independent lines of investigation. Duhem may have been right about physics, but he is not right about our knowledge of the social world.

Social science or social studies?

A genuinely difficult question is this: does the idea of a rigorous “social science” really make sense, given what we know of the nature of the social world, the nature of human agency, and the nature of historical change?

There are of course large areas of social inquiry that involve genuine observation and measurement: demography, population health statistics, survey research, economic activity, social statistics of various kinds. Part of science is careful observation of a domain and analysis of the statistical patterns that emerge; so it is reasonable to say that demography, public health, and opinion research admit of rigorous empirical treatment.

Second, it is possible to single out complex historical events or processes for detailed empirical and historical study: the outbreak of WWI, the occurrence and spread of the Spanish influenza epidemic, the rise of authoritarian populism in Europe. Complex historical events like these admit of careful evidence-based investigation, designed to allow us to better understand the sequence of events and circumstances that made them up. And we can attempt to make sense of the connections that exist within such sequences, whether causal, cultural, or semiotic.

Third, it is possible to identify causal connections among social events or processes: effective transportation networks facilitate the diffusion of ideas and germs; price rises in a commodity result in decreases in consumption of the commodity; the density of an individual’s social networks influences the likelihood of career success; etc. It is perfectly legitimate for social researchers to attempt to identify these causal connections and mechanisms, and further, to understand how these kinds of causal influence work in the social world. A key goal of science is explanation, and the kinds of inquiry mentioned here certainly admit of explanatory hypotheses. So explanation, a key goal of science, is indeed feasible in the social realm.

Fourth, there are “system” effects in the social world: transportation, communication, labor markets, electoral systems — all these networks of interaction and influence can be seen to have effects on the pattern of social activity that emerge in the societies in which they exist. These kinds of effects can be studied from various points of view — empirical, formal, simulations, etc. These kinds of investigation once again can serve as a basis for explanation of puzzling social phenomena.

This list of legitimate objects of empirical study in the social world, resulting in legitimate and evidence-based knowledge and explanation, can certainly be extended. And if being scientific means no more than conducting analysis of empirical phenomena based on observation, evidence, and causal inquiry, then we can reasonably say that it is possible to take a scientific attitude towards empirical problems like these.

But the hard question is whether there is more to social science than a fairly miscellaneous set of results that have emerged through study of questions like these. In particular, the natural sciences have aspired to formulating fundamental general theories that serve to systematize wide ranges of natural phenomena — the theory of universal gravitation or the theory of evolution through natural selection, for example. The goal is to reduce the heterogeneity and diversity of natural phenomena to a few general theoretical hypotheses about the underlying reality of the natural world.

Are general theories like these possible in the social realm?

Some theorists have wanted to answer this question in the affirmative. Karl Marx, for example, believed that his theory of the capitalist mode of production provided a basis for systematizing and explaining a very wide range of social data about the modern social world. It was this supposed capacity for systematizing the data of the modern world that led Marx to claim that he was providing a “science of society”.

But it is profoundly dubious that this theory, or any similarly general theory, can play the role of a fundamental theory of the social world, in the way that perhaps electromagnetic theory or quantum mechanics play a fundamental role in understanding the natural world.

The question may seem unimportant. But in fact, to call an area of inquiry “science” brings some associations that may not be at all justified in the case of study of the social world. In particular, science is often thought to be comprehensive, predictive, and verifiable. But knowledge of the social world falls short in each of these ways. There is no such thing as a comprehensive or foundational social theory, much as theorists like Marx have thought otherwise. Predictions in the social realm are highly uncertain and contingent. And it is rare to have a broad range of social data that serves to “confirm” or “verify” a general social theory.

Here is one possible answer to the question posed above, consistent with the points made here. Yes, social science is possible. But what social science consists in is an irreducible and pluralistic family of research methods, observations, explanatory hypotheses, and mid-level theories that permit only limited prediction and that cannot in principle serve to unify the social realm under a single set of theoretical hypotheses. There are no grand unifying theories in the social realm, only an open-ended set of theories of the middle range that can be used to probe and explain the social facts we can uncover through social and historical research.

In fact, to the extent that the ideas of contingency, heterogeneity, plasticity, and conjuncturality play the important role in the social world that I believe they do, it is difficult to avoid the conclusion that there are very narrow limits to the degree to which we can aspire to systematic or theoretical explanation in the social realm. And this in turn suggests that we might better describe social inquiry as a set of discrete and diverse social studies rather than unified “social science“. We might think of the domain of social knowledge better in analogy to the contents of a large and diverse tool box than in analogy to an orrery that predicts the “motions” of social structures over time.

 

The soft side of critical realism

Critical realism has appealed to a range of sociologists and political scientists, in part because of the legitimacy it renders for the study of social structures and organizations. However, many of the things sociologists study are not “things” at all, but rather subjective features of social experience — mental frameworks, identities, ideologies, value systems, knowledge frameworks. Is it possible to be a critical realist about “subjective” social experience and formations of consciousness? Here I want to argue in favor of a CR treatment of subjective experience and thought.

First, let’s recall what it means to be realist about something. It means to take a cognitive stance towards the formation that treats it as being independent from the concepts we use to categorize it. It is to postulate that there are facts about the formation that are independent from our perceptions of it or the ways we conceptualize it. It is to attribute to the formation a degree of solidity in the world, a set of characteristics that can be empirically investigated and that have causal powers in the world. It is to negate the slogan, “all that is solid melts into air” with regard to these kinds of formations. “Real” does not mean “tangible” or “material”; it means independent, persistent, and causal.

So to be realist about values, cognitive frameworks, practices, or paradigms is to assert that these assemblages of mental attitudes and features have social instantiation, that they persist over time, and that they have causal powers within the social realm. By this definition, mental frameworks are perfectly real. They have visible social foundations — concrete institutions and practices through which they are transmitted and reproduced. And they have clear causal powers within the social realm.

A few examples will help make this clear.

Consider first the assemblage of beliefs, attitudes, and behavioral repertoires that constitute the race regime in a particular time and place. Children and adults from different racial groups in a region have internalized a set of ideas and behaviors about each other that are inflected by race and gender. These beliefs, norms, and attitudes can be investigated through a variety of means, including surveys and ethnographic observation. Through their behaviors and interactions with each other they gain practice in their mastery of the regime, and they influence outcomes and future behaviors. They transmit and reproduce features of the race regime to peers and children. There is a self-reinforcing discipline to such an assemblage of attitudes and behaviors which shapes the behaviors and expectations of others, both internally and coercively. This formation has causal effects on the local society in which it exists, and it is independent from the ideas we have about it. It is by this set of factors, a real part of local society. (If is also a variable and heterogeneous reality, across time and space.) We can trace the sociological foundations of the formation within the population, the institutional arrangements through which minds and behaviors are shaped. And we can identify many social effects of specific features of regimes like this. (Here is an earlier post on the race regime of Jim Crow; link, link.)

Here is a second useful example — a knowledge and practice system like Six Sigma. This is a bundle of ideas about business management. It involves some fairly specific doctrines and technical practices. There are training institutions through which individuals become expert at Six Sigma. And there is a distributed group of expert practitioners across a number of companies, consulting firms, and universities who possess highly similar sets of knowledge, judgment, and perception.  This is a knowledge and practice community, with specific and identifiable causal consequences.

These are two concrete examples. Many others could be offered — workingclass solidarity, bourgeois modes of dress and manners, the social attitudes and behaviors of French businessmen, the norms of Islamic charity, the Protestant Ethic, Midwestern modesty.

So, indeed, it is entirely legitimate to be a critical realist about mental frameworks. More, the realist who abjures study of such frameworks as social realities is doomed to offer explanations with mysterious gaps. He or she will find large historical anomalies, where available structural causes fail to account for important historical outcomes.

Consider Marx and Engels’ words in the Communist Manifesto:

All fixed, fast-frozen relations, with their train of ancient and venerable prejudices and opinions, are swept away, all new-formed ones become antiquated before they can ossify. All that is solid melts into air, all that is holy is profaned, and man is at last compelled to face with sober senses his real conditions of life, and his relations with his kind.

This is an interesting riff on social reality, capturing both change and persistence, appearance and reality. A similar point of view is expressed in Marx’s theory of the fetishism of commodities: beliefs exist, they have social origins, and it is possible to demystify them on occasion by uncovering the distortions they convey of real underlying social relations.

There is one more perplexing twist here for realists. Both structures and features of consciousness are real in their social manifestations. However, one goal of critical philosophy is to show how the mental structures of a given class or gender are in fact false consciousness. It is a true fact that British citizens in 1871 had certain ideas about the workings of contemporary capitalism. But it is an important function of critical theory to demonstrate that those beliefs were wrong, and to more accurately account for the underlying social relations they attempt to describe. And it is important to discover the mechanisms through which those false beliefs came into existence.

So critical realism must both identify real structures of thought in society and demystify these thought systems when they systematically falsify the underlying social reality. Decoding the social realities of patriarchy, racism, and religious bigotry is itself a key task for a critical social sciences.

Dave Elder-Vass is one of the few critical realists who have devoted attention to the reality of a subjective social thing, a system of norms. In The Causal Power of Social Structures: Emergence, Structure and Agency he tries to show how the ideas of a “norm circle” helps explicate the objectivity, persistence, and reality of a socially embodied norm system. Here’s is an earlier post on E-V’s work (link).

Discovering the nucleus

In the past year or so I’ve been reading a handful of fascinating biographies and histories involving the evolution of early twentieth-century physics, paying attention to the individuals, the institutions, and the ideas that contributed to the making of post-classical physics. The primary focus is on the theory of the atom and the nucleus, and the emergence of the theory of quantum mechanics. The major figures who have come into this complex narrative include Dirac, Bohr, Heisenberg, von Neumann, Fermi, Rutherford, Blackett, Bethe, and Feynman, along with dozens of other mathematicians and physicists. Institutions and cities played a key role in this story — Manchester, Copenhagen, Cambridge, Göttingen, Budapest, Princeton, Berkeley, Ithaca, Chicago. And of course written throughout this story is the rise of Nazism, World War II, and the race for the atomic bomb. This is a crucially important period in the history of science, and the physics that was created between 1900 and 1960 has fundamentally changed our view of the natural world.

       

One level of interest for me in doing this reading is the math and physics themselves. As a high school student I was fascinated with physics. I learned some of the basics of the story of modern physics before I went to college — the ideas of special relativity theory, the hydrogen spectrum lines, the twin-slit experiments, the puzzles of radiation and the atom leading to the formulation of the quantum theory of electromagnetic radiation, the discoveries of superconductivity and lasers. In college I became a physics and mathematics major at the University of Illinois, though I stayed with physics only through the end of the first two years of course work (electricity and magnetism, theoretical and applied mechanics, several chemistry courses, real analysis, advanced differential equations). (Significantly for the recent reading I’ve been doing, I switched from physics to philosophy while I was taking the junior level quantum mechanics course.) I completed a mathematics major, along with a philosophy degree, and did a PhD in philosophy because I felt philosophy offered a broader intellectual platform on questions that mattered.

 
So I’ve always felt I had a decent layman’s understanding of the questions and issues driving modern physics. One interesting result of reading all this historical material about the period of 1910-1935, however, is that I’ve realized what large holes there are in my mental map of the topics, both in the physics and the math. And it is genuinely interesting to realize that there are deeply fascinating questions in this terrain which I haven’t really got an inkling about. It is energizing to know that it is entirely possible to open up new areas of knowledge and inquiry for oneself. 
 
Of enduring interest in this story is the impression that emerges of amazingly rapid progress in physics in these few decades, with major discoveries and new mathematical methods emerging in weeks and months rather than decades and centuries. The intellectual pace in places like Copenhagen, Princeton, and Göttingen was staggering, and scientists like Bohr, von Neumann, and Heisenberg genuinely astonish the reader with the fertility of their scientific abilities. Moreover, the theories and mathematical formulations that emerged had amazingly precise and unexpected predictive consequences. Physical theory and experimentation reached a fantastic degree of synergy together. 
 
The institutions of research that developed through this period are fascinating as well. The Cavendish lab at Cambridge, the Institute for Advanced Studies at Princeton, the Niels Bohr Institute in Copenhagen, the math and physics centers at Göttingen, and the many conferences and journals of the period facilitated rapid progress of atomic and nuclear physics. The USSR doesn’t come into the story as fully as one would like, and it is intriguing to speculate about the degree to which Stalinist dogmatism interfered with the development of Soviet physics. 
 
I also find fascinating in retrospect the relations that seem to exist between physics and the philosophy of science in the twentieth century. In philosophy we tend to think that the discipline of the philosophy of science in its twentieth-century development was too dependent on physics. That is probably true. But it seems that the physics in question was more often classical physics and thermodynamics, not modern mathematical physics. Carnap, for example, gives no serious attention to developments in the theory of quantum mechanics in his lectures, Philosophical Foundations of Physics. The philosophy of the Vienna Circle could have reflected relativity theory and quantum mechanics, but it didn’t to any significant degree. Instead, the achievements of nineteenth-century physics seem to have dominated the thinking of Carnap, Schlick, and Popper. Logical positivism doesn’t seem to be much influenced by modern physics, including relativity theory, quantum theory, and mathematical physics.  Post-positivist philosophers Kuhn, Hanson, and Feyerabend refer to some of the discoveries of twentieth-century physics, but their works don’t add up to a new foundation for the philosophy of science. Since the 1960s there has been a robust field of philosophy of physics, and the focus of this field has been on quantum mechanics; but the field has had only limited impact on the philosophy of science more broadly. (Here is a guide to the philosophy of physics provided to philosophy graduate students at Princeton; link.)

On the other hand, quantum mechanics itself seems to have been excessively influenced by a hyper version of positivism and verificationism. Heisenberg in particular seems to have favored a purely instrumentalist and verificationist interpretation of quantum mechanics — the idea that the mathematics of quantum mechanics serve solely to summarize the results of experiment and observation, not to allow for true statements about unobservables. It is anti-realist and verificationist.

I suppose that there are two rather different ways of reading the history of twentieth-century physics. One is that quantum mechanics and relativity theory demonstrate that the physical world is incomprehensibly different from our ordinary Euclidean and Kantian ideas about ordinary-sized objects — with the implication that we can’t really understand the most fundamental level of the physical world. Ordinary experience and relativistic quantum-mechanical reality are just fundamentally incommensurable. But the other way of reading this history of physics is to marvel at the amount of new insight and clarity that physics has brought to our understanding of the subatomic world, in spite of the puzzles and anomalies that seem to remain. Mathematical physical theory made possible observation, measurement, and technological use of the microstructure of the world in ways that the ancients could not have imagined. I am inclined towards the latter view.

It is also sobering for a philosopher of social science to realize that there is nothing comparable to this history in the history of the social sciences. There is no comparable period where fundamental and enduring new insights into the underlying nature of the social world became possible to a degree comparable to this development of our understanding of the physical world. In my view as a philosopher of social science, that is perfectly understandable; the social world is not like the physical world. Social knowledge depends on fairly humdrum discoveries about actors, motives, and constraints. But the comparison ought to make us humble even as we explore new theoretical ideas in sociology and political science.

If I were asked to recommend only one out of all these books for a first read, it would be David Cassidy’s Heisenberg volume, Beyond Uncertainty. Cassidy makes sense of the physics in a serious but not fully technical way, and he raises important questions about Heisenberg the man, including his role in the German search for the atomic bomb. Also valuable is Richard Rhodes’ book, The Making of the Atomic Bomb: 25th Anniversary Edition.

Inductive reasoning and the philosophy of science

I’ve just finished reading Sharon Bertsch McGrayne’s book on Bayesian statistics, The Theory That Would Not Die: How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy. McGrayne presents a very interesting story of the advancement of a scientific idea over a very long period (1740s through the 1950s). As she demonstrates at length, the idea that “subjective prior beliefs” could enhance our knowledge about causation and the future was regarded as paradoxical and irrational by mathematicians and statisticians for well over a century.

McGrayne’s book does a very good job of highlighting the scientific controversies that have arisen with respect to Bayesian methods, and the book also makes a powerful case for the value of the methods in many important contemporary problems. But it isn’t very detailed about the logic and mathematics of the field. She gives a single example of applied Bayesian reasoning in appendix b, using the example of breast cancer and mammograms. This is worth reading carefully, since it makes clear how the conditional probabilities of a Bayesian calculation work.

As McGrayne demonstrates with many examples, Bayesian reasoning permits a very substantial ability to draw novel conclusions based on piecemeal observations and some provisional assumptions about mechanisms in the messy world of complex causation. Examples can be found in epidemiology (the cause of lung cancer), climate science, and ecology. And she documents how Bayesian ideas have been used to enhance search processes for missing things — for example, lost hydrogen bombs and nuclear submarines. Here is an important example of the power of Bayesian reasoning to identify causal linkages to lung cancer, including especially cigarette smoking.

In 1951 Cornfield used Bayes’ rule to help answer the puzzle. As his prior hypothesis he used the incidence of lung cancer in the general population. Then he combined that with NIH’s latest information on the prevalence of smoking among patients with and without lung cancer. Bayes’ rule provided a firm theoretical link, a bridge, if you will, between the risk of disease in the population at large and the risk of disease in a subgroup, in this case smokers. Cornfield was using Bayes as a philosophy-free mathematical statement, as a step in calculations that would yield useful results. He had not yet embraced Bayes as an all-encompassing philosophy. Cornfield’s paper stunned research epidemiologists. 

More than anything else, it helped advance the hypothesis that cigarette smoking was a cause of lung cancer. Out of necessity, but without any theoretical justification, epidemiologists had been using case studies of patients to point to possible causes of problems. Cornfield’s paper showed clearly that under certain conditions (that is, when subjects in a study were carefully matched with controls) patients’ histories could indeed help measure the strength of the link between a disease and its possible cause. Epidemiologists could estimate disease risk rates by analyzing nonexperimental clinical data gleaned from patient histories. By validating research findings arising from case-control studies, Cornfield made much of modern epidemiology possible. In 1961, for example, case-control studies would help identify the antinausea drug thalidomide as the cause of serious birth defects. (110-111)

One fairly specific thing that strikes me after reading the book concerns the blindspots that existed in the neo-positivist tradition in the philosophy of science that set the terms for the field in the 1960s and 1970s (link). This tradition is largely focused on theories and theoretical explanation, to the relative exclusion of inductive methods. It reveals an underlying predilection for the idea that scientific knowledge takes the form of hypothetico-deductive systems describing unobservables. The hypothetico-deductive model of explanation and confirmation makes a lot of sense in the context of this perspective. But after reading McGrayne I’m retrospectively surprised at the relatively low priority given within standard philosophy of science curriculum to probabilistic reasoning — either frequentist or Bayesian. Many philosophers of science have absorbed a degree of disregard for “inductive logic”, or the idea that we can discover important features of the world through careful observation and statistical analysis. The basic assumption seems to have been that statistical reasoning is boring and Humean — not really capable of discovering new things about nature or society. But in hindsight, this disregard for inductive reasoning is an odd distortion of the domain of scientific knowledge, and, in particular, of the project of sorting out causes.

Some philosophers of science have indeed given substantial attention to Bayesian reasoning. (Here is a good article on Bayesian epistemology by Bill Talbott in the Stanford Encyclopedia of Philosophy; link.) Ian Hacking’s textbook An Introduction to Probability and Inductive Logic provides a very accessible introduction to the basics of inductive logic and Bayesian reasoning, and his The Emergence of Probability: A Philosophical Study of Early Ideas about Probability, Induction and Statistical Inference provides an excellent treatment of the history of the subject from a philosophy of science point of view. Another philosopher of science who has treated Bayesian reasoning in detail is Michael Strevens. Here Strevens provides a good brief treatment of the subject from the point of view of the philosophy of science (link). And here is a first-rate unpublished manuscript by Strevens on the use of Bayesian ideas as a theory of confirmation (link). Strevens’ recent Tychomancy: Inferring Probability from Causal Structure is also relevant. And the research program on causal reasoning of Judea Pearl has led to a flourishing of Bayesian reasoning in the theory of causality (link).

What is the potential relevance of Bayesian reasoning in sociology and other areas of the social sciences? Can Bayesian reasoning lead to new insights in assessing social causation? Several features of the social world seem particularly distinctive in the context of a Bayesian approach. Bayesianism conforms very naturally to a scenario-based way of approaching the outcomes of a system or a complicated process; and it provides an elegant and rigorous way of incorporating “best guesses” (subjective probability estimates) into the analysis of a given process. Both features are well suited to the social world. One reason for this is the relatively narrow limits of frequency-based estimates of probabilities of social events. The social sciences are often concerned with single-instance events — the French Revolution, the Great Depression, the rise of ISIS. In cases like these frequency-based probabilities are not available. Second, there is the problem of causal heterogeneity in many social causal relations. If we are interested in the phenomenon of infant mortality, we are led immediately to the realization that there are multiple social factors and conditions that influence this population characteristic; so the overall infant mortality rate of Bangladesh or France is the composite effect of numerous social and demographic causes. This means that there is no single underlying causal property X, where X can be said to create differences in infant mortality rates in various countries. And this in turn implies that it is dubious to assume that there are durable objective probabilities underlying the creation of a given rate of infant mortality. This is in contrast to the situation of earthquakes or hurricanes, where a small number of physical factors are causally relevant to the occurrence of the outcome.

Both these factors suggest that subjective probabilities based on expert-based assessment of the likelihood of various scenarios represent a more plausible foundation for assigning probabilities to a given social outcome. This is the logic underlying Philip Tetlock’s approach to reliable forecasting in Superforecasting: The Art and Science of Prediction and Expert Political Judgment: How Good Is It? How Can We Know? (link). Both points suggest that Bayesian reasoning may have even more applicability in the social world than in the natural sciences.

The joining of Monte Carlo methods with Bayesian reasoning that McGrayne describes in the case of the search for the missing nuclear submarine Thresher (199 ff.) is particularly relevant to social inquiry, it would seem. This is true because of the conjunctural nature of social causation and the complexity of typical causal intersections in the social domain. Consider a forecasting problem similar to those considered by Tetlock — for example, the likelihood that Russia will attempt to occupy Latvia in the next five years. One way of analyzing this problem is to identify a handful of political scenarios moving forward from the present that lead to consideration of this policy choice by Russian leadership; assign prior probabilities to the component steps of each scenario; and calculate a large number of Monte Carlo “runs” of the scenarios, based on random assignment of values to the component steps of each of the various scenarios according to the prior probabilities assigned by the experts. Outcomes can then be classified as “Russia attempts to occupy Latvia” and “Russia does not attempt to occupy Latvia”. The number of outcomes in the first cell allows an estimate of the overall likelihood of this outcome. The logic of this exercise is exactly parallel to the calculation that McGrayne describes for assigning probabilities to geographic cells of ocean floor for the final resting spot of the submarine, given the direction and speed scenarios considered. And the Bayesian contribution of updating of priors is illuminating in this analysis as well: as experts’ judgments of the probabilities of the component steps change given new information, the overall probability of the outcome changes as well.

Here is a very simple illustration of a scenario analysis. The four stages of the scenario are:

A: NATO signals unity

B: LATVIA accepts anti-missile defense

C: US signals lack of interest

D: KREMLIN in turmoil

Here is a diagram of the scenarios, along with hypothetical “expert judgments” about the likelihoods of outcomes of the branch points:

 


This analysis leads to a forecast of an 7.8% likelihood of occupation (O1, O10, O13). And an important policy recommendation can be derived from this analysis as well: most of the risk of occupation falls on the lower half of the tree, stemming from a NATO signal of disunity. This risk can be avoided by NATO giving the signal of unity instead; then the risk of occupation falls to less than 1%.


Predicting, forecasting, and superforecasting

I have expressed a lot of reservation about the feasibility of prediction of large, important outcomes in the social world (link, link, link). Here are a couple of observations drawn from these earlier posts:

We sometimes think that there is fundamental stability in the social world, or at least an orderly pattern of development to the large social changes that occur…. But really, our desire to perceive order in the things we experience often deceives us. The social world at any given time is a conjunction of an enormous number of contingencies, accidents, and conjunctures. So we shouldn’t be surprised at the occurrence of crises, unexpected turns, and outbreaks of protest and rebellion. It is continuity rather than change that needs explanation. 

Social processes and causal sequences have a wide range of profiles. Some social processes — for example, population size — are continuous and roughly linear. These are the simplest processes to project into the future. Others, like the ebb and flow of popular names, spread of a disease, or mobilization over a social cause, are continuous but non-linear, with sharp turning points (tipping points, critical moments, exponential takeoff, hockey stick). And others, like the stock market, are discontinuous and stochastic, with lots of random events pushing prices up and down. (link)

One reason for the failure of large-scale predictions about social systems is the complexity of causal influences and interactions within the domain of social causation. We may be confident that X causes Z when it occurs in isolated circumstances. But it may be that when U, V, and W are present, the effect of X is unpredictable, because of the complex interactions and causal dynamics of these other influences. This is one of the central findings of complexity studies — the unpredictability of the interactions of multiple causal powers whose effects are non-linear.

 

Another difficulty — or perhaps a different aspect of the same difficulty — is the typical fact of path dependency of social processes. Outcomes are importantly influenced by the particulars of the initial conditions, so simply having a good idea of the forces and influences the system will experience over time does not tell us where it will wind up.

Third, social processes are sensitive to occurrences that are singular and idiosyncratic and not themselves governed by systemic properties. If the winter of 1812 had not been exceptionally cold, perhaps Napoleon’s march on Moscow might have succeeded, and the future political course of Europe might have been substantially different. But variations in the weather are not themselves systemically explicable — or at least not within the parameters of the social sciences.

Fourth, social events and outcomes are influenced by the actions of purposive actors. So it is possible for a social group to undertake actions that avert the outcomes that are otherwise predicted. Take climate change and rising ocean levels as an example. We may be able to predict a substantial rise in ocean levels in the next fifty years, rendering existing coastal cities largely uninhabitable. But what should we predict as a consequence of this fact? Societies may pursue different strategies for evading the bad consequences of these climate changes — retreat, massive water control projects, efforts at atmospheric engineering to reverse warming. And the social consequences of each of these strategies are widely different. So the acknowledged fact of global warming and rising ocean levels does not allow clear predictions about social development. (link)

When prediction and expectation fail, we are confronted with a “surprise”.

So what is a surprise? It is an event that shouldn’t have happened, given our best understanding of how things work. It is an event that deviates widely from our most informed expectations, given our best beliefs about the causal environment in which it takes place. A surprise is a deviation between our expectations about the world’s behavior, and the events that actually take place. Many of our expectations are based on the idea of continuity: tomorrow will be pretty similar to today; a delta change in the background will create at most an epsilon change in the outcome. A surprise is a circumstance that appears to represent a discontinuity in a historical series. 

It would be a major surprise if the sun suddenly stopped shining, because we understand the physics of fusion that sustains the sun’s energy production. It would be a major surprise to discover a population of animals in which acquired traits are passed across generations, given our understanding of the mechanisms of evolution. And it would be a major surprise if a presidential election were decided by a unanimous vote for one candidate, given our understanding of how the voting process works. The natural world doesn’t present us with a large number of surprises; but history and social life are full of them. 

The occurrence of major surprises in history and social life is an important reminder that our understanding of the complex processes that are underway in the social world is radically incomplete and inexact. We cannot fully anticipate the behavior of the subsystems that we study — financial systems, political regimes, ensembles of collective behavior — and we especially cannot fully anticipate the interactions that arise when processes and systems intersect. Often we cannot even offer reliable approximations of what the effects are likely to be of a given intervention. This has a major implication: we need to be very modest in the predictions we make about the social world, and we need to be cautious about the efforts at social engineering that we engage in. The likelihood of unforeseen and uncalculated consequences is great. 

And in fact commentators are now raising exactly these concerns about the 700 billion dollar rescue plan currently being designed by the Bush administration to save the financial system. “Will it work?” is the headline; “What unforeseen consequences will it produce?” is the subtext; and “Who will benefit?” is the natural followup question. 

It is difficult to reconcile this caution about the limits of our rational expectations about the future based on social science knowledge, with the need for action and policy change in times of crisis. If we cannot rely on our expectations about what effects an intervention is likely to have, then we can’t have confidence in the actions and policies that we choose. And yet we must act; if war is looming, if famine is breaking out, if the banking system is teetering, a government needs to adopt policies that are well designed to minimize the bad consequences. It is necessary to make decisions about action that are based on incomplete information and insufficient theory. So it is a major challenge for the theory of public policy, to attempt to incorporate the limits of knowledge about consequences into the design of a policy process. One approach that might be taken is the model of designing for “soft landings” — designing strategies that are likely to do the least harm if they function differently than expected. Another is to emulate a strategy that safety engineers employ when designing complex, dangerous systems: to attempt to de-link the subsystems to the extent possible, in order to minimize the likelihood of unforeseeable interactions. (link)

One person who has persistently tried to answer the final question posed here — the conundrum of forming expectations in an uncertain world as a necessary basis for action — is Philip Tetlock. Tetlock’s decades-long research on forecasting and judging is highly relevant to this topic. The recent book Superforecasting: The Art and Science of Prediction provides an excellent summary of the primary findings of the research that he and senior collaborators have done on the topic.

Tetlock does a very good job of tracing through the sources of uncertainty that make projections and forecasts of the future so difficult. The uncertainties mentioned above all find discussion in Superforecasting; and he supplements these objective sources of uncertainty with a volume of recent work on cognitive biases leading to over- or under-confidence in a set of expectations. (Both Daniel Kahneman and Scott Page find astute discussions in the book.)

But in spite of these reasons to be dubious about pronouncements about future events, Tetlock finds that there are good theoretical and empirical reasons for believing that a modest amount of forecasting of complex events is nonetheless possible. He takes very seriously the probabilistic nature of social and economic events, so a forecast that “North Korea will perform a nuclear test within six months” must be understood as a probabilistic statement about the world (there is a specific likelihood of such a test in the world); and a Bayesian statement about the forecaster’s degree of confidence in the prediction. And good forecasters aim to be specific about both probabilities: for example, “I have a 75% level of confidence that there is a 55% likelihood of a North Korean nuclear test by date X”.

Moreover, Tetlock argues that it is possible to evaluate individual forecasters on the basis of their performance on specific tasks of forecasting and observation of the outcome. Tetlock would like to see the field of forecasting to follow medicine in the direction of an evidence-based discipline in which practices and practitioners are constantly assessed and permitted to improve their performance. (As he points out, it is not difficult to assess the weatherman on his or her probabilistic forecasts of rain or sun.) The challenge for evaluation is to set clear standards of specificity of the terms of a forecast, and then to be able to test the forecasts against the observed outcomes once the time has expired. This is the basis for the multiple-year tournaments that the Good Judgment Project has conducted over several decades. The idea of a Brier score serves as a way of measuring the accuracy of a set of probabilistic statements (link). Here is an explanation of “Brier scores” in the context of the Good Judgment Project (link); “standardized Brier scores are calculated so that higher scores denote lower accuracy, and the mean score across all forecasters is zero”. As the graph demonstrates, there is a wide difference between the best and the worst forecasters, given their performance over 100 forecasts.

So how is forecasting possible, given all the objective and cognitive barriers that stand in the way? Tetlock’s view is that many problems about the future can be broken down into component problems, some of which have more straightforward evidential bases. So instead of asking whether North Korea will test another nuclear device by November 1, 2016, the forecaster may ask a group of somewhat easier questions: how frequent have their tests been in the past? Do they have the capability to do so? Would China’s opposition to further tests be decisive?

Tetlock argues that the best forecasters do several things: they avoid getting committed to a single point of view; they consider conflicting evidence freely; they break a problem down into components that would need to be satisfied for the outcome to occur; and they revise their forecasts when new information is available. They are foxes rather than hedgehogs. He doubts that superforecasters are distinguished by being of uniquely superior intelligence or world-class subject experts; instead, they are methodical analysts who gather data and estimates about various components of a problem and assemble their findings into a combined probability estimate.

The author follows his own advice by taking conflicting views seriously. He presents both Daniel Kahneman and Nassim Taleb as experts who have made significant arguments against the program of research involved in the Good Judgment Project. Kahneman consistently raises questions about the forms of reasoning and cognitive processes that are assumed by the GJP. More fundamentally, Taleb raises questions about the project itself. Taleb argues in several books that fundamentally unexpected events are key to historical change; and therefore the incremental forms of forecasting described in the GJP are incapable in principle of keeping up with change (The Black Swan: Second Edition: The Impact of the Highly Improbable: With a new section: “On Robustness and Fragility” (Incerto) as well as the more recent Antifragile: Things That Gain from Disorder). These are arguments that resonate with the view of change presented in earlier posts and quoted above, and I have some sympathy for the view. But Tetlock does a good job of establishing that the situation is not nearly so polarized as Taleb asserts. Many “black swan” events (like the 9/11 attacks) can be treated in a more disaggregated way and are amenable to a degree of forecasting along the lines advocated in the book. So it is a question of degree, whether we think that the in-principle unpredictability of major events is more important or the incremental accumulation of many small causes is a preponderance of historical change. Processes that look like the latter pattern are amenable to piecemeal probabilistic forecasting.

Tetlock is not a fan of pundits, for some very good reasons. Most importantly, he argues that the great majority of commentators and prognosticators in the media and cable news are long on self-assurance and short on specificity and accountability. Tetlock argues several important points: first, that it is possible to form reasonable and grounded judgments about future economic, political, and international events; second, that it is crucial to subject this practice to evidence-based assessment; and third, that it is possible to identify the most important styles, heuristics, and analytical approaches that are used by the best forecasters (superforecasters).

(Here is a good article in the New Yorker on Tetlock’s approach; link.)

%d bloggers like this: