Experimental sociology of norms and decision-making

The discipline of experimental economics is now a familiar one. It is a field that attempts to probe and test the behavioral assumptions of the theory of economic rationality, microeconomics, and game theory. How do real human reasoners deliberate and act in classic circumstances of economic decision-making? John Kagel and Alvin Roth provide an excellent overview of the discipline in The Handbook of Experimental Economics, where they identify key areas of research in expected utility theory, game theory, free-riding and public goods theory, bargaining theory, and auction markets.

Behavioral economics is a related field but is generally understood as having a broader definition of subject matter. It is the discipline in which researchers use the findings of psychology, cognitive science, cultural studies, and other areas of behavioral sciences to address issues of economics, without making the heroic assumptions of strict economic rationality concerning the behavior and choices of the agents. The iconoclastic writings of Kahneman and Tversky are foundational contributions to the field (Choices, Values, and Frames), and Richard Thaler’s work (Nudge: Improving Decisions About Health and Wealth, and Happiness and Misbehaving: The Making of Behavioral Economics) exemplifies the approach.

Here is a useful description of behavioral and experimental economics offered by Ana Santos:

Behavioural experiments have produced a substantial amount of evidence that shows that human beings are prone to systematic error even in areas of economic relevance where stakes are high (e.g. Thaler, 1992; Camerer, 1995). Rather than grounding individual choice on the calculus of the costs and benefits of alternative options so as to choose the alternative that provides the highest net benefit, individuals have recourse to a variety of decisional rules and are influenced by various contextual factors that jeopardise the pursuit of individuals’ best interests. The increased understanding of how people actually select and apply rules for dealing with particular forms of decision problems and of the influence of contexts on individual choices is the starting point of choice architecture devoted to the study of choice setups that can curb human idiosyncrasies to good result, as judged by individuals themselves, or by society as a whole (Thaler and Sunstein, 2003, 2008).

Researchers in experimental and behavioral economics make use of a variety of empirical and “experimental” methods to probe the nature of real human decision-making. But the experiments in question are generally of a very specialized kind. The goal is often to determine the characteristics of the decision rule that is used by a group of actual human decision-makers. So the subjects are asked to “play” a game in which the payoffs correspond to one of the simple games studied in game theory — e.g. the prisoners’ dilemma — and their behavior is observed from start to finish. This seems to be more a form of controlled observation than experimentation in the classical sense — isolating an experimental situation and a given variable of interest F, and then running the experiment in the presence and absence of F.

It is intriguing to ask whether a similar empirical approach might be applied to some of the findings and premises of micro-sociology. Sociologists too make assumptions about motivation, choice, and action. Whether we consider the sociology of contention, the sociology of race, or the sociology of the family, we are unavoidably drawn to making provisional assumptions about what makes the actors in these situations tick. What are their motives? How do they evaluate the facts of a situation? How do they measure and weigh risk in the actions they choose? How do ambient social norms influence their action? Whether explicitly or implicitly, sociologists make assumptions about the answers to questions like these. Could some of the theoretical ideas of James Coleman, Erving Goffman, or Mark Granovetter be subjected to experimental investigation? Even more intriguingly, are there supra-individual hypotheses offered by sociologists that might be explored with experimental methods?

Areas where experimental and empirical investigation might be expected to pay dividends in sociology include the motivations underlying cooperation and competition, Granovetter’s sociology of social embeddedness, corruption, the theories of conditional altruism and conditional fairness, the dynamics of contention, and the micro-social psychology of race and gender.

So is there an existing field of research that attempts to investigate questions like these using experiments and human subjects placed in artificial circumstances of action?

To begin, there are some famous examples of experiments in the behavioral sciences that are relevant to these questions. These include the Milgram experiment, the Stanford Prison experiment, and a variety of altruism experiments. These empirical research designs aim at probing the modes of behavior, norm observance, and decision-making that characterize real human beings in real circumstances.

Second, it is evident that the broad discipline of social psychology is highly relevant to this topic. For example, the study of “motivated reasoning” has come to play an important role within the discipline of social psychology (link).

Motivated reasoning has become a central theoretical concept in academic discourse across the fields of psychology, political science, and mass communication. Further, it has also entered the popular lexicon as a label for the seemingly limitless power of partisanship and prior beliefs to color and distort perceptions of the political and social world. Since its emergence in the psychological literature in the mid- to late-20th century, motivated reasoning theory has been continuously elaborated but also challenged by researchers working across academic fields. In broad terms, motivated reasoning theory suggests that reasoning processes (information selection and evaluation, memory encoding, attitude formation, judgment, and decision-making) are influenced by motivations or goals. Motivations are desired end-states that individuals want to achieve. The number of these goals that have been theorized is numerous, but political scientists have focused principally on two broad categories of motivations: accuracy motivations (the desire to be “right” or “correct”) and directional or defensive motivations (the desire to protect or bolster a predetermined attitude or identity). While much research documents the effects of motivations for attitudes, beliefs, and knowledge, a growing literature highlights individual-level variables and contexts that moderate motivated reasoning.

See Epley and Gilovich (link) for an interesting application of the “motivated reasoning” approach.

Finally, some of the results of behavioral and experimental economics are relevant to sociology and political science as well.

These ideas are largely organized around testing the behavioral assumptions of various sociological theories. Another line of research that can be treated experimentally is the investigation of locally relevant structural arrangements that some sociologists have argued to be causally relevant to certain kinds of social outcomes. Public schools with health clinics have been hypothesized to have better educational outcomes than those without such clinics. Factory workers are sometimes thought to be more readily mobilized in labor organizations than office workers. Small towns in rural settings are sometimes thought to be especially conducive to nationalist-populist political mobilization. And so forth. Each of these hypotheses about the causal role of social structures can be investigated empirically and experimentally (though often the experiments take the form of quasi-experiments or field experiments rather than randomly assigned subjects divided into treatment and control populations).

It seems, then, that the methods and perspective of behavioral and experimental economics are indeed relevant to sociological research. Some of the premises of key sociological theories can be investigated experimentally, and doing so has the promise of further assessing and deepening the content of those sociological theories. Experiments can help to probe the forms of knowledge-formation, norm acquisition, and decision-making that real social actors experience. And with a little ingenuity, it seems possible to use experimental methods to evaluate some core hypotheses about the causal roles played by various kinds of “micro-” social structures.

ABM fundamentalism

 

I’ve just had the singular opportunity of participating in the habilitation examination of Gianluca Manzo at the Sorbonne, based on his excellent manuscript on the relevance of agent-based models for justifying causal claims in the social sciences. Manzo is currently a research fellow in sociology at CNRS in Paris (Centre National de la Recherche Scientifique), and is a prolific contributor to analytical sociology and computational social science. The habilitation essay is an excellent piece of work and I trust it will be published as an influential monograph. Manzo has the distinction of being expert both on the philosophical and theoretical debates that are underway about social causation and an active researcher in the field of ABM simulations. Pierre Demeulenaere served as a generous and sympathetic mentor. The committee consisted of Anouk Barberousse, Ivan Ermakoff, Andreas Flache, Olivier Godechot, and myself, and reviewer comments and observations were of the highest quality and rigor. It was a highly stimulating session.

One element of our conversation was especially enlightening to me. I have written a number of times in Understanding Society and elsewhere about the utility of ABM models, and one line of thought I have developed is a critique of what I have labeled “ABM fundamentalism” — the view that ABM models are the best possible technique for constructing social explanations for every possible subject in the social sciences (link). This view is expressed in Joshua Epstein’s slogan, “If you didn’t grow it, you didn’t explain it.” I maintain that ABM is a useful technique, but only one of many methods appropriate to the problem of constructing explanations of interesting sociological outcomes (link). So I advocate for theoretical and methodological pluralism when it comes to the ABM program.

I asked Gianluca whether he would agree that ABM fundamentalism is incorrect, and was surprised to find that he defends the universal applicability of ABM as a tool to implement any sociological theory. According to him, it is a perfectly general and universal modeling platform that can in principle be applied to any sociological problem. He also made it clear that he does not maintain that the use of ABM methods is optimal for every sociological problem of explanation. His defense of the universal applicability of ABM simulation techniques therefore does not imply that Manzo privileges these techniques as best for every sociological problem. But as a formal matter, he holds that ABM technology possesses the resources necessary to represent any fully specified social theory within a simulation.

The subsequent conversation succeeded in clarifying the underlying source of disagreement for me. What I realized in the discussion that ensued is that I was conflating two things in my label of ABM fundamentalism: the simulation technology and the substantive doctrine of generative social science. Epstein is a generativist, in the sense that he believes that social outcomes need in principle to be generated from a representation of facts about the individuals who make it up (Generative Social Science: Studies in Agent-Based Computational Modeling). Epstein is also an advocate of ABM techniques because they represent a particularly direct way of implementing a generativist explanation. But what Gianluca showed me is that ABM is not formally committed to the generativist dogma, and that an ABM simulation can perhaps incorporate factors at any social level. The insight that I gained, then, is that I should separate the substantive view of generativism from the formal mathematical tools of ABM simulations techniques.

I am still unclear how this would work — that is, how an ABM simulation might be created that did an adequate job of representing features at a wide range of levels — actors, organizations, states, structures, and ideologies. For example, how could an ABM simulation be designed that could capture a complex sociological analysis such as Tilly’s treatment of the Vendée, with peasants, protests, and merchants, the church, winegrowers’ associations, and the strategies of the state? Tilly’s historical narrative seems inherently multi-stranded and irreducible to a simulation. Similar points could be made about Michael Mann’s comparative historical account of fascisms or Theda Skocpol’s analysis of social revolutions.

So there is still an open question for me in this topic. But I think I am persuaded that the fundamentalism to which I object is the substantive premise of generativism, not the formal computational methods of ABM simulations themselves. And if Gianluca is correct in saying that ABM is a universal simulation platform (as a Turing machine is a universal computational device) then the objection is misplaced.

So this habilitation examination in Paris had exactly the effect for me that we would hope for in an academic interaction — it led me to look at an important issue in a somewhat different way. Thank you, Gianluca!

Modeling the social

One of the most interesting authorities on social models and simulations is Scott Page. This month he published a major book on this topic, The Model Thinker: What You Need to Know to Make Data Work for You, and it is a highly valuable contribution. The book corresponds roughly to the content of Page’s very successful Coursera course on models and simulations, and it serves as an excellent introduction to many different kinds of mathematical models in the social sciences.

Page’s fundamental premise in the book is that we need many models, and many intellectual perspectives, to make sense of the social world. Mathematical modeling is a way of getting disciplined about the logic of our theories and hypotheses about various processes in the world, including the physical, biological, and social realms. No single approach will be adequate to understanding the complexity of the world; rather, we need multiple hypotheses and models to disentangle the many concurrent causal and systemic processes that are under way at a single time. As Page puts the point:

As powerful as single models can be, a collection of models accomplishes even more. With many models, we avoid the narrowness inherent in each individual model. A many-models approach illuminates each component model’s blind spots. Policy choices made based on single models may ignore important features of the world such as income disparity, identity diversity, and interdependencies with other systems. (2)

Social ontology supports this approach in a fundamental way. The way I would put the point is this: social processes are almost invariably heterogeneous in their causes, temporal characters, and effects. So we need to have a way of theorizing society that is well suited to the forms of heterogeneity, and the many-models approach does exactly that.

Page proposes that there are multiple reasons why we might turn to models of a situation (physical, ecological, social, …): to “reason, explain, design, communicate, act, predict, and explore” (15). We might simplify this list by saying that models can enhance theoretical understanding of complex phenomena (explanation, discovery of truth, exploration of hypotheses) and they may also serve practical purposes involving prediction and control.

Especially interesting are topics taken up in later chapters of the book, including the discussion of network models and broadcast, diffusion, and contagion models (chapters 9-10). These are all interesting because they represent different approaches to a common social phenomenon, the spread of a property through a population (ideas, disease, rebellion, hate and intolerance). These are among the most fundamental mechanisms of social change and stability, and Page’s discussion of relevant models is insightful and accessible.eihdcchuljhknnlgkbrjtkhudklhinuvduffhghktubn

Page describes the constructs he considers as models, or abstract representations analogous to mathematical expressions. But we might also think of them as mini-theories of social mechanisms. Many of these examples illustrate a single kind of process that is found in real social situations, though rarely in a pure form. Games of coordination are a good example (chapter 15): the challenge of coordinating behavior with another purposive actor in order to bring about a beneficial outcome for both is a common social circumstance. Game theory provides an abstract analysis of how coordination can be achieved between rational agents; and the situation is more complicated when we consider imperfectly rational actors.

Another distinction that might be relevant in sorting the models that Page describes is that between “micro” and “macro”. Some of the models Page presents have to do with individual-level behavior (and interactions between individuals); whereas others have to do with transitions among aggregated social states (market states, political regimes, ecological populations). The majority of the models considered have to do with individual choice, decision rules, and information sharing — a micro-level approach comparable to agent-based modeling techniques. Several of the systems-dynamics models fall at the macro-end of the spectrum. Page treats this issue with the concept of “granularity”: the level of structure and action at which the model’s abstraction is couched (222).

The book closes with two very interesting examples of important social phenomena that can be analyzed using some of the models in the book. The first is the opioid epidemic in the United States, and the second is the last four decades’ rapid increase in economic inequality. Thomas Schelling’s memorable phrase, “the inescapable mathematics of musical chairs”, is relevant to both problems. Once we recognize the changing rates of prescription of opioids, clustering of opioid users, and probability of transitioning from usage to addiction, the explosion of addition rates and mortality is inevitable.

Early in the book Page notes the current vogue for “big data” as a solution to the problem of understanding and forecasting large social trends and changes. He rightly argues that the data do not speak for themselves. Instead, it is necessary to bring analytical techniques to bear in order to identify relevant patterns, and we need to use imagination and rigor in creating hypotheses about the social mechanisms that underlie the patterns we discover. The Model Thinker is indeed a model of an approach to analyzing and understanding the complex world of social action and interaction that we inhabit.

Machine learning

The Center for the Study of Complex Systems at the University of Michigan hosted an intensive day-long training on some of the basics of machine learning for graduate students and interested faculty and staff. Jake Hofman, a Microsoft researcher who also teaches this subject at Columbia University, was the instructor, and the session was both rigorous and accessible (link). Participants were asked to load a copy of R, a software package designed for the computations involved in machine learning and applied statistics, and numerous data sets were used as examples throughout the day. (Here is a brief description of R; link.) Thanks, Jake, for an exceptionally stimulating workshop.

So what is machine learning? Most crudely, it is a handful of methods through which researchers can sift through a large collection of events or objects, each of which has a very large number of properties, in order to arrive at a predictive sorting of the events or objects into a set of categories. The objects may be email texts or hand-printed numerals (the examples offered in the workshop), the properties may be the presence/absence of a long list of words or the presence of a mark in a bitmap grid, and the categories may be “spam/not spam” or the numerals between 0 and 9. But equally, the objects may be Facebook users, the properties “likes/dislikes” for a very large list of webpages, and the categories “Trump voter/Clinton voter”. There is certainly a lot more to machine learning — for example, these techniques don’t shed light on the ways that AI Go systems improve their play. But it’s good to start with the basics. (Here is a simple presentation of the basics of machine learning; link.)

Two intuitive techniques form the core of basic machine learning theory. The first makes use of the measurement of conditional probabilities in conjunction with Bayes’ theorem to assign probabilities of the object being a Phi given the presence of properties xi. The second uses massively multi-factor regressions to calculate a probability for the event being Phi given regression coefficients ci.

Another basic technique is to treat the classification problem spatially. Use the large number of variables to define an n-dimensional space; then classify the object according to the average or majority value of its m-closest neighbors. (The neighbor number m might range from 1 to some manageable number such as 10.)

There are many issues of methodology and computational technique raised by this approach to knowledge. But these are matters of technique, and smart data science researchers have made great progress on them. More interesting here are epistemological issues: how good and how reliable are the findings produced by these approaches to the algorithmic treatment of large data sets? How good is the spam filter or the Trump voter detector when applied to novel data sets? What kind of errors would we anticipate this approach to be vulnerable to?

One important observation is that these methods are explicitly anti-theoretical. There is no place for discovery of causal mechanisms or underlying explanatory processes in these calculations. The researcher is not expected to provide a theoretical hypothesis about how this system of phenomena works. Rather, the techniques are entirely devoted to the discovery of persistent statistical associations among variables and the categories of the desired sorting. This is as close to Baconian induction as we get in the sciences (link). The approach is concerned about classification and prediction, not explanation. (Here is an interesting essay where Jake Hofman addresses the issues of prediction versus explanation of social data; link.)

A more specific epistemic concern that arises is the possibility that the training set of data may have had characteristics that are importantly different from comparable future data sets. This is the familiar problem of induction: will the future resemble the past sufficiently to support predictions based on past data? Spam filters developed in one email community may work poorly in an email community in another region or profession. We can label this as the problem of robustness.

Another limitation of this approach has to do with problems where our primary concern is with a singular event or object rather than a population. If we want to know whether NSA employee John Doe is a Russian mole, it isn’t especially useful to know that his nearest neighbors in a multi-dimensional space of characteristics are moles; we need to know more specifically whether Doe himself has been corrupted by the Russians. If we want to know whether North Korea will explode a nuclear weapon against a neighbor in the next six months the techniques of machine learning seem to be irrelevant.

The statistical and computational tools of machine learning are indeed powerful, and seem to lead to results that are both useful and sometimes surprising. One should not imagine, however, that machine learning is a replacement for all other forms of research methodology in the social and behavioral sciences.

(Here is a brief introduction to a handful of the algorithms currently in use in machine-learning applications; link.)

ANT-style critique of ABM

A short recent article in the Journal of Artificial Societies and Social Simulation by Venturini, Jensen, and Latour lays out a critique of the explanatory strategy associated with agent-based modeling of complex social phenomena (link). (Thanks to Mark Carrigan for the reference via Twitter; @mark_carrigan.) Tommaso Venturini is an expert on digital media networks at Sciences Po (link), Pablo Jensen is a physicist who works on social simulations, and Bruno Latour is — Bruno Latour. Readers who recall recent posts here on the strengths and weaknesses of ABM models as a basis for explaining social conflict will find the article interesting (link). VJ&L argue that agent-based models — really, all simulations that proceed from the micro to the macro — are both flawed and unnecessary. They are flawed because they unavoidable resort to assumptions about agents and their environments that reduce the complexity of social interaction to an unacceptable denominator; and they are unnecessary because it is now possible to trace directly the kinds of processes of social interaction that simulations are designed to model. The “big data” available concerning individual-to-individual interactions permits direct observation of most large social processes, they appear to hold.

Here are the key criticisms of ABM methodology that the authors advance:

  • Most of them, however, partake of the same conceptual approach in which individuals are taken as discrete and interchangeable ‘social atoms’ (Buchanan 2007) out of which social structures emerge as macroscopic characteristics (viscosity, solidity…) emerge from atomic interactions in statistical physics (Bandini et al. 2009). (1.2)
  • most simulations work only at the price of simplifying the properties of micro-agents, the rules of interaction and the nature of macro-structures so that they conveniently fit each other. (1.4)
  • micro-macro models assume by construction that agents at the local level are incapable to understand and control the phenomena at the global level. (1.5)

And here is their key claim:

  • Empirical studies show that, contrarily to what most social simulations assume, collective action does not originate at the micro level of individual atoms and does not end up in a macro level of stable structures. Instead, actions distribute in intricate and heterogeneous networks than fold and deploy creating differences but not discontinuities. (1.11) 

This final statement could serve as a high-level paraphrase of actor-network theory, as presented by Latour in Reassembling the Social: An Introduction to Actor-Network-Theory. (Here is a brief description of actor-network theory and its minimalist social ontology; link.)

These criticisms parallel some of my own misgivings about simulation models, though I am somewhat more sympathetic to their use than VJ&L. Here are some of the concerns raised in earlier posts about the validity of various ABM approaches to social conflict (linklink):

  • Simulations often produce results that appear to be artifacts rather than genuine social tendencies.
  • Simulations leave out important features of the social world that are prima facie important to outcomes: for example, quality of leadership, quality and intensity of organization, content of appeals, differential pathways of appeals, and variety of political psychologies across agents.
  • The factor of the influence of organizations is particularly important and non-local.
  • Simulations need to incorporate actors at a range of levels, from individual to club to organization.
And here is the conclusion I drew in that post:
  • But it is very important to recognize the limitations of these models as predictors of outcomes in specific periods and locations of unrest. These simulation models probably don’t shed much light on particular episodes of contention in Egypt or Tunisia during the Arab Spring. The “qualitative” theories of contention that have been developed probably shed more light on the dynamics of contention than the simulations do at this point in their development.

But the confidence expressed by VJ&L in the new observability of social processes through digital tracing seems excessive to me. They offer a few good examples that support their case — opinion change, for example (1.9). Here they argue that it is possible to map or track opinion change directly through digital footprints of interaction (Twitter, Facebook, blogging), and this is superior to abstract modeling of opinion change through social networks. No doubt we can learn something important about the dynamics of opinion change through this means.

But this is a very special case. Can we similarly “map” the spread of new political ideas and slogans during the Arab Spring? No, because the vast majority of those present in Tahrir Square were not tweeting and texting their experiences. Can we map the spread of anti-Muslim attitudes in Gujarat in 2002 leading to massive killings of Muslims in a short period of time? No, for the same reason: activists and nationalist gangs did not do us the historical courtesy of posting their thought processes in their Twitter feeds either. Can we study the institutional realities of the fiscal system of the Indonesian state through its digital traces? No. Can we study the prevalence and causes of official corruption in China through digital traces? Again, no.

 
In other words, there is a huge methodological problem with the idea of digital traceability, deriving from the fact that most social activity leaves no digital traces. There are problem areas where the traces are more accessible and more indicative of the underlying social processes; but this is a far cry from the utopia of total social legibility that appears to underlie the viewpoint expressed here.
 
So I’m not persuaded that the tools of digital tracing provide the full alternative to social simulation that these authors assert. And this implies that social simulation tools remain an important component of the social scientist’s toolbox.
 

John von Neumann and stochastic simulations

source: Monte Carlo method (Wikipedia)

John von Neumann was one of the genuine mathematical geniuses of the twentieth century. A particularly interesting window onto von Neumann’s scientific work is provided by George Dyson in his  book, Turing’s Cathedral: The Origins of the Digital Universe. The book is as much an intellectual history of the mathematics and physics expertise of the Princeton Institute for Advanced Study as it is a study of any one individual, but von Neumann plays a key role in the story. His contribution to the creation of the general-purpose digital computer helped to lay the foundations for the digital world in which we now all live.

There are many interesting threads in von Neumann’s intellectual life, but one aspect that is particularly interesting to me is the early application of the new digital computing technology to the problem of simulating large complex physical systems. Modeling weather and climate were topics for which researchers sought solutions using the computational power of first-generation digital computers, and the research needed to understand and design thermonuclear devices had an urgent priority during the war and post-war years. Here is a description of von Neumann’s role in the field of weather modeling in designing the early applications of ENIAC  (P. Lynch, “From Richardson to early numerical weather prediction”; link):

John von Neumann recognized weather forecasting, a problem of both great practical significance and intrinsic scientific interest, as ideal for an automatic computer. He was in close contact with Rossby, who was the person best placed to understand the challenges that would have to be addressed to achieve success in this venture. Von Neumann established a Meteorology Project at the Institute for Advanced Study in Princeton and recruited Jule Charney to lead it. Arrangements were made to compute a solution of a simple equation, the barotropic vorticity equation (BVE), on the only computer available, the ENIAC. Barotropic models treat the atmosphere as a single layer, averaging out variations in the vertical. The resulting numerical predictions were truly ground-breaking. Four 24-hour forecasts were made, and the results clearly indicated that the large-scale features of the mid-tropospheric flow could be forecast numerically with a reasonable resemblance to reality. (Lynch, 9)

image: (link, 10)

A key innovation in the 1950s in the field of advanced computing was the invention of Monte Carlo simulation techniques to assist in the invention and development of the hydrogen bomb. Thomas Haigh, Mark Priestley, and Crispin Rope describe the development of the software supporting Monte Carlo simulations in the ENIAC machine in a contribution to the IEEE Annals of the History of Computing (link). Peter Galison offers a detailed treatment of the research communities that grew up around these new computational techniques (link). Developed first as a way of modeling nuclear fission and nuclear explosives, these techniques proved to be remarkably powerful for allowing researchers to simulate and calculate highly complex causal processes. Here is how Galison summarizes the approach:

Christened “Monte Carlo” after the gambling mecca, the method amounted to the use of random, numbers (a la roulette) to simulate the stochastic processes too complex to calculate in full analytic glory. But physicists and engineers soon elevated the Monte Carlo above the lowly status of a mere numerical calculation scheme; it came to constitute an alternative reality–in some cases a preferred one–on which “experimentation” could be conducted. (119) 

At Los Alamos during the war, physicists soon recognized that the central problem was to understand the process by which neutrons fission, scatter, and join uranium nuclei deep in the fissile core of a nuclear weapon. Experiment could not probe the critical mass with sufficient detail; theory led rapidly to unsolvable integro-differential equations. With such problems, the artificial reality of the Monte Carlo was the only solution–the sampling method could “recreate” such processes by modeling a sequence of random scatterings on a computer. (120)

The approach that Ulam, Metropolis, and von Neumann proposed to take for the problem of nuclear fusion involved fundamental physical calculations and statistical estimates of interactions between neutrons and surrounding matter. They proposed to calculate the evolution of the states of a manageable number of neutrons as they traveled from a central plutonium source through spherical layers of other materials. The initial characteristics and subsequent interactions of the sampled neutrons were assigned using pseudo-random numbers. A manageable number of sampled spaces within the unit cube would be “observed” for the transit of a neutron (127) (10^4 observations). If the percentage of fission calculated in the sampled spaces exceeded a certain value, then the reaction would be self-sustaining and explosive. Here is how the simulation would proceed:

Von Neumann went on to specify the way the simulation would run. First, a hundred neutrons would proceed through a short time interval, and the energy and momentum they transferred to ambient matter would be calculated. With this “kick” from the neutrons, the matter would be displaced. Assuming that the matter was in the middle position between the displaced position and the original position, one would then recalculate the history of the hundred original neutrons. This iteration would then repeat until a “self-consistent system” of neutron histories and matter displacement was obtained. The computer would then use this endstate as the basis for the next interval of time, delta t. Photons could be treated in the same way, or if the simplification were not plausible because of photon-matter interactions, light could be handled through standard diffusion methods designed for isotropic, black-body radiation. (129)

Galison argues that there were two fairly different views in play of the significance of Monte Carlo methods in the 1950s and 1960s. According to the first view, they were simply a calculating device permitting the “computational physicist” to calculate values for outcomes that could not be observed or theoretically inferred. According to the second view, Monte Carlo methods were interpreted realistically. Their statistical underpinnings were thought to correspond exactly to the probabilistic characteristics of nature; they represented a stochastic view of physics.

King’s view–that the Monte Carlo method corresponded to nature (got “back of the physics of the problem”) as no deterministic differential equation ever could–I will call

stochasticism

. It appears in myriad early uses of the Monte Carlo, and clearly contributed to its creation. In 1949, the physicist Robert Wilson took cosmic-ray physics as a perfect instantiation of the method: “The present application has exhibited how easy it is to apply the Monte Carlo method to a stochastic problem and to achieve without excessive labor an accuracy of about ten percent.” (146)

This is a very bold interpretation of a simulation technique. Rather than looking at the model as an abstraction from reality, this interpretation looks at the model as a digital reproduction of that reality. “Thus for the stochasticist, the simulation was, in a sense, of apiece with the natural phenomenon” (147).

One thing that is striking in these descriptions of the software developed in the 1950s to implement Monte Carlo methods is the very limited size and computing power of the first-generation general-purpose computing devices. Punch cards represented “the state of a single neutron at a single moment in time” (Haigh et al link 45), and the algorithm used pseudo-random numbers and basic physics to compute the next state of this neutron. The basic computations used third-order polynomial approximations (Haigh et al link 46) to compute future states of the neutron. The simulation described here resulted in the production of one million punched cards. It would seem that today one could use a spreadsheet to reproduce the von Neumann Monte Carlo simulation of fission, with each line being the computed result from the previous line after application of the specified mathematical functions to the data represented in the prior line. So a natural question to ask is — what could von Neumann have accomplished if he had Excel in his toolkit? Experts — is this possible?

An evolutionary view of research frameworks

social science methods

gene comparison

It was noted in a prior post that there is a great diversity of research frameworks in the social sciences, and that there is much to be gained by attempting to understand the social processes through which these frameworks change and develop over time (link).

Is it possible to get more specific about how the various research frameworks are related to each other, how similar or different they are, and how far apart they are in the space of the total universe of scientific research frameworks?

Here is an intriguing idea. We might think of a research framework as the compound of a set of “genes” in the form of a collection of ideas and practices through which scientists approach their studies of the world. And we might define the similarity or difference between two research frameworks in the way that geneticists define the similarity of two mammalian species, in terms of the degree of shared genetic material they possess.

On this approach, the research framework (methodology) constitutes a “code” which the young scientist learns through advanced training in the discipline. The resulting body of theory and research findings is the phenotype that results from the expression of the genotype through the activities of individuals and groups of scientists within the social and political environment of the research community.

The genotype of a methodology might consist of items like these:

  • ontological assumptions
  • investigatory strategies
  • practices of scientific collaboration
  • experimental designs
  • ideas about good explanation
  • procedures for evaluating evidence

On this view, scientific innovation takes place when researchers modify one or more elements of the code — background ontological beliefs, investigative practices, modes of confirmation, regulative ideas about theory and explanation. A given innovation may confer greater efficacy on the research framework by leading to more discoveries, more publications, or more funding.

By invoking the ideas of genotypes, phenotypes, and ecologies into the domain of research methodologies it is tempting to consider whether other aspects of the evolutionary paradigm are applicable as well, including the idea of evolution through selection.

Key here is the question of the mechanisms of fitness and selection that might be at work in the field of science. Do methodologies compete for survival and reproduction? Are there selection pressures at work in the domain of epistemology? Do scientific research frameworks “evolve” through small adjustments at the “gene” level (components of methodology at the level of theory and practice)? If so, do those selection processes lead to the evolution of methodologies better suited to discovering truth, or do they lead instead to methods that better serve the proximate interests of researchers and micro research communities?

One might argue that the two possibilities — selection pressure and veridicality — converge. Researchers use and refine methodologies based on their effort to generate publications, influence, and funding. One methodology is more “fit” than another insofar as it contributes to comparative advantage in these related outcomes. But methodologies generate influence, publication, and funding in proportion to the collective judgment of the expert community that they are creating new insight and more truthful representations of the world. So selection processes lead to greater veridicality.

This would be an agreeable outcome — the concrete practice of science leads generally to greater levels of truthful representation of the natural and social world. But is it plausible? Or does the history and sociology of science suggest that the factors that select for research methodologies are less epistemic and more situational or political? Does the process of science favor innovations that align with received views? Do scientific careers depend more on non epistemic factors than epistemic qualities? Does the process of science favor politically acceptable findings (“fracking is harmless” rather than “fracking causes micro-seismic activity”)? Are there contrarian “predator” practices at foot that actively contrive to push scientific findings away from truth (climate deniers, smoking industry advocates; Naomi Oreskes, Merchants of Doubt: How a Handful of Scientists Obscured the Truth on Issues from Tobacco Smoke to Global Warming)?

There are visible differences between biological evolution and the evolution of complexes of ideas and methods, of course. Biological evolution takes place through random variation at the genetic level, whereas theories and methods are subject to design by the scientists who develop them. Functional adaptation is the result of a blind process of selection in the biological world, whereas it is an intended consequence in the realm of theories and methods. And it is possible to define ecological fitness more precisely and mathematically in the biological realm than it is in the realm of science and culture.

It will be noted that this approach has a lot in common with the evolutionary approach to economics and the evolution of firms. A key text in this approach is Richard Nelson and Sidney Winter, An Evolutionary Theory of Economic Change. Richard Dawkins explored a similar view in The Selfish Gene when he considered “memes” as cultural counterparts to genes in the biological world.

Mechanisms, experiments, and policies

The social mechanisms approach to the social sciences aligns well with two key intellectual practices, experiments and policies. In an experiment we are interesting in testing whether a given factor has the effect it is thought to have. In a policy design we are interested in affecting an outcome of interest by manipulating some of the background conditions and factors. In both instances having a theory of the mechanisms in play in a domain permits us to frame our thinking better when it comes to designing experiments and policies.

Let’s say that we are interested in reducing the high school dropout rate in a high-poverty school. We may have a hypothesis that one important causal factor that leads to a higher likelihood of dropping out is that high-poverty students have a much greater burden of family and social problems than students in low-poverty populations. We might describe the mechanism in question in these terms:

H1: (a) high burden of social/familial problems => (b) student has higher likelihood of becoming discouraged => (c) student has higher likelihood of stopping attending => (d) student has a higher likelihood of dropping out of high school
We can evaluate this hypothesis about one of the mechanisms of dropping out of high school in several ways. First, we note that each clause invokes a likelihood. This means that we need to look at sets of students rather than individual students. Single cases or individual pairs of cases will not suffice, since we cannot make any inference from data like these:
A. Individual X has high burden of social/familial problems; Individual X does not become discouraged; Individual X does not drop out of high school.
B. Individual Y has a low burden of social/familial problems; Individual Y does become discouraged; Individual Y does drop out of high school.
Observations A and B are both compatible with the possible truth of the mechanisms hypothesis. Instead, we need to examine groups of individuals with various configurations of the characteristics mentioned in the hypothesis. If H1 is true, it can only be evaluated using population observations.
In theory we might approach H1 experimentally: randomly select two groups G1 and G2 of individuals; expose G1 to a high burden of social/familial problems while G2 is exposed to a low burden of social/familial problems; and observe the incidence of dropping out of high school. This would be to treat the hypothesis through an experiment based on the logic of random controlled trials. The difficulty here is obvious: we are harming the individuals in G1 in order to assess the causal consequences of the harmful treatment. This raises an irresolvable ethical problem. (Here is a discussion of Nancy Cartwright’s critique of the logic of RCT methodology in Evidence Based Policy; link.)
A slightly different experimental design would pass the ethics test. Select two schools S1 and S2 with comparable levels of high-poverty students and high burdens of social/familial problems for the individuals at the schools and comparable historical dropout rates. Now expose the students at S1 to a “treatment” that reduces the burden of social/familial problems (provide extensive social work services in the school that students can call upon). This design too conforms to the logic of a random controlled trial. Continue the treatment for four academic years and observe the graduation rates of the two schools. If H1 is true, we should expect that S1 will have a higher graduation rate than S2.
A third approach takes the form of a “quasi-experiment”. Identify pairs of schools that are similar in many relevant respects, but differ with respect to the burden of social/familial problems. This is one way of “controlling” for the causal influence of other observable factors — family income, race, degree of segregation in the school, etc. Now we have N pairs of matched schools and we can compute the graduation rate for the two components of the matches; that is, graduation rates for “high burden school” and “low burden school”. If we find that the high burden schools have a lower graduation rate than the low burden schools, and if we are satisfied that the schools do not differ systematically in any other dimension, then we have a degree of confirmation for the causal hypothesis H1. But Stanley Lieberson in Making It Count poses some difficult challenges for the logic of this kind of experimental test; he believes that there are commonly unrecognized forms of selection bias in the makeup of the test cases that potentially invalidates any possible finding (link).
So far we have looked at ways of experimentally evaluating the link between (a) and (d). But H1 is more complex; it hypothesizes that social/familial problems exercise their influence through two behavioral stages that may themselves be the object of intervention. The link from (b) to (c) is an independent hypothetical causal relation, and likewise the link from (c) to (d). So we might attempt to tease out the workings of these links in the mechanism as well. Here we might design our experiments around populations of high burden students, but attempt to find ways of influencing either discouragement or the link from discouragement to non-attendance (or possibly the link from non-attendance to full dropping out).
Here our intervention might go along these lines: the burden of social/familial problems is usually exogenous and untreatable. But within-school programs like intensive peer mentoring and encouragement might serve to offset the discouragement that otherwise results from high burden of social/familial problems. This can be experimentally evaluated using one or another of the designs mentioned above. Or we might take discouragement as a given but find an intervention that prevents the discouraged student from becoming a truant — perhaps a strong motivational incentive dependent on achieving 90% attendance during a six-week period.
In other words, causal hypotheses about causal mechanisms invite experimental and quasi-experimental investigation.

What about the other side of the equation; how do hypotheses about mechanisms contribute to policy intervention? This seems even more straightforward than the first question. The mechanism hypothesis points to several specific locations where intervention could affect the negative outcome with which we are concerned — dropping out of high school in this case. If we have experimental evidence supporting the links specified in the hypothesis, then equally we have a set of policy options available to us. We can design a policy intervention that seeks to do one or more of the following things: reduce the burden of social/familial problems; increase the level of morale of students who are exposed to a high burden; find means of encouraging high-burden students to persevere; and design an intervention to encourage truants to return to school. This suite of interventions touches each of the causal connections specified in the hypothesis H1.

Now, finally, we are ready to close the circle by evaluating the success of interventions like these. Does the graduation rate of schools where the interventions have been implemented work out to be higher than those where the interventions were not implemented? Can we begin to assign efficacy assessments to various parts of the policy? Can we arrive at secondary hypotheses about why this policy intervention (“reduce the burden of social/familial issues”) doesn’t succeed, whereas another policy intervention (“bolster morale among high-risk students”) does appear to succeed?
The upshot is that experiments and policies are opposite sides of the same coin. Both proceed from the common assumption that social causes are real; that we can assess the causal significance of various factors through experimentation and controlled observation; and that we can intervene in real-world processes with policy tools designed to exert influence at key junctures in the causal process.

George and Bennett on case study methodology

 

Establishing causal relationships within the fabric of the social world is more challenging than in the biological or physical-chemical domains. The reasons for this difficulty are familiar — the high degree of contextuality and contingency that is characteristic of social change, the non-deterministic character of social causation, and the fact that most social outcomes are the result of unforeseen conjunctions of independent influences, to name several.

Alexander George and Andrew Bennett argue for the value of a case-study method of social research in Case Studies and Theory Development in the Social Sciences. The idea here is that social researchers can learn about the causation of particular events and sequences by examining them in detail and in comparison with carefully selected alternative examples.

Here is how they describe the case-study method:

The method and logic of structured, focused comparison is simple and straightforward. The method is “structured” in that the researcher writes general questions that reflect the research objective and that these questions are asked of each case under study to guide and standardize data collection, thereby making systematic comparison and cumulation of the findings of the cases possible. The method is “focused” in that it deals only with certain aspects of the historical cases examined. The requirements for structure and focus apply equally to individual cases since they may later be joined by additional cases. (67)

George and Bennett believe that the techniques and heuristics of the case study approach permit the researcher to arrive at rigorous and differentiated hypotheses about underlying social processes. In particular, they believe that the method of process-tracing has substantial power in social research, permitting the researcher to move from the details of a particular historical case to more general hypotheses about causal mechanisms and processes in other contexts as well (6). They discourage research strategies based on the covering-law model, in which researchers would seek out high-level generalizations about social events and outcomes: “highly general and abstract theories … are too general to make sharp theoretical predictions or to guide policy” (7). But they also note the limits of policy relevance of “independent, stable causal mechanisms” (7), because social mechanisms interact in context-dependent ways that are difficult or impossible to anticipate. It is therefore difficult to design policy interventions based on knowledge of a few relevant and operative mechanisms within the domain of behavior the policy is expected to govern, since the workings of the mechanisms in concrete circumstances are difficult to project.

Fundamentally they align with the causal mechanisms approach to social explanation. Here is how they define a causal mechanism:

We define causal mechanisms as ultimately unobservable physical, social, or psychological processes through which agents with causal capacities operate, but only in specific contexts or conditions, to transfer energy, information, or matter to other entities. In so doing, the causal agent changes the affected entity’s characteristics, capacities, or propensities in ways that press until subsequent causal mechanisms act upon it. (137)

And they believe that the case-study method is a suite of methodological approaches that permit identification and exploration of underlying causal mechanisms.

The case study approach – the detailed examination of an aspect of a historical episode to develop or test historical explanations that may be generalizable to other events – has come in and out of favor over the past five decades as researchers have explored the possibilities of statistical methods … and formal models. (5)

The case study method is designed to identify causal connections within a domain of social phenomena.

Scientific realists who have emphasized that explanation requires not merely correlational data, but also knowledge of intervening causal mechanisms, have not yet had much to say on methods for generating such knowledge. The method of process-tracing is relevant for generating and analyzing data on the causal mechanisms, or processes, events, actions, expectations, and other intervening variables, that link putative causes to observed effects. (214)

How is that to be accomplished? The most important tool that George and Bennett describe is the method of process tracing. “The process-tracing method attempts to identify the intervening causal process–the causal chain and causal mechanism–between an independent variable (or variables) and the outcome of the dependent variable” (206). Process tracing requires the researcher to examine linkages within the details of the case they are studying, and then to assess specific hypotheses about how these links might be causally mediated. 

Suppose we are interested in a period of violent mobilization VM in the countryside at time t, and we observe a marked upswing of religious participation RP in the villages where we have observations. We might hypothesize that the surge of religious participation contributed causally to the political mobilization that ensued. But a process-tracing methodology requires that we we consider as full a range of alternative possibilities as we can: that both religious and political activism were the joint effect of some other social process; that religious participation was caused by political mobilization rather than caused that mobilization; that the two processes were just contingent and unrelated simultaneous developments. What can we discover within the facts of the case that would allow us to disentangle these various causal possibilities? If RP was the cause of VM, there should be traces of the influence that VM exerted within the historical record — priests who show up in the interrogation cells, organizational linkages that are uncovered through archival documents, and the like. This is the work of process tracing in the particular case. And I agree with George and Bennett that there is often ample empirical evidence available in the historical record to permit this kind of discovery.

Finally, George and Bennett believe that process-tracing can occur at a variety of levels:

The simplest variety of process-tracing takes the form of a detailed narrative or story presented in the form of a chronicle that purports to throw light on how an event came about…. A substantially different variety of process-tracing converts a historical narrative into an analytical causal explanation couched in explicit theoretical forms…. In another variety of process-tracing, the investigator constructs a general explanation rather than a detailed tracing of a causal process. (210-211)

One of the strengths of the book is an appendix presenting a very good collection of research studies that illustrate the case study methodology that they explore. There are examples from American politics, comparative politics, and international relations. These examples are very helpful because they give substance to the methodological ideas presented in the main body of the book.

Geddes on methods

Earlier posts have examined some recent thinking about social science methods (link, link). Here I will examine another recent contributor to this field, Barbara Geddes.

Geddes is a specialist in comparative politics, and her 2003 Paradigms and Sand Castles: Theory Building and Research Design in Comparative Politics is a thoughtful contribution to the debate about how the social sciences should proceed. Her central concern is with the topic of research design in comparative politics. How should a comparative researcher go about attempting to explain the varying outcomes we observe within the experiences of otherwise similar countries? How can we gain empirical grounds for validating or rejecting causal hypotheses in this field? And how do general theories of politics fare as a basis for explaining these concrete trajectories — the rise of authoritarianism in one country, the collapse of communism in the USSR, an outbreak of democracy in that country, or a surprising populism in another? Geddes finds that the theories that guided comparative politics in the sixties, seventies, and eighties proved to be inadequate to the task of explaining the twists and turns the political systems of the world took during those decades and argues that the discipline needs to do better.

Geddes’s proposed solution to this cul de sac is to bring theory and research design closer together. She wants to find a way of pursuing research in comparative politics that permits for more accumulation of knowledge in the field, both on the side of substantial empirical findings and well grounded theoretical premises. Theoretical premises need to be more carefully articulated, and plans for data collection need to be more purposefully guided so the resulting empirical findings are well suited to evaluating and probing the theoretical premises. Here is a good summary paragraph of her view:

The central message of this book is that we could steer a course through that narrow channel between untested theory and atheoretical data more successfully, and thus accumulate theoretical knowledge more rapidly, if certain research norms were changed. Although research norms are changing, basic principles of research design continue to be ignored in many studies. Common problems include inappropriate selection of cases from which to draw evidence for testing theories and a casual attitude towards nonquantitative measurement, both of which undermine the credibility of evidence gathered to support arguments. The failure to organize and store evidence in ways that make it accessible to others raises the cost of replication and that also slows theoretical progress. Uncritical acceptance by readers of theories that have not undergone systematic empirical test exacerbates the problem. (5)

What does Geddes mean by “theory” in this context? Her examples suggest that she thinks of a theory as a collection of somewhat independent causal hypotheses about a certain kind of large social outcome — the emergence of democracy or the occurrence of sustained economic development, for example. So when she discusses the validity of modernization theory, she claims that some components were extensively tested and have held up (the correlation between democracy and economic development, for example; 9), whereas other components were not adequately tested and have not survived (the claim that the diffusion of values would rapidly transform traditional societies; 9).

Geddes does not explicitly associate her view of social science inquiry with the causal mechanisms approach. But in fact the intellectual process of inquiry that she describes has a great deal in common with that approach. On her view of theory, the theory comes down to a conjunction of causal hypotheses, each of which can in principle be tested in isolation. What she refers to as “models” could as easily be understood as schematic descriptions of common social mechanisms (33). The examples she gives of models are collective action problems and evolutionary selection of social characteristics; and each of these is a mechanism of social causation.

She emphasizes, moreover, that the social causal factors that are at work in the processes of political and economic development generally work in conjunction with each other, with often unpredictable consequences.

Large-scale phenomena such as democratic breakdown, economic development, democratization, economic liberalization, and revolution result from the convergence of a number of different processes, some of which occur independently from others. No simple theory is likely to explain such compound outcomes.  Instead of trying to “explain” such compound outcomes as wholes, I suggest a focus on the various processes that contribute to the final outcome, with the idea of theorizing these processes individually. (27)

What Geddes’s conception of “theory” seems to amount to is more easily formulated in the language of causal mechanisms. We want to explain social outcomes at a variety of levels of scale — micro, meso, macro. We understand that explanation requires discovery of the causal pathways and processes through which the outcome emerged. We recognize that social outcomes have a great deal of contingency and path dependency, so it is unlikely that a great outcome like democratization will be the result of a single pervasive causal factor. Instead, we look for mid-level causal mechanisms that are in place in the circumstances of interest — say the outbreak of the Bolshevik uprising; and we attempt to discern the multiple causal factors that converged in these historical circumstances to bring about the outcome of interest. The components of theories to which Geddes refers are accounts of reasonably independent causal mechanisms and processes, and they combine in contingent and historically specific ways.

And in fact she sometimes adopts this language of independent mid-level causal mechanisms:

To show exactly what I mean, in the pages that follow I develop a concrete research strategy that begins with the disaggregation of the big question — why democratization occurs — into a series of more researchable questions about mechanisms. The second step is a theorization of the specific process chosen for study — in this case, the internal authoritarian politics that sometimes lead to transition. The third step is the articulation of testable implications derived from the theorization. (43)

And later:

I argued that greater progress could be made toward actually understanding how such outcomes [as democratization and authoritarian rule] by examining the mechanisms and processes that contribute to them, rather than through inductive searches for the correlates of the undifferentiated whole. (87)

(This parallels exactly the view taken by McAdam, Tarrow, and Tilly in Dynamics of Contention, where they argue systematically for a form of analysis of episodes of contention that attempts to identify recurring underlying processes and mechanisms.)

It emerges that what Geddes has in mind for testing mid-level causal hypotheses is largely quantitative: isolate a set of cases in which the outcome is present and examine whether the hypothesized causal factor varies appropriately across the cases. Do military regimes in fact persist with shorter average duration than civilian authoritarian regimes (78)? Like King, Keohane, and Verba in Designing Social Inquiry: Scientific Inference in Qualitative Research, Geddes is skeptical about causal methods based on comparison of a small number of cases; and like KKV, she is critical of Skocpol’s use in States and Social Revolutions: A Comparative Analysis of France, Russia and China of Mill’s methods in examining the handful of cases of social revolution that she examines. This dismissal of small-N research represents an unwelcome commitment to methodological monism, in my view.

In short, I find Geddes’s book to be a useful contribution that aligns more closely than it appears with the causal mechanisms approach to social research. It is possible to paraphrase Geddes’s approach to theory and explanation in the language of causal mechanisms, emphasizing meso-level analysis, conjunctural causation, and macro-level contingency. (More on this view of historical causation can be found here.)

Geddes’s recommendations about how to probe and test the disaggregated causal hypotheses at which the researcher arrives represent one legitimate approach to the problem of giving greater empirical content to specific hypotheses about causal mechanisms. It is regrettable, however, that Geddes places her flag on the quantitative credo for the social sciences. One of the real advantages of the social mechanisms approach is precisely that we can gain empirical knowledge about concrete social mechanisms through detailed case studies, process tracing, and small-N comparisons of cases that is not visible at the level of higher-level statistical regularities. (A subsequent post will examine George and Bennett, Case Studies and Theory Development in the Social Sciences (Belfer Center Studies in International Security), for an alternative view of how to gain empirical knowledge of social processes and mechanisms.)

%d bloggers like this: