Michael Lister has responded to my critique of his work with what Lakatos (1980) called a “problem shift”. In his original paper, Lister (2007) presented empirical evidence that selective welfare state institutions undermine norms of solidarity, which in turn depresses turnout in general elections. In my comment (Arzheimer 2008), I demonstrate that, while the hypothesis is certainly plausible, the evidence itself is inconclusive at best. From my reading of Lister’s reply, I take it that we are in agreement on this point at least. Our disagreements (or mutual misunderstandings?) relate to three different issues, which I will address in turn.
The Status of Social Norms and the Problem of Ecological Fallacy
Lister (2009) argues that I do “great violence” to his position by oversimplifying his presentation of the causal chain that links a) institutions, b) social norms, c) individual (internalised) norms, d) individual voting behaviour and e) turnout, and also by falsely accusing him of committing an ecological fallacy. In my bid to develop a stylised representation of his argument (Arzheimer 2008: 682), I do indeed simplify his sophisticated theoretical account and omit social norms, and I do so for two good reasons: first, in the reductionist spirit of Almond and Verba (1965), it is not clear how social norms are different from the distribution of individual attitudes; and second, even if we assume that social norms do have an independent ontological status at the macro level, we have no independent measure for them and cannot, therefore, model their relationship with institutions. As for my “false accusation”, Lister (2007) studies a correlation between two aggregate variables, inequality, as a proxy for institutions, and turnout, which he then interprets in terms of individual agency. That approach, by definition, constitutes an ecological fallacy.
A quantitative impasse?
In my comment on Lister’s original article, I point out that institutional variables tend to vary a lot between countries but are often fairly stable over time within countries. As a result, it is often extremely difficult to disentangle institutional and country effects. Responding to this point, Lister (2009) concludes that a “statistical evaluation of this [i.e. his] hypothesis is virtually impossible”, but this is not major problem because his original results “should be seen as ‘indicative’” anyway. I could not disagree more strongly.
First, new statistical methods have been developed that are better suited to dealing with “weak” datasets (see the references in my comment). Second, over the last three decades or so, waves of democratisation have given us many more (varied) observations, which most studies have so far ignored. Moreover, political scientists and welfare economists have begun to tap into the power of experimental surveys that allow us to confront citizens with a whole range of alternative hypothetical welfare-state arrangements. The quantitative analysis of such experiments at the micro level is a useful complement to macro-quantitative information.
Finally, and perhaps most importantly, we should remember that the whole purpose of quantitative analysis is to test competing hypotheses so that the most plausible can be identified, given the data at hand. If one is not interested in this type of reality-check, there is little point in conducting quantitative analysis.
Qualitative vs Quantitative Analysis of the Institutions-Attitudes link – a false dichotomy
In his response to my comment, Lister suggests that the alleged link between institutions and attitudes should be studied by applying “more qualitative techniques, such as case studies”. He then briefly compares the development of the welfare state and levels of social trust in Sweden and the United States.
Following King, Keohane and Verba (1994: 4), qualitative research does not rely on numerical measurement but aims at providing more detailed information on the phenomenon in question. However, the same logic of inference associated with quantitative research and the same general rules of scientific inquiry apply. In this instance, an appropriate qualitative research design might include a detailed analysis of welfare-state legislation (at the macro level), accompanied by some carefully selected in-depth interviews with Swedish and American citizens that illustrate their experience of the welfare state and their more-or-less-trusting relationship with other citizens.
What we get instead is the information that some benefits introduced under President Johnson were effectively abolished under the Reagan and Clinton administrations so that the system remained highly selective, that cuts were made in Sweden during the 1990s without changing the universal nature of the system, that social trust has fallen in the US from 53% in 1963 to 35% in 1999 and that levels of social trust in Sweden are extremely high at present and were presumably slightly lower in the past.
How is this different from the original quantitative analysis? The institutional information could easily be translated into an ordinal or quasi-interval measure for universalism (from low to extremely low in the United States, and from extremely high to quite high in Sweden), while the information on social trust comes directly from quantitative population surveys such as the World Values Survey.
The dependent variable has thus changed from turnout to social trust, yet we are still left with an opaque aggregate correlation that could have been caused by any number of processes at the micro level. The biggest difference between this new information and the original analysis is that Lister has dropped all control variables and reduced the number of observations from about 130 to 4.
Lakatos famously differentiated between “progressive” and “degenerative” problem shifts. Whether the switch from a time-series cross-sectional design to a case-study design falls into the former or the latter category is a judgement best left to the reader.
Almond, Gabriel A. and Verba, Sidney, 1965: The Civic Culture. Political Attitudes and Democracy in Five Nations. Boston, Little, Brown and Company.
Arzheimer, Kai, 2008: ‘Something old, something new, something borrowed, something true? A comment on Lister’s ‘Institutions, Inequality and Social Norms: Explaining Variations in Participation’’, British Journal of Politics and International Relations, 10, 681-697
King, Gary, Keohane, Robert O. and Verba, Sidney, 1994: Designing Social Inquiry. Scientic Inference in Qualitative Research. Princeton, Princeton University Press.
Lakatos, Imre, 1980: The Methodology of Scientific Research Programs. Cambridge, Cambridge University Press.
Lister, Michael, 2007: ‘Institutions, inequality and social norms: Explaining variations in participation’, British Journal of Politics and International Relations, 9, 20–35.