Using experiments in comparative policy analysis: from policy evaluation to the policy process

TY - CHAP

T1 - Using experiments in comparative policy analysis

T2 - from policy evaluation to the policy process

AU - John, Peter

N1 - Publisher Copyright: © B. Guy Peters and Guillaume Fontaine 2020. All rights reserved.

PY - 2020/1/1

Y1 - 2020/1/1

N2 - This chapter sets out the advantages (and some disadvantages) of random allocation or the randomized controlled trial (RCT) as a method to test interventions and evaluate casual processes in the social sciences. Randomization allows the evaluation of the counterfactual as both observed and unobserved factors are equally present in both treatment and control groups except for the variable of interest or intervention. While RCTs were present at the founding of the social sciences in the early twentieth century, the method languished until its enthusiastic rediscovery by development economists and students of political behaviour in the late 1990s and early 2000s. Now both economics and political science have a profusion of experiments and many scholars adopt the framework of causal inference and use identified designs in the form of regression discontinuity designs, difference-in-differences, comparative interrupted time series, and instrumental variable models, which may be called natural experiments. Trials have been used increasingly in public policy evaluations since the 1960s. But studies of the policy process and implementation remain largely untouched. This contrasts with their rapid adoption in the sister discipline of public administration, which shares an interest in decision making and the role of institutions and organizations, as shown by a recent wave of experiments with bureaucrats and studies under the banner of behavioural public administration. The chapter seeks to explain this lack of interest as a result of social networks and traditions of study in public policy, which have led the predominance of case study and interview-based methods (which can of course be integrated into randomized designs). The conclusion to draw is that there is a massive gap to be filled in the study of public policy. Experimental studies should be carried out so public policy studies can keep pace with advances in other disciplines.

AB - This chapter sets out the advantages (and some disadvantages) of random allocation or the randomized controlled trial (RCT) as a method to test interventions and evaluate casual processes in the social sciences. Randomization allows the evaluation of the counterfactual as both observed and unobserved factors are equally present in both treatment and control groups except for the variable of interest or intervention. While RCTs were present at the founding of the social sciences in the early twentieth century, the method languished until its enthusiastic rediscovery by development economists and students of political behaviour in the late 1990s and early 2000s. Now both economics and political science have a profusion of experiments and many scholars adopt the framework of causal inference and use identified designs in the form of regression discontinuity designs, difference-in-differences, comparative interrupted time series, and instrumental variable models, which may be called natural experiments. Trials have been used increasingly in public policy evaluations since the 1960s. But studies of the policy process and implementation remain largely untouched. This contrasts with their rapid adoption in the sister discipline of public administration, which shares an interest in decision making and the role of institutions and organizations, as shown by a recent wave of experiments with bureaucrats and studies under the banner of behavioural public administration. The chapter seeks to explain this lack of interest as a result of social networks and traditions of study in public policy, which have led the predominance of case study and interview-based methods (which can of course be integrated into randomized designs). The conclusion to draw is that there is a massive gap to be filled in the study of public policy. Experimental studies should be carried out so public policy studies can keep pace with advances in other disciplines.

UR - http://www.scopus.com/inward/record.url?scp=85125824884&partnerID=8YFLogxK

U2 - 10.4337/9781788111195.00016

DO - 10.4337/9781788111195.00016

M3 - Chapter

AN - SCOPUS:85125824884

SN - 9781788111188

SP - 153

EP - 166

BT - Handbook of Research Methods and Applications in Comparative Policy Analysis

PB - Edward Elgar Publishing Ltd

ER -