Assessing impacts of social policy innovation in the EU:
A methodological framework rooted on complex systems theories and dynamic simulation modelling - i-FRAME 2.0

Gianluca Misuraca
European Commission, Joint Research Centre, Directorate Growth & Innovation, ESP

Cristiano Codagnone
Università degli Studi di Milano, ITA
Universitat Oberta de Catalunya, Barcelona, ESP

Abstract

This paper discusses the challenges for evaluating social innovation policies, and the potential of alternative approaches to measure and assess social and economic impacts. It outlines the rationale and approach for developing a methodological framework for simulation modelling of impacts of policy interventions promoting social investment developed by the European Commission's Joint Research Centre (JRC) as part of the research 'ICT-Enabled Social Innovation' (IESI). This framework, in short i-FRAME, is rooted in complex systems theory and dynamic simulation modelling methodologies. It supports the implementation of reforms suggested under the Social Investment Package (SIP) which encourages Member States to prioritize social investment approaches and modernize welfare systems.

Disclaimer: The views expressed in this paper are purely those of the authors and may not in any circumstances be regarded as stating an official position of the European Commission.

Acknowledgments: This article is based on research conducted by the European Commission's Joint Research Centre (JRC) as part of the project 'ICT-Enabled Social Innovation in support to the Implementation of the Social Investment Package' (IESI) - link

Policy context and research background

The Social Investment Package (SIP), adopted by the EU (European Commission, 2013) to fight poverty and reduce polarization by stimulating social innovation, was the first tangible result of the public debate exploring the economic and social dimensions of capitalism, spurred by the 2007/2008 recession. In support of social policy innovation and social innovation as ways to engage social investment, the European Commission's Joint Research Centre conducted a multi-year research project: the final result is the methodological framework known as i-FRAME. This framework is designed to help policy makers and practitioners assess and evaluate interventions in the area of social policy innovation and social innovation with a specific focus on the use ICT-Enabled initiatives.

Such issues as employment, youth, ageing, housing and social inclusion are not discrete domains; policies and services addressing such issues need to be integrated and mutually reinforced rather than siloed (Kvist, 2014). They interact with each other and with budgetary constraints and macro-economic conditions. Hence i-FRAME, as an evaluation framework, engages with the complexity of several sub-systems in constant interaction and adaptation with each other and the overall socio-economic system. The various reviews and case studies conducted in this multi-year research project showed that the lack of understanding of such complexity is one of the main factor explaining the failure of interventions in this domain. In policy-making there are countless examples of interventions that are defeated by the system's response to the intervention itself (Sterman, 2006). From the perspective of policy, complexity is a two-fold challenge. First, it poses tangible barriers and constraints; for example budgetary trade-offs when facing competing social problems to address competing needs. Second, and possibly most importantly, complexity is an intellectual challenge as it shows the inadequacy of the intellectual framework adopted to design and evaluate policy interventions. Even when available, evidence may often be ignored or misused. Many interventions, for instance, are ineffective for lack of an adequate understanding of motivations, behaviors and the cognitive bias and heuristics (bounded rationality) for both potential beneficiaries of social policies and of those designing and enacting them. So, the goal and inspiration of the i-FRAME is to recognize and deal with the complexity surrounding social policy innovation and the social innovations themselves.

In a recent essay reviewing three books that bring complexity theory into policy evaluation (Forss et al., 2011; Patton, 2011; Wolf-Branigin, 2013), Gerrits and Verweij (2015) warn that 'complexity theory' is not always used in an internally consistent fashion. Complexity theory, in fact, includes several loosely grouped concepts, mechanisms and models and is often brought into narratives in generic way. The authors raise "a dual concern: that the complexity of social reality is often ignored, leading to misguided evaluations and policy recommendations, and that the current methodological toolbox is not particularly well-suited to deal with that complexity" (Gerrits & Verweij, 2015: 485). In the elaboration of the i-FRAME the focus on complexity has been more on the second dimension, the methodological toolbox, for two reasons. First, in relation to the choice of a modelling simulation tool to be included as one of the operational component of the framework we compared models as to their capacity to deal with complexity. Second, complexity underlies our critique of the more orthodox version of Evidence Based Policy (EBP) that characterizes the reviewed literature. Actually, these two aspects can be seen as 'emergent outcomes' in an open and non-linear complex process of research and discovery.

This paper, presents the rational and the final results of i-FRAME framework elaboration, whose underlying epistemology is presented in Section 2. Section 3 then presents the final proposal for the methodological framework, while in Section 4, we draw conclusions and explain the implications these have for policy, outlining directions for future research.

Dealing with complexity: The need for pluralistic approaches

We observed that traditional economic modelling simulations based on Computable General Equilibrium (CGE) or Dynamic Stochastic General Equilibrium (DSGE) models have proven very ineffective at both predicting the recession of 2007-2008 and, especially, at prescribing the way out of it (Fagiolo & Roventini, 2016). The reason for this failure was the overly simplistic use of representative agents and the idea that at any given point in time we will find quantities and prices for a clearing equilibrium. So, moving through the comparative analysis of different modelling approaches we came to the conclusion that models should be able to consider economy and society as complex evolving system. An ecology populated by heterogeneous agents, whose far-from-equilibrium interactions continuously change the structure of the system. In this respect the concepts of 'emergent properties/ effect' and Complex Adaptive Systems (CAS) are fundamental and are worth discussing a bit further for they underpin the choice of the proposed modelling tool.

From an epistemological point of view, the concept of emergent properties/effects outlines an anti-reductionist view of reality. A set of entities at a certain level owes its existence to lower-level entities but also presents a set of states / properties / regularity of its own that can be studied independently. For all of the above reasons and the possibility of emergent effects the interaction between economy, society, and policy intervention presents a level of complexity that should be modelled as the result of the interaction between heterogeneous agents undertaking heterogeneous activities. Hence, the macro-socio-economic system should be conceived as a Complex Adaptive System (CAS), in which myriads of agents with their different interests and behavioral characteristics interacts on interrelated domains (Arthur, 2013; Kirman, 2010). CAS are highly non-linear (often due to interaction) and are organized on many spatial and temporal scales. This built in non-linearity leads to endogenous fluctuations and sudden regime changes from one type of regime to another. The properties of the system are modifiable as a result of interactions. The notion of CAS is particularly useful when one conceives innovation in evolutionary terms (Dosi, 1988; 2013; Nelson & Winter, 1982). Recently, in fact, the evolutionary theory of innovation has been embedded in Agent Based Models with remarkable results (Dosi et al., 2013; Dosi et al., 2015; Dosi et al., 2010).

It was evident how current approaches based on measurement indicators or on quasi-experimental counterfactual evaluations, lacked not only the instruments but also the underlying epistemology needed to address the complexity surrounding social policy innovation and social innovation. Many of the reviewed approaches suffered from a deterministic understanding of Evidence Based Policy (EBP), from which the i-FRAME took distance as illustrated below. EBP in its more extreme and orthodox variants, adopting a linear and simplistic positivist epistemology, seeks to keep politics and deliberation out of policy-making adopting. But in so doing, it falls into three main pitfalls (e.g., McMillin, 2012; Sanderson, 2011; Cartwright & Hardie, 2012)

First, Randomized Controlled Trials (RCTs) do not provide sufficient evidence to indicate that a policy, which is successful 'there', will work 'here'. Even when best implemented, RCTs have limited external validity, unless replicated in different settings and contexts. This is particularly true in social sciences, given the complexity that characterizes socio-economic ecosystems. The attractiveness of RCTs is that they are self-validating if implemented correctly (i.e., they ensure internal validity) and they do not require that we understand why a certain cause produces a certain effect. The How and Why can apparently be ignored. Policymakers 'can take a free ride and have confidence that this caused that, without wondering how it happened, because somebody knows' (Cartwright & Hardie, 2012: 124). In practice, however, all the times what worked there turns out not to work here shows that things are more complex and we cannot get away from theory, and eventually deliberation. This is so, because we need to search for consequences of non-redundant pieces of non-necessary but sufficient contextual causes (Bogliacino et al., 2015). In practice, a policy intervention can rarely cause an effect by itself but does so in combination with other contextual support factors.

Second, and related to the first critique, EBP by stressing the importance of RCTs betrays an inadequate view of causality. Policy remains a domain of competing interests where deliberation cannot be ruled out. Hence, before simply replicating an intervention here because an RCT showed that it worked there, policymakers and their advisers must always ask how and why questions, rather than following the deterministic prescriptions of orthodox EBP. A horizontal search of support factors and a vertical search of causal principles at the correct level of abstraction should always be carried out. But above all causality should be re-interpreted in view of complexity.

Third, EBP's claim to objectivity run against the limits of measurement in the context of studying socio-economic systems where concepts and measurement in the social sciences can be socially constructed, value-laden, and politically motivate. At time the concepts and measurements used in empirical research and ranked as best evidence by EBP are in fact chosen and framed by particular groups, especially by those in power. This may bias the research and prevent us from considering the views of less powerful social groups.

i-FRAME: Methodological framework and operational approach for Evidence Informed Social Policy Innovation

Figure 3 below presents the final proposal of the i-FRAME 2.0 as a generic meta-framework which can be applied at different levels.

Figure3

Figure 3: i-FRAME 2.0: Diamond for Evidence-Informed Social Policy Innovation

Departing from orthodox EBP, the starting point is a 'deliberation and design phase' that shapes the design of interventions to inform the following evaluation phases. It will produce gradual but constant improvements across the cycle depicted in Figure 3 and called the Diamond for Evidence-Informed Social Policy Innovation. In our approach, we are not offering 'a magic bullet' but a set of options inspired by methodological pluralism. Discretion of deliberative processes will be needed, albeit informed by pluralist, open, and transparent evidence.

For instance, in order to check, in the deliberation phase, whether a successful policy can work in different settings, we must look for causal explanations for the 'how' and 'why' questions. This can be done econometrically using structural evaluation (Heckman, 2010), or alternatively using broadly defined theory of change or theory-based evaluation (for an application see Carvalho & White, 2004). In theory-based policy evaluation interventions are not seen as monoliths. Beneficiaries and stakeholders are not simply passive recipients and takers of the 'treatments': their views are crucial to the evaluation. These views are collected through interviews or from relevant documents and are treated as 'theories' of change and action. They are used as hypotheses to be tested empirically. The tools for the proposed deliberation phase of i-FRAME are inspired by this latter approach. They include: Quantitative Story Telling (QST); Problem tree; Ex ante failure scenario with simplified causal model; Step-by-Step and backward theory-based evaluation thinking; and Quick exit tree. In this phase, hypotheses and theories can be derived from RCTs, meta-analysis, systematic reviews, and other mixed methods of evaluation. They should then be tested in the settings where a policy is being considered, having gathered all the relevant evidence using mixed methods and triangulating different sources of evidence. The theory has to describe what conditions are needed and what causal principles may be at work for a policy to achieve the desired outcomes. Assumptions may be developed during the different steps of an intervention and checked with available evidence. This process may identify ex-ante where conditions may be lacking or the assumed causal principle may not work and it may lead to the planned policy being discarded or redesigned. The same logic can also be used ex-post to understand why a policy succeeded or failed.

In the ex-ante evaluation phase, traditional Impact Assessment tools can be used as in most evaluation approaches. In our approach however we proposed to use Agent Based Modelling simulations. To this end we developed and tested a specific module in the field of Active Labor Market Policy of the Macro-economic Agent Based Model Keynes plus Schumpeter (Dosi et al., 2010) with explicitly decentralized interactions among firms and workers in the labor market (Dosi et al., 2017).

Clearly, for the ex-post phase there are experimental and quasi-experimental methods, which are well known. However, the latter methods are not among the operational components proposed as part of our approach because they are already recognized and methodological guidance on their use is widely available in literature. Finally, an operational tool that can be used across ex-ante, in itinere, and ex-post evaluations is a system of monitoring indicators of inputs, outputs, and outcomes.

Building on this proposed set of instruments, as part of the research, some of the operational components have been developed as computer based prototypes. These include the Evidence Informed Social Policy Innovation Warehouse (EISPI-W), which is a web tool showcasing research evidence on the impact of a variety of social policy innovations on main outcome areas. Whether it is to assess policy options, to understand how an initiative was previously implemented, or to assess which initiatives were or were not successful in a given context, knowing the evidence is essential for better delivery and greater impact.

With the EISPI-W, tasks that previously required hours of desk research can be accomplished in a matter of minutes. Due to its broad appeal, policy makers and practitioners can use the warehouse for a wide variety of purposes, including (1) developing policy and programs, (2) informing strategic decisions and (3) keeping up to date with the latest research.

The prototype has been developed to collect and assembles scientific research investigating the impact of active labor market policies (ALMP) on main outcome areas, such as employment rates and earnings, in order to understand what is known and unknown on ´what works´ by mapping available evidence. It is customizable, allowing users to add or remove data points and can be easily transferred for using in other policy domains.

Another operational component that has been developed as prototype is a generic Quick Exit Tree (QET), which is a simple tool to eliminate policy options by answering with evidence binary Yes/No questions. It provides clear-cut answers and may save efforts if the NO comes up at the very beginning. The instrument, in fact, entails a series of question on the presence or absence of a condition needed for the policy to work. If the first answer is YES, then one proceeds to the next, but if the answer is NO then one can stop and discard the option. The advantage is that it provides an unequivocal answer whether a policy will work here. This, however, requires that all the possible conditions are laid down in the tree and that evidence is gathered to answer all the questions. On the other hand, requiring a dichotomous answer does not allow for more nuanced answers or scores. This requires that evidence is very robust to back the YES or NO answer.

To summarize, the i-FRAME approach can be used in a policy lab context, or in any other form of co-design and co-production approach, following a structured sequence of steps:

  1. Problem and ecosystem functioning: Define the problem the intervention aims to address within the functioning of a given ecosystem > Start with QST >use the EISPI-W (Evidence Informed Social Policy Innovation Warehouse, to draw a problem tree;
  2. Interactive discussion: Engage stakeholders and experts into a discussion on possible causal logic and the suitability for the given intervention of what worked elsewhere > Start with QST > use the EISPI-W and Quick Exit Tree (QET) to develop theory-based thinking open to the views of different constituencies;
  3. Decide and design the intervention: Obviously, the final deliberation and design of an intervention will have to be done following formal and prescribed rule, peculiar to the context of any given country. Yet, in a policy lab context this step could be done as an exercise and should strategically embed the suitable measurement and evaluation methods/tools in the very design of the intervention always following a QST approach;
  4. Identify key variables: Define the input, output, outcomes, and impacts to be measured and evaluated> logic model, intervention logic, CLDs, etc.
  5. Run simulations and/or measure/evaluate (ex ante, in itinere, ex post): Depending on the previous steps and on the nature of the intervention simulations, measurements, or evaluations could be run using the insights from the earlier mentioned warehouse. In this case there are countless possible tools.

Conclusions: Future research directions and policy implications

In this paper we have proposed a methodological framework to assess the social and economic impacts of social policy innovations, which promote social investment. For this purpose, we carried out an iterative series of extensive and exhaustive reviews of sources and set the basis for the development of an open, interactive, and pluralistic platform for evidence-informed social policy innovation.

To this end the proposed i-FRAME considers both formalized quantitative empirical approaches, theory based evaluation, modelling simulations, and qualitative methods. Such epistemological and methodological pluralism, and the set of tools assembled under the umbrella of i-FRAME, are best suited to support collaborative interactions within policy labs, as well as part of more traditional consultation and capacity building initiatives, with the aim to co-design and co-produce with policy makers and stakeholders the instruments needed to inform decisions with evidence rather than simply deriving such decisions mechanically from experiments, composite indicators, and modelling simulations.

In this respect our proposal lays the foundations for both system-oriented formalized modelling simulations and for theory-based approaches, based on the critique of orthodox Evidence-Based Policy.

We have also started setting the basis for developing a blueprint for conceptual modelling and have proposed and tested some of the operational components of the i-FRAME 2.0. Though these are still depicted in a high level / generic fashion, they have been developed and tested as a 'prototype' of a computer-based simulation model which include:

In this connection various domains, levels of governance of service integration implemented in diverse welfare systems and social service delivery models, drawn from different EU Member States should be considered for further testing and validating the approach proposed.

In a future perspective, the i-FRAME has the potential for making a significant contribution especially in setting standards for the use of models for the evaluation of impacts of social policy initiatives. A number of proof-of-concept use cases shall be chosen together with policymakers, analysts and evaluators and data from the knowledge repository could be used for calibrating simulation experiments for diverse scenarios of use. Results of experiments and test validation shall be then discussed together with policymakers to assess their utility. At the same time, and from a policy perspective, the final version of the i-FRAME (Comprehensive Web Platform and Simulator) shall be complemented by concrete technical and policy recommendations for its use and how to scale up the methodological approach.

To achieve these objectives and carry out this challenging, but very much needed and timely work, an ambitious research plan should be envisaged. This should be embedded into a specific high-level science for policy agenda. Experts and representatives of stakeholders should be closely involved on an ongoing basis: researchers from relevant scientific communities, and practitioners and policymakers should be called on directly to address concrete and specific complex policy challenges.

References

Arthur, W. (2013). Complexity Economics: A Different Framework for Economic Thought, Working Paper No. 2013-04-012, link.

Bogliacino, F., Codagnone, C., and Veltri, G. (2015). "The behavioral turn in consumer policy: perspectives and clarifications," Intereconomics, ISSN 0020-5346, 50(2): 108-114, link.

Cartwright, N. and Hardie, J. (2012). Evidence Based Policy: A Practical Guide to Doing It Better, ISBN 9780199841622.

Carvalho, S. and White, H. (2004). "Theory-based evaluation: The case of social funds," American Journal of Evaluation, ISSN 1098-2140, 25(2): 141-160, link.

Dosi, G. (1988). "Sources, procedures, and microeconomic effects of innovation," Journal of Economic Literature, ISSN 0022-0515, 26(3): 1120-1171.

Dosi, G. (2013). Economic Organization, Industrial Dynamics and Development, Selected Essays, ISBN 9781782540151.

Dosi, G., Fagiolo, G., Napoletano, M., and Roventini, A. (2013). "Income distribution, credit and fiscal policies in an agent-based Keynesian model," Journal of Economic Dynamics and Control, ISSN 0165-1889, 37(8): 1598-1625, link.

Dosi, G., Fagiolo, G., Napoletano, M., Roventini, A., et al. (2015). "Fiscal and monetary policies in complex evolving economies," Journal of Economic Dynamics and Control, ISSN 0165-1889, 52: 166-189, link.

Dosi, G., Fagiolo, G., and Roventini, A. (2010). "Schumpeter meeting Keynes: A policy-friendly model of endogenous growth and business cycles," Journal of Economic Dynamics and Control, ISSN 0165-1889, 34(9): 1748-1767, link.

Dosi, G., Pereira, M., Roventini, A., and Virgillito, M. E. (2017). "When more flexibility yields more fragility: The microfoundations of Keynesian aggregate unemployment," Journal of Economic Dynamics and Control, ISSN 0165-1889, 81: 162-186.

European Commission (2013). Towards Social Investment for Growth and Cohesion: Including Implementing the European Social Fund 2014-2020, Brussels, COM(2013) 83 final.

Fagiolo, G., and Roventini, A. (2016). "Macroeconomic policy in DSGE and agent-based models redux: New developments and challenges ahead," Pisa, Italy: Working Paper Series 2016/17, Laboratory of Economics and Management (LEM), Scuola Superiore Sant’Anna, link.

Forss, K., Marra, M., and Schwartz, R. (eds.). (2011). Evaluating the Complex: Attribution, Contribution, and Beyond, ISBN 9781412818469.

Gerrits, L., and Verweij, S. (2015). "Taking stock of complexity in evaluation: A discussion of three recent publications," Evaluation, ISSN 1356-3890, 21(4): 481-491, link.

Heckman, J. J. (2010). "Building bridges between structural and program evaluation approaches to evaluating policy," Journal of Economic Literature, ISSN 0022-0515, 48(2): 356-398.

Kirman, A. (2010). Complex Economics: Individual and Collective Rationality, ISBN 9780415594240.

Kvist, J. (2014). "A framework for social investment strategies: Integrating generational, life course and gender perspectives in the EU social investment strategy," Comparative European Politics, ISSN 1740-388X, 13(1).

McMillin, S. E. (2012). "Proposing a test for policy-based evidence making: A content analysis of the don't ask, don't tell review," Journal of Policy Practice, ISSN 1558-8742, 11(4): 255-272.

Nelson, R., and Winter, S. (1982). An Evolutionary Theory of Economic Change, ISBN 9780674272286.

Patton, M. (2011). Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, ISBN 9781606238721.

Sanderson, I. (2011). "Evidence-based policy or policy-based evidence? Reflections on Scottish experience," Evidence and Policy, ISSN 1744-2648, 7(1): 59-76.

Sterman, J. D. (2006). "Learning from evidence in a complex world," American Journal of Public Health, ISSN 1541-0048, 96(3): 505-514.

Wolf-Branigin, M. (2013). Using Complexity Theory for Research and Program Evaluation, ISBN 9780199829460.