Max Nathan, Deputy Director What Works Centre for Local Economic Growth (WWG), shared some experiences and lessons from the UK. What Works came about to meet the need for a systematic assessment of policy outcomes. He pointed to two main problems. First, the policy cycle does not work as a nice theoretical schema, there is a gap between idea and reality. Second, evaluations are incomplete. There are different types of evaluation to measure effect, process and monitoring. According to Nathan, all three aspects are needed but many evaluations often miss measuring impact, the actual gain from implementing a policy. Therefore, What Works created a minimum standard for impact evaluation.
Our minimum standard for impact evaluation measures the change in outcome for those included in a program versus those not included in the program.
The What Works approach is a process that start with scoping the issue, followed by research, score findings and write up an evaluation report. They try to illustrate the findings with graphical illustration and include findings in their capacity building offer, which also includes for example tool kits, “how to”- guides, workshops and industrial strategy support. The idea is to offer help with improving the use of evidence.
Max Nathan
What are some of the lessons from the What Works project thus far? What we can see, Nathan said, is that overall policy success rate is around 50 % and success varies on key outcomes. We need more clarity on what programs want to achieve, and how. We also see that targeting is usually effective and that there is sometimes confusion about the rationales for a policy, something may be pitched as having economic wins but actually delivers social wins.
He also pointed to the need to consider limitations of evaluations – institutions are different, bodies of evidence are incomplete, and not least, there is a political economy to evaluations.
The interests of policy evaluators and policy makers do not always align. Not everyone wants to know whether a policy works.
The What Works concept is about cultural and institutional change, from traditional ways to a test – learn – adapt approach, Nathan concluded.
This is a discussion we really need in Sweden, Marita Ljung, Former State Secretary at the Ministry of Enterprise, said. We tend to favor “hard facts” and neglect dynamic effects, which are just as important. I would have needed the What Works findings in my discussions with other departments when I worked in government. We need a broader perspective on policy impact in Sweden.
Growth and regional policies should be addressed by more departments within the government. Then we could start talking about the important dynamic effects. ‘
Marita Ljung
Anders Wagner, Public Affairs The West Sweden Chamber of Commerce, questioned the tendency to always look at what does not work as oppose to what does when planning a project. It may be more difficult but sets the right benchmark.
It’s not exactly helpful to look at the disastrous project Hallandsåsen when building new infrastructure.
Max Nathan, Anders Wagner and Marita Ljung
From the audience, Pontus Braunerhjelm, Professor and Research Director Swedish Entrepreneurship Forum, addressed the problem that policy makers often regard policy impact evaluations as threats and therefore focus tend to shift towards critique of methodology and process.
As a grand finale, moderator Per Tryding, Chamber of Commerce and Industry of Southern Sweden, asked Max Nathan for a one-minute Brexit bonus:
In the Brexit project we looked at possible effects of soft and hard Brexit. Our analysis showed that some areas of southern London would be hit hardest. However, other analysis has projected that some of the “leave” regions would be most affected. Time will tell what the exact effects are but it’s clear that there will challenges for the labor market as well as for sectors with cross-border value chains.
Max Nathan and Per Tryding