In the world of policymaking, understanding the impact of initiatives is crucial for two main reasons: learning and accountability.

But it’s also critical that evaluations are high-quality, drawing on the latest thinking and methods to provide robust insights.

UK Research and Innovation (UKRI) commissioned Frontier Economics to conduct a rapid evidence assessment to examine different counterfactual methodologies used to evaluate Research and Innovation (R&I) programmes. They wanted to know what approaches have been used for different types of R&I intervention, and whether there are emerging novel methods that have or could be applied to future R&I evaluations to help them stay on the cutting-edge.  

Evaluations play a key role in policy design, helping to understand what changes have occurred as a result of a policy intervention, mechanisms of impact, and the context for what works. These insights can inform subsequent decision-making, and help evidence the value for money of public spending.

Evaluating the impact of a policy intervention robustly is difficult, as we need to consider impacts relative to the counterfactual of what would have happened in its absence. The counterfactual cannot be observed, and so must be estimated.

Our research covered a broad range of counterfactual methodologies across quantitative, mixed methods and qualitative approaches. 30 studies were reviewed, covering the full spectrum of R&I policies including knowledge generation, capacity-building, collaboration, knowledge exchange and commercialisation.

Key questions we Explored:

• Counterfactual Methods: What methods do funders use to establish the counterfactual in R&I evaluation, and why?
• Strengths & Challenges: What are the strengths and challenges associated with different methods? How do these impact reliability, time, and costs?
• Innovative Techniques: Have any groundbreaking techniques emerged in recent years for establishing counterfactuals in R&I evaluation?
• Lessons for UKRI: What insights can we gather from our analysis to help UKRI conduct more robust impact evaluations?

Where’s the innovation in evaluating innovation?

Our report spotlights a wide range of methods relevant for R&I evaluation. In particular we identify the potential of methods such as Machine Learning, Qualitative Comparative Analysis and Synthetic Controls as high potential methods for R&I policy evaluation, given the complex nature of interventions in this area and the often limited sample sizes available to evaluators.

Click here to read RAPID EVIDENCE REVIEW OF COUNTERFACTUAL METHODOLOGIES TO EVALUATE RESEARCH & INNOVATION PROGRAMMES.