The concept of Value for Money (VfM) in appraisal and evaluation is straightforward. VfM is concerned with the good use of public funds and with demonstrating the relationship between the costs and benefits of an intervention – whether a policy, a project or a programme. It is an important tool for the accountability of public spending.

The practice of demonstrating VfM is more difficult. Particular policies or programmes often seek to deliver a range of different benefits, some of which may take many years to materialise, in complex environments influenced by a host of external factors. In recent years theory-based evaluation has grown in popularity as an effective way of evaluating policy in such circumstances. But there is less guidance on how to think about VfM in this context.

This bulletin explores the issue of VfM in theory-based evaluation. We examine some of the key principles of VfM and why it may be challenging to demonstrate VfM in complex policy settings. We also look at the lessons learned from our work on how to address those challenges.

Value for money is conceptually straightforward…

VfM is concerned with the optimal use of public funding and is a critical tool for the accountability of public spending. A significant body of guidance has been published on VfM, notably the Treasury Green Book as well as departmental-specific guidance. The National Audit Office’s ‘four Es’ framework – economy, efficiency, effectiveness and equity – is also frequently used to think about different aspects of VfM.

VfM is not about minimising cost. The focus is the relationship between costs and benefits – ‘bang for the buck’ – to get the most out of public spending. The benefits assessment is holistic, taking into account social as well as economic benefits. As the Green Book puts it:

“Value for Money … is a judgment about the optimal use of public resources to achieve stated objectives … based on consideration of … present value to society of all social, economic and environmental benefits – these may be qualitative or quantitative – [and] … present public resource costs.” (p. 52)

In other words, a VfM assessment considers the relationship between the social benefits of a policy intervention and its costs, in current value terms, while also weighing up the risks to these costs and benefits.

…with considerable guidance on VfM in appraising new interventions

The most common use of VfM is in the appraisal of new projects, policies or programmes. Once a range of possible interventions to achieve intended goals has been developed, costs and benefits are considered as part of the decision-making process to help rank options in terms of VfM. The Green Book provides detailed guidance on how to use VfM in this way. It stresses the importance of good evidence to inform assumptions made about the likely scale, profile and certainty of benefits over time.

Importantly, VfM is never the only basis on which options are compared and decisions made, though it is recognised as an important factor in the process. Recent updates to the Green Book have particularly emphasised the importance of Strategic Fit – how policy options contribute to a wider range of government objectives such as net zero or levelling up:

“…from the start proposals must be designed to ensure they provide a supportive strategic fit with wider public policies … [that is] how well the option provides holistic fit and synergy with other strategies, programmes and projects.” (p. 23 and 32, emphasis added)

It is important to note that the strategic case comes first and articulates an overall rationale for intervention. A shortlist of options that fits this case is then drawn up and undergoes appraisal, including a social cost benefit assessment of potential VfM.

In principle, VfM focuses on monetised benefits, which allows for easy comparison with the costs. In practice, valuing benefits in monetary terms is not always straightforward, though over time research has been commissioned and advice published on best practice in valuing a range of social benefits, including environmental health and well-being benefits. Sometimes benefits cannot easily be quantified or monetised, or it would not be proportionate to attempt to do so. In such cases guidance on appraisal recommends that VfM considers:

• Quantified benefits which can be measured but are hard to value; and
• Qualitative summaries of benefits which are hard to measure and quantify at all.

There is renewed emphasis on demonstrating VfM as part of policy evaluation, albeit with more limited guidance

The public finances are likely to be under pressure in the coming years, as departments grapple with the impact of higher inflation, slower growth and post-Covid levels of debt. Against this background, it is perhaps unsurprising that we see renewed emphasis on the importance of demonstrating VfM as part of an ex post policy evaluation.

The Magenta Book stresses that VfM (sometimes called ‘economic evaluation’) sits alongside process and impact assessment as part of a comprehensive, best practice evaluation. The aim of the exercise is to know not just whether a policy ‘worked’ and was delivered well, but also whether it represented a good use of public funds:[7]

“While impact demonstrates and quantifies outcomes, it cannot on its own assess whether those outcomes are justified. Value-for-money evaluation considers such issues, including whether the benefits of the policy are outweighed by the costs, and whether the intervention remains the most effective use of resources.” (p.15)

The basic principle in applying VfM as part of an evaluation is therefore to demonstrate that an intervention was a good use of public resources – that benefits outweigh costs. This contrasts with VfM in appraisals, where it is an explicit tool to help choose between policy or project options. In an evaluation, where the option has already been chosen, VfM is about showing that the benefits realised are worth the costs incurred.

Another distinction is that a VfM assessment in an evaluation should be informed by the evidence collected about additional benefits generated by the intervention as part of the impact evaluation. By contrast, in conducting an appraisal policymakers may not have much direct evidence about potential benefits. In that case they will need to draw on theory, modelling, previous literature and reasoned assumptions to estimate VfM.

The relationship between impact evaluation and VfM evaluation can be summarised by the diagram below.

Figure 1: Link between impact evaluation and Value for Money evaluation

Source: Frontier Economics

Put simply:

• Impact evaluation is concerned with attribution – what changes do we observe as the result of an intervention, how large and enduring are the changes, and how far has the intervention itself caused them? The main issues here are around measurement of benefits and additionality relative to what would have happened otherwise.
• Value for Money evaluation builds on the impact evaluation and is concerned with valuation – what is the societal value of the outcomes and impacts we attribute to the intervention and how do they compare with costs? Here the main issue is whether benefits can credibly be valued at all and represent genuine net gains to society. A particular challenge is to identify and value separate impacts and the interdependence between them in order to avoid double counting – that is, attributing value to the same benefits more than once when thinking about the overall value of a policy intervention.

Strikingly, there is much less formal guidance on how to conduct an ex post VfM evaluation than on how to consider VfM ex ante as part of a policy appraisal. The main Magenta Book and associated methodological pointers refer back to Green Book appraisal guidance to inform the approach.

There are, of course, overlaps between forward- and backward-looking VfM assessments. The principles for valuing social benefits can be applied equally to evidence of the scale of those gains gathered from an evaluation and to ex ante forecasts of potential benefits. Indeed, so long as the underlying impact evaluation evidence is robust, we can be much more confident in the VfM assessments made. But detailed guidance on how to use impact evaluation evidence to inform a VfM evaluation is lacking.

There may be particular challenges in evaluating VfM for complex, long-running programmes or policies…

It’s often tempting to think of a policy as a discrete intervention to achieve a particular objective, with a start and an end both clearly defined. In practice, this is rarely the case. Governments seek to generate a range of different benefits from a single intervention. Policies are executed in complex environments and their outcome is influenced by the existence of a large set of stakeholders, interest groups and outside factors. Benefits can be long-lasting, but they can take time to materialise. That means long-term commitments to public investment may be required if policies are to succeed.

Policies seeking to promote productivity and innovation are a prime example. Innovation is often a slow, risky process of discovery, testing, scaling and commercialisation. Policy support may be needed at each stage of the process to deal with a range of market and systems failures, and may have to be sustained over many years before the ultimate benefits are reaped. Success will be affected by a huge range of factors beyond the direct control of science and innovation ministers: migration rules, the regulatory and planning environments, patent and IP frameworks, the tax system, the policies of other countries and institutional and investor risk aversion, to name just a few. Policy support may also have wider ambitions: to nurture local supply chains, improve collaboration, boost investment, increase employee skills, produce more effective and influential research, generate IP, and so on.

Policymakers will need to make an evidence-based case for renewing investment. Recent years have seen increased use of theory-based approaches to impact evaluation as a way to provide such evidence. It is often extremely difficult, or at least not proportionate, to assess such interventions using randomised control trials or other counterfactual-based approaches. Instead, an overall theory of change for the policy is developed, and hypotheses or questions suggested by the theory are tested using evidence gathered as part of the evaluation process.

Despite the growing popularity of theory-based approaches, there are at least three key challenges in carrying out both impact and VfM evaluations in situations where theory-based assessments are most often used:

1. Benefits realised with long lags may not be observed in the course of an evaluation carried out while the policy is still being executed, or shortly after. There is often greater reliance on (uncertain) predicted future benefits or proxy leading indicators to feed into a VfM assessment.
2. The range of intended benefits, many of which are intangible and hard to measure, let alone value, means a VfM assessment might be possible only for a part (even a small part) of the overall policy.
3. Theory-based evaluation does not lend itself to a robust quantification of additionality and a precise estimation of the scale of benefits generated. This makes it challenging to size the benefits to include a VfM evaluation.

These difficulties notwithstanding, Treasury stakeholders will understandably expect to see evidence of VfM from past outlays when considering further investment in policies which are implemented over many years. It will be hard to rely on general estimates from the wider literature to make the economic case for renewed funding. It is therefore important that questions of VfM are not put aside in delivering theory-based evaluation.

What have we learned about these challenges?

Frontier has been heavily engaged in carrying out theory-based evaluation in recent years. We have been involved in high-profile policy areas including Industrial Strategy Challenge Funds and subsidies for purchasing zero- and low-emission vehicles. We have also worked closely with Innovate UK and the Catapult network recently to consider how to incorporate VfM in a theory-based evaluation.

Some key insights from our work are:

• Build VfM into the design of a theory-based evaluation. Thinking about VfM when developing an assessment framework will help departments consider what data or evidence might be needed not only to measure certain outcomes or impacts, but also to monetise them. For example, having figures on the number of jobs created or safeguarded by an intervention may be a useful indicator of economic impact, but evidence of the skills or wages that go with those jobs is needed to assign an economic value to them for a VfM assessment. Importantly, it also means that the collection of cost data should be built into an evaluation, in particular public service delivery costs. These are often relatively easy to identify.
• Consider impact evaluation and VfM evaluation together. These are not separate initiatives: the evidence from an impact evaluation will directly inform the VfM exercise. In theory-based assessments, it is common to develop a set of thematic evaluation questions which will be tested against the evidence gathered. Having a question focused on VfM (“Has the policy represented good value for money for taxpayers?”) will help to focus attention on the issue, and setting out how the VfM assessment will draw on the wider evidence collected in practice is critical.
• Know the audience for the VfM findings. If the evaluation results will also be underpinning the case for future funding, the scope of a VfM assessment will be broader than simply demonstrating that benefits outweigh costs. It will also include evidence for the strategic fit of the policy. In addition, greater emphasis on the uncertainties and scenarios around a central VfM assessment might be needed to provide a more robust evidence base for a future appraisal. This conclusion relates to previous Frontier research for the Department for Transport that emphasises the value of strong links between appraisal and evaluation in building an evidence base for making effective policy decisions.
• Avoid double counting. The biggest risk with VfM evaluation in theory-based analysis comes from counting benefits more than once. Policies may have multiple objectives and a range of different mechanisms to achieve them. In our experience, it is helpful to use the underlying theory of change to identify distinct benefit pathways which can be measured and potentially monetised. By this means, and by focusing typically on the final intended impacts of a policy, the risks of double counting are minimised. Erring on the side of caution is likely to be sensible. It is better to make the case that benefits exceed costs by monetising a few distinct pathways than trying to put a value on a lot of them. Doing so will much reduce the risk of double counting.
• Be realistic and transparent about the limits of what can be monetised. A VfM evaluation only needs to demonstrate that the benefits of an intervention exceed the costs. In a theory-based evaluation, not all benefits can be quantified or monetised. If it is possible to show a net gain from those benefits which can be monetised, then a narrative account of wider benefits, drawn from the impact evaluation, will add to the persuasiveness of the case. Putting speculative or spurious values on a broader range of benefits, by contrast, may detract from the perceived robustness of the VfM assessment.
• Don’t neglect social benefits. Best practice guidance is increasingly focusing on how to include a wider range of social benefits in a cost-benefit assessment. Environmental benefits (e.g. the value of carbon saving or air quality improvements), health benefits (e.g. QALYs) and well-being benefits (e.g. WELLBYs) are now routinely being included in appraisals. They can be applied to evaluation evidence as well. The most recent Green Book also contains detailed guidance on demonstrating local economic impact, including the use of regional multiplier analysis, consistent with the government’s levelling-up agenda. Evaluators should consider the range of relevant social impacts where the valuation of benefits might be possible.
• Make use of ‘what if’ questions or scenarios where needed. There are likely to be considerable limits on how far the evidence from a theory-based evaluation can be used to attach credible values to benefits generated. That is either because the degree of additionality is hard to quantify, or because the benefits are hard to measure or are distant. Persuasive evidence of VfM may therefore need to rely on questions such as ‘how much value would we have to place on non-monetised benefits in order for costs to be at least met?’ or ‘how large is the degree of additionality we need to assume in order that benefits exceed costs?’ In this regard, the broader evaluation evidence used to demonstrate that these thresholds are plausibly met will be critical.

In conclusion…

Ensuring that a policy or project provides value for money is common sense, particularly at a time of stretched budgets. Yet proving VfM in theory-based evaluations is anything but simple. Benefits can be hard to measure and value; the impact of interventions can be difficult to disentangle in complex, long-running programmes; and the risk of double counting is ever present. But Frontier’s client work shows that these are problems that can be overcome, allowing VfM in theory-based assessments to play an important role in the stewardship of the public finances.

This piece was also co-authored with academic expert Tom Worsley, Visiting Fellow at the Institute for Transport Studies at the University of Leeds.

We hosted an in-person event on 'New thinking in appraisal and evaluation' on 4th October. You can view the presentation by clicking here.