Recent growth in the capabilities of Artificial Intelligence (AI) technologies has led to increased interest in the use of AI in Government. 

Frontier’s guidance, commissioned by the Government’s Evaluation Task Force and the Department for Transport ,  outlines key principles of best practice in carrying out a robust impact evaluation of AI interventions in Government: programmes and initiatives using AI systems in central Government or the delivery of public services. This guidance complements HM Treasury’s guidance on evaluation in Central Government (the Magenta Book).

Evaluation of AI use in government (including process, impact and value for money questions) is essential to understand the impact of AI systems compared to the status quo, improve current interventions, inform future policy development and ensure the Government is accountable to the public.

The key principles of robust impact evaluation are no different for AI interventions than for any other type of government programme. However, AI interventions can present additional opportunities and challenges for evaluation.  

Our guidance describes some of the key challenges and opportunities for evaluating AI interventions, including using experimental methods, providing advice on how to best address them. This includes:

• Strongly consider the use of experimental approaches, leveraging opportunities to control assignment of the intervention;
• Take into account the iterative process of developing and deploying AI interventions, by using rapid evaluaton methods at early stages, evaluating regularly, and building in flexibility in evaluation plans;
• Use theory-based approaches where the intervention is part of a complex system, the impacts of the intervention are hard to predict, and/or where it is important to understand how and for whom the intervention works;
• Give explicit consideration to variation in the impact of the intervention for different groups;
• Consider the role of public attitudes and perceptions in influencing the impact of the intervention;
• Consider how best to gather baseline evidence on complex or subjective processes that are being replaced or enhanced by AI.

For those involved in designing or delivering an AI intervention, it is crucial to consult with evaluation experts in the relevant departments to ensure that the impact of the intervention is evaluated robustly.

Please click here to read the full guidance.