Top ten tips for conducting an evaluation

Real-world evaluation aims to generate knowledge about how something works in practice – in a real-world setting, with real people. Evaluation findings can be used to inform decision making for services and policy. Within the context of innovations in health and social care, real-world evaluation needs to be flexible and rapid to align with funding and commissioning cycles and to support service transformation. Often, ‘pragmatic evaluations’ are needed, which is an approach that balances the need for a robust and rigorous method of data collection and analysis with the priorities of the people requiring the evidence, such as short time scales and limited resource. In other words the 4 ‘Rs’ model of evaluation: Rigour, Rapidity, Responsiveness, and Relevance (Smith et al., 2023) is key.

The real-world evaluation team at Health Innovation East supports the evaluation of health and social care innovations including new technologies, new pathways and new ways of working. We generate and share evidence and insights from our evaluations that can help lead to adoption and spread of useful and impactful innovations in health and care systems. We also support capacity building across the system for evaluation and evidence-informed innovation. We have conducted numerous evaluations of innovations in different health and care settings across our region and nationally, of which some of the reports feature on our website.

Based on our experiences of conducting real-world, and often rapid and pragmatic, evaluations we have summarised our top ten tips for evaluation:

1. Evaluation needs to be realistic (achievable by those doing it, in the time frame and in the budget available) and relevant to the users of the evaluation – consider what are the most important elements to understand, and what you should prioritise to evaluate.
2. Plan the evaluation before implementation and plan well – the initial planning phase is an important part of any evaluation to ensure you have a well thought-out plan for how you are going to achieve the evaluation aims and objectives. Pinning down the finer details of data collection requirements in the early stage will set you up well for the evaluation itself.
3. Recognise the scale, scope and investment in evaluation will vary depending on the project. Each project is likely to benefit from a different approach depending on the aims and objectives, the populations, settings and the priorities of those seeking the evaluation. Evaluation needs to be planned according to the resources available.
4. Take a collaborative approach – involve key stakeholders early in scoping the evaluation so that differing priorities can be managed and agreed. It is important to be independent in your evaluation, however there is value in collaborating with your key stakeholder group throughout the project to ensure you deliver something that is in line with their expectations and needs.
5. Be agile and take an iterative approach that allows findings to be used in real-time and responds to emerging needs. Conducting evaluations in real-world settings means that you will almost always encounter unexpected challenges along the way. Prepare to be flexible and adapt to the context and project which you are working in
6. Agree frequency and forms of communication so that everyone has a mutual understanding of the project and has the same expectations
7.  Use pre-existing and validated tools and frameworks if possible – this will minimise work in creating new data collection tools, facilitates a structured methodology and increase the rigour of the data.
8. Consider specific aspects of your intervention in the evaluation design, for example the digital nature of the project (IT capacity, patient uptake etc), and how these may influence implementation, outcomes and health inequalities – build measures to evaluate this into your evaluation plan.
9. Where possible use data that will be routinely generated and agree data access early, this will reduce the burden of needing extra data collection specifically for the evaluation.
10. Consider how the findings will be used and communicated at the planning stage and ensure this is done within the time frame of the project so that momentum is not lost. Sharing the findings (knowledge exchange) is critical if the evaluation is to have an impact and inform practice. Think of your audience when developing your final report and be creative.

Fit for the future: 10 Year Health Plan outlines how it will transform the NHS through three shifts (analogue to digital, treatment to prevention and secondary care to community-based care) – innovation is essential to realising this. Effective evaluation that can provide feedback and insights in real time will be critical in ensuring proven innovations are implemented.