Rapid evaluation: our ethos and approach

Evaluation is an important adjunct to the delivery of health and social care. Service and treatment models are always evolving, and insights into the success of these changes (or innovations) supports improvement. Evaluation provides these insights, exploring whether:

• innovations are enacted as envisaged;
• benefits are achieved as anticipated;
• benefits are offset by risks and unintended consequences;
• changes are cost-effective, acceptable and equitable.

To be of maximal value, evaluation findings must provide the right intelligence needed for decision making at the right time. Evaluation timeliness is particularly important for service delivery innovation. Changes to service delivery are common and implementation can be rapid, especially when compared with treatments like drugs – which have stringent and time-consuming evidence hurdles to navigate before licensing. 

Health systems need rapidity in evaluation to maximally inform decision making about the shape and success of the services they deliver. Here, we broadly conceptualised rapid evaluation as delivering empirical formative or summative findings efficiently, and in line with evidence user’s decision needs, which may not be met using more standard evaluation timelines.

Evaluations, however, cannot always be easily divided into categories of ‘rapid’ and ‘not rapid’. While thresholds have been discussed, the reality is that there is no arbitrary evaluation duration that supports this distinction. Rather, evaluations sit on a spectrum of rapidity. An evaluation’s position on this spectrum is decided by how approaches to rapidity are used in some or all of the evaluation stages including its:

• commissioning;
• development and design;
• inclusion of the public voice;
• required approvals, data collect and analytic approaches;
• formats of dissemination and knowledge mobilisation.

Our collaborative rapid evaluation ethos

Our overarching ethos is one of working with stakeholders to identify the right questions; collect the right data to address these questions and then report this information in the right way to ensure knowledge is actionable. At every evaluation stage, we critically explore how rapidity will shape our approach, assessing the impact on the evaluation’s scope, breadth, depth of findings and, crucially, how it fits to the end-user’s needs.  

While the foundation of our evaluations is methodological rigour, our evaluations are structured around a process and operational framework to support timely conduct. Features of this framework are:

• Co-production at the heart of everything we do.
• Considered process and governance.
• Excellence in planning, preparation, and project management.
• Clear, consistent, and considered communication.
• Considered use of formal, structured meetings.
• Engaging with all stakeholders and ‘finding a way forward’.
• Consideration of health inequalities in all evaluation.

Co-production is integral to our approach, and we view this as an on-going process during evaluation. Co-production involves:

• researchers, professionals, and the public working together over the course of a programme or project including the use of participatory research methods;
• the co-creation of new knowledge by researchers working alongside other often professional stakeholders;
• working to design and develop innovations with intended users and recipients.

NIHR INVOLVE: Guidance on co-producing a research project (PDF), 2019 (accessed 10 March 2022).

Our evaluations use a range of health services research methodologies, and most of our evaluations are anticipated to have a ‘mixed methods’ approach. Often no single type of evidence on its own is sufficient to make judgements on whether an innovation is worth pursuing further. Each evaluation we conduct includes several questions that drive data collection and synthesis.