Online event co-hosted by the Behavioural Science Community of Practice for Wales and Public Health Wales’ Research and Evaluation Community of Practice

7^th March 2024

Over 100 colleagues joined our online event on the 7^th March 2024 which included the following presentations:

Adaptation of the Welsh National Exercise Referral Scheme to virtual delivery: Evaluation of impact and opportunities  – Dr Katie Newby, Associate Professor, University of Hertfordshire 
Using behavioural science to increase uptake of childhood immunisations  - Dr Lesley Lewis, Behavioural Science Specialist, Somerset County Council  
Social and behavioural research at Cancer Research UK: A case study of the development and evaluation of a Health Marketing Intervention - Dr Charlotte Ide-Walters and Dr Vicky Whitelock, Research and Evaluation Managers, Cancer Research UK   
Feasibility studies of public health interventions: which questions can they answer and how do we design and conduct them?  – Dr Jeremey Segrott, Senior Lecturer, Centre for Trials Research, DECIPHer (Centre for Development, Evaluation, Complexity and Implementation in Public Health Improvement), Cardiff University
 
Launch of ‘Identifying Behaviour Change Techniques’ and ‘Testing and Evaluating Behaviour Change Interventions’ Two Deep Dive Tools  - Dr Alice Cline, Senior Behavioural Science Specialist, Public Health Wales

The event also included the launch of two new, practical tools developed by the Behavioural Science Unit:

Tool 5: Identifying and Applying Behaviour Change Techniques  

Tool 6: Evaluating Behaviour Change Interventions

Below are the written responses to the questions asked of Professor Katherine Brown and Dr Tim Chadborn during the event.

Q&A with Professor Katherine Brown and Dr Tim Chadborn

How do you know the intervention worked through the mechanism you expected it to work?

Understanding whether an intervention works as expected relies on a structured approach. By using the Theory of Change or a logic model informed by behavioural science theory, we can outline the expected mechanisms of action and their effects. This helps prioritize evaluation questions and shape the evaluation approach.

For simpler interventions, an experimental approach might suffice, possibly with a factorial design. However, for more complex interventions, a theory-based impact evaluation would be more appropriate. This tailored approach ensures we capture the nuances of the intervention’s effectiveness accurately.

How do you know which bit of the intervention worked if there are multiple components?

Determining which part of an intervention worked in a complex system is one of our biggest challenges. Breaking down the intervention and isolating specific components (such as Behaviour Change Techniques and the associated Mechanisms of Action) can help us to understand each one in more detail. Being clear on the components at the start of the intervention development and evaluation planning stage can help us to do this more effectively.

You can also consider combining evidence from different type of evaluations to help understand the likely effects of the behaviour change components or their combination. This means looking at the behavioural outcomes, but also the active components of the intervention: the content (the smallest identifiable components that can potentially change behaviour), how was the behaviour change intervention delivered and in what context.

This is typically a challenge in evaluating complex behaviour change intervention, as many behaviour change components occur together, interact with each other to reduce or amplify effectiveness, the effectiveness will also depend on how they are delivered and this will vary across different population and setting.

With so many elements that can be evaluated, how do you focus your aims and evaluation questions down to something that can be done within the time and resources available while managing expectations?

Narrowing down evaluation aims and questions amidst various elements can be challenging. Practically speaking on PHIRST (Public Health Intervention Responsive Studies Team), an iterative approach is key. We begin by constructing a logic model of the intervention, outlining its components and expected outcomes. Through iterative cycles of drafting and refining research questions, involving diverse stakeholders such as local partners and the public, we gradually hone in on feasible and relevant evaluation aims.

This iterative process allows for flexibility and responsiveness to emerging insights and constraints. By engaging stakeholders throughout, we ensure alignment with their priorities and perspectives, thereby managing expectations effectively. Ultimately, this collaborative approach enables us to focus evaluation efforts on aspects that are most feasible and meaningful within the available time and resources.

Key to any evaluation, is a well-defined evaluation question. It is advisable to make them as specific as possible, in other words, make them answerable. As outlined above, this will depend on the priorities (e.g. what is it you need to understand about the intervention, or how do you hope to use the information from the evaluation). You can consider also looking at APEASE criteria to help structure your questions. Theory of Change can also be a particularly useful tool to help identify priority areas for the evaluation, inform the development of meaningful evaluation questions and identify key indicators for monitoring changes in your outcomes.

The other considerations that will influence your research questions will also depend on the stage of your intervention, and what you would like to explore (e.g. efficacy, effectiveness, how the intervention works to produce the desired change, how does it interact with the context).

How do you overcome researcher bias with behaviour change evaluation – seeing things and in populations you want to?

To counteract researcher bias in behaviour change evaluation, it’s crucial to acknowledge potential biases upfront (e.g. independence, or not) and plan accordingly. Strategies include diversifying the research team, using transparent methodologies like pre-registration of your proposed evaluation approach, and communicating openly about limitations and biases in findings.

You should also carefully consider firstly, your recruitment strategy and utilising standardised questions/surveys, and also a thorough understanding of the evidence base and whether your findings support or are different to those of other researchers and why? This proactive approach helps ensure the credibility and validity of the evaluation. Also, this links to evaluation standards and good evaluation principles, described in an answer to a question below.

How do you overcome researcher bias with behaviour change evaluation – seeing things and in populations you want to?

What are the gaps in behaviour change evaluations that us as practitioners might be able to help contribute to?

One gap lies in the evaluation’s focus on short-term outcomes, often neglecting the sustainability of behaviour change over time. By designing evaluations that assess behaviour change trajectories and long-term impacts, practitioners ensure interventions are sustainable beyond the initial implementation phase.

Another area for improvement is the integration of theory into evaluations. Behaviour change interventions are frequently implemented without a clear theoretical foundation, hindering our understanding of underlying mechanisms of change. Practitioners can contribute by incorporating theoretical frameworks into evaluation designs, facilitating a deeper understanding of how and why behaviour change occurs.

Evaluations can often overlook the broader context in which behaviour change takes place, including environmental, social, and cultural factors. Practitioners can help address this gap by adopting a more holistic approach to evaluation, considering the multifaceted influences on behaviour change and designing interventions that address these complexities.

Equity and inclusion are also important considerations. Behaviour change interventions may unintentionally exacerbate disparities or fail to reach marginalized populations. Practitioners can contribute by ensuring evaluations consider equity implications, actively involving diverse stakeholders throughout the evaluation process, and designing interventions that prioritize inclusivity and accessibility.

What are your suggestions, on overcoming some of the common challenges in evaluation of behaviour change intervention (e.g. tendency to consider short term impacts rather than whether the behaviour was sustained over time)?

Longitudinal studies offer insights into behaviour changes over time, ensuring a comprehensive understanding of sustainability.

Employing a mixed-methods approach, blending quantitative and qualitative methods, allows for a nuanced examination of behaviour dynamics.

Grounding evaluations in a robust theoretical framework, such as the Theory of Change, provides clarity and guidance throughout the process.

Engaging participants and stakeholders can help to provide a deeper understanding of their perspectives, enriching the evaluation process.

Utilizing validated measures for both short-term and long-term impacts ensures accurate assessment.

Careful planning of data collection points enables capturing changes over time effectively.

Additionally, acknowledging and incorporating contextual factors like environmental and social influences enriches the evaluation, offering valuable insights into behaviour change processes.

Will you be talking about the behaviours of those delivering the interventions?

The behaviours of those delivering interventions, such as healthcare workers and other professionals, are crucial considerations in behaviour change evaluations. While it’s common to focus primarily on service users or patients, broadening the scope to include the behaviours of intervention deliverers offers valuable insights.

In healthcare, for instance, understanding and addressing healthcare worker behaviours can significantly impact patient outcomes and the overall effectiveness of interventions. By incorporating a broader perspective that encompasses the behaviours of all relevant actors, we can better understand the complexities of behaviour change and develop more comprehensive and effective interventions.

From an evaluation point of view – it is always useful to look at the whole system in which the intervention is being delivered, as these components do not exist in isolation but interact and in combination lead to the results. So, in other words – exploring: ‘Is the intervention being delivered as intended?’ And if so/if not, why?

This can be undertaken, for example, as part of process evaluation, so looking at the implementation of the intervention in practice and fidelity to the intervention. This can commonly include data collection methods such as observations of delivery of the intervention in practice, qualitative interviews or focus groups with those delivering the intervention on the ground in order to gather their viewpoints, identify any challenges, assess what worked well and what worked less well, what can be done differently in the future.

Is there a sliding scale of rigour in evaluation and a level of rigour we should avoid sliding below?

Evaluation standards identify how the quality of an evaluation will be judged, rather than a linear ‘sliding scale’. Many organisations have guidelines addressing issues of quality for evaluation, largely centred around the need for accessible, transparent and reproducible evaluation. These guidelines often include good evaluation principles such as:

The evaluation has followed applicable standards in the design, planning and conduct and governance.
There is consistency in data collection, methodology, reporting and interpretation of findings; applicable methodological guidelines/standards followed.
The evaluation is credible, grounded ion independence, impartiality and with rigorous methodology.
The evaluation is demand-driven, fair, impartial, transparent, timely & used.
Impartiality contributes to the credibility of evaluation and the avoidance of bias in findings, analyses and conclusions.