On Monday 29th April 2019, the Centre for the Evaluation of Complexity Across the Nexus (CECAN) hosted a day workshop on Commissioning Complex Evaluations at Prince Phillip House, London. The event brought commissioners from across government and policy evaluation contractors to share their experiences on complexity-appropriate approaches and thoughts on how to overcome their key challenges.
Kelly Beaver, Managing Director of Public Affairs at Ipsos MORI, provided tips on successful commissioning and delivery of complex evaluations as a part of a final panel discussion. Consultants in attendance from Ipsos MORI’s Policy and Evaluation Unit here share key themes from the day and thoughts on approaching complex evaluations going forward.
What is a complex evaluation?
Complex evaluations, or ‘complexity-appropriate’ evaluation, embody two key characteristics. First, they consist of multiple components and interacting variables (and actors). Such examples include multiple schemes within a single programme with varying priorities and objectives; multiple stakeholders and beneficiaries; and a dynamic and unpredictable policy landscape. Second, they are usually developmental in nature, either testing unproven concepts at scale or sitting within uncertain policy environments. Our experience in complex evaluation work is increasingly expanding particularly within the Energy & Environment, Health & Social Care, and Innovation policy areas.
A common preconception when approaching complex evaluations is the need for them to include “non-traditional”, advanced, or data-intensive methods, however in reality this is not always the case. A complex evaluation bears similar traits to any standard evaluation, but the challenges that can impact on the expenditure and the quality of outputs - for example within the commissioning stage, the tendering process, project management and effective delivery - are significantly heightened.
What are the key challenges?
The event provided an opportunity to hear first-hand the challenges experienced by both commissioners and contractors of complex evaluations, many of which were clearly interrelated and mutualistic.
For commissioners …
Commissioners highlighted challenges from the outset of complex evaluation: procurement. Uncertainty over appropriate methods, their quality and credibility, usability of findings, time constraints and available budgets all contribute to a challenging procurement process.
Subsequently, commissioners voiced their difficulty in striking the right balance within evaluation specifications between being too prescriptive, and therefore limiting the scope for novel or flexible approaches to be proposed, and being too vague, which may elicit suboptimal responses.
Commissioners also raised concerns of internal resource constraints having a negative impact on their ability to oversee and ensure appropriate project management. This seemed to be exacerbated by an apparent scepticism around adequate project management of complex evaluations by external evaluators.
Contractors expressed frustrations with contracts and frameworks not designed to be adaptive and flexible to allow for changes around a programme’s activity, such as emerging findings or even changes to policy. Restrictive contracts or less scope for change where variables arise which were unknown at the beginning of the contracting process can result in unfeasible or unsuitable methodologies and stifle evaluation findings.
In line with this, it is becoming increasingly important to structure project teams with the right mix of skills, expertise, and curiosity to easily adapt to such changes. The inability to quickly draw expertise from across the evaluation team and their institutions to respond to emergent findings, for example, can significantly risk meaningful and transferrable conclusions from being drawn.
Contractors also referred to the challenge of truly understanding the subject matter from the outset, particularly where little information is provided in evaluation briefs. This presents further difficulties, for example when designing an appropriate methodology and pricing structure.
Contractors are also faced with finding the right balance between complex-appropriate methods and easily digestible findings. Complex evaluation outputs are inherently narrative focused or over-complicated with solutions not fit-for-purpose. This limits the extent to which learnings can be actioned or transferred.
What can we do?
Overall, commissioners and contractors viewed the future of complex evaluations with optimism. There was a shared vision that future policy decisions, specifically within an increasingly connected and dynamic world, will necessitate flexible solutions, and that collaboration and transparency between commissioners and contractors were key to successful delivery and meaningful conclusions.
Key takeaways for both parties to keep in mind going forward included the following:
- Commissioners should be strategic and flexible in their procurement and contracting. Having open conversations with contractors pre-tender is crucial to ensure an accurate understanding of requirements, and what can be feasibly delivered within the budget and timescales. Commissioners could use scoping studies or reviews to test the feasibility of evaluation approaches, research methods and the availability of supporting data. Other possible elements of flexible contracting could include include staggered tenders; flexible consortia contracts; use of expert opinion panels to quality assure scoping outputs.
- Contracting should be both holistic and adaptive. Contracts should allow space for testing of novel methodologies in tandem with robust “traditional” methods as much as viably possible. Where possible, commissioners should welcome proposals with multiple methodological suggestions or allow for changes in methods to apply post award of contract.
- There is great value in softer skills. Project teams should be equipped with the right set of skills and level of curiosity to adapt well to change. A team that is reflective and responsive to change, as well as strong communicators and able to manage complex stakeholder groups is as important as technical evaluation capability.
- A mutual understanding of integrity, communication and open management is vital. Commissioners and contractors should recognise their shared responsibility in the management of complex evaluations. Continuous communication and transparency from both sides is required for an effective management process, from providing policy updates, to changes in priorities, to the time and resource committed to the evaluation from both parties.
- Shared learning is an iterative and predictive process. Encouraging contractors to present and document interim evaluation findings and using these to predict longer-term outcomes provides a useful way to ensure final conclusions are meaningful.
Getting the balance right …
The commissioner-contractor relationship and operating as a unified team is increasingly important for complex evaluations. A holistic approach, teams possessing the right mix of technical expertise and soft skills, and open and transparent management are the core requirements needed to respond to inevitable scope movement. A better mutual understanding of these requirements will ease the commissioning process and support an evaluation which is bespoke, cost-effective, efficient and ultimately delivers the most impact.
Ipsos MORI’s dedicated Policy and Evaluation team leads a range of evaluation projects for UK, European and International clients across the public and not-for-profit sectors. Our multi-disciplinary team of evaluation professionals, economists, and policy experts harness Ipsos MORI’s strength in conducting robust social research across the full spectrum of policy areas to deliver high quality and impactful end-to-end evaluation solutions for our clients.