Applying Experimental and Quasi-experimental Methods in Cocoa Sustainability Program Evaluations

Quantifying the impact of your sustainability program is tricky. Download our case studies testing a range of approaches.

The author(s)

  • Meghann Jones Senior Vice President, US, Public Affairs
  • Robert Petrin, MAS, PhD Director of Data Analytics & Applied Sciences, US, Public Affairs
Get in touch

Working out whether or not your sustainability program is having the impact you intend is more complex than you might think. There’s heated debate among academics and practitioners around how to do it best, much of it pointing out that designing a perfect measurement methodology in theory is a lot more straightforward than applying it in practice. That said, most can agree that it’s a necessary exercise. With this blog, we add the Cocoa Life perspective to the debate.

Impact is important for many reasons. One recalls the quote often attributed to management guru Peter Drucker: “if you can’t measure it, you can’t manage it.” This is just as applicable in development interventions as it is with corporate performance. Only by measuring and evaluating what’s working and what is not can those responsible for a program gain the information needed to fine-tune the program to make greater effect on target communities.

In a similar vein, impact reporting can be essential for justifying the money spent to funders who are eager to know they are putting their money behind something that works. Achieving and reporting on impact is an important factor in ensuring the sustainability of funding for most programs. For Cocoa Life, Mondelēz International’s cocoa sustainability program, we measure and evaluate the impact of our interventions so we can understand what is working and how to improve for our own purposes. But just as important as the internal reasons is sharing our insights with others, to help them learn from our experiences and overcome the same challenges we faced. The central challenge in evaluating the impact of any program is working out what would have happened in the absence of the intervention. This is called the counterfactual. By its very nature, you can never know what it actually is. Thankfully, economists and statisticians have developed some clever ways of estimating the counterfactual. Once you have this estimate, you compare it with the data for people who took part in your program. The difference between the two can be attributed to the effects of your program.

Cocoa Life works with Ipsos to run impact evaluation in cocoa origin countries. We have trialled a range of approaches, from the experimental randomized control trial (RCT) – considered by many as the gold standard for impact evaluation – to quasi-experimental methods, such as propensity score matching, for occasions when it’s too resource-intensive to run a successful experimental evaluation.

As anyone experienced in impact evaluations will acknowledge, a lot can go wrong during the process. To help others learn from our experiences, we have prepared a short report containing two case studies on M&E projects in cocoa communities: an RCT in Ghana and propensity score-matching in Lampung, Indonesia. Download it today, and send us your feedback!

For more information about the Cocoa Life impact assessment methodology, please download our methodology factsheet.

The author(s)

  • Meghann Jones Senior Vice President, US, Public Affairs
  • Robert Petrin, MAS, PhD Director of Data Analytics & Applied Sciences, US, Public Affairs

Society