Burgess COMMENTARY

Peter Burgess

MY THOUGHTS SHARED ... 'Impact' is the word

Interesting enough that within 24 hours of initially drafting this post, I had three separate conversations with three different organizations that all told me that “impact” is the new word-of-the-day. Crowd-sourcing, Volunteers, Online Mapping, Innovation, and Resilience have all been touted and discussed. But, now management is starting to ask “So, what is the impact?”. With that in mind, I am super excited to release the post below and hopefully aid the Impact conversation move forward quickly.

One of the clear lessons that I learned after collaborations with the Volunteer & Technical Communities (V&TCs) for Libya and Japan in 2011 (see Lessons Learned) was that being able to measure and articulate the impact that large groups of volunteer have is critical. Not only do we need to commend them for their work, but we need to be able to show that they are having maximum impact (or be able to adjust their efforts to do so). Sounds simple. Yet, even the Humanitarian Innovation Fund (citing Karen Proudlock and Ben Ramalingam) note that “impact is hard to measure”.

image

The first challenge for me is timeliness. The limiting factor of the “real-time evaluations” that most humanitarian groups currently conduct is that they are not designed to impact the ongoing emergency. They are often started a few weeks after the emergency started with a report released only a few months later. They are not timely. But, as we enter the information and networked age, we can evaluate much faster if we are setup to do so. And, with volunteers likely only being heavily engaged for the first few weeks of sudden onset emergencies, we need a framework that could be setup and used in live time.

The second challenge is objectiveness. As Daniel Kahneman points out in Thinking, Fast and Slow, evaluations post facto introduce hindsight bias which “has pernicious effects on the evaluations”. He continues noting that “it leads observers to assess the quality of a decision not by whether the process was sound but by whether its outcome was good or bad. …..When the outcomes are bad, the clients often blame their agents for not seeing the handwriting on the wall—forgetting that it was written in invisible ink that became legible only afterward. Actions that seemed prudent in foresight can look irresponsibly negligent in hindsight”. I have seen this problem first hand. We need a framework that lets us evaluate based on what is happening and what is known “now”.

In that regard, Annie Waldman, Quentin Nicaise and I set out to develop a real-time impact evaluation framework and tool. In 1816, French physician Rene Laennec rolled up sheets of paper into a tube and touched the end to a patient’s chest. To his surprise, the beats were clearer. He was looking for something more objective than “I felt hot and then threw up”. The rolled paper was the first version of the stethoscope which moved doctors away from relying on purely subjective information*. We see our framework version as a very first version of something that can make evaluating impact practical, timely and objective.

image

There were different discussions and attempts over the past couple of years. But, in this version, we pulled ourselves back to the basics and tried to deliver something simple. Even with that approach in mind, the resulting process and tool take a bit of time to understand and customize. We hope that you take the time and thus maximize the impact of your collaboration efforts going forward.

For our purposes, we started with six general categories to gauge collaboration: Effectiveness, Efficiency, Relevance, Impact, Coordination, and Sustainability. (See the framework for our definition of each). Within each of these categories we defined a set of appropriate questions which can be scored on a 1-5 basis. Then, by weighting the importance of each question we can calculate their score and an overall summary per category. Plotting the summaries over time then allows us to monitor the impact of an ongoing collaboration. All of this have been setup in the template tool which you can take a copy, learn and customize to your needs.

The efforts to develop and release the framework have been part of the Impact Evaluation Community-of-Interest. A special mention to Kenny Meesters, Jennifer Chan and Ryan Burns who provided guidance and comments along the way. If you are interested in improving the framework, I would recommend that you get involved with the COI to discuss what has been done (e.g. unsuccessful application to HIF) and what has not been done. Andrej * Story as outlined in Michael Saylor’s The Mobile Wave: How Mobile Intelligence Will Change Everything