Burgess COMMENTARY
I am glad I have found this piece from The Atlantic in 2012. I was atracted to the title because of my concern that 'data' is a tool, and a very powerful tool, and like any tool, the value of the tool is in how it is used. In my experience in development (which is not inconsequential) data has often been an end in itself, and done 'by rote' and with little understanding that the data are simply part of a process or system that needs the right decisions and the right actions in order to have any value at alll. The Randomized Controlled Trials (RCT) advocated for by Esther Duflo and promoted by J-PAL at MIT is a wonderful technique but way too expensive in the development context. The idea that the other ways of using data are not as rigorous and therefore of little utility is purely academic ... the key is to upgrade the 'understanding' of issues that constrain development and a better understanding of the system itself. This is not a data issue ... it is more an case of applying advanced common sense and having experts who have this competence!
Peter Burgess

Can Better Data End Global Poverty?

A leading development economist speaks on the virtues and limitations of a data-driven approach to healing the world's most intractable problems

615 malaria net.jpg Reuters

Do free bed nets in some countries lead to more cases of malaria? Could anti-parasite pills raise school attendance in one country and have no effect in another? How cheap does preventative care have to be for low-income families to see the doctor?

There might not be a perfect way to answer these thorny questions on a country-by-country basis. But some leading scientists think the most rigorous answer comes from what they call 'randomized controlled trials.'

Esther Duflo is widely recognized as the world's leading advocate of randomized controlled trials in development economics. As a methodology, RCTs have been used for over a half-century in clinical medicine, where the effect of a drug or medical procedure is confirmed or denied in scientific experiments involving control and treatment groups. The use of RCTs to address global poverty is a phenomenon of the last decade, but it has caught on with the force of a paradigm shift in economics, public policy, and other disciplines.

Last year, Duflo* and her co-conspirator at J-PAL, Abhijit Banerjee, published a book called Poor Economics: A Radical Rehinking of the Way to Fight Global Poverty. The book overviews much of what they have learned through RCTs and otherwise, and it stakes a larger claim against 'grand universal answers' and 'sweeping conclusions' about poverty. Instead, they recommend a data-driven approach that seeks specific, targeted answers to what actually works, what works better, and what works cost-effectively.

Having been trained as a physicist and an engineer, I appreciate and support Duflo's scientific approach to fighting poverty. (Full disclosure: I'm on the board of Innovations for Poverty Action, a close J-PAL partner.) Yet, reading the book, two things repeatedly came to mind: First, the best science requires theory as much as experimentation. Data without good theory is only measurement, not knowledge, and powerful theory is often sweeping. Purely as a practical matter, theory helps to sustain us when we lack data. Second, Duflo and Banerjee seem uncomfortable with their own stance against grand answers. Most of the book's chapters conclude with sections that make general pronouncements about public health, education, microfinance, and entrepreneurship, often not entirely backed up by experiment. Their final chapter, titled 'In Place of a Sweeping Conclusion,' nevertheless offers five broad statements about poor people worldwide.

I asked Duflo some questions about RCTs and her book over e-mail...

KT: What do you think is the greatest contribution of RCTs so far in international development?

ED: First, we have made a lot of progress on debates that we would probably still be endlessly debating: Does giving bed nets away for free discourage usage? Do non-zero prices discourage use of preventive care? Second, we have identified some promising interventions, and those are starting to be scaled up, with deworming being a prime example. [KT: Deworming refers to anti-parasite pills which when given to children have been shown to increase school attendance.] Third, we have accumulated enough knowledge in some domains (primary education, for example) that we are beginning to have a broad systemic view of problems and solutions. Fourth, perhaps most importantly, the importance of trying things out under rigorous observation, of leaving open the possibility of failure, is now more recognized within governments and organizations. I think this shift in culture could turn out to be the biggest contribution in the long run.

KT: What is the greatest challenge in the way that RCTs are actually conducted, interpreted, or talked about in practice? Or, to ask the question a different way, what dominant practice within the community of people who run or read about RCTs would you most want to change?

ED: I don't have much to complain about actually. I think at the moment, people are being very creative, trying to push the boundaries in many directions: incorporating more theory, combining structural models and experiment, combining 'lab experiment' with RCT, trying really bold things such as randomizing the wages for civil servants.

I am also encouraged by how the use of RCTs is entering policy circles, but there I have one problem: I think policy makers are willing to use experiments to justify doing something, but what we are not seeing yet, is a serious use of experiment to prune policies.

KT: When I took economics courses as an undergraduate, I remember thinking that much of econometrics was an attempt to extract causal explanations from data exactly in those cases when RCTs were impractical. The implicit assumption seemed to be that RCTs would have been ideal, but in their absence, complex econometrics was necessary. Is this a reasonable interpretation of economic methodology, and if so, why do you think it's taken so long for development economics to embrace RCTs?

ED: I think, partly, we as a field may have assumed it was more difficult or more expensive than it really was. Michael Kremer's true stroke of genius was to try RCTs on a small budget. Then, when many of us, encouraged by his example, started doing similar work, we progressively learnt what people have long known in other scientific fields: that there are possibilities to raise money for academic studies. So we started raising more money. These two insights combined opened the entire field.

KT: In your book, you argue strongly against overarching theories of international development, which could be interpreted as an implicit acknowledgement of the sheer complexity of development. Yet, RCTs in development economics are often RCTs of human behavior in particular contexts. Are you, then, implying that there can't be overarching theories of human behavior?

ED: There can definitely be theoretical insights, we conclude with some in the book. But there is not one big framework that is going to be able to explain everything, or a big solution for everything.

KT: Although there is rough agreement on the goals of international development, individuals often have different ideas as to the ultimate ends. Some people think it's some level of economic achievement for every country; others think it's universal healthcare and education; still others think it's social justice; etc. Do you have a final goal in mind that you work towards?

ED: I am not sure I have really an opinion on this, or that I should. I think the goals should be set by the political process, by what a society at some point wish is important. I may have my own view on what is important in life, but I am not a philosopher, and I don't feel it is my job to tell people what they should think is important. I think the job of an economist should be to help individuals or societies (via their policy makers) reach these goals, once they have been set.

***

In any case, Poor Economics is a must-read for anyone interested in international development, and it has just been released in paperback. Whether you believe in grand theories or not, it's hard to deny the value of data and careful observation. Banerjee and Duflo fill their book with hard-won insights from personal experience and scientific study. And, despite Duflo's ceding of her moral objectives to philosophy and the public sphere, her rationally tempered passion for helping poor communities around the world is felt on every page.

______

*Duflo generously credits her colleague, Harvard economist Michael Kremer, with kick-starting the recent wave of RCTs in international development, but it's her tireless efforts, with a boost from the eponymous donor of the Abdul Latif Jameel Poverty Action Lab (J-PAL) at MIT that have made RCTs all but mainstream.

ABOUT THE AUTHOR

KENTARO TOYAMA is the W.K. Kellogg Chair Associate Professor at the University of Michigan School of Information. He is the author of Geek Heresy: Rescuing Social Change from the Cult of Technology. Twitter