This summer the Journal of Development Studies accepted a manuscript by Jonathan Morduch and myself laying out our critique of an influential microcredit study from the 1990s by Mark Pitt of Brown University and Shahidur Khandker of the World Bank. Our article should appear in the journal this year or next. The acceptance is milestone for Jonathan and me, for it represents a ratification of our work, and is very long in coming.
It was 15 years ago that Jonathan first laid out his doubts about Pitt and Khandker (P&K). Pitt retorted the next year. And there the dispute rested, never adjudicated by journals, until I entered the picture 6 years ago by writing a program that, for the first time, allowed an exact replication of P&K’s math.
Jonathan and I have played a sort of doubles match with Mark and Shahid. They’ve written four defenses against our criticisms since 1999. We’ve responded every time, sometimes in a paper, sometimes on my blog for the Center for Global Development, where I worked until July.
Every return volley taught us something and led us to improve our work. But in our view, most of P&K’s points are weak in logic, if strong in wording. (That goes in particular for the paper P&K wrote last November, which is why our take hasn’t changed much substantively since 2011.) The one exception—and surely the dramatic high point in this history—was Mark’s identification in 2011 of two key discrepancies between the original study and our first-attempt “replication.” P&K had documented one of the key choices we had failed to copy; the other they had not. Together the discrepancies explained why we had contradicted their headline result, finding microcredit for women to be associated with higher poverty, not lower.
So we revamped our paper. The final version, as accepted by JDS, still has much in common with the first, but also deeper insights. The continuity lies in our emphasis that whatever the correlation between microcredit and poverty in this data set from early-1990s Bangladesh, it is impossible to convincingly determine causation. Does microcredit lower poverty, or does lower poverty lead to more microcredit borrowing? That’s why we so value the new generation of randomized studies, which run experiments in order to pin down cause and effect.
The deeper insights are fairly technical. They can be partially summarized as a discovery that the P&K data violate certain assumptions in the P&K analysis. In particular, there are too many really well-off households in the data. And this seems to make the complex P&K statistical method unstable. That in turn helps explain why in our first-attempt replication, arguably small discrepancies in method led to large discrepancies in results—that flip in the apparent relationship between microcredit for women and household poverty. In our final paper, we match P&K quite well thanks to Mark’s corrections; but we show that dropping the 14 most anomalously affluent households, out of 1,798, suffices to eliminate the instability and causes the microcredit-poverty association to settle near zero. That is called fragility.
I draw a few lessons from this saga:
- One virtue of randomized studies is their mathematical simplicity, which adds transparency. Roughly, you randomly offer some people a treatment and others not, wait a while, then see whether treated or untreated households are doing better on average. Primary school children learn the fundamental concepts used here: averages and differences. In contrast, the complex math of P&K, meant to compensate for the lack of experimentation, created new problems, like instability. The complexity also made it extremely hard for others to rerun and reexamine the math.
- Openness is risky for scientists but good for science. As was the norm in the 1990s, P&K never shared their computer code. They shared the complete data set used in analysis in early 2011, 3.5 years after I first contacted Mark Pitt and 13 years after the publication of P&K. In contrast, we posted all of our data and code at each step. This exposed us to attack and embarrassment—but by the same token allowed Mark to detect our errors and helped us improve our work. That served the greater good. (To be fair, Mark has posted data and code since 2011.)
- Replication is underappreciated in academia. Before being welcomed by JDS, several other journals rejected the manuscript without review. Why? They don’t publish replications. It is unclear to me why an article that analyzes data set X and concludes Y is more worthy of publication than a second one that analyzes data set X and concludes not-Y.
- Microcredit shouldn’t be expected to reduce poverty on average. The discovery of fragility in P&K reconciles the apparent contradiction between its optimistic findings and the muted results from the newer randomized studies.
No doubt the saga will continue. Pitt & Khandker may defend themselves once more, perhaps even in the pages of JDS (precedent). Still, Jonathan and I are happy to have reached a certain closure with the journal acceptance.
After 11 years at the Center for Global Development, David Roodman became a Senior Economic Advisor at the Bill & Melinda Gates Foundation in July. All views expressed are his alone.