What’s wrong with this “evidence of impact” for high-profile charities?
- The Millennium Villages project’s main evidence of impact so far appears to be increasing crop yields by a factor of 4-15x in 3 villages. (H/T to the recent coverage of Millennium Villages on Aid Watch)
- A KickStart evaluation (available on our website (DOC)) reports that “Family/household incomes of 3,000 farmers increased by 100% within 24 months following acquisition of micro-irrigation pumps.”
- The Carter Center’s agriculture program “provides the tools and means to help farmers in sub-Saharan Africa increase their crop yields through agricultural development, sometimes two- or even threefold.”
Among other possible problems, two major issues jump out:
1. No context on what “normal” variation in incomes looks like for poor farmers. Some years have more favorable weather - and local economic situations - than others. Enough that one year’s income or crop yield could be double another’s? 4x? 20x?
Unfortunately, one of the better pieces of “evidence” that jumps to mind is a 75-year-old novel, The Good Earth, whose farmer protagonist is comfortable one year and has literally zero income the next, for no other reason than the weather. If a given year’s yield were close enough to zero, the next year could be a huge increase (2x, 4x, 20x or more) simply by returning to normal.
I have seen little information on the local year-to-year volatility that poor farmers can experience, but I imagine that it (a) varies greatly from region to region and (b) could easily involve incomes falling and jumping by enormous amounts.
None of the above reports provide any context on this question, beyond qualitative statements about how favorable the rains were in each year examined. None of them employ any sort of “comparison group” of farmers (aside from one vague reference to “farms not using improved seeds and fertilizers” in the Malawi Millennium Village). Ultimately, none accomplish one of the most basic goals of an evaluation: giving a sense of how likely the “gains” they describe are to have arisen by pure chance.
With larger sample sizes, we might be able to use country-level volatility for context. But that brings me to the next problem.
2. We have no assurance that the described gains are representative, as opposed to “cherry-picked.”
All of the above organizations have reputations for consistent and thorough monitoring and evaluation, yet in all cases, we find ourselves looking for “impact” from a tiny subset of their projects.
- The Millennium Villages evaluation “includes results from the Sauri baseline survey, annual reports from Sauri and Koraro and crop-harvest data from Malawi.” How were these three chosen from among the 78 villages mentioned in the abstract?* The report doesn’t say.
- KickStart implies that it is constantly and consistently tracking impact, yet the report linked above discusses a single project over a single year.
- The Carter Center is somewhat explicit that its number is a “high-end” one (”sometimes two- or even threefold”). Examining the most recent annual report for its agriculture program shows a very small upward trend, if any, in crop yields across the countries in which it is working.
Some ways to produce more compelling evidence of impact
- Be clear about what is being measured and what is being published, and when. It seems to us that in this area, charity evaluation lags far behind clinical trials, which are constantly registered before they are complete so people can track their progress. (The Poverty Action Lab is similarly transparent with its own ongoing projects.)
- More sample size; more context; use of comparison groups. Discussed above.
- Look for more sustained improvements in people’s lives. One measure I find superior to straight “income” or “crop yields” is asset accumulation. A jump in income could be temporary; if someone upgrades their roof or sanitation, it’s likely that at least they expect the gain to be a real and lasting one. The Village Enterprise Fund’s evaluation is one of the better charity evaluations I’ve seen in the area of economic empowerment, partly because it focuses on standard of living rather than a simple measure of income.
*It’s possible that the yields mentioned are for “clusters” of villages rather than individual villages; there are only 12 clusters. However, the source documents available for Sauri and Koraro appear to be at the village rather than the cluster level, and the details of how the measurements were made are unclear.
Regarding the Millenium Villages, it should be noted that the project is adding roughly $100 (*) per person per year to villages’ local economy. Note that GDP per capita in these villages would be somewhere around $150 (*) per year, again raising the specter of how these changes would compare to just handing out cash.
(*) Note that figures named in this comment are measured at market exchange rates, not PPP.
Also, regarding sustainability, there is really only one measure that you can use to find out if a project is making a sustainable difference: Withdraw the funds and see if the subject thrive or wither. Even checking to see if roofs are built doesn’t tell you whether or not they will be maintained.
These facts are unfortunate, partly because withdrawing funds is hard, and partly because it takes years before you get usable data.
Ian, it depends somewhat on what is meant by “sustainability.” I agree that your test is the only one for long-term impact of a project on a community. However, I believe we can fairly infer a major change in an individual’s life from less. Asset accumulation could be misleading in many ways, but I believe the measure has fewer pitfalls than the measures discussed in this post.
Holden,
Suppose that you take someone who is earning $150 per year and give them an extra $100 per year in cash for five years. Wouldn’t you expect them to build up assets every year during those five years? But in the sixth year, when you stop giving, this person will survive on $150 plus whatever assets they saved in the prior five years. By the time you get to year 15, is there any reason to think that the person’s life will be much better off than it was in year zero? Perhaps, but it’s far from obvious to me.
My problem with the millennium villages is that it’s impossible to tell what part of their success comes from injecting $100 per person into the local economy, and what part of their success comes from improving crop yields, reducing malaria, etc.
Cheers,
–Ian
Ian, I think it’s hard to tell whether Millennium Villages has had “success” in any area, and harder to break it down as you describe.
I have much more confidence that lives were improved if I see asset accumulation than if I see a one-time bump in incomes/crop yields, though less confidence than if I see a major health improvement (particularly a life-saving one) or - as you say - a long-term, sustained improvement in living standards.
Holden,
On economic empowerment long term success metrics, in addition to typical income gain, and asset accumulation that you prefer, what other indicators have you found be used in practice?
Sam, speaking off the cuff, I’d say the types of evidence we’ve seen collected fall into the following broad categories:
The last category is the one we find most compelling.