The GiveWell Blog

We have completed the process of allocating $250,000 in grants for economic empowerment organizations in sub-Saharan Africa.

49 organizations applied for our grant, out of 157 invited. From these, we felt that two stood out in making a strong, evidence-based case that they are improving the financial situations of low-income people in the developing world. We have awarded $125,000 to each.

The Village Enterprise Fund is a microenterprise organization providing cash grants as well as business training and mentoring services to extremely poor business owners in rural Kenya, Tanzania and Uganda. As we have written before, we feel that cash transfers are a relatively direct and promising way of helping people, a reasonable default approach in an area where little evidence is available on what works. Village Enterprise Fund stands out for its clear targeting of extremely poor clients (and data collection to ensure that the targeting is working), addressing a key concern about cash grants.

The Small Enterprise Foundation is a microfinance institution in South Africa. We have written before that we see great promise in microfinance charity, but also have serious concerns about it; these concerns led us to investigate individual microfinance institutions overseas, rather than sticking with large U.S. charities. The Small Enterprise Foundation stands out for its ability to answer our key questions about microlending. (Note: SEF is not a U.S.-registered charity.)

Within the next few weeks, we’ll be publishing everything we’re allowed to publish about how this decision was made: which organizations applied, what materials they submitted, what criteria we used, and how we made the final call.

For now, see the audio and documents from the board meeting where we made the grant decision.

In a previous post, we asked whether Haiti earthquake relief has “room for more funding” and concluded that, in general, it isn’t clear. (For more on the general topic of “room for more funding,” see our 5-post series on the topic.)

Of course, it probably makes a big difference which organization we’re talking about. We’ve seen a lot of different charities soliciting funds in the context of the Haiti earthquake, and some of them may have greater abilities than others to translate funding into relief assistance. The problem for donors is that by and large, these charities aren’t accompanying their appeals with the information we’d need to have a sense of their “room for more funding.”

The key questions we feel charities should be answering

We assert that any organization raising funds in the context of Haiti earthquake relief should provide clear, prominent answers to the following questions:

1. How much are you trying to raise?
2. Roughly speaking, what activities are you seeking to fund?
3. How much have you raised so far?
4. If you raise more than your target, what will you do with the remaining funds?

Answers to such questions would make it possible to hold organizations accountable (well after the fact) for how they actually spent money raised in their relief appeals.

Charities’ answers to these questions

We’ve examined the online appeals of charities that (a) have raised $10 million or more in the context of Haiti relief, according to the Chronicle of Philanthropy’s tally or (b) have come up via Google Adwords on relevant searches (”Haiti”, “Haiti relief”, “Haiti donations”, etc.) (We also included Yele Haiti due to the attention it’s received, even though it did not meet either of these criteria.) It seems fair to say that all of these charities have done significant fundraising in the context of the Haiti relief effort.

For each, we looked on their website for answers to the above questions, both by clicking relevant links and by Google-searching the site. (Note that this work was mostly done last week.) We provide our results in Excel format.

Most (not all charities) provide at least an overview of what sort of aid they are looking to provide. However, only 6 out of 17 appear to give any target for how much they’re looking to raise; only three appear to be sharing how much they’ve raised to date (although many more appear to be sharing this information via the Chronicle of Philanthropy’s tally); and only two (the two mentioned in our previous post, Heifer International and Doctors Without Borders) are explicit about what they will do with “extra” funds.

We don’t believe that a top-down pooling of funds is the only way to make sure money gets to the organizations that can use it productively. We see potential in the idea of different charities’ raising funds for different efforts, as long as they make it clear how much they are looking for and why; make it possible to hold them accountable for how they spend the funds they raise; and are explicit about the point at which they’d spend funds on other potentially worthy causes. However, from what we can see, charities aren’t currently sharing enough information along these lines.

Donors have given more than $560 million to charities “to help earthquake relief efforts in Haiti.”

How much of that money has funded/will fund earthquake relief efforts in Haiti? How much should?

Money isn’t the only thing needed to deliver relief

Reports from the relief effort have stressed logistical challenges, such as blocked roads and limited access for planes and boats. See, for example, this interview on Reuters AlertNet from the 18th:

” The capacity of the Port-au-Prince airport is about to be increased but it is still a small airport. It’s very congested … The seaport is not operational and needs to be fixed in the coming days … It’s a big traffic jam of vehicles carrying humanitarian assistance and carrying people who want to leave Port-au-Prince … A massive effort is needed for Haiti but it needs to be done in a coordinated way. We need more realism about how long it takes to get an operation of this magnitude in such an intensely complicated environment running. A feeding operation for 2 million people is the goal. We know that in every case it takes time in the beginning and more time when every structure on which we can rely has been so appallingly hit.”

We’re not sure of the extent to which these issues have improved since. But we think it’s important to note that they do not sound like the kind of issues that can be solved directly by more supplies or even more money.

In fact, it is easy to see how deliveries of supplies and arrivals of volunteers could make things worse, if they are not carried out in full consideration of conditions within the country. We speculate that charities that are desperate to help, but without a strong on-the-ground presence, could be dropping off supplies without a clear plan for their distribution, and thus worsening congestion. This dynamic is described (for the case of the 2004 tsunami) in a Global Post article (H/T Good Intentions Are Not Enough):

although officials didn’t request any medicine, they received 4,000 metric tons of it, or more than 4 pounds for each person in the tsunami-affected area. There were multiple-year supplies of antibiotics, and palette loads of drugs unknown to health care providers. Seventy percent of it was labeled in a language that locals did not understand … In the end, most of the drugs had to be incinerated — you can’t simply send such a stock to the dump, where it would seep into the ground water and create another health hazard. That cost donors and the Indonesian government millions.

Other charities may be reluctant to overspend on the relief effort, for exactly these reasons. Doctors without Borders, which has a major prior presence in Haiti (over $15 million spent there in 2008, according to its activity report), is explicit that it is no longer seeking to use additional donations for the relief effort:

We are incredibly grateful for the generous support from our donors for the emergency in Haiti … We are now asking our donors to give to our Emergency Relief Fund. These types of funds ensure that our medical teams can react to the Haiti emergency and humanitarian crises all over the world, particularly neglected crises that remain outside the media spotlight. Your gift via this website will be earmarked for our Emergency Relief Fund.

Just because charities are soliciting doesn’t mean they need more funding for Haiti earthquake relief

Heifer International, is explicit about the possibility that its Haiti appeal will be overfunded, leading to funds’ going to its other activities:

Funds raised in this appeal will be used in the recovery and rebuilding effort in Haiti in the wake of the earthquake. Any funds that exceed the level needed to provide relief in this rebuilding effort will go toward the disaster relief fund and for the entire mission of Heifer International.

Other charities may be less explicit. (We have argued before that even if donations are formally earmarked for a project, they may effectively end up funding the charities’ general activities.) Worse, some charities may in fact be soliciting for Haiti and intend to spend the money in Haiti, even if they can’t do so productively.

In an ideal world, charities that had no strong and underfunded presence in the relief effort would not be soliciting donations in the context of the relief effort. But it seems to us that this crisis (and the media coverage around it) arouses people’s emotions in a way everyday suffering can’t. That gives charities reasons to “capitalize on the opportunity” - reasons that may be

• Straightforwardly selfish: fundraisers are evaluated by how much money they bring in.
• Based on morality: donors want to give more to a relief effort that can be usefully spent, but they give less to other humanitarian initiatives than can be usefully spent.

This dynamic could explain why so many different charities are clamoring for donations under the heading of “Haiti earthquake relief” (details forthcoming in a future post).

Can you make sure your money is going to the Haiti relief effort? Should you?

There are strong arguments that the gift you make for Haiti, in the heat of the moment, would be better applied to (for example) a bednet distribution program in Africa.

However, in our view, the fact that there are many other worthwhile activities doesn’t absolve charities of responsibility for being clear about how funds are spent, and if anything it increases the responsibility of donors to understand what they’re paying for and how it compares to their other options for accomplishing good.

We believe that “room for more funding” questions do not get enough attention. In future posts, we’ll be discussing how those questions might be answered in the context of the Haiti relief effort.

The disaster in Haiti - and the media coverage of it - pull at the emotions in a way that everyday suffering in the developing world does not. However, our rough calculations suggest that in fact, a donor can have a bigger impact for less money by funding top charities’ everyday activities to reduce unnecessary death and debilitation.

We estimate a “generous” cost-effectiveness figure for a donation to Haiti is by considering (a) the total amount given and (b) the total number of people affected by the disaster.

Total amount given: It’s hard to find definitive figures for the amount of money already donated to the relief efforts, but ReliefWeb provides what appears to be a reasonable lower-bound (i.e., conservative) estimate. As of January 27, ReliefWeb reports that $1.2 billion had been given or pledged to Haiti relief efforts. (See document A on ReliefWeb’s Haiti Earthquake: Appeals and Funding page.) Note that its numbers clearly do not include all charities or all revenue sources: for example, it lists only $2.8 million raised by Partners in Health, whereas the Chronicle of Philanthropy’s roundup - up as of two days ago but seemingly down at this writing - lists $40 million (and also includes many charities that don’t appear at all in Reliefweb’s stats.)

Number of people affected: Haitian authorities estimated that 1 million people were left homeless by the quake (according to ABC World News) and perhaps 3 million were “affected” (according to the Guardian). For the sake of our calculuation, we’ll assume a range of 1-3 million people affected.

Expenses per person affected: using the Reliefweb number (assuming that nothing is excluded from it and that no more money is forthcoming, both assumptions that clearly understate the funds) yields a current level of $395-$1,185 pledged or donated per person affected. (For some interesting context, we’ve seen estimates that the 2004 Indian Ocean Tsunami affected 200,000 people and donors gave approximately $14 billion — funding per person of $7,000.)

We estimate that our top-rated charities save a life for approximately that amount. While it’s clearly not possible to directly compare the impact of relief efforts to the impact of saving an individual’s life, our feeling is that saving an individual’s life is likely to have a significantly larger effect on that person than the future change likely to occur in the average life of a Haitian due to the relief efforts.

Reader Evan writes:

I’ve been thinking about how best to donate to Haiti, and I reviewed some of the materials on your website and found them pretty helpful and persuasive. So thank you! But then my law firm announced that it would match donations to the Red Cross or Doctors Without Borders. Given that, I think I have to donate to one of those orgs: even if my money would probably be better spent elsewhere, it’s hard to imagine that it would be more than twice as well spent. Do you disagree?

My intuition here is different than Evan’s. My guess would be that giving to one of our top-rated charities could easily accomplish more than twice as much good as supporting the efforts of the Red Cross or Doctors Without Borders in Haiti.

This guess is largely based on two factors:

1. The large divergence in relative cost-effectiveness of different programs (which can approach a factor of 1,000, not just a factor of 2) combined with the reasonable position that disaster relief is not among the most cost-effective avenues for charitable funds.
2. A back-of-the-envelope calculation for cost-effectiveness of efforts in Haiti which puts it well below the cost-effectiveness of our top charities.

In this post, I’ll look at the first factor. I’ll post more on the second issue in a future post.

Cost-effectiveness for different approaches to helping people varies widely

The most cost-effective programs are so much more impactful per dollar than other programs that a much smaller donation to a top program will likely help significantly more people. We’re careful about our use of cost-effectiveness figures, and the Disease Control Priorities Report’s (DCP) in particular (which we think constitute “best-case” scenarios rather than what a donor can expect from his donation), but we do think they give a reasonable basic sense of the differences between different kinds of programs.

Figures 2.2 and 2.3 on pages 41-2 of the DCP report provide cost-effectiveness estimates for many common programs charities run. (These are all presented using the $/DALY metric. For more information on what this is, see our overview for interpreting the DALY metric.) Some of the most cost-effective programs are deworming programs ($3/DALY), expanding immunization coverage ($7/DALY), and bednets to prevent malaria ($11/DALY).

Some of the least cost-effective (but common among charities) programs are improved water and sanitation to prevent diarrhea ($4,185/DALY), some types of maternal and neonatal care packages ($1,060/DALY), and Antiretroviral therapy to treat HIV/AIDS ($922/DALY).

These examples are not meant to demonstrate that the less cost-effective programs are necessarily less worthy, but they do illustrate that the impact per dollar a donor can expect from his gift can easily vary by 2-3 orders of magnitude, even under assumptions that programs are essentially being carried out as intended. Of course, if some programs are poorly executed or simply ineffective, the difference can be much larger still.

With that context, when choosing which charity to support, I wouldn’t trade much confidence-in-an-organization to merely double the size of my donation.

As we’ve discussed before, with limited information we’d tentatively guess that disaster relief funds are closer to the less-cost-effective end of the range rather than the most-cost-effective end. With that in mind, I’d guess that a gift to VillageReach or Stop TB could easily accomplish more than twice as muc good as a gift supporting the Red Cross or Doctors without Borders in Haiti.

The above is very general and though relevant, not at all specific to the situation in Haiti. In a future post, I’ll post more on some specifics regarding Haiti and why I think it offers further support to the notion that donors can accomplish more good by giving to our top charities, even if they give less.

Two other small notes

There are a couple other factors that contribute (though in a relatively small way) to my conclusion here:

• It doesn’t seem appropriate to consider causing one’s company to give a donation to be equivalent to doubling one’s cost-effectiveness. The firm may have taken matching funds from a pool already allocated to charitable giving, or the partners may have given the funds to charities themselves. Even if the funds wouldn’t otherwise go to charity, the firm likely has another motive for giving, which should lead you to consider how this program differs from other embedded giving programs, which we think are of dubious additional value.
• Giving to a charity because it has demonstrated effectiveness has the additional benefit of signaling to other charities that effectiveness matters to donors. A core belief of ours at GiveWell is that rewarding charities for effectiveness in changing lives will incentivize other charities to improve their programs to compete for those donor funds. Proactive giving (i.e., trying to choose the best charity available) furthers this dynamic; passive giving (choosing from a predefined list) hampers it. It’s also quite possible that in a very direct sense telling your company that you’ve chosen to give in this way could influence them to adjust their own giving towards more considered and effective charities.

We are concerned about the way repayment rates are often reported. We’ve written about this issue before, arguing that different delinquency indicators can easily be misleading and pointing to one example we found where a microfinance institution’s reported repayment rate substantially obscures the portion of its borrowers that have repaid loans.

Following the links from David Roodman’s recent post about Richard Rosenberg, we found another paper Mr. Rosenberg authored making all the same points, much better than we did. The paper is Richard Rosenberg’s. “Measuring microcredit delinquency: ratios can be harmful to your health.” CGAP Occasional Paper #3. 1999. Available online here (pdf).

Relevant quotes from Mr. Rosenberg’s paper

The importance of using the “right” delinquency measure:

MFIs use dozens of ratios to measure delinquency. Depending on which of them is being used, a “98 percent recovery rate” could describe a safe portfolio or one on the brink of meltdown. (Pg 1)

The measure we’ve been asking for seems to be equivalent to what he calls the “collection rate.”

Most of the discussion will be devoted to three broad types of delinquency indicators: (a) Collection rates measure amounts actually paid against amounts that have fallen due. (b) Arrears rates measure overdue amounts against total loan amounts. (c) Portfolio at risk rates measure the outstanding balance of loans that are not being paid on time against the outstanding balance of total loans. (Pg 2)

It’s essential to not only know which measure is being used, but precisely how an MFI calculates its version of the measure:

But the reader must be warned that there is no internationally consistent terminology for portfolio quality measures—for instance, what this paper calls a “collection rate” may be called a “recovery rate,” a “repayment rate,” or “loan recuperation” in other settings. No matter what name is used, the important point is that we can’t interpret what a measure is telling us unless we understand precisely the numerator and the denominator of the fraction. (Pg 2)

Mr. Rosenberg describes different tests to which MFIs should subject various delinquency measures to determine which is most appropriate. For GiveWell’s purposes, one of the key tests is the “smoke and mirrors” test:

Can the delinquency measure be made to look better through inappropriate rescheduling or refinancing of loans, or manipulation of accounting policies? This is our smoke and mirrors test. (Pg 3)

The practice of rescheduling and renegotiating loans:

When a borrower runs into repayment problems, an MFI will often renegotiate the loan, either rescheduling it (that is, stretching out its original payment terms) or refinancing it (that is, replacing it—even though the client hasn’t really repaid it—with a new loan to the same client). These practices complicate the process of using a collection rate to estimate an annual loan loss rate. Before exploring those complications and suggesting alternative solutions for dealing with them, the author needs to issue a warning: any reader looking for a perfect solution will be disappointed. The suggested approaches all have drawbacks. It is important to recognize that heavy use of rescheduling or refinancing can cloud the MFI’s ability to judge its loan loss rate. This is one of many reasons why renegotiation of problem loans should be kept to a minimum—some MFIs simply prohibit the practice. (Pg 10)

The strengths of PAR (”portfolio at risk”) as a measure:

The international standard for measuring bank loan delinquency is portfolio at risk (PAR). This measure compares apples with apples. Both the numerator and the denominator of the ratio are outstanding balances. The numerator is the unpaid balance of loans with late payments, while the denominator is the unpaid balance on all loans The PAR uses the same kind of denominator as an arrears rate, but its numerator captures all the amounts that are placed at increased risk by the delinquency. (Pg 13)

And its weaknesses:

Like many other delinquency measures, the PAR can be distorted by improper handling of renegotiated loans. MFIs sometimes reschedule—that is, amend the terms of—a problem loan, capitalizing unpaid interest and set- ting a new, longer repayment schedule. Or they may refinance a problem loan, issuing the client a new loan whose proceeds are used to pay off the old one. In both cases the delinquency is eliminated as a legal matter, but the resulting loan is clearly at higher risk than a normal loan. Thus a PAR report must age renegotiated loans separately, and provision such loans more aggressively. If this is not done, the PAR is subject to smoke and mirrors distortion: management can be tempted to give its portfolio an artificial facelift by inappropriate renegotiation. (Pg 16)

PAR can also be misleading in a situation where an MFI is growing rapidly (a key argument of our past posts):

Another potential distortion in PAR measures is worth mentioning. Arguably the PAR denominator should include only loans on which at least one payment has fallen due, so that late loans in the numerator are compared only to loans that have had a chance to be late. Nevertheless, it is customary to use the total outstanding loan balance for the denominator. The distortion involved is usually not large for MFIs, because the period before the first payment is a small fraction of the life of their loans. For instance, for a stable portfolio of loans paid in 16 weekly installments with no grace period, a PAR of 5.0 percent measured with the customary denominator (total outstanding portfolio) would rise only to 5.3 percent using the more precise denominator (excluding loans on which no payment has yet come due.) However, if a portfolio is growing very fast, or if there is a grace period or other long interval before the first payment is due, then the customary PAR denominator can seriously understate risk. Pg 17

Table 6 on Pg 19 summarizes the strengths of weaknesses of different measures:

Why is this important?

Given how complicated this all is, we think that MFIs need to be clear and transparent about (a) which measures they use and (b) precisely how they calculate them.

However, this isn’t the case. For example, we aren’t confident that most MFIs normally report rescheduled and renegotiated loans as at-risk in PAR measures.

On the one hand, Commenter Ben writes, “Best practice is to treat all loans that have been rescheduled as PAR.” (This is consistent with MixMarket’s glossary, which indicates that, “[A PAR measure] also includes loans that have been restructured or rescheduled.”

Nevertheless, “best practice” may not correlate with “in practice.”

• This Kiva document (its “Partnership Application”) is explicit in the definition of PAR 30: “The value of loans outstanding that have one or more repayments past due more than 30 days. This includes the entire unpaid balance of the loan, including both past due and future installments, but not accrued interest or renegotiated loans.” (emphasis mine) Note that, to Kiva’s credit, it explicitly asks for renegotiated loans separately in the application.
• As Holden recently commented, “At least one MFI has indicated to us that it does not report [renegotiated loans in its PAR measures].”

The definition you read today isn’t necessarily the one that MFIs are using.

What measure do we use and why?

We’ve written before that our preferred measure is what the paper discussed above calls the collection rate. While the collection rate measure fails to provide a warning to MFIs that their portfolio is in danger, it is the strongest on Mr. Rosenberg’s “Bottom-line” test because it simply and clearly measures failed repayments. It’s therefore less susceptible to obfuscation and manipulation.

For GiveWell’s purposes, we need a delinquency measure that most clearly reports borrowers’ situations. While PAR measures provide information, it’s clear that PAR measures are more valuable to evaluating the risk of an MFI’s portfolio, which while relevant is not our key concern.

Update: see our official page on Haiti earthquake relief, which consolidates advice from us and a few other sources we have high opinions of.

Reader Brigid writes:

“I would love to hear any thoughts you have on contributions in light of the crisis in Haiti. My sense is that now of all times is when people give significantly without due diligence into a charity’s impact and that donor have more illusions than generally (i.e., My gift is going directly to a hurt Haitian). It seems inevitable the news piece several months after the event when donors are surprised/angry to learn their gifts were not used as they believed they were.

Is there any way for an average donor to help the crisis in Haiti right now? Is there a way to “capture” the generosity that these events inspire while still focusing on impact? Specifically to GiveWell: would your team consider focusing quick efforts on analyzing charities that are addressing the crisis in Haiti (i.e. would you shift your mission at this moment)? Or, would you say: despite the current crisis in Haiti, any contribution an individual donor wants to make will still impact more people if, for example, given to fighting tuberculosis through Stop TB Partnership.”

A few notes:

• Unfortunately, “disaster relief” is not an area we’ve researched. When we’ve (very briefly) looked into the cause of “disaster relief” in the past, we haven’t found substantive, well-supported, specific recommendations for where to give. More at two posts from May 2008, following the cyclone in Burma:
  • Cyclone Relief: Recommendation and Questions
  • Emergency Assistance for Donors
• According to the Disease Control Priorities Project, emergency aid is one of the worst uses of donations, despite being one of the most emotionally compelling. For more, see our post on the cost-effectiveness of disaster-focused aid.
• Our intuition then, as it is now, is that the most effective way to help those in need is to give to effective organizations. A few of our recommended organizations have a presence in Haiti:
  • The Stop TB Partnership provides tuberculosis drugs in Haiti.
  • Population Services International has worked in Haiti since 1989.
  • Partners in Health’s flagship clinic is located in Haiti.

Before we start giving our answers to the questions of this post, I wanted to share some raw data that we look at to gauge how things are going.

The charts/tables below cover the following:

• “Money moved,” i.e., donations made to GiveWell-recommended charities due to GiveWell’s research
• Website traffic.

This is just a subset of the information we have. We’ll be releasing a more complete set of charts/tables/data shortly.

The table below shows the support each of our recommended charities received in 2009. Update, 1/8/2010: VillageReach sent us an updated file that includes donors through the end of 2009. The updated table is below. Note: VillageReach’s total fell as we discovered that we had erroneously double-counted some funds.

You can view the original table we posted here.

• “Pledgers” refers to people who made GiveWell Pledges (advance commitments to give based on our research) in 2008, before our recent report was completed, and followed through on these commitments in 2009.

• “Large gifts” refers to donors who made large gifts, and directly told us (and the charities they were giving to) that GiveWell’s research had been the key factor in where they gave.
• “Economic empowerment grant” refers to a grant made directly by GiveWell, with funds from a single donor.
• “Through website” refers to gifts made through the “Donate Now” buttons on GiveWell.net (some through Google and some through Network for Good (NFG)).
• “Grants” refers to grants made directly by GiveWell, mostly with funds that were restricted by donors for regranting (we also granted just under $15,000 in unrestricted funds).
• “Direct to charity” refers to donations that VillageReach received, not through GiveWell’s website, and believes it can confidently attribute to GiveWell (this is due to the fact that VillageReach is a relatively small organization that does not get many donations from unfamiliar individuals). We are still awaiting data from 12/22/09-12/31/09, so we expect the final version of this number be higher.

The following two charts show the amount donated and number of donors through the GiveWell site, comparing 2007-2009 (and the beginning of 2010).

Finally, we show monthly web traffic to the GiveWell site and blog. Two notes: (1) we unfortunately lost tracking for much of 2008 — that explains the lack of data during that period. (2) We had an immense spike on 12/20/2007 due to media coverage; we’ve purposefully set the left-axis as it is to make it easier to view the rest of the chart.

Our current top priority is assessing the state of GiveWell: what we’ve accomplished, where we stand, and where we should focus our limited resources next. Over the coming weeks, we’ll be trying to examine ourselves as dispassionately and critically as possible, and sharing our self-review in something close to real time via this blog.

GiveWell’s mission is to find outstanding charities and publish the full details of our analysis to help donors decide where to give. The ultimate goal is to have significant impact on the flow of donations, moving toward a world in which donors reward charities for success in improving lives. The major questions about GiveWell, as I see them, are as follows.

Questions for “customers,” i.e., people considering using GiveWell’s research to decide which charities to support

• Does GiveWell provide quality research that highlights truly outstanding charities in the areas it has covered?
• Is it practical for donors to evaluate and use GiveWell’s research in the areas it has covered?
• Has GiveWell covered enough areas to be useful?

Additional questions for stakeholders, i.e., people considering giving their time, money and other support directly to GiveWell (these include the GiveWell Board and staff)

• Is GiveWell’s research process “robust,” i.e., can it be continued & maintained without relying on the co-Founders?
• Does GiveWell present its research in a way that is likely to be persuasive and impactful (i.e., is GiveWell succeeding at “packaging” its research)?
• Does GiveWell reach a lot of potential customers (i.e., is GiveWell succeeding at “marketing” its research)?
• Is GiveWell a healthy organization with an active Board, staff in appropriate roles, appropriate policies and procedures, etc.?
• What is GiveWell’s overall impact, particularly in terms of donations influenced? Does it justify the expense of running GiveWell?

For all of these questions, we intend to discuss

• The progress we’ve made since November 2008 (when we last laid out a business plan)
• Where we stand today, relative to where we need/hope to be to consider GiveWell a success
• What we can do to improve

Our self-review won’t be entirely comparable to the reviews we perform of other organizations. The latter tend to be focused on the “end product,” as we stay agnostic on progress other organizations have made and how they can improve. When evaluating ourselves, it is essential that we examine “intermediate indicators” as well as our ultimate impact, and think critically about the different paths we can take to improve.

Philanthropedia has responded to our take on its microfinance report:

• Philanthropedia’s response to GiveWell’s analysis - Part I
  
• Philanthropedia’s response to GiveWell’s analysis - Part II

We find the response to be encouragingly straightforward. For the most part, it agrees with the concerns we have raised and commits to addressing them. Good Intentions are Not Enough argues that the response raises more questions than it answers, and we agree in substance, though to us the most important part of the response is an honest recognition of shortcomings and an expressed intent to improve.

A couple of responses on points where there is some disagreement:

Re: incentives. Our original post argued that Philanthropedia, if it became highly influential under its current model, would create bad incentives for experts (allowing them to continue to keep their thoughts under wraps) and charities (encouraging them to win over experts in ways unrelated to improving their social impact).

• Philanthropedia states, “Even if only 31 experts agreed to publish their bios and stand behind the results, that is exactly 31 more than before.” We disagree that this is a good thing. The experts have not been linked with recommendations (as far as we can tell) - all we know is that they shared their votes, and we know the aggregate result. The problem here is that no one is individually accountable for their own recommendations and reasoning. If it became influential, this model would allow experts to have the benefits of transparency (i.e., influence over individual donations) without the accountability, which we feel would be a net negative impact on the incentives for experts to be transparent.
• Philanthropedia states that it intends to rigorously guard against “gaming of the system.” But the dynamics we are concerned about are less related to “rule-breaking” or outright conflict of interest (for example, a charity pays an expert for a positive rating) than to distortion of ratings in a softer sense. Becoming “popular in a certain crowd” can be accomplished in a lot of ways that have nothing to do with improving impact. That’s why it’s so important that people who recommend charities put as much as possible of their reasoning out in public, where others can ask, “Are these reasons strong enough to explain this person’s support?”

Should donors use Philanthropedia’s current report?

In its current form, we feel that Philanthropedia amounts to a set of recommendations unlinked to either people or reasoning. That’s essentially asking donors to trust anonymous people, which we feel is a bad idea and not helpful for impact-focused donors. (And if donors need a “Who’s Who in microfinance,” that information is already available.)

For context: we are generally sympathetic to the “It’s not perfect, but it’s better than nothing” argument. We are acutely aware that donors have very limited resources today, and we ourselves have generally erred on the side of publishing/sharing our research as quickly as possible. Our research process and our own knowledge have enormous room for improvement (and this was more true when we first published research in November 2007). But we have not gone as far as publishing recommendations while in the middle of our process - we wanted to make sure we could clearly present our criteria and the charities considered, so that people could check our process and hold us accountable if they wished to do so.

There are good arguments for Philanthropedia’s sharing/promoting its research even in its current state, and it’s ultimately Philanthropedia that makes that call.

Now that giving season has ended, we will be shifting our priorities and slowing down the pace of blog posts. Here’s a quick overview of the highlights from our last few months:

• You Can Save A Life: GiveWell in a nutshell. Why give to a charity with a great story and nebulous impact, when by giving to a top charity you can actually save a life?
• Frank discussions of our top two charities, VillageReach and Stop Tuberculosis Partnership.
• Celebrated charities we don’t recommend: a roundup of recent critiques of non-recommended charities.
• Our take on microfinance:
  • 6 myths about microfinance, debunked.
  • “High repayment rates” don’t mean what you think.
  • Summary of our full 17-post series.
• Our take on other approaches to “economic empowerment”:
  • Agricultural training/technology/knowledge sharing
    
  • Gifts of livestock (e.g., Heifer International)
    
  • Social enterprise investment (e.g., Acumen Fund)
• 5-post series on “room for more funding”, the essential question about charities that no one seems to ask.
• Poor in the U.S. = rich: why U.S. poverty and overseas poverty are no comparison. (Also see our post on different definitions of “hunger”.)
• Some thoughts on GiveWell’s general approach:
  • We focus on gifts that help people, not gifts that feel good. We think the latter currently gets too much attention.
  • We don’t focus on “how much a charity spends on overhead.” A broad consensus is emerging that this is the worst way to pick a charity.
    
  • We believe that the burden of proof is on the charities to show that they’re having an impact. Our “default assumption” about a charity is that it isn’t living up to its story, and we think great charities are the exception rather than the rule.

  • As such, we are willing to give a charity a negative rating based on a failure to share meaningful information, even if we don’t have hard proof that it’s failing. This is probably the single most controversial aspect of our process, and we defend it at length in this post.

I believe that Philanthropedia has a promising model. We were glad to join with them on a recent press release urging donors to look beyond administrative expense ratios. We are potentially interested in collaborating with them and/or incorporating their work in the future. I examined their new report on microfinance with great interest.

At this point, I feel that Philanthropedia’s execution of the model has too many serious problems to make it of use to a donor, or to make it a positive influence on the sector. Philanthropedia is a very young organization and we expect/hope to see major improvement, which is why we are sharing these detailed thoughts (we have also discussed them with Philanthropedia).

Overview:

• The definition of a Philanthropedia “expert” is unclear, and we are concerned that the distinction between experience and expertise is being neglected.
• Only 31 of 131 “experts” even appear to have their names accessible on the site. The little we know about the “experts” gives us substantial concerns about their representativeness and credibility.
• Because of concerns about who the experts are, how they are chosen, and what criteria they are applying, we find that we can’t make sense of the final output (i.e., recommended microfinance organizations). We disagree strongly with the recommendations, but cannot see enough of what went into them to examine the sources of the disagreement.
• We would not want donors to give based on this report, not only because of the problems listed above, but because we believe that doing so would reinforce bad incentives.

Details follow. This is a long post, but if you are a nonprofit professional interested in the Philanthropedia model - or a donor considering following its recommendations - I urge you to read it all. There are more substantial concerns than I can present briefly.

The definition of a Philanthropedia “expert” is unclear

How does Philanthropedia define an “expert”? The most specific answer I can find to this question is via the Philanthropedia blog: “To find these experts, we do a variety of things such as research thought leaders in the space, look for program officers at foundations who specialize in this area, research universities who have faculty focused on these issues, identify journalists who write extensively about the topic, look for executive directors or heads of nonprofits working in the space, etc.” The Philanthropedia FAQ adds that “We target experts in a social cause through a combination of cold emails and warm referrals (on the basis of professional and personal connections).”

But the numerical breakdown of experts cites 8% academics, 8% funders, 53% nonprofit executives, and 24% unspecified “others.” (We’re not sure why the percentages add up to <100% but consider this a minor issue.)

• Why such a large proportion of nonprofit executives and “others”? Is this proportion deliberate, based on any particular view of the right proportions, or has it simply “fallen out” of which people Philanthropedia was able to reach and which chose to participate? Is it a function of the “personal connections” mentioned by the Philanthropedia FAQ?
• It seems clear that these numbers don’t come close to the total numbers of academics, nonprofit executives, etc. who would have relevant perspectives. So why these “experts” and not others? Philanthropedia’s description of its process is too vague to give an answer to this question.
• Who are the “others?” Are they all “journalists who write extensively on the topic” or do many fall into the “etc.” category?

Philanthropedia is all about trust. It provides very little information that donors can assess for themselves - the premise is that experts are better positioned to make recommendations. So it is essential to be crystal clear about just what constitutes the “expertise” that makes someone’s opinion so much more valuable and credible than an individual donor’s. We feel that Philanthropedia needs to add a substantial amount of information before it can be considered clear enough.

And as we have written before, we believe there is a huge difference between “experience” and “expertise” - particularly in the nonprofit sector with its broken feedback loops.

Who are the experts?

When we look at what information is available about the “experts” behind the microfinance report, we note:

• The report appears to have surveyed 131 total “experts.” As mentioned above, the vast majority of these are nonprofit executives and “other” and only 8% (~10) are academics.
• This leaves very little room for representation of microfinance skeptics. Nearly everyone, if not everyone, surveyed is likely coming from a worldview that embraces the core assumptions behind the pro-microfinance mentality, and thus embraces microfinance the way it is currently carried out. It seems fairly clear how this issue can lead to bias: for example, people who are already committed enough to microfinance’s potential to be funders/nonprofit executives seem likely to focus more on scale (reaching as many people as possible) rather than social impact (asking critical questions about microfinance’s effect on clients).
• Names and affiliations are provided for only 31 of the 131 “experts.” For 21 of the 31, the only information given is the name, position and organization name (and not all of these organizations can easily be found on the web - for example, is this the WAVE Foundation that employs Anwar Hossain, who is listed as one of the “experts”?)
• What we can see about these “experts” sheds very little light on how they were chosen and how representative they are.
  • Some are top-level executives and some are not.
  • 3 of the 31 represent Unitus; one represents Tearfund, a very large charity working in diverse areas; but other major U.S. microfinance-focused charities such as Grameen Foundation and FINCA have no representatives in this set. (Perhaps Unitus’s heavy representation has something to do with its low rank, since Philanthropedia requires that nonprofit professionals not vote for their own organization.)
  • In what regions have these “experts” done their work? With what kinds of programs? In what ways might their experience and allegiances be biased? Do they have valuable experience and impressive accomplishments or merely many years of being employed in this area? We don’t have the information that could address these questions.

The recommended charities

As stated above, Philanthropedia provides very little information on the recommended charities aside from their names, links to their websites, and figures whose exact meaning is unclear but which seem to indicate the level of “expert support.”

Our main observation is that the list reads like a “who’s who in large U.S. microfinance charities.” This pertains to our above observation that microfinance skeptics don’t seem well represented.

Most of the ratings are clustered fairly close together (7%-11% range). Accion is at the top with 19%, and CARE, FINCA and Unitus are at the bottom. It’s unclear exactly what these numbers mean - should we take from this that Accion is particularly outstanding or that Unitus has problems? Probably not, since Philanthropedia states that it doesn’t rank nonprofits.

We would not, at this point, recommend any of the charities listed:

• Accion, Grameen, Opportunity International, Women’s World Banking and Unitus all fall into the category of “consultant” charities that we’ve expressed reservations about. For all of these organizations, we have serious concerns about their social-impact-focused due diligence.
• Freedom from Hunger, BRAC and CARE all run a wide variety of programs including but not limited to microfinance. It’s not clear that your donation to these organizations will even fund expansion of microfinance (regardless of whether you restrict it).
• We know less about Pro Mujer and FINCA. We have not looked deeply into either as part of our ongoing grant application (Pro Mujer does not work in Africa and FINCA declined to apply).
• MIX Market is an information source (and an excellent one, in my view), not a provider of direct services. I would be open to recommending it but would need to know what it would do with more funds.
• As for Kiva, we have written a lot about it. At this point our basic take is that it’s effectively getting your money to partner microfinance institutions, and we don’t have enough information, or confidence in, its due diligence process on partners from a social impact point of view (our posts on Kiva list a couple particular, substantial points of concern).

The “expert quotes” provided by Philanthropedia do not address these concerns, and in fact raise questions about whether the “experts” are even trying to help donors accomplish good. Take the Kiva page for an example. (Click “Expert assessment” from that link to view quotes.) Nearly all strengths listed pertain to Kiva’s marketing ability: “They have a cute model,” “They are appealing to donors,” and more. In other words, you are being advised to give to a charity because it’s good at making you want to give to it.

The few comments that pertain to other aspects of Kiva raise other concerns:

• “They are truly innovative as a capital source for microfinance institutions.” The exact nature of the innovation is unspecified, but probably amounts to more praise for the marketing.
• “Kiva charges 0% interest rate on capital.” Kiva charges 0% to its partners, but borrowers have to pay interest. Is that a good thing, and if so, why?
• “They provide a direct link from funder to borrower” - this is simply false.
• “They are not neglecting the US domestic poverty issue” - this comment comes up in more than one organization’s profile. We have argued that it is better not to focus on US domestic poverty; in any case, this is clearly a difficult, value-laden judgment call.

The “expert quotes” on other organizations raise similar concerns. Overall, it is not apparent that the “experts” are giving advice based on how an individual donor can have the most impact. It is not clear what their recommendations mean.

As for our own take on which microfinance charities are best, we have deep concerns about this sector that have led us to go beyond U.S. charities and seek out outstanding microfinance institutions. We’ll be writing about more about the best we’ve found, the Small Enterprise Foundation.

We are coming from an attitude of skepticism toward microfinance in general. Our skepticism is certainly debatable, but we have been explicit about it and discussed our thoughts on microfinance at length. A donor who wants to know where our advice is coming from and what it’s based on can find out. We feel it is important that donors have similar context on the people cited by Philanthropedia.

Reinforcing bad incentives

We pay a lot of attention to incentives. When discussing how we should rate charities, we constantly ask, “If charities all knew we were handing out ratings on this basis and expected donors to give based on our ratings, how would they work the system? What would we expect to change?” (For example, we give negative reviews to charities that publish no information, not just charities whose materials raise concerns; we don’t want to provide incentives to hide information.)

We feel that if Philanthropedia stuck to its current approach and donors used it:

• “Experts” would not have any incentives to increase their own transparency above current, unacceptable levels. They could influence others’ giving by giving recommendations without substantive reasons and without even disclosing their names.
• Charities would have strong incentives to do whatever got them recommended by Philanthropedia’s set of experts. This could include networking, flattery, wining/dining, outright conflicts of interest, and learning to speak of their programs in ways that would appeal to Philanthropedia experts, all of which might end up being more cost-effective from the charities’ perspective than optimizing and demonstrating their social impact.

Bottom line
I urge Philanthropedia to:

• Publish exhaustive details of how “experts” are defined, selected and invited.
• Publish names and affiliations of all experts who are invited to participate, whether or not they accept.
• It is particularly important to publish names and affiliations for those who do participate because donors should not be asked to trust anonymous people.
• Address the concerns we have raised about the representativeness and credibility of experts, if these concerns still appear valid once the identities and process are disclosed.
• Publish the templates for all surveys sent to experts. Be explicit about the criteria by which experts are asked to recommend charities.
• Ask experts to make their best case publicly and to be clear about whether they’re recommending charities to impact-focused donors, praising charities for marketing abilities, or something else.

We believe the Philanthropedia model has a lot of potential. We would find great use, for donors and for ourselves, in a site that could credibly claim to have assembled the charities with the most support from people with relevant experience.

Philanthropedia has accomplished a lot in a short time, but we don’t believe its content is yet appropriate for distribution to individual donors looking for help with their giving decisions.

We ask you, as a donor, to turn down some great pitches – “Your interest-free loan will help this person escape poverty forever,” “You can give a cow to a poor family for Christmas,” etc. – and give instead to charities that aren’t terribly good at storytelling. Why?

It comes down to this. We think that most of those stories are just that – stories. (For more, see our summary of recent posts on “big-name” charities, which we feel are representative of the full set of charities we’ve reviewed.) But if you give to one of our top charities, you really can save (or dramatically change) a human life.

It hasn’t been easy to find charities that we can honestly say this about. That’s what our process is built around and where most of our energy goes. This week we’ve blogged about the best we’ve found, VillageReach and Stop Tuberculosis Partnership. There is plenty of room for doubt even with them, but overall we think there is a strong case - even for the skeptic - that your donation to them can save a life.

What do we mean when we say “save a life?”

By “you can save a life,” we don’t mean anything as simple, concrete, or easy to grasp as the stories charities usually tell.

• Your gift can’t literally be linked to an individual. It will help an organization that, all things considered, is achieving a lot of impact for what it spends.
• If you must know what “your” dollars are doing, it’s likely that they’ll be sitting in reserves to ensure financial stability, or enabling a slightly larger travel budget for evaluators, or something similarly unexciting.
• It’s even highly possible that your donations will be wasted, and that the charity you give to – even the best you can find – will fail. We don’t think there are true guarantees in aid.
• Even if these charities are succeeding, it’s very likely that your donation won’t ultimately result in the charity doing anything differently. It’s pretty hard to think about how $1000, by itself, could really change anything about Stop TB Partnership’s plans for next year.
• Yet donations add up. 50-100 of these donations could mean a significantly larger grant, more people getting tuberculosis treatment … and that could mean families staying intact instead of being struck by sudden death.

The truth is that it takes a lot of abstraction and analytical thinking to really think about how your donation saves a life. The life you can save is an “expected” life (”expected” in the sense of probabilistic expected value) - it isn’t a person we can point to or show you a picture of. More than with typical charities, you have to use your imagination. But more than with typical charities, your impact is real.

With opportunity comes responsibility

In The Life You Can Save (which prominently features GiveWell and which we have reviewed), Peter Singer writes:

By donating a relatively small amount of money, you could save a child’s life … we all spend money on things we don’t really need, whether on drinks, meals out, clothing, movies, concerts, vacations, new cars, or house renovation. Is it possible that by choosing to spend your money on such things rather than contributing to an aid agency, you are leaving a child to die, a child you could have saved? (pg 5)

Our corollary: is it possible that you are leaving a child to die when you choose to donate to a charity with a “feel-good” story rather than a charity with a great case for real impact?

It is true that, as our critics often point out, a charity can be impactful without being demonstrably impactful. But when one charity proves itself and another leaves you guessing, it seems clear to us which one offers the “better bet” – and more “expected lives saved” – given the information available. When you have the option of giving to an outstanding charity that demonstrably can save a life, how do you justify giving to a charity whose true impact is essentially a big question mark?

I’ll leave this blog’s last words for 2009 to Natalie, a relatively recent GiveWell hire (she has been working full-time on research since July).

Sometimes I’m almost tempted to give to a charity I know less about. I’ve been over VillageReach and I’ve seen how complex the situation is and how many questions there are. If I gave to some charity I know nothing about, I could just think about the story they tell and feel good and not have these nagging doubts. But I’m not going to do that – in the end it’s more important to me that I really make a difference.

VillageReach currently holds the top slot in our top charities list. While I will be donating to the Stop Tuberculosis partnership (discussed yesterday), Elie’s donation this year went to VillageReach.

VillageReach consults on health system logistics in high-poverty, remote areas to help life-saving supplies get to those who need them. It may not be the most attention-grabbing story (”improving operational efficiency” doesn’t have the same ring to it as giving a cow or fixing a smile), but

• It’s directly addressing a key issue - limited health system capacity - that may limit the amount of good donations to any health-related cause are able to do.
• Immunizations in sub-Saharan Africa are a top global health priority (ranked #4 by the Copenhagen Consensus, for example and have been estimated to avert infant deaths for ~$200 each (estimate includes all costs).

In other words, VillageReach would be accomplishing tremendous good if it could improve logistics to deliver more vaccines. Can it?

Pilot project

VillageReach assessed its pilot project in Mozambique (Cabo Delgado province) with more thoroughness than we’ve seen almost anywhere outside of JPAL and IPA. It is a great demonstration that just because a charity is relatively small, and experimental, doesn’t mean it can’t collect substantive and compelling evidence regarding its impact.

VillageReach collected (and has now published) data on trends in vaccine/supply deliveries, “stock-outs” of vaccines (the single most compelling indicator, in our view) and vaccination coverage rates. It also conducted vaccination coverage surveys in both the province where it worked and a nearby province for comparison. After examining this evidence along with some independent analysis with our own, we’ve concluded that the case for impact (despite not including a randomized controlled trial) is quite compelling: VillageReach caused a significant jump in vaccination rates, which we believe can be fairly extrapolated to lives saved at well under $1000 each.

VillageReach hoped to transfer responsibility for the modified logistics system (which has a dedicated team responsible for delivering supplies, rather than each clinic responsible for its own collection) to the government, but available evidence suggests that the transfer failed. The fact that vaccination rates have since fallen is further evidence that VillageReach made a difference while they were there, but obviously discouraging relative to what they had hoped for. We have expressed doubts in the past about donors’ getting their hopes up for “sustainability” in aid, and think it’s worth noting that even VillageReach’s project – with its strong case for impact as well as a case that it actually would save the government money – has so far failed at the ambitious goal of changing government practices.

More details at the “Does it work?” section of our review.

The future of VillageReach

There are a lot of questions about the future of VillageReach. As a small organization, it sees frequent changes in its plans and engagements. It has taken on a variety of consulting projects, some of which we have little information about and could be significantly different from where it has succeeded before.

However, it has provided an unusual amount of clarity and specificity about the likely use of additional donations. We feel that VillageReach can productively absorb up to $2.5 million in unrestricted donations over the next year and that these funds will be used for work very similar to its pilot project: reactivating its program in the original province, in addition to possibly expanding to one or more neighboring provinces.

Pros and cons

VillageReach is very different from the “ideal charity” we had in mind early in our international aid research. (Our “ideal charity” is much closer to the charity I discussed yesterday, the Stop Tuberculosis Partnership). It is relatively small and still represents something of an experiment, which has both pros and cons.

VillageReach is a small charity that has struggled at times to raise enough funding, not a large charity built to absorb huge sums of money.

• This adds a certain kind of uncertainty to its situation, as its plans hinge heavily on how fundraising goes as well as other factors (such as government cooperation) outside its control.
• But being small also comes with advantages for a donor. The need for more funding is extremely concrete and some extra funding could make a huge difference to VillageReach’s ability to perform up to its potential. In addition, it is feasible – if difficult – to have a complete view of VillageReach’s work. Where a large charity like Stop TB Partnership relies heavily on audits and spot-checks – leaving open a lot of room for unobservable error – VillageReach is measuring, and accountable for, the results of everything it does.

VillageReach has only one demonstrable success to its credit. It is still establishing the potential of its model.

• VillageReach is “experimental” in a sense. It doesn’t have the same consistency, accomplishing similar results in many times and places, as Stop TB Partnership.
• However, we think this is a very different level of “experiment” from an organization with no demonstrated impact. We think there is enough information about VillageReach’s past impact for an individual donor to be confident that its future activities are a “good bet.”
• And with the risk of an “experiment” comes upside. If VillageReach continues to demonstrate success, it could affect the way health logistics works all over the world, through the spread of information rather than through its own funds. The problem we have with most “experiments” in international aid is that we aren’t convinced that they will produce the information needed to (a) determine whether they’ve succeeded and (b) bring about widespread adoption if they have. By contrast, VillageReach has demonstrated a willingness and ability to produce credible evidence of impact.

VillageReach does not fully share GiveWell’s priorities.

• We are most interested in “doing more of something that demonstrably improves lives.” VillageReach has expressed great interest – and allocated funds to - the more ambitious goal of “advocacy” (broadly defined), changing the way governments and larger funders operate.
• Yet even as it aims for these more ambitious goals, we believe that VillageReach is in fact saving lives at a level of cost-effectiveness that rivals that of any other charity. So we see its more ambitious goals almost as a “bonus” – if it succeeds at them, we’ll have helped fund a truly incredible level of impact, and if it fails at them, we still expect to have gotten our “money’s worth” (lives saved for under $1000 each).

Bottom line

The pros and cons listed above are largely mirror images of each other. Stop TB Partnership is an outstanding “large charity” – with both the advantages and concerns that accompany great size – and VillageReach is an outstanding “small charity,” with a different set of advantages and concerns. Both give good answers to all our major questions, leaving only the kind of concerns that are likely inevitable with charities of their size.

Which you have more confidence in, and prefer to donate to, is a judgment call. We have ranked VillageReach as our top charity because Elie and I happen to agree that it seems like a “better bet” overall (to approach the impact implied by the academic cost-effectiveness estimates) – we think the ability to hold it fully accountable outweighs the risks that come with being small and experimental.

Elie and I do part ways when it comes to the ultimate donation decision. I am giving to Stop TB Partnership for the personal values-related reasons discussed yesterday. Elie doesn’t share the values I mentioned and has donated to VillageReach.

The Stop Tuberculosis Partnership is one of our top-rated charities and the one I will personally be giving to this year.

Like most charities, the Stop Tuberculosis Partnership has an admirable goal. It uses donations to fund grants of tuberculosis drugs to governments, and also assists them with (and monitors their implementation of) the WHO-recommended tuberculosis control strategy. It’s an appealing activity because:

• Tuberculosis is an infectious disease that is frequently fatal, yet highly treatable.
• Standard tuberculosis control is estimated at $150-$750 per death averted (independent estimate including all costs – more on the organization providing this estimate).
• The authors of the Copenhagen Consensus disease paper rate tuberculosis control as the #1 priority for donors (pg 51).
• The more drug-resistant forms of tuberculosis may represent a major catastrophe risk, making it particularly important to avoid medical mismanagement and inappropriate use of anti-tuberculosis therapy. We haven’t looked into the “catastrophic risk” aspect of TB control much, but I see this as an added benefit that would come from supporting a good TB control organization.

As with most charities, the story doesn’t end there – there is a lot that can go wrong with tuberculosis control, and we came at the Stop Tuberculosis Partnership with a lot of questions. But unlike most charities, the Stop Tuberculosis Partnership has provided stellar answers that have increased our confidence in what it’s doing. A sample:

Do governments use the drugs for treatment, or sell/lose/misuse the drugs?

• In order to receive the drugs, governments must agree to provide a certain level of coverage, i.e., free treatment for patients, as well as adhering to multiple other conditions of the WHO-recommended tuberculosis control strategy.
• Stop TB then performs spot checks of medical facilities (audits at locations chosen by Stop TB, not by the governments) as well as examination of country-wide data (particularly treatment success rates) and data reporting systems.
• Treatment success rates are generally high.

The answer to the above question relies on audits. Does Stop TB carry out these audits consistently and produce substantive information on country compliance? We have positive though not fully conclusive indications.

• We have seen four audit reports – those of the highest-burden countries – and found them to be quite substantive and straightforward about strengths and weaknesses of the TB control programs.
• We believe that the audits are carried out consistently, as Stop TB’s annual reports from 2003-2005 summarized the outcomes of audit reports from all drug grant recipients. Stop TB has changed its reporting format, but has told us that it still does audit all grantees.
• We have seen specific instances of funding being cut off for noncompliance. (See Stop TB’s progress reports). It is very rare to see a charity publicly disclosing a shortcoming like this, and to be able to confirm that it was handled appropriately.
• Details at the “Does it work?” section of our Stop TB review.

But even if the governments are carrying out quality tuberculosis control programs, how much of Stop TB’s support is really resulting in more coverage – as opposed to simply substituting for funds the government would have spent anyway? We would guess that there is a degree of “fungibility,” but feel that Stop TB takes significant steps to maximize the impact of its funds.

• Pre-grant, it conducts detailed analysis of governments’ other sources of funding.

• Post-grant, it examines trends in the coverage of treatment before and after it became a funder. Though these trends do not have the rigor of a randomized controlled trial, we feel they provide reasonable evidence that Stop TB’s funding results in additional coverage.
• Details at the discussion of this issue in our review.

Does Stop TB, itself, have room for more funding?

• Stop TB has provided us with analysis of projected revenues and expenses, showing a significant gap (about $10 million a year over the next four years). We are not cleared to post this analysis publicly, but Stop TB will consider sharing it with interested donors who contact them.
• We have also tried to dig into the exact meaning of this “gap,” and have only been partially satisfied with the response.
  • Stop TB has cited specific examples of underfunded countries whose “buffer stock” has had to be cut (so that they have enough drugs to treat the projected number of patients, but little margin for error).
  • It has also stated to us that more funding would enable it to approve the treatment of more patients.
  • It has not provided quantitative scenario analysis, so it’s hard for us to be fully concrete about what additional donations can be expected to lead to (in terms of # people approved for treatment and in terms of treating more people vs. providing “buffer stock”).
  • Details at the discussion of this issue in our review.

Bottom line

• Big picture: Stop TB is the best international example we’ve found of a “proven, cost-effective, scalable, transparent” charity. It focuses on one highly cost-effective and proven intervention, collects and shares the information needed to address the questions we’ve come up with (and then some), and projects a significant funding gap. It is both large (significant ability to absorb more funding) and programmatically focused. It has both an excellent goal/program (tuberculosis treatment) and outstanding reporting on how its specific activities are going.
• That doesn’t mean we have no concerns. There are a lot of moving pieces here and a lot of complexity (as with just about any charity working internationally). Stop TB takes all the measures we find reasonable to assess its impact, and provides a strong picture of impact overall, but we would guess that there are many problems that its audits simply aren’t comprehensive enough to catch, as well as questions about “fungibility of money” that can’t be fully answered.
• Tomorrow we will discuss VillageReach, a very different charity that we are ultimately slightly more confident in, although there are major pros and cons to both. I am personally giving to Stop TB because (a) I put more weight on preventing adult deaths than on preventing child deaths (and tuberculosis affects adults more than vaccine-preventable diseases do); (b) VillageReach works in the most remote areas, while Stop TB works in a great variety of areas. I prefer (all else equal) to help people in less remote areas, who I believe have more opportunities (for themselves and for helping others).