What makes a good teacher? The question to date has largely been answered with a nod toward mysticism. In a recent New York Times Magazine article, for instance, a director at the Urban Institute said successful teaching depends on a kind of “voodoo;“ in the same piece, Sylvia Gist, a dean at the college of education at Chicago State University said there is an “innate ability for teaching.“

If this is true—if good teaching really is just an accident of birth, not linked to any specific things that good teacher do (and that others can be taught)—then, frankly, a lot of money is being wasted on teacher training, funded graduate education programs and the like, since studies have shown that successful teachers are no more likely than the unsuccessful (as measured by student performance) to have gone to graduate school, to be outgoing, have prior experience working in impoverished environments, or posses many other qualities that experts have long thought causal to teaching success.

And yet a group of consultants and researchers actually do believe that good teachers are made—and that our current ways of educating aspiring teachers is not what makes them. The same New York Times article quoted above profiles the different efforts of two individuals, Doug Lemov and Deborah Loewenberg Ball. Lemov works for an organization called Uncommon Schools, a nonprofit that starts and runs urban charter schools. As part of an effort to improve the effectiveness of his teachers, Lemov conducted a multi-year effort to observe and record what the most successful teachers do in the classroom. The result of that effort is a taxonomy of effective teaching practices that outlines a series of “classroom management” techniques used to ensure kids are paying attention and doing the things they should to learn the lesson.

For her part, Deborah Ball, an assistant professor at Michigan State, believes classroom management is not enough. A teacher can have a great control of her classroom, but that won’t get her far if she has nothing to teach them. As a result, she has focused her work on identifying what teachers need to know. Her work shows that teachers need not only strong knowledge of their subject but strong understanding of how others think about the information, and why a student might some up with the wrong answer (she interestingly found that teachers who scored well on a multiple choice test designed to assess their ability to judge student math work, as distinct from their own math knowledge, were among the more effective instructors).

Lemov and Ball are not alone in their efforts to identify what makes great teachers. This January article from The Atlantic on what makes a great teacher highlights an effort by Teach for America to deconstruct the actions of their best teachers, those who bring their students forward by one or more years during a school year. Far from a mystical ability or any abstract ‘care’ they show for students, that effort has likewise uncovered some shared actions that effective teachers consistently do. For example, successful teachers are constantly re-vamping their teaching methods, techniques and materials to hit on something better. The classrooms of successful teachers also operate in a predictable and rhythmic manner, so that kids know what comes next and can transition from one activity to the next without confusion or lost time. Teach for America has been taking what it learns and using it to guide how it selects its fellows and how it trains them. Internal metrics show that these efforts have resulted in a dramatic increase in the number of TfA fellows bringing their kids forward (the piece reported that there is a study underway by research nonprofit Mathematica Policy Research to see if those results hold).

The larger conversation in education today seems to focus heavily on the “market driven” experiments such as school choice and incentive programs. Little evidence exists for or against these initiatives, so there is not reason to embrace or dismiss them out of hand. But there is significant evidence that shows that the teacher can do great things or inflict great damage. From the outside, it seems logical and desirable to simply get rid of demonstrably bad teachers, but plenty of experience from New York shows how difficult that is to achieve in practice. And even if that could be done, what can school systems do with the ones who are simply mediocre? Viewing teaching as a set of skills and actions that can be taught—not as sentimental magic—offers a good beginning.

Last week we hosted a conference call with Sean Stannard-Stockton of Tactical Philanthropy and David Roberts of New Dominion Philanthropy Metrics discussing the use of qualitative metrics and some approaches to measuring the unmeasureable. We recorded the call so that those who could not attend can listen in. MP3 files are available for download here.

On the topic of measuring the unmeasureable, let me also point you to two newly available articles. First, Alliance Magazine has now made available an article from last year where various heavy hitters in the philanthropy industry (including Paul Brest of the Hewlett Foundation, Michael Weinstein of the Robin Hood Foundation and Brian Trelstad of Acumen Fund) discuss their approaches to measuring impact. Second, the Wall Street Journal published a special report yesterday that includes an overview of social return on investment from a European perspective, including thoughts from New Philanthropy Capital and the Bertelsmann Foundation.

Editor’s Note: This is a guest post from David Roberts, Executive Director of New Dominion Philanthropy Metrics.

In the first two weeks after the Haitian earthquake, a great deal of the news coverage focused on search and rescue efforts of international teams. Approximately 62 international search and rescue teams (totaling something approaching 1,800 people) traveled to Haiti. The footage of their efforts was gripping. The teams were able to rescue 132 people from the rubble over an 11 day period. President Obama mentioned the efforts of some of the American teams in his State of the Union address; a recent OpEd in the New York Times argued the United States should invest more in SAR teams as a form of international aid and goodwill generation.

I think that’s exactly the wrong lesson from the results of the highly specialized SAR effort in Haiti. I cannot say with certainty how dollars should have been allocated in Haiti, or in future urban disasters. But as a public health practitioner, and someone dedicated to outcomes measurement, it seems likely to me that SAR-focused relief efforts improperly subordinate the good of the many to the good of a very few. In other words, there are much better ways to direct disaster relief than SAR.

While there is not good data on costs available, I’ve put together some rough figures based on what I could find.

What did it cost those 1,800 rescue workers to go to Haiti? Given that SARs workers bring in all their own supplies and equipment, $6,000 per person seems like a reasonable figure, though I understand there can be significant variations in costs. The International Rescue Corps, in explaining why they did not go to Haiti, reported that it cost them $60,000 to send a team of 10 to Indonesia after the October, 2009, earthquake. Using this number, we estimate that the 1,800 SARs workers cost approximately $10.8 million USD, or $81,818 per person rescued. 

What does a well-equipped and well-staffed field hospital cost? Israel was the only country to send a field hospital to Haiti immediately. While it is difficult to determine the cost of such a hospital, I did find a news report suggesting that the Canadians allocated $1 million to set up a similar field hospital in Haiti. The Israelis sent 200 people to staff the hospital, which we can assume cost another $1 million. Over the first five days of their arrival, there were 1,000 people treated, 300 operations and 16 babies delivered. This translates into $1,519 per beneficiary. Realistically, these hospitals continue to benefit people longer than five days, which lowers the cost-per-beneficiary number.

If we stick with the $1,519 per beneficiary number, and assume that the money spent on the SARs had instead been spent on field hospitals, we move from 132 total beneficiaries to 7,110 beneficiaries.  In theory, therefore, over 53 times the number of quake survivors could have been helped if all the SAR money was diverted to establishing, equipping and maintaining field hospitals in the affected zone.

The base figures also don’t consider opportunity costs and dependencies. Some of the SAR efforts were wasted because the people rescued soon died from their injuries—because their was no medical care available to treat them. For instance, an eleven-year old girl rescued two days after the earthquake later died because the first aid station she was rushed to was “not equipped to deal with her injuries.“ There are, unfortunately, many more stories like hers.

Since the damaged Port-au-Prince airport could handle only so many inbound relief flights, the SAR-focused relief effort also carried a very real opportunity cost. Those survivors who had suffered injury but escaped the falling buildings were just as imperiled as those who were entombed in the rubble.  While the world watched footage of dramatic rescue efforts, they faced death due to shock and infections that could easily have been avoided with adequate medical care. Some have estimated that as many as 20,000 survivors died each day as a result of inadequate medical care. Every plane carrying a SAR team could have been carrying medical supplies, food, water and other equipment that may have helped stave off infections that led to a great deal of limb amputations.

Keep in mind too that thousands, if not tens of thousands of people were rescued from the rubble by their families and neighbors, not by international rescue teams. Indeed this is the case in all such disasters. While we typically only see the efforts of the professionals, average citizens do far more of the immediate response and rescue work.

Ultimately the point is not that we don’t need SAR in urban disasters, but rather to argue that the subject of SAR, like many aspects of disaster relief, is not as simple as many might think. All disaster relief efforts would benefit from an analysis of the relative costs and benefits of different support approaches. Put simply, we need to learn from what happened in Haiti. My former School of Public Health dean at Johns Hopkins, Alfred Summers, recently compared the rescue efforts in Port-au-Prince to the “girl in the well” phenomenon—in which, while the world is fixated on reports of a multi-day rescue, tens of thousands of children die unnecessarily from malnourishment or disease.

I admit that, on one level at least, this is nothing more than an academic exercise. However, the numbers are compelling enough that when the next disaster strikes, governments and relief agencies should seriously consider that focusing on SAR teams is the wrong response.

Update: The Chile earthquake happened shortly after I wrote this post. I found this quote from the Chilean Foreign Minister: “Experience over the years and in prior earthquakes, as well as from international cooperation efforts like in Haiti, have left us lessons,” Foreign Minister Mariano Fernández told reporters. “We have to be very precise about what our needs are in order for the assistance to be of any use.” Hopefully, the world will listen to what Chile requests which at this point includes field hospitals, water purification plants and communications equipment.

In my post from last week I highlighted the attention given in the press to adoptions of Haitian children in the wake of the January earthquake, and pointed out why adoption for most children in need in the country is not currently an option—and won’t be for at least a year. So what should people do who want to help Haiti’s children?

Address basic needs for children and caregivers
Profiles like the one by Jon Lee Anderson that appeared in the New Yorker and the video blog by the New York Times give a glimpse into the every day reality for survivors on the ground. The injured and uninjured alike struggle to find basics such as water, food and health services. Given that children are particularly vulnerable, the instinct at this time might be to donate to child-centered organizations—orphanages or child protection agencies. Just don’t forget the caregivers. Desperate adults who struggle to find food every day may be more likely to abandon their children to an on-the-ground aid agency in the belief that their kids will be better cared for. Observe how more than 20 of the 33 children detained with the 10 missionaries from Idaho were sent willingly by their parents. Such choices can only increase feelings of fear and trauma and put additional pressure on aid workers.

Send money, not stuff
The Haitian health minister put it politely a few weeks ago when he asked Americans to stop sending shipments of baby formula, as it was encouraging Haitian mothers to stop breast-feeding their babies. The women believed that their trauma was polluting the milk and the babies would be better off with formula—but what happens with the formula runs out? Or they can’t get clean water to mix it with. Even innocuous-seeming contributions can be wrong for the context. Unless you are sure, send money to an organization you trust.

Support reunification efforts
Organizations such as Unicef, Save the Children and others have been actively building child-friendly spaces and shelters and working to find any unaccompanied children still in the streets. These are short term solutions, however. The best case for most of these children is to find surviving parents or relatives and reunite the families. These organizations are creating registries for the purpose so that the children in their care are registered and the adults have a place to go to look for them.

Plan for ongoing psychological support
As relief efforts focus on providing food, clothing and shelter, it might be easy to overlook the ongoing psychological needs, especially of the country’s children. Yet post-traumatic stress can linger for a long time and have negative implications on a child’s physical and cognitive development. The Peter C. Alderman Foundation is working with Partners in Health and the Haitian government to identify a long-term plan for offering psychological services to Haiti’s victims.

Surveys are probably the most common data gathering approach in use by non-profits and foundations. Surveys are easier and cheaper (especially since the advent of free online survey tools) than most other forms of evaluation.

Unfortunately, that means that many organizations rely on surveys to measure their programs and that could be steering them in the wrong direction. David Roberts of New Dominion Philanthropy Metrics goes so far as to call surveys dangerous in his article for Philanthropy Action. Says David, “Surveys can easily lead to bad data and false conclusions that can result in future missteps.“ For instance, a recent study by Dean Karlan and Jonathan Zinman in South Africa found that roughly 40 percent of survey respondents provided erroneous information about their use of credit.

To help organizations use surveys effectively, we’re hosting a conference call on February 22nd at 1pm Eastern/10am Pacific with David and Sean Stannard-Stockton of Tactical Philanthropy. The call is free. To register for the call and get call-in details, send an email to info@ndpmetrics.com.

Hope to see you there!

The 10 Central Valley Baptist Church in Idaho volunteers who have been detained in Port-au-Prince for trying to take 33 Haitian children into the Dominican Republic have had the positive effect of raising the profile of children in Haiti in the aftermath of the earthquake. Though the group’s actions at best grew out of extraordinary hubris and poor judgment, at worst out of something more sinister, their act at least raises a larger question: What are some real and effective ways to help children in Haiti now that the immediate post-disaster days and weeks have passed?

The discussion has been dominated by the subject of adoption. This seems to have come about partly because of actions of US government, which is working to expedite adoptions of Haitian children already underway.  The New York Times last week dedicated its “Room for Debate” to the Haitian adoption question, and CNN ran a story on an on the effort to push through cross-border adoptions that have been stalled because of the earthquake.

Although the adoption issue is very real and poignant both for parents whose adoptions were stalled by the quake and for those who are heartbroken to see images of Haitian children in orphanages, adoption is nonetheless a remote possibility for so many of the affected children and complicated for even more. The Haitian government is rightly balking at any efforts to expand the adoption options. Haiti reportedly had more than 380,000 children in orphanages before the quake. The experts on the Times blog for the most part agree that cross-border adoptions that were underway before the earthquake should be allowed to proceed—so long as the adoptive parents have gone through the series of checks necessary to ensure they are fit parents. But the stickier question of whether adoption should be expedited for children who were not already paired seems impossible to address independent of larger issues of disaster recovery and poverty alleviation for the country as a whole.

A photo montage in the Wall Street Journal makes the same point addressed by Times commentator EJ Graff of Brandeis University, which is that children in orphanages in developing countries are not necessarily orphans. Instead, destitute families sometimes place their children with social services in an effort to get them three squares and an education. This is less an act of abandonment than of desperation—witness what some of our country’s own desperate parents did when Nebraska passed its “safe haven” law without putting a limit on the child’s age.

In the aftermath of a disaster the confusion between orphans and “unaccompanied children” is exacerbated, because children are often separated from parents or caregivers. Save the Children reported that its efforts to create a child registry in Aceh Province after the Asian tsunami resulted in reuniting nearly half the children with their families. Graff proposes that Haitian and American officials allow a year for surviving parents and family members to be reunited, a reasonable span given the need to make sure that the children are in fact in need of placement and that all parties—the orphanages and the adopting parents—are acting in the child’s best interests. This says nothing, of course, to the broader implications of placing children traumatized by tragedy with families that may not share the same ethnic and cultural background.

If cool heads prevail, the solution for many of Haiti’s children will not be international adoption. So what can donors otherwise do to help? In my next post, some humble ideas.

There have been a number of articles and reports published in the past few weeks pointing toward the value of clear evidence of what works in the field of education.

The impetus of much of the education discussion seems to be, in the case of US-focused pieces, the announced intention of by the Obama administration to rewrite the No Child Left Behind law (NCLB) to include a greater variety of measures for assessing school achievement. Currently the law relies on student achievement scores in reading and math to determine a school’s score. Education Secretary Arne Duncan said the proposed accountability measures would “factor in student growth, progress in closing achievement gaps, proficiency towards college and career-ready standards, high school graduation and college enrollment rates.”

Speaking of Arne Duncan, the New Yorker published a profile of the education secretary in the February 1 edition (subscription only access). The piece touches briefly on Duncan’s proposed rewrite of NCLB discussed above and Duncan is quoted vaguely in the piece as saying that “NCLB had loose standards and tight prescriptions on what you have to do, and we want to flip that so instead we have very clear goals and a high bar but flexibility on how to get there.“ More interesting was the way the article highlighted the rift in education taking place right now between the “market forces” proponents (which Duncan largely supports) and the liberal traditionalists. The market reformers want to inject life and competition in the education space by offering more school choice and a broader range of extrinsic incentives for teachers and students (such as merit pay and student incentive programs). Liberal traditionalists are pretty much rallying around teachers’ unions and university education programs. This piece did not add any new information to that debate, per se, but it did inject a bit of reality into the market enthusiasm for charter schools and merit pay, namely by confirming that they are only “best guesses” at how to proceed. There isn’t a lot of evidence for any of them.

In the current battle for Race for the Top funding underway right now, wouldn’t a useful requirement of funding be that the state will conduct rigorous impact measurement?

In support of measurement, a December story on 60 Minutes highlighted Geoffrey Canada’s Harlem Children’s Zone and the Promise charter school associated with it. The Harlem Children’s Zone is an area in central Harlem in New York City where Canada has made health services, parent training, literacy tutoring as well as a charter school ingredients in a effort to bring a generation of Harlem kids to college. This program was set up to be measured, and Harvard’s Roland Fryer published a paper in November, the results from which were highlighted on the spot. His findings show that primary school students of Canada’s Promise Academy had the same achievement scores in math and reading as New York’s white students, while middle school students closed the gap in math and narrowed it by 50 percent in English. The problem, Fryer says, is he still isn’t sure why. Community investments alone cannot account for the difference, so the magic pill may lie in some or many aspects of the Promise School (longer days? limited summer vacation? Saturday lessons? motivated teachers? student pay-for-performance?). Solving the mystery is a critical question given that Obama and Duncan have signaled their desire to ‘scale’ the HCZ model to other localities.

Turning to the developing world, a new UNESCO report on education in the developing world shows a slow-down in getting more of the world’s children enrolled in school in keeping with the Millennium Development Goal of universal education. More concerning is the real issue—the kids aren’t learning whether enrolled or not. Tests show developing world kids consistently scoring well below age-related markers in the poorest countries. Again, researchers are investigating whether school choice for parents, in the form of the inexpensive private schools promoted by James Tooley, or teacher incentives, in the form of merit pay, can help lift of the quality of the education. In the case of the latter, two studies in India, one on incentive pay by USC’s Karthik Muralidharan and another on attendance incentives by JPAL’s Esther Duflo, showed positive impact.

This is a guest post from Jeff Raderstrong, who blogs at Change Charity. Jeff’s views are his own and do not necessarily reflect the views of Philanthropy Action.—eds.

Mario Morino’s recent column at Venture Philanthropy Partners’ blog underscores the concerns I have about focusing too much on donors in the effort to improve philanthropy. In an earlier post I argued that donors pushing for better measurement and accountability might not be as effective as non-profit employees pushing for those same reforms internally, but Morino takes it one step further and says external pressure might actually be detrimental.

Morino says that when dealing with measurement at non-profits, one should always ask “to what end?“ are those measures are being taken.  A focus on measurements is only beneficial if it helps the non-profit fulfill its mission. If non-profits become disconnected from the purpose of measurements, major problems can occur. “[I]f the metrics are overly simplistic and unmoored from mission, then organizations will go racing in the wrong direction. To paraphrase Yogi Berra, they’ll get lost, but they’ll be making good time.“ If a non-profit is told its funding will be based on external measurement indicators, the organization could spend more time worrying about showing results, rather than producing them. (See “teaching to the test,” No Child Left Behind Act, which Morino discusses.)

Therefore, Morino argues, measurements and changes based on those measurements should be directed internally. “If we were to take that nonprofit-centric—rather than funder-centric—approach, we would all have a much higher likelihood of achieving what we’re really setting out to accomplish. Nonprofit leaders would not be navigating with intuition alone. They would gain powerful tools to determine where they’re headed, chart a logical course, and course-correct when they’re off.“

I think Morino’s underlying assumption, which he never explicitly states, is that a lack of measurement does not necessarily mean a lack of results. There are a lot of non-profits out there that don’t have the capacity for social-outcome measurement and they shouldn’t be punished for that. If donors want non-profits to do measurement right, they will need to provide a lot of “encouragement and support” to allow non-profits to find the best ways to measure, along with appropriate adaptations to those measures.

Morino says this process will come over a long period of time and represent a dramatic shift in our thinking:“We’ve approached this challenge as if it’s about numbers when it’s really about changing cultures. Changing culture requires large and persistent investments of time, talent, and money.“

Update: I’ll be on Science Friday with Ira Flatow (NPR) at 2pm today, Jan. 22nd, discussing giving to Haiti and donor advice.

I’ve been pleasantly surprised to see a steady flow of high quality advice for donors responding to the Haitian earthquake. The New York Times profiled the efforts of two intrepid bloggers, Alanna Shaikh and Saundra Schimmelpfennig. Part of Ms. Schimmelpfennig’s effort has been to compile good advice from others, in addition to her own expertise, from around the world (if you haven’t seen the comprehensive advice compiled by Ms. Schimmelpfenning at Good Intentions Are Not Enough, it is highly recommended). Still, even she, I don’t think, has been able to keep track of all the advice columns and stories.

As I see more and more of these stories appear, I’ve begun to wonder: how will we know if this proliferation of good advice has had an impact on the Haitian relief and recovery effort? What metrics will tell us that donors to Haiti and the nonprofits working there learned the lessons of the tsunami, Katrina, and Nargis? I have a few ideas:

* the ratio of gifts-in-kind to cash donations is significantly lower
* the amount of money given to disaster relief organizations that is earmarked for Haiti is lower
* the percentage of funds given to organization that didn’t already have a long-term presence in Haiti is lower
* the percentage of funds given to disaster relief organizations (as opposed to long-term development organizations) is lower
* the number of “mission” trips to perform unskilled labor to “help” Haitians decreases dramatically
* giving to Haiti shows a “long tail”—in other words donations don’t abruptly fall off after the immediate crisis is over and donors give to recovery efforts for months into the future

Of course, all of these are comparative metrics. I’ve never seen the figures for other disasters which would make-up the baseline for such comparisons. Perhaps they are out there.

What other metrics should we be looking at to assess the impact of donor advice? Have you seen any data on these issues from prior disasters?

I’ll confess I’ve even begun to wonder if there is too much advice from too many sources out there. Would the message be heard more clearly if it was coming from so many places, and with so many variations, at once?

This is a guest post from Jeff Raderstrong, who blogs at Change Charity. Jeff’s views are his own and do not necessarily reflect the views of Philanthropy Action. Actually, we do in fact disagree with Jeff’s view, but we’re happy to have the debate.—eds.

Maybe it’s my homegrown American sense of individualism, or maybe it’s those NBC public service announcements, but for some reason, I’ve always thought that with the right information, people could accomplish anything. I’ve applied this logic to philanthropic reform: If people just had access to information about their favorite charity’s impact, or just simply requested that information, non-profits would succumb to the wishes of their funders and start evaluating and revising based on the effectiveness of their programs.

Philanthropy Action and other  blogs also operate—to varying degrees—under that assumption. But as Tim and Laura have written here recently, good evidence does not always get people to change their behavior. Examples of readily accessible information being ignored in favor of the status quo are too numerous: Climate change, fair trade, etc., etc.

The good information directed at donors about effective philanthropy may also be falling on deaf ears. Charity Navigator, one of the biggest charity evaluators out there, recently announced a complete overhaul to its evaluation system to one based on impact and effectiveness, which has the potential to offer good charity evaluation information to more people than at any other point in history. However, based on my rough and generous calculations, Charity Navigator has a market penetration of about 2% of all US donors. I worry that most of its new information will not be accessed, or that if it is, it will be ignored.

So focusing on giving information to donors—especially casual donors—might not create the industry-wide reform needed. Instead, if we want non-profits to think about effectiveness and work off of accountability, information should be directed at the non-profit organizations themselves. Many organizations, like the Independent Sector, the Social Impact Exchange and the Acumen Fund are showing non-profits that operating under effectiveness-based measurements is better for them and how it will ultimately better serve their clients.

Ultimately, pressure from both sides is needed. But since most donations come from individuals and not a small group of major donors, expecting the masses of donors to pressure non-profits to change might not be feasible. Instead, reform could come sooner if non-profits begin to realize that measures of accountability will create better programs and increased capacity to leverage donations. If more outreach and information about reform is directed towards those working in non-profits, organizations might change before all their funders know why it’s necessary.