Judges’ Reflections

Some 100&Change judges provided feedback to individual applicants as part of the evaluation and also to MacArthur about the judging process and the structure of the competition. We will collect more formal feedback from the judges as MacArthur conducts a formal evaluation of 100&Change, but these informal, unsolicited observations are already informing our thinking about the next iteration of the competition. Often, these suggestions seem applicable not only to 100&Change, but to any broad, open, competitive call.

We initially asked each judge to read at least 10 proposals, but some read less and some read 15 or 20. Hence, each judge saw a tiny slice of the pool of 801 submissions, less than 2.5 percent. And, we heard from only a sample of our 413 panel of wise heads, so these early comments must be considered in that context.

A common theme: we underestimated the time required to read and score the proposals. Others worried whether they were applying the criteria fairly:

I wonder if I added a bit of my own values to the criteria in that I was also looking for a proposal that was transformative and scalable and worthy of $100 million. …I am also intrigued by the reviewer calibration, though after reading my proposals I realize that this may be challenging given the small number of proposals we each read. From my scores, you could either conclude that I am a very tough scorer or that I had very weak proposals. — Steven Bucksbaum

Several judges wanted the opportunity to provide a summary assessment of the proposals in addition to the assessment based on the four criteria. They likely would have agreed with Carl Pabo’s suggestion:

It might be interesting, perhaps after each judge has filed his or her more detailed review, to come back with just one overarching question about each application and ask:

"Would you be excited and proud if MacArthur spent $100 million to fund this program?"

And perhaps you could give people three choices:

Yes, I’d feel good
I’d have mixed feelings, but it seems to be in a plausible range
No, this would feel odd

Judges felt that, in many cases, they lacked sufficient information to evaluate team capacity:

I was worried that a critical question was the competency of the proposer to carry out the proposal. We had little way, outside of the proposal itself, to make such evaluations. I did a little bit of exploration on the internet to see whether or not there was the type of ancillary evidence that would show such competence. In several cases the lack of information made me dubious, although the projects themselves sounded spectacular. — George Akerlof

Overall, I think that a question that needs to be addressed in all of these proposals is capacity and infrastructure, even though an applicant may have an innovative and bold idea, a $100 million grant may not be effective without adequate structure, financial acumen, and project management capacity. — Ellen Barry

Once again, Carl Pabo offered a very specific suggestion about how to address capacity before individuals submitted proposals:

Finally, there were a number of cases—perhaps 25 to 30 percent in which groups applied without having any of the kind of substantive background, experience, outside validation, or proven initial track record that would be expected of any program that is a serious contender for funding at the $100 million dollar level.

If you run the program again in a few years, I wonder: Can you try adding some paragraph, some warning that would—in effect—encourage everyone to pause for a minute and try to see their own efforts in a slightly more critical, impartial way? In other words, could you, without discouraging any of the best groups, help cut down on the number of people who seem to have applied almost as if they thought they were buying some kind of lottery ticket? It could read:

Over the years, as we have had the privilege of helping and supporting so many passionate, talented groups, we have seen a few characteristic patterns of “organizational development” that may be worth bearing in mind as you consider submitting an application. We welcome any application, but—as a practical step—you might want to pause and think for a moment about the level of competition that may be involved here and about potential expectations from the reviewers. As you will know from your own experience: developing an organization takes time. And most groups—if you really want to be competitive for funding at the $100 million dollar level—you will be applying with a team a) that has been in place for three to five years b) has already worked with funding at a level of several million dollars per year c) already has some demonstrated way of making a completely outsized impact on the world.

Another category of comments concerned MacArthur’s choice to offer a single grant of $100 million. One judge recommended two competitions, one for organizations based in emerging economies and one for organizations based in high-income countries. Another suggested support for smaller organizations that might lack the resources to conduct literature reviews to provide the evidentiary base for their work or whether we might consider grants of smaller increments.

A $100 million award raises interesting issues. Almost all of the proposals I read consisted of scaling up existing smaller-scale efforts. The best one I judged established the efficacy of an existing program and laid out a very systematic approach to step-by-step, check-as-you-go scale up efforts. But by their very nature, these kinds of proposals would produce benefits more or less proportional to the money provided to them—$35 million would provide about one-third of the benefits of $100 million. — Greg J. Duncan

As this sample of comments illustrates, our judges read applications with empathy and diligence. We are not yet ready to offer responses to these specific suggestions regarding the competition. However, the judges will be an important source of information as MacArthur refines and improves 100&Change. We greatly appreciate their generous donation of time and energy to give careful consideration to each valid submission.