RCTs: Really Crap Things or Really Cool Things?

Credit: http://cplanicka.blogspot.com.es/

I’ve spent the last couple of months enrolled on two MOOCs (massive open online courses), ‘The Challenges of Global Poverty’, and ‘Evaluating Social Programs’, both effectively run by J-PAL, a research centre at MIT. I know what you’re thinking “Wow there Charlie, this year abroad’s taking you off the RAILS”. Well screw you. Anyway this has meant three things: 1) I’ve been exposed to a LOT of chat about Randomised Control Trials (RCTs); 2) I find the meme on the right much funnier than it actually is; and 3) I kind of wish I were an economist.

I want to talk about the first of those three things. Specifically: what are the downsides of RCTs?

First what they are (and I’m going to try to do this in my own layman words): a randomised control trial is a particular method of impact evaluation for an intervention (eg in development). It gets together a load of people, randomly assigns half of the group (the treatment group) to receive the particular intervention (e.g. access to microfinance) and half (the control group) to not receive the intervention, then at the end of the program measures the difference between treatment and control groups with respect to an appropriate indicator (e.g. income) to get the ‘impact’ of the intervention. Basically, the fact that the two groups were randomly chosen means that they were statistically the same, and so the cause of any difference in outcome between them at the end must be the intervention. Phew.

They’re argued to be pretty good at demonstrating the effectiveness (or not) of programs because they allow you to get to the heart of the causality, getting us closer to the ‘impact seeker’s Nirvana’.

Now after over two months of Esther Duflo and her ‘randomista’ (proponents of randomisation) babes enlightening me about all the cool things they’ve found out using their economist ‘gold standard’ tricks (like if you bribe them with dal, mothers are more likely to get their kids vaccinated. Or providing teenage girls with school uniforms can reduce rates of teen pregnancy more than sex-ed programs), I was ready to read some damning critique. But although RCTs to take a lot of flak, much of the arguments against seem weirdly bad. There’s a bit of a non-debate going on in many quarters.

1. Here’s one: it’s wrong to play God and experiment on people.
Reply: this one’s crap. As Howard White says, “all new policies are essentially experiments”. Now if we knew all the answers in development, then doing RCTs would be, at worst, expensive and useless. We don’t, so trialling and evaluating programmes as best we can to find out what does seems like the opposite of wrong.

2. And another, slightly better: RCTs are immoral because you’re rolling a dice to see who gets the bednet and who doesn’t.
Reply: you always have to choose who receives the intervention and who doesn’t; doing it explicitly and by chance its better than doing it implicitly and by virtue of the distance of the village from the nearest four-star hotel. RCTs never lead to “fewer people getting a service than they would if we haven’t been working on the evaluation” as Rachel Glennester, a randomista, puts it. Also, when you do an evaluation you don’t know whether the intervention works, so you might be better off not receiving it. Which is a bit of a backhanded benefit, but anyway.

Wait, so if the arguments against are rubbish, is this the answer for finding what works in development? Hah no, cause development is BORING, as thinkers ranging from Francisco Toro to my younger brother have argued (with slightly different meanings), and so there’s never one exciting answer, always lots of prosaic partial answers. As Lant Pritchett expresses it “RCTs are one hammer in the development toolkit and previously protruding nails were ignored for lack of a hammer, but not every development problem is a nail”. Some more thorny problems:

3. External validity: you can’t generalise from your RCT because the context matters. Owen Barder: “[in] the obvious example of the de-worming program, it clearly makes sense in communities that suffer from that kind of worm. But you clearly couldn’t generalize to communities where there aren’t worms”. ‘Solutions are slippery’ and you have to acknowledge the ‘hyper-dimensionalityof the design space’. Fuck. Sounds difficult and is. In fact so difficult that they do RCTs to see if the RCTs can be generalised.

4. Too much what not enough why: learning the impact of an intervention doesn’t tell you why it works or not. Also true.

5. Ironically, there is little rigorous evidence to suggest that rigorous evidence is used in policy. Lant Pritchett, chief development troller describes RCT-use as a ‘faith-based activity’. Meanwhile Philip Krause makes the obvious but valid point that “today’s rich countries didn’t get rich by using evidence systematically”.

6. Leaving out the big questions: you can’t run an RCT on whether aid works. You can’t use an RCT to determine what stimulates economic growth. You can’t use an RCT to test the effects of fixed vs. flexible exchange rates. Randomistas would accept this, and perhaps suggest that big ‘macro’ questions are made up of little ‘micro’ questions which they can answer. Debatable. But even WORSE, RCTs can’t answer many ‘micro’ questions either. Ben Ramalingam argues that many of the thorniest problems facing the world today are problems of ‘organised complexity’. With these, all the randomista assumptions about the intervention being the only difference between experiment and control group, linear cause and effect, etc. just don’t hold.

Basically, RCTs are good – so definitely read Poor Economics -... but not that good. According to the hype cycle, they are currently sunning themselves at the Peak of Inflated Expectations, soon to tumble down into the Trough of Disillusionment. Development is boring and there are no quick answers. Hooray!

The hype cycle, credit Gartner

DISCLAIMER: Having said all this, the above issues are slightly more nuanced than a short, deliberately-flippant blogpost would allow. Here are some extra resources (pick a couple) in the unlikely event you want to find out more.

A nice summary piece on RCTs in Slate

An in-depth document explaining RCTs by International Initiative for Impact Evaluation 'Big Cheese' Howard White

Some criticism of RCTs in the New York Times and a response from Jessica Goldberg

A development drums podcast on ‘Randomized Evaluations’

Some great blogposts questioning the relevance and impact of RCTs/raining on the parade

A more detailed look at the ethics of RCTs on the World Bank blog. Plus responses #1 #2 and #3

The Rebirth of Education (part 2): best bits

In part 1 I gave a quick run-down of Lant Pritchett's 'The Rebirth of Education'. In six words, learning not schooling, starfish not spider. Now I thought I'd highlight some important and/or interesting take-home messages. They mostly buzz around the same theme but hopefully will crystallise things a little.

On the problem:

1) You can't solve the education problem if you don't understand it.

We need to grasp that schooling systems adopted in developing countries were not designed to enable children to meet learning goals, so more of the same (more inputs alone) doesn't cut it. As with an addiction to Twitter, so with education policy, the first step is admitting you have a problem. "You cannot search [for solutions] if you are convinced you have already found what you need" says Lant. This also means that arguments on the public-private axis (as fun as they are) are also tangential to the debate in the developing country context.

2) 'More inputs help a bit so we might as well carry on doing that for now' = not advisable 

 “Taking a placebo can be dangerous if it prevents you from seeking out the right diagnosis”. Probably enough said there.


3) Western education experts need to eat some humble pie

Usefully-named 70s band Humble Pie

"No living Western education expert has led, participated in, or lived through a truly major national improvement in measured student achievement". All currently high scoring educational systems in OECD countries already had high-scoring systems 40 years ago, so we 'over here' don't seem to know how to transform an education system either. More generally the problems for education in developing countries are (unsurprisingly) different to those in developed countries, so even with a system proven to be successful in the Netherlands, for example, we may not be any closer to our goal.





4) This isn't technical, it's incentives, it's political

The problem is in the dynamic of the system and the incentives it produces, not just a lack of a certain missing X. This reminds me of a conversation I had with a colleague when working in a school in Uganda last summer, when it dawned on us that everyone was 'playing their role' more or less effectively (teachers mostly present, pupils sat behind desks, adequate classrooms, timetable, register etc) but a lot of the time, nothing was actually happening. There was no learning. Pritchett's cover shows a child walking through a doorway of a façade of a school with nothing behind and this captures the point perfectly. Translating this into practical advice for someone who works for a charity trying to improve schools in developing countries (like me), when you think you've found the perfect initiative for the school, you need to ask yourself two questions: why isn’t this being done already? And why will it be done that way in the future? Incentives. And panning back out to the level of the whole system, this means politics: switching to a starfish system means ceding power and giving up central control to some measure, and there lies the difficulty.

Towards a solution...

5) There is no blueprint and there is no silver bullet

If what we are trying to do is learn the principles of design rather than how to design a particular house (Lant's metaphor), and we take into account architecture degrees are like six years (me stretching Lant's metaphor), it's clear there is no quick blueprint for success. Even within his six key characteristics for a successful system, you need these all functioning together. If you look at point 2, 'locally-operated' and think that the panacea for poor education systems is decentralisation, then you are likely to end up with a decentralised poor education system. You need all of these characteristics together, with the details right. Then adding in the inevitable caveat that political, social and educational context will determine the precise formula, and the surreal, incomprehensible bit of Zen teaching Pritchett adds in "If you should meet Buddha on the road, kill him"(?!), it's clear that there is no one formula for success.

*searching for something positive...*

6) There's loads of innovation at the micro-level, so change is possible

Particularly for those working on individual schools rather than an entire education system, take inspiration from the fact that there are plenty of actors doing great, innovative work in difficult contexts. We know that some interventions (remedial classes, curriculum reform, and others) are really effective at improving learning. The problem is more that the spider system often prevents these innovations from scaling-up.

And the irritating, super-keen development student's choice...

7) Systems-thinking is the way forward

Complex adaptive systems did this

Trying to take heed of the development ideas hype cycle and not explode with wonk-y excitement, but systems-thinking, and particularly complex adaptive systems theory, is the future. Lant has a great passage on why we're crap at thinking in terms of systems rather than agents (which is also an example of his excellent prose style) but all the cool kids in policy are talking about systems-thinking and I'll soon be blogging about it (once I've read this and this and this and this and this). Getting back to the concrete example of education, thinking in terms of systems explains why absent teachers and obstructive teachers' unions don't mean that teachers are the baddies and all we need to do is make teachers be less rubbish. If the system is set up to treat teachers like workers not professionals, and fails to provide the right incentives, then the outcomes will be inevitably be disappointing.

In conclusion, read the book because it's great and will change the way you think.

The Rebirth of Education (part 1): Spider vs. Starfish

As the call comes in for more money for education in developing countries, so too, does the news that current performance on education is appalling . Shockingly appalling.  If you need any more convincing, an Educational Initiatives survey covering 74% of the Indian population found that schoolchildren’s performance on conceptual questions is often worse than random guessing. Meanwhile at the current rate of improvement, Indonesian schools will take more than 1000 years to reach current OECD-country levels of quality. Other countries, at the current rate, would never get there.

Lant Pritchett’s book ‘The Rebirth of Education’ (in which a whole chapter is dedicated to crushing our spirits about education in developing countries with statistics like the last two above) attempts to explain this dire situation and what to do about it, and in doing so he kind of blew my mind. I started reading the book in part to supplement the lit. review I am currently doing for Education Partnerships Africa in which we initially aimed to find a set of basic inputs (in terms of infrastructure and resources) which would add up to a ‘good school’, and thus produce a schematic ‘blueprint’ for our partner schools. It turns out it doesn’t work that way.

Governments across the world are great at getting children physically into school. The population of labour force age in the developing world has now completed three times more years of schooling than in 1950 when 60% had no schooling at all. And in Ghana, for example, the average 7.8 years of schooling in 2010 for each child was not attained by the UK until 1970. The problem is, that the type of system which is great at sorting the logistics of schooling isn’t good at ensuring children learn. The problem is the system, meaning that 'more of the same' (= more schools, more books, more desks, more computers) won't solve the problem. 

These centralised, top-down, bureaucratic, inflexible, unresponsive, closed, ‘spider’ systems which exist across the world arose out of a mixture of demand, driven by a modernising economy (education has good returns in terms of income), but more from a drive for the ideological control of socialisation (indoctrinate those kids) and isomorphic mimicry (if we acquire the outer trappings of a responsible, well-functioning state then we seem more legitimate).  So the whole dynamic of the system has nothing to do with effective learning, and more to do with political patronage and rent-seeking in many cases; even if you assume that intentions are benevolent (strong assumption) then the spider systems just aren’t set up to deliver good education*. Pritchett even finds that ‘spider’ education systems in developing countries are value-subtracting. That is to say, as well as being more expensive than alternatives, spider systems deliver absolutely worse education than if the same teachers were teaching the same kids outside the system.

Kill that spider!

So what beats a spider in rock-paper-scissors? A starfish, you got it. Whereas in a spider system, all information across the web filters up to the brain at the centre of the web, in a starfish there is no centralised nervous system; each part is constantly taking in and processing information, and the overall movement of the starfish comes about as an aggregate of tiny little movements of each part. Translated back into education, this means a system which is

• Open:  many different types of school and approaches allowed
• Locally-operated: local actors have the autonomy to operate, explore and discover innovative approaches
• Performance pressured: schools are accountable on the basis of actual, measurable learning outcomes not just schooling input
• Professionally networked: a professional ethos among educators and organisational learning across the system facilitated and promoted
• Technically supported: schools are equipped with the capabilities to succeed
• Flexibly financed: finance follows students and performance with local control

Happy cause he's learning

So there we have it: Pritchett comes up with the solution? Erm, kind of. He does give a few examples of starfish systems (the US/UK higher education system; the International Baccalaureate system; Brazil’s basic education system) but this is no concrete blueprint like I was hoping for. In one of his many analogies (Pritchett’s way of explaining things, both in the book and in podcasts I’ve heard, is brilliantly lucid and witty) he likens what he is doing as not showing how to build a specific house, but giving the principles of design. Having said this, he could definitely have expanded more on the positive, starfish-side of things (five of six chapters are dedicated to taking down the current spider system), and I’m sure research/policy discussions will push on in a more practical direction, particularly as we reach 2015 and the post-MDG apocalypse.

I was going to follow that up with my impressions/key take-home messages, but this has got really long. Still haven’t got the succinctness of a pro-blogger. So I’m going to call this ‘part 1’ and write an accompanying ‘part 2’ on the ambitious assumption that people would actually read two consecutive posts of mine. In the meantime, to consolidate, gorge yourself on review/blog/blurb/podcast about the book from people more intelligent than I.

*if you’re wondering why spider systems in developed countries do a good job, Pritchett’s explanation is that they took over pre-existing local starfish systems