RCTs: Really Crap Things or Really Cool Things?

Credit: http://cplanicka.blogspot.com.es/

I’ve spent the last couple of months enrolled on two MOOCs (massive open online courses), ‘The Challenges of Global Poverty’, and ‘Evaluating Social Programs’, both effectively run by J-PAL, a research centre at MIT. I know what you’re thinking “Wow there Charlie, this year abroad’s taking you off the RAILS”. Well screw you. Anyway this has meant three things: 1) I’ve been exposed to a LOT of chat about Randomised Control Trials (RCTs); 2) I find the meme on the right much funnier than it actually is; and 3) I kind of wish I were an economist.

I want to talk about the first of those three things. Specifically: what are the downsides of RCTs?

First what they are (and I’m going to try to do this in my own layman words): a randomised control trial is a particular method of impact evaluation for an intervention (eg in development). It gets together a load of people, randomly assigns half of the group (the treatment group) to receive the particular intervention (e.g. access to microfinance) and half (the control group) to not receive the intervention, then at the end of the program measures the difference between treatment and control groups with respect to an appropriate indicator (e.g. income) to get the ‘impact’ of the intervention. Basically, the fact that the two groups were randomly chosen means that they were statistically the same, and so the cause of any difference in outcome between them at the end must be the intervention. Phew.

They’re argued to be pretty good at demonstrating the effectiveness (or not) of programs because they allow you to get to the heart of the causality, getting us closer to the ‘impact seeker’s Nirvana’.

Now after over two months of Esther Duflo and her ‘randomista’ (proponents of randomisation) babes enlightening me about all the cool things they’ve found out using their economist ‘gold standard’ tricks (like if you bribe them with dal, mothers are more likely to get their kids vaccinated. Or providing teenage girls with school uniforms can reduce rates of teen pregnancy more than sex-ed programs), I was ready to read some damning critique. But although RCTs to take a lot of flak, much of the arguments against seem weirdly bad. There’s a bit of a non-debate going on in many quarters.

1. Here’s one: it’s wrong to play God and experiment on people.
Reply: this one’s crap. As Howard White says, “all new policies are essentially experiments”. Now if we knew all the answers in development, then doing RCTs would be, at worst, expensive and useless. We don’t, so trialling and evaluating programmes as best we can to find out what does seems like the opposite of wrong.

2. And another, slightly better: RCTs are immoral because you’re rolling a dice to see who gets the bednet and who doesn’t.
Reply: you always have to choose who receives the intervention and who doesn’t; doing it explicitly and by chance its better than doing it implicitly and by virtue of the distance of the village from the nearest four-star hotel. RCTs never lead to “fewer people getting a service than they would if we haven’t been working on the evaluation” as Rachel Glennester, a randomista, puts it. Also, when you do an evaluation you don’t know whether the intervention works, so you might be better off not receiving it. Which is a bit of a backhanded benefit, but anyway.

Wait, so if the arguments against are rubbish, is this the answer for finding what works in development? Hah no, cause development is BORING, as thinkers ranging from Francisco Toro to my younger brother have argued (with slightly different meanings), and so there’s never one exciting answer, always lots of prosaic partial answers. As Lant Pritchett expresses it “RCTs are one hammer in the development toolkit and previously protruding nails were ignored for lack of a hammer, but not every development problem is a nail”. Some more thorny problems:

3. External validity: you can’t generalise from your RCT because the context matters. Owen Barder: “[in] the obvious example of the de-worming program, it clearly makes sense in communities that suffer from that kind of worm. But you clearly couldn’t generalize to communities where there aren’t worms”. ‘Solutions are slippery’ and you have to acknowledge the ‘hyper-dimensionalityof the design space’. Fuck. Sounds difficult and is. In fact so difficult that they do RCTs to see if the RCTs can be generalised.

4. Too much what not enough why: learning the impact of an intervention doesn’t tell you why it works or not. Also true.

5. Ironically, there is little rigorous evidence to suggest that rigorous evidence is used in policy. Lant Pritchett, chief development troller describes RCT-use as a ‘faith-based activity’. Meanwhile Philip Krause makes the obvious but valid point that “today’s rich countries didn’t get rich by using evidence systematically”.

6. Leaving out the big questions: you can’t run an RCT on whether aid works. You can’t use an RCT to determine what stimulates economic growth. You can’t use an RCT to test the effects of fixed vs. flexible exchange rates. Randomistas would accept this, and perhaps suggest that big ‘macro’ questions are made up of little ‘micro’ questions which they can answer. Debatable. But even WORSE, RCTs can’t answer many ‘micro’ questions either. Ben Ramalingam argues that many of the thorniest problems facing the world today are problems of ‘organised complexity’. With these, all the randomista assumptions about the intervention being the only difference between experiment and control group, linear cause and effect, etc. just don’t hold.

Basically, RCTs are good – so definitely read Poor Economics -... but not that good. According to the hype cycle, they are currently sunning themselves at the Peak of Inflated Expectations, soon to tumble down into the Trough of Disillusionment. Development is boring and there are no quick answers. Hooray!

The hype cycle, credit Gartner

DISCLAIMER: Having said all this, the above issues are slightly more nuanced than a short, deliberately-flippant blogpost would allow. Here are some extra resources (pick a couple) in the unlikely event you want to find out more.

A nice summary piece on RCTs in Slate

An in-depth document explaining RCTs by International Initiative for Impact Evaluation 'Big Cheese' Howard White

Some criticism of RCTs in the New York Times and a response from Jessica Goldberg

A development drums podcast on ‘Randomized Evaluations’

Some great blogposts questioning the relevance and impact of RCTs/raining on the parade

A more detailed look at the ethics of RCTs on the World Bank blog. Plus responses #1 #2 and #3

Down and Out in Paris and London from Madrid

General thoughts as a piece of writing: Down and Out in Paris and London, Orwell

A strange read: I’m not quite sure what genre it was, somewhere between travel writing (but instead of moving geographically, moving economic situation and social class), short story, autobiography, and anthropological study. Orwell dons the 'beggar's cloak' (there's something slightly uncomfortable ethically now I reflect - using people's lives as material before hopping back out of poverty) and spends some months living the life of someone at the bottom of the pile, economically and socially-speaking. It’s written in such a way, with a cast of characters that seem more literary than real that it’s hard to believe that it is (at least partly) non-fiction. There is ‘Charlie’, who discourses on his discovery of ‘love’ with a terrified prostitute, and ‘Bozo’ the astronomer-screever (street artist), among many, many others. Meanwhile the description of life as a plongeur (the bottom rung of employee in a Parisian hotel-restaurant), the colourful and slightly disorientating account of Saturday night at Orwell’s preferred Parisian tavern, and the Russian secret society operating out of a laundrette similarly test the suspension of disbelief. Which also made it incredibly readable and compelling; I gobbled it up in about 4 hours.

Poverty 'over here'

The cover of my own 1984 edition

Anyway, we’re all here to witness my journey of intellectual enlightenment on development matters. Or for The Thick of It lovers, in Phil’s words, “my quest to become the man I know I can be". And in that context, it’s a useful tonic as someone who spends a fair amount of their life reading about and acting on poverty located outside their own society. It’s a cliché but poverty is a massive problem in ‘developed’ countries too, perhaps even reaching levels of a 'humanitarian crisis', and though it was published in 1933 the number of homeless people I awkwardly avoid eye contact with on a daily basis in London or Madrid (especially) suggests it’s still unfortunately relevant. On top of that, not only does the whole story take place in my own geographical space, the language and style of the writing means that it’s coming from my own cultural and class world. It shouldn’t be this way, but inevitably as a card-carrying public schoolboy (sounds awful doesn’t it?), reading an account of extreme poverty as ‘discovered’ by an ‘Etonian’ about 90 years ago is even more potent for me in some ways than having lived among extremely poor people in a rural area of Uganda for two months. The fact that writing all that has made me feel uncomfortable is probably another credit to the book. Too many middle class white people with no idea what poverty is like in ‘development’. ANYWAY.

A simple representation of poverty

Leaving aside nights spent in a postcolonial-theory-induced cold sweat, Orwell is super eloquent at illustrating the daily hardships and impossible decisions that people without money have to go through. It’s kind of the literary counterpart to reading Poor Economics which similarly aims to get away from grand theories and down to the realities of poor people, whatever your opinion of RCTs. We see the reality of the people on the other side of the wall in the fancy restaurant, who are basically invisible in contemporary media or politics (though this is a good article if you read Spanish). Going without food for days at a time feels “as though all one’s blood had been pumped out and luke-warm water substituted”. Life moving from one lodging house to another is “a squalid, eventless life of crushing boredom.”

One especially relevant point for this blog is the indignity of poverty and the power relations charity hides: on more than one occasion Orwell and his homeless friend Paddy are forced to accept a religious service in exchange for food, which he describes as a humiliating experience. In development land, it kind of links into Hattori's idea on giving as a means of legitimating the ethical hegemony of capitalism if that isn't too pretentious to mention (if you can stomach it, gated paper is here). Money, and especially charity, gives power at a very basic level. Orwell notes,“[i]t is curious how people take it for granted that they have a right to preach at you and pray over you as soon as your income falls below a certain level”. The parallels with international development here are obvious. Structural adjustment imposed by the IMF and World Bank in the 1980s and 1990s would do nicely, or, playing devil’s advocate you might well draw parallels with some donors’ recent suspension of aid over Uganda’s anti-homosexuality bill. Don’t quote me on that.

Though Orwell also describes little acts of resistance which are kind of brilliant. At one point a religious group spontaneously enters a hostel to give a service to the lodgers, presumably to ‘save their souls’. “No more notice was taken of them than f they had been earwigs”. Taking up the structural adjustment parallel again, you can kind of see this in the way that some developing-country governments made a complete mockery of the imposed conditionality. Easterly's great for this: in The Elusive Quest for Growth, he states that Pakistan was given no less than 22 adjustment loans conditional on reducing its budget deficit... which was steady at 7% of GDP throughout. Van der Walle also depicts this amazingly.

Anyway before I end up summarising the entire book, the message in general is (if there is one) that the poor are people like you and me. Orwell says it better: “the rich and poor are differentiated by their incomes and nothing else, and the average millionaire is only the average dishwasher dressed in a new suit”. There’s a really poignant illustration of this (can’t help myself) quite late in the book, when Orwell, clearly in low spirits, suggests to Bozo, a disabled homeless person that “[i]t seems to me that when you take a man’s money away he’s fit for nothing from that moment”. And Bozo replies with moving defiance, “You’ve just got to say to yourself, ‘I’m a free man in here’ – he tapped his forehead – and you’re all right”.

The broader picture

Oh and Orwell is humble. Self-deprecating (British) and humble. And that’s the beauty: that there is no stylised, trite narrative or direct attempt at mobilisation, just a plain presentation of experiences. The political implications are left for the reader to decide. It starts without back-story – “The rue du Coq d’Or, Paris, seven in the morning.” – and ends just as simply -“My story ends here. It is a fairly trivial story”. There’s definitely a moral to be drawn in the development world: the importance of decentring the role of ‘outsiders’ in defining the issues. Actually quite a nice illustration of this in this interview with Bill Easterly (about two-thirds of the way through I think).

Having said that, Orwell does venture some tentative opinions about the underlying dynamics of poverty and the injustice of the system. His explanation of the utter social uselessness of a plongeur and of fancy restaurants and hotels, for example, is persuasive (all that strife for a “cheap, shoddy imitiation [of luxury]”). In general the attack on the materialism and superficiality of society is probably even more relevant now than then, even if the politics then were different, as we now unfortunately don’t have any politics in the UK at least ("Q&A centred on people lamenting the state of the British Left or even whether such a thing still exists"). But the Marxist-inspired ‘keeping-the-mob-occupied-for-fear-of-revolution’ argument was probably more relevant in the 1930s, so I won’t dwell on the theorising too much, be cause neither does Orwell – just “a sample of the thoughts that are put into one’s head by working in a hotel”. Humble.

Much to think about, and development-related too, without there being a single line of argument. It’s a great read, and I’ll definitely be moving on to The Road to Wigan Pier and Homage to Catalonia. I’ll leave you with the man himself for a modest conclusion:

“I shall never again think that all tramps are drunken scoundrels, nor expect a beggar to be grateful when I give him a penny, nor be surprised if men out of work lack energy, nor subscribe to the Salvation Army, nor pawn my clothes, nor refuse a handbill, nor enjoy a meal at a smart restaurant. That is a beginning.”

[note: I feel obliged to mention that Orwell is weirdly racist and anti-Semitic for someone I think of as progressive, but I choose not to dwell for the purposes of this blog]

Are the extreme poor hungry?

An obvious question maybe. The WFP reports that there are 842 million undernourished people in the world today. Policymakers, too, seem to think so: at Davos last month global leaders committed to the zero hunger challenge. MDG 1 refers to 'eradicating extreme poverty and hunger'. Ayuda en Acción also have a cool campaign on hunger right now which reminds me of Where the Wild Things Are. So the extreme poor are hungry, right?

But are the extreme poor hungry?

This question is the the central problem posed by Banerjee & Duflo (B&D) in Chapter 2 of Poor Economics, and of the second week of the 'The Challenges of Global Poverty' MOOC of theirs I'm currently taking. So, inspired by Development Intern's weekly report on Sachs' rival course on Sustainable Development, I'm going to do something similar, hopefully mashing up course material with ideas from their book and other sources sufficiently to sidestep anticipated legal issues (it's a free course available to anyone so not sure 'revealing course content' has any meaning but whatever I'm a coward).

Getting back to the question, B&D suggest that we could be looking at a food-based 'poverty trap'. I.e. poor people don't have enough food to be productive, meaning they earn less and have less money to buy food, and so the cycle continues. Meanwhile someone with more money can buy more food, be more productive, and thus escape from poverty. In the class B&D use concepts like the S-shape graph and capacity curves and elasticity so you feel like a real economist: it's fun.

But if we actually look at the actual behaviour of the poor in developing countries, we get a story different to the WFP and poverty trap narrative. B&D find that in Udaipur, the poor could spend up to 30% more on food than they actually do by cutting expense on festivals, alcohol and tobacco. Another study cited by B&D found that a 1% increase in income only translates into a 0.67% increase in food expenditure. Significance: there's room for the poor to be buying more food. A further study in China even found that when the price of rice declined, people ate less rice (that is to say, fuck you conventional economics). In general, Deaton and Dreze find that poor Indians have been eating less and less over the last few decades while the percentage of people complaining that they do not have enough food dropped from 17% in 1983 to 2% in 2004. Not what you would expect from hungry people stuck in a food-based poverty trap. So maybe poor people in general have enough food. As B&D put it, “most adults, even the very poor, are outside of the nutrition poverty trap zone”. We can offer some explanations - technological change means that people aren't doing as much physical labour and so require less food; people are healthier so lose fewer calories to hungry parasites or to fighting illness.

But the Freakonomics-style journey doesn't end there, because it's not just that the WFP is totally wrong and everyone is fine food-wise. Roughly half the children under five in India are stunted, a horrific stat which suggests something is going seriously wrong with their nutrition. So, skipping a few steps, we arrive at our 'solution': it's quality not quantity. When 40% of pregnant women are anaemic according to the WHO, it doesn't matter how much rice you give them, they need iron. And this is particularly bad in India because many people are vegetarian and the staple is rice (which actually inhibits iron absorption) creating a perfect, iron-free, storm. The results of addressing nutrient deficiencies are huge - B&D cite a study which found that deworming children for two years (equivalent of giving them the nutrients and calories taken by the worms - and yes it does work like that) led to a yearly increase in income of 20% and a lifetime gain of $3269, compared to a cost of $1.36 per deworming treatment. Bam.

Sweet potatoes: yummy and nutritious

So we need to get kids dewormed (already happening in Kenya), nutrients to pregnant women and to young children, because that's where the real 'hidden hunger' is, say B&D. Another nail in the coffin of USAID's food aid policy based on getting US-produced staples to supposedly needy countries (not that it needed another one). And bad news too for the Campaign for Boring Development with its focus on increasing incomes: money isn't the issue here, so having more of it won't make a big difference. Fortified fish-sauce is already cheap and within reach of the poor in Indonesia (relative to its benefits and average income). Instead we need to directly target mothers-to-be during pregnancy with fortified foods, perhaps sprinkle children's food at school with micronutrient-rich salts, and more sustainably, develop nutritious foods that people actually want to eat (see DFID bloggers' on exactly this topic.
Orange fleshed sweet potatoes and iron-rich beans are where we're heading apparently).

The most interesting part of this I've left until last. If the benefits are so great and the costs so small, why aren't poor people doing this anyway? And this is where B&D are at their best, taking us away from high-level, often introspective debates driven by Western academics (whatever its merits, I still find the fact that Nina Munks just wrote a book on Jeff Sachs kind of surreal) and towards the realities of the poor. As they drily put it, “the natural place to start to unravel the mystery is to assume that the poor must know what they are doing”. Unpacking all the reasons why the poor might not take to foreign experts' advice would, and probably will, take another blog post (think mistrust of the state and NGOs, difficulty of verifying supposed benefits, human psychology). But one aspect of the explanation becomes obvious when you think of the extreme poor as normal people. Because people aren't machines made to maximise their productivity or long-term income - I'm sure I'd be better off productivity-wise giving up coffee, sugar and alcohol but I'm not going to. Now imagine you work a lot harder than I do, and on top of that don't have the internet, books, podcasts and a load of leisure time to spend with friends, you might be justifiably tempted to spend your money buying tastier but less nutritious food, or even saving up for a TV, instead of investing in things that might, in the long-term, be objectively 'beneficial'. B&D quote George Orwell's The Road to Wigan Pier:

“When you are unemployed, you don’t want to eat dull wholesome food. You want to eat something a little tasty”.

So a couple of conclusions from this week's class: 1) quality not quantity. And 2) the extreme poor in developing countries know what they're doing more than we give them credit.

Some MOOCs to be getting on with

It's at this time of year that many people seek to overburden their lives with unnecessary and unrealistic aims and projects. I am like any other in this respect, and have signed up to two of the MOOCs (Massive Open Online Courses) listed below - Banerjee and Duflo, of Poor Economics fame were far too much for me to resist, and 'The European Union in Global Governance' is far enough away for me not to care about what I might be doing then. Not sure what I make of MOOCs as a concept yet (see here and here and here for some opinions on them) but it'll be interesting to see what they're like whilst brushing up on my global poverty and global governance knowledge at the same time.

Anyway here are five MOOCs on development-related subjects starting in the near future, along with brief information about their content, duration, conveners, etc. Oh and thanks to Anna Spinks for the inspiration for this. Enjoy!

1. Introduction to Sustainability

By: University of Illinois
Start date: 20 Jan 2014
Course length: 10 weeks
Effort: 8-10 hours per week
Superstar instructor: Jonathan Tomkin

A sneak preview...

"[The study of sustainability seeks to] uncover the principles of the long-term welfare of all the peoples of the planet". One for the inter-disciplinarians; as the description says: "understanding our motivations requires the humanities, measuring the challenges of sustainability requires knowledge of the sciences (both natural and social), and building solutions requires technical insight into systems (such as provided by engineering, planning, and management)"

2. The Age of Sustainable Development

By: Columbia University
Start date: 21 Jan 2014
Course length: 14 weeks
Effort: 5-7 hours per week
Superstar instructor: Jeffrey Sachs

A sneak preview...

"The fundamental question is how the world economy can continue to develop in a way that is socially inclusive and environmentally sustainable. The course describes the complex interactions between the world economy and the Earth's physical environment. Ecological processes and constraints (climate, disease ecology, physical resources such as soils and energy sources, topography and transport conditions) significantly shape the patterns of economic development, demography, and wealth and poverty. At the same time, human activities (farming, land use, urbanization, demographic change, and energy use) change the physical environments, increasingly in dangerous ways"

3. The Challenges of Global Poverty

By: MITx
Start date: 5 Feb 2014
Course length: 15 weeks
Effort: at least 6 hours per week
Superstar instructors: Banerjee and Duflo

A sneak preview...

"[I]s extreme poverty a thing of the past? What is economic life like when living under a dollar per day? Are the poor always hungry? How do we make schools work for poor citizens? How do we deal with the disease burden? Is microfinance invaluable or overrated? Without property rights, is life destined to be "nasty, brutish and short"? Should we leave economic development to the market? Should we leave economic development to non-governmental organizations (NGOs)? Does foreign aid help or hinder? Where is the best place to intervene? At the end of this course, you should have a good sense of the key questions asked by scholars interested in poverty today, and hopefully a few answers as well."

4. International Human Rights

By: LouvainX
Start date: 17 Feb 2014
Course length: 10 weeks
Effort: 6-8 hours per week
Superstar instructor: Olivier De Schutter

A sneak preview...

A course dealing with, among other issues, "the sources of human rights, the rights of individuals and the duties of States, and the mechanisms of protection...religious freedom in multicultural societies, human rights in employment relationships, economic and social rights in development, [and] human rights in the context of the fight against terrorism."

5. The European Union in Global Governance

By: KU Leuven
Start date: April 2014
Course length: 6 weeks
Effort: n/a
Superstar instructor: Prof. Dr. Jan Wouters (among others)

A sneak preview...

"The study of the EU as an international actor has become a key element in European and International Law, European Studies and International Relations. The EU represents the world’s largest trade power and aid donor, has a diplomatic service larger than that of most states, and has launched more than 20 civil-military operations. It has presented itself as a normative, global actor, and its emergence as a legal entity that is neither a state nor a classic international organization has both puzzled and fascinated legal scholars and political scientists alike. We invite students to delve into and help solve the most intriguing of these puzzles."