REVIEW:
Intention-to-treat (ITT) Effects
Differences between treatment and control group averages
ITT = E(Y|Z = 1) – E(Y|Z = 0)
Y - outcome
Z - assigned treatment
Effect of Treatment on the Treated (TOT)
Differences between treatment and control group means
Differences in compliance for treatment and control groups
TOT = “Wald Estimator” = E(Y|Z = 1) – E(Y|Z = 0)
E(X|Z = 1) – E(X|Z = 0)
Y - outcome
Z - assigned treatment
X - actual treatment (i.e. compliance)
Example:
If a kid takes some sort of good health inducing pill, he’ll grow taller. How much taller?
Say we give the treatment group a coupon for free drugs, but they still have to walk to the store to redeem the coupon. So some aren’t going to bother. Also, some kids in the control group that don’t get coupons are going to happen to be hanging out with their friends when the friend goes to the pharmacy, and the treatment (coupon) kid says “everybody’s doing it, just once won’t hurt you, it’ll make you feel good,” and the control (no-coupon kid) ends up taking the drug.
Height of average coupon kid: 48 inches
Height of average non-coupon kid: 46 inches
Probability that a coupon kid actually took the drug: 0.90
Probability that a non-coupon kid took the drug: 0.15
What does that make the Intention to treat (ITT) effect? (2 inches)
What does that make the effect of the treatment on the treated? 2/(.9-.15)=2.67
Which one of these measures is easier for us to calculate (both in terms of math and practicality)?
(ITT easier to measure—how are we ever really going to know how many of the no-coupon kids bought drugs on their own? Also, in this example, we could go around and collect the coupons to get the probability of there being redeemed, but how do we know how many kids actually swallowed the pill?)
Why might we be interested in one of these measures over the other?
Another example from a natural experiment: Quarter of Birth (Josh Angrist QJE Nov ‘91)
Back in the day when dropping out of high school was a pretty common thing for all sorts of people to do, the compulsory schooling laws had a peculiar effect. Imagine a state that said you had to be 5 years old before the beginning of the year you wanted to start school, and that you had to stay in school until you turned 16 years old.
Say Kid 1 was born on Dec 31, 1959, and Kid 2 was born on Jan 1, 1960. Kid 1 is older, by a day. That’s essentially random. The parents were planning for a baby, and they probably had a few-week window in which they expected the baby, but there’s unlikely to be large innate differences between kids born one day and kids born the next.
So a few years later, when school starts in September 1965, Kid 1 is allowed to enroll. Kid 2 stays at home for another year, and starts school in September 1966. Assume both kids don’t excel at school and would prefer to drop out and start working as soon as legally possible. Kid 1 can drop out on Dec 31, 1975, and Kid 2 can drop out on Jan 1, 1976. That’s only one day apart, but how much schooling will they have when they both drop out?
Kid 1 enrolled in 1965 and will be in the middle of his 11th year of school when he drops out.
1:1965-66
2:1966-67
3: 67-68
4:68-69
5:69-70
6:70-71
7:71-72
8:72-73
9:73-74
10:Sep 74-June 75
11:June 75-quit on his birthday.
Kid 2 enrolled in 1966 and will be in the middle of his 10th year of school when he drops out.
So, basically randomly, some people were compelled to get more schooling than others.
Let’s pretend we collected the following data:
Average kid born in December earnings: $40,000
Average kid born in January earnings: $39,000
But clearly, not everyone drops out exactly on their birthday.
Average December baby schooling: 11 years
Average January baby schooling: 10.8 years
What’s the effect of a year of schooling on earnings?
(Yd-Yj)/(Sd-Sj)
It makes sense to think about this in terms of fractions: The top is $/age, and the bottom is school/age, so you end up with $/school, which is the useful thing here.
Another quick example from Angrist’s Vietnam Lottery paper?
Average earnings of people with a low draft number (low is bad): $35,000
Average earnings of high draft number: $36,000
Fraction of low draft numbers to go to Vietnam: 0.80
Fraction of high draft numbers to go to Vietnam 0.40
What’s the effect of serving in Vietnam on earnings?
(earnings/draft)/(draft/serve)=(earnings/serve)=(1,000/.4)=$2,500
WORMS: IDENTIFYING IMPACTS ON EDUCATION AND HEALTH
IN THE PRESENCE OF TREATMENT EXTERNALITIES
BYEDWARDMIGUEL ANDMICHAELKREMER1
Worms in general. 1 in 4 worldwide have some sort of intestinal worm. Used to be prevalent in USA until 1910-1920 Rockefeller Sanitary Commission did massive deworming program in the South (Bleakley (2002)).
Hookworm, roundworm, whipworm, and schistosomiasis. 1.3 billion with Hook/Round, 900m with Whip, and 200 with Shisto. They don’t reproduce inside humans, just grow and make you sick. Anemia, abdominal pain, listlessness. Shisto can be much worse (liver/spleen). Easily treated with low-cost single dose Albendizole (every six months) and Praziquantal (annually, for shisto). Minimum of side effects (upset stomach, probably worse on an empty stomach). WHO says mass treatment in highly effected areas is great—eliminates the need for individual screening. Their cutoff is 50% prevalence for the first three types of worms (called geohelminths) and 30% for Shisto. Brings cost down to as little as $0.49 per person per year in Africa.
Let’s just say we took kids at 75 primary schools, randomly selected half of the kids (say there’s 400 kids at each school), so we had 30,000 kids, and 15,000 kids get deworming drugs, and 15,000 don’t. At any school, since we randomized, we’d expect to find around 200 kids getting the drugs and 200 not.
How do you get worms? (feces for Hook/Round/Whip, swimming in contaminated water for Shisto, namely Lake Victoria)
So if half of the friends that you go to school with, and thus probably share a latrine with, and maybe go swimming with all the sudden don’t have worms, are you less likely to get worms yourself? (Yes!)
This is an EXTERNALITY. An externality occurs when two parties engage in a transaction of some sort, and they cause harm or benefit to happen to a third party who wasn’t involved in the original transaction.
When a rancher sells his meat to a customer, neither of them are usually thinking about the water pollution or erosion that the cattle cause for people downstream, and definitely they’re not accounting for the methane the cattle produce that increases global warming.
When a truck-driver buys diesel from the gas station, nobody’s really accounting for the sulfides that are getting released into the atmosphere and causing asthma for all sorts of other people.
DRAW GRAPH WITH PRIVATE BENEFIT, PRIVATE COST, HIGHER SOCIAL COST
Externalities can just as easily be good. People buying honey from beekeepers don’t pay attention to the fact that the bees pollinate all the nearby fruit trees.
And when your mom took you to the doctors when you were a kid to get you vaccinated, she was only thinking about your health, and not the fact that because you got vaccinated you made it less likely for everyone else in your pre-school to get sick.
DRAW GRAPH WITH PRIVATE BENEFIT, PRIVATE COST, HIGHER SOCIAL BENEFIT
So back to our example of a school with 200 kids getting drugs and 200 not. We’d typically measure the effect of the drugs by just taking the difference between the control means and the treatment means of whatever we were trying to measure, say height or days of school attended. But are those 200 kids at the school really a pure control group? (No.)
Why? When they go run around and play with kids that have been treated. If some of them have been treated, their feces are less likely to have worms and give you worms, and the fresh water you both swim and wash in is less likely to be contaminated. So the treated kids are less likely to get new infections.
So both groups have fewer worm infections thanks to the positive externality, and if you do the standard thing, Yt-Yc, then Yc (some good outcome) will be higher than it would have been without the treatment group being treated, and you will thus understate the benefits from deworming.
But this is in fact what most people did.
A 2000 review of 30 randomized trials found the following results:
Dickson, British Medical Journal
Thirty randomised controlled trials in more than 15 000 children were identified. Effects on mean weight were unremarkable, and heterogeneity was evident in the results. There were some positive effects on mean weight change in the trials reporting this outcome: after a single dose (any anthelmintic) the pooled estimates were 0.24 kg (95% confidence interval 0.15 kg to 0.32 kg; fixed effects model assumed) and 0.38 kg (0.01 kg to 0.77 kg; random effects model assumed). Results from trials of multiple doses showed mean weight change in up to one year of follow up of 0.10 kg (0.04 kg to 0.17 kg; fixed effects) or 0.15 kg (0.00 to 0.30; random effects). At more than one year of follow up, mean weight change was 0.12 kg (-0.02 kg to 0.26 kg; fixed effects) and 0.43 (-0.61 to 1.47; random effects). Results from studies of cognitive performance were inconclusive.
Conclusions: There is some limited evidence that routine treatment of children in areas where helminths are common has effects on weight gain, but this is not consistent between trials. There is insufficient evidence as to whether this intervention improves cognitive performance.
Our interpretation of these findings is that the evidence of benefit for mass treatment of children related to positive effects on growth and cognitive performance is not convincing. In the light of these data, we would be unwilling to recommend that countries or regions invest in programmes that routinely treat children with anthelmintic drugs to improve their growth or cognitive performance.
------
So instead, Miguel and Kremer evaluated a program that was randomized by school. We’ve mentioned this sort of clustering before, but now we’ll talk about it in some more detail.
SHOW MAP OF KENYA, BUSIA DISTRICT
Miguel and Kremer evaluated the Primary School Deworming Project, which was operated by a Dutch NGO called ICS-Africa.
75 schools, almost all of the rural schools in the district.
Over 30,000 pupils.
January 1998, stratify by zone and involvement in other programs, arranged alphabetically, then counted off by threes.
Group 1: treated starting in 1998.
Group 2: treated starting in 1999.
Group 3: treated starting in 2001.
That makes group 1 treatment in 1998 and 2 and 3 control, and for 1999, 1 and 2 were treatment and 3 was control.
WRITE A FEW LINES FROM TABLE 1 on board
Key Point: The observables are the same across all the groups, and thus it’s reasonable to assume that the unobservables are the same too.
Prior to treatment, 92% of a sub-sample of children who provided stool samples had at least one infection, and 37% had at least one moderate-to-heavy infection. All school met the WHO geohelminth cut-off in both years (’98 and ’99), and 6 of 25 and 16 of 50 for Shisto in ’98 and ’99.
ITT v. TOT.
Which are we really interested in? (ITT)
By keeping track o f the names of kids that received medication on any given day, we know that 78% of the kids in the ’98 treatment group received some treatment, and we have a survey saying only 5% of comparison school kids got treatment from some outside source. So we might be able to piece some sort of TOT estimate together, but especially because of the externalities we’re dealing with, we’re really more interested in ITT.
TABLE 5: SIMPLE DIFFERENCES
Any moderate infection, 25 percentage points lower,
Height for age Z-score .09 higher.
They’re nice, but what’s wrong with these numbers?
(We’ve sort of fixed the within-school externality by randomizing by school, but we’ve also got the potential for across-school externalities.)
There are a lot of primary schools, and nearly 1/4 of kids attend a primary school that’s actually not the closest one to their house. So again, control-school kids are actually healthier because they walk by a treatment school on the way to their control school, and they encounter fewer eggs/feces/whatever on the way, and we’d like to be able to measure that part of health improvement too.
CROSS-SCHOOL EXTERNALITIES
(We’re really confident about this part because a student’s proximity to treatment schools is exogenous because it was controlled by the randomization procedure).
EQUATION 1, PAGE 175
Explain all terms.
Y=outcome
T=treatment
X=observable characteristics (The whole idea of randomization means we could leave these out, but including them increases statistical precision. If you run the regression with and without these and your other estimates change, that’s likely a problem.)
What were they? (Average school test score, pre-treatment infection rates, grade in school, etc.)
Nt=Number of treatment pupils in a given distance
N=number of total pupils
Sum of coeff’s on T and Nt are what we’re interested in.
WITHIN-SCHOOL EXTERNALITIES:
We can’t solve both within and across-school externalities with our experimental method, so we’ve got to use some sort of statistical method to try and solve the with-in school question. That is, how much of the health improvements at a treatment school are due to kids taking the drugs and getting healthier themselves, and how much is due to them being less-likely to infect their friends at the same school, and those friends being healthier as a result?
We can hope for something close:
“Group 1 pupils who did not receive treatment in 1998 are compared to Group 2 pupils who did not receive treatment in 1999, the year that Group 2 schools were incorporated into treatment, to at least partially deal with potential bias due to selection into medical treatment. For the health outcomes, we compare these two groups as of January to February 1999, when Group 1 schools had already been treated (in 1998) but Group 2 schools had not.”
Baseline characteristics show no differences.
TABLE VI, Panel B
This comparison shows 21% less likely to have an infection.
Compare that to the treated in 1998 (24% chance) to the people who would get treated in 1999 (51% chance) That’s a 27% difference. Now compare 21% to 27%. 3/4 of the benefit comes from externalities!
EQUATION 3, DOING BOTH AT ONCE
12% with-in school
14% actually being treated
9% cross-school
So 26% for being in treated school and taking.
SCHOOLING.
7 percentage point increase in attendance at treatment schools. (25% reduction in absenteeism.)
0.14 more school years
No test score differences
EFFECTIVENESS:
49 cents per student per year
649 DALY averted.
$5 per DALY. (Measles, Diptheria, Pertussis, Tetanus cost 12 to 17 per DALY averted)
Externalities are 76% of that gain. 99% of the gain comes from Shisto. Without Shisto the geohelminth DALY averted is $280 per. Not worth it.
Education cost effectiveness:
0.49 dollars per student per year/.14 years more schooling=$3.50 per year of schooling.
Compare to ICS Child Sponsorship Program (giving uniforms), $99 per year of schooling.
Return to schooling:
$570 wages, 7% return to a year of schooling—increase net present value of wages by $30, for only $0.49.
But what if the increased attendance required more teachers? Teacher compensation is $1942. Times 0.14, divide by 30 students = $9.06 worth of an extra teacher, so full cost is $9.55
“When local treatment externalities are expected, field experiments can be purposefully designed to estimate externalities by randomizing treatment at various levels.”
THE ILLUSION OF SUSTAINABILITY
3/4 of the benefit could be from other people taking it. Does that make you likely to want to shell out for the drug, or does it make you likely to free-ride?
POTENTIAL SOURCES OF BIAS
Asymmetric flows of pupils into treatment schools. (Didn’t happen)
Differential Attrition.