Making Potential Outcomes Concrete

In this exercise, I will describe a simple data science project, and your job is to map the elements of that project onto the Potential Outcomes Framework.

The Study

You have been approached by the Duke Office of Student Wellbeing. They are worried that due to the COVID pandemic, students are spending too much time sitting in front of their computers and not being active, leading to an increase in student Body Mass Index (BMI) scores (a measure of whether someone’s weight is more or less than one might expect given their height) which might, in extreme cases, have negative health consequences.

One policy they’re considering implementing to address this issue is to replace all sodas in the dining halls with diet (sugar-free) sodas. But they aren’t sure if this will actually make a difference.

To help them decide, they wanted to measure the effect of diet soda on BMI, so they collected data on all Duke students, and compared student BMI scores for students who drink diet soda with those who drink regular soda (they’re students – they all drink some soda :)).

They found that students who drink diet soda actually have higher BMIs (suggesting they may actually be less healthy). They interpreted this as evidence that diet soda is making students less healthy, so they changed their plans, and instead of removing sugary soda, they’ve decided to remove all diet soda from the campus.

Mapping to the Potential Outcomes Framework

In words, describe exactly what quantities in this study context would correspond to the different components of the Potential Outcomes Framework we’ve been studying. When defining these, remember to define both the thing being measured and the population being measured.

As you do so, avoid using terms like “treatment”, “control group”, or “potential outcome”. The goal of this exercise is to move from the abstract conceptual derivations we’ve read to the specifics of this study. For example, I’ve put in an answer to Question 1 below:

1: \(E(Y_{T=0})\)

The average BMI of all Duke students in a world where no one drinks diet soda.

2: \(E(Y_{T=1})\)

The average BMI of all Duke students in a world where everyone drinks diet soda.

3: \(E(Y_{T=1}) - E(Y_{T=0})\)

The average difference in the BMI of all Duke students between a world where all Duke students drink diet soda and a world where no Duke students drink diet soda (i.e. the average treatment effect of diet soda on Duke students).

4: \(E(Y_{T=1}| D=1)\)

The average BMI of Duke students who we actually observe drinking diet soda (in the world where they drink diet soda).

5: \(E(Y_{T=0}| D=0)\)

The average BMI of Duke students who we actually observe not drinking diet soda (in the world where they don’t drink diet soda).

6: \(E(Y_{T=1}|D=0)\)

The average BMI of Duke students who we actually observe not drinking diet soda in a world where they do drink diet soda.

7: \(E(Y_{T=0}| D=1)\)

The average BMI of Duke students who we actually observe drinking diet soda in a world that they don’t drink diet soda.

8: \(E(Y_{T=1}| D=0) - E(Y_{T=0}|D=0)\)

For Duke students who we actually observe not drinking diet soda, this is the average difference in BMI between a world where those students do drink diet soda and a world where they do not (i.e., it is the effect of diet soda on BMI for students who don’t actually drink diet soda, also known as the “Average Treatment on the Control (ATC)”).

9: \(E(Y_{T=0}| D=1) - E(Y_{T=0}|D=0)\)

The average difference in BMI between Duke students who we actually observe drinking diet soda and Duke students who we actually observe not drinking diet soda in a world whether neither group drinks diet soda (i.e., the baseline difference in BMI between the two groups).

Observability

10: Now, which of the quantities above can be directly observed?

The quantities in in Q4 & Q5

Causal Inference

In order for the difference in BMIs found in the report – that those who drink diet soda have higher BMIs – to be a true estimate of the average effect of drinking diet soda, we know that it must be the case that:

\(E(Y_{T=0}| D=1) - E(Y_{T=0}| D=0) = 0\)

and

\(E(Y_{T=1}| D=0) - E(Y_{T=0}| D=0) = E(Y_{T=1}| D=1) - E(Y_{T=0} | D=1)\).

In the context of this study, what do those two conditions mean in plain english? As above, avoid using abstract terms (“treatment”, “baseline”, etc.) and try and be as concrete as possible.

11 \(E(Y_{T=0}| D=1) - E(Y_{T=0}| D=0) = 0\):

This condition means that in a world were nobody drinks diet soda, the students who we actually observe drinking diet soda and the students who we actually observe not drinking diet soda would have the same average BMIs. In other words, there are no baseline differences in BMI.

12 \(E(Y_{T=1}| D=0) - E(Y_{T=0}| D=0) = E(Y_{T=1}| D=1) - E(Y_{T=0}| D=1)\):

This condition means that the differences in BMI between a world where students drink diet soda and a world where students don’t drink diet soda is the same for the students we actually observe drinking diet soda and the students we actually observe not drinking diet soda. In other words, the difference in BMI for these two groups of students is the same between the world where they drink diet soda and the world where they don’t drink diet soda.

Or, in different other words, both groups of students’ BMIs respond to drinking diet soda the same way.

Now, for each of the conditions above, please give one reason—in plain English—why those conditions may not be met in the context of this study?

As you do so, be specific! Tell me a story about why in the case of this study you think one of these conditions may hold. One can always say things like “people in the two groups may have been different”, but I want a specific reason you think they might have been different in a way that meets the conditions.

13 It may be the case that \(E(Y_{T=0}| D=1) - E(Y_{T=0}| D=0) \neq 0\) because…:

People who have more difficulty controlling their weight (and thus BMI) may also be more likely to drink diet soda as a way of further regulating their weight. Thus, even if the people we observe drinking diet soda weren’t drinking diet soda, they might still tend to have higher BMIs.

14 It may be the case that \(E(Y_{T=1}| D=0) - E(Y_{T=0}|D=0) \neq E(Y_{T=1}| D=1) - E(Y_{T=0}| D=1)\) because…:

The people who drink soda may be doing so precisely because they find it helps them control their weight (i.e., they find that they have lower BMIs when drinking diet soda then when they don’t drink diet soda). This could be because they drink lots of soda, so the impact of drinking diet soda is larger on their calorie intake.

At the same time, those who drink regular soda may be choosing to do so because drinking diet soda doesn’t help them control their weight, and so they drink the beverage they prefer (regular soda). This could be because they only drink soda once or twice a week, and so the sugar content of the soda doesn’t matter much.

As a result, the “effect” of drinking diet soda may be bigger for the people we observe drinking diet soda than the effect we’d see for the people we don’t observe drinking diet soda.