{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Resume Experiment Analysis\n",
    "\n",
    "How much harder is it to get a job in the United States if you are Black than if you are White? Or, expressed differently, what is the *effect* of race on the difficulty of getting a job in the US?\n",
    "\n",
    "In this exercise, we will be analyzing data from a real world experiment designed to help answer this question. Namely, we will be analyzing data from a randomized experiment in which 4,870 ficticious resumes were sent out to employers in response to job adverts in Boston and Chicago in 2001. The resumes differ in various attributes including the names of the applicants, and different resumes were randomly allocated to job openings. \n",
    "\n",
    "The \"experiment\" part of the experiment is that resumes were randomly assigned Black- or White-sounding names, and then watched to see whether employers called the \"applicants\" with Black-sounding names at the same rate as the applicants with the White-sounding names.\n",
    "\n",
    "(Which names constituted \"Black-sounding names\" and \"White-sounding names\" was determined by analyzing names on Massachusetts birth certificates to determine which names were most associated with Black and White children, and then surveys were used to validate that the names were perceived as being associated with individuals of one racial category or the other). \n",
    "\n",
    "You can get access to original article [here](https://www.aeaweb.org/articles?id=10.1257/0002828042002561). \n",
    "\n",
    "**Note to Duke students:** if you are on the Duke campus network, you'll be able to access almost any academic journal articles directly; if you are off campus and want access, you can just go to the [Duke Library](https://library.duke.edu/) website and search for the article title. Once you find it, you'll be asked to log in, after which you'll have full access to the article. You will also find this pattern holds true at nearly any major University in the US.\n",
    "\n",
    "- Download the data set `resume_experiment.dta` from [github here](https://github.com/nickeubank/MIDS_Data/tree/master/resume_experiment), or by doing to `www.github.com/nickeubank/MIDS_Data` and opening the `resume_experiment` folder.\n",
    "- For `python` users, use `read_stata` in `pandas` to load the data set; For `R` users, use `read_dta` in `haven` to load the data set\n",
    "- `black` is the treatment variable in the data set (whether the resume has a Black-sounding name). \n",
    "- `call` is the dependent variable of interest (did the employer call the fictitious applicant for an interview)\n",
    "\n",
    "In addition, the data include a number of variables to describe the other features in each fictitious resume, including applicants education level (`education`), years of experience (`yearsexp`), gender (`female`), computer skills (`computerskills`), and number of previous jobs (`ofjobs`). Each resume has a random selection of these attributes, so on average the Black-named fictitious applicant resumes have the same qualifications as the White-named applicant resumes. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking for Balance\n",
    "\n",
    "The first step in analyzing any experiment is to check whether you have *balance* across your treatment arms—that is to say, do the people who were randomly assigned to the treatment group look like the people who were randomly assigned to the control group. Or in this case, do the resumes that ended up with Black-sounding names look like the resumes with White-sounding names. \n",
    "\n",
    "Checking for balance is critical for two reasons. First, it's always possible that random assignment will create profoundly different groups—the *Large of Large Numbers* is only a \"law\" in the limit. So we want to make sure we have reasonably similar groups from the outset. And second, it's also always possible that the randomization wasn't actually implemented correctly—you would be amazed at the number of ways that \"random assignment\" can go wrong! So if you ever do find you're getting unbalanced data, you should worry not only about whether the groups have baseline differences, but also whether the \"random assignment\" was actually random!\n",
    "\n",
    "### Exercise 1\n",
    "\n",
    "\n",
    "Check for balance in terms of the average values of applicant gender (`female`), computer skills (`computerskills`), and years of experience (`yearsexp`) across the two arms of the experiment (i.e. by `black`). Calculate both the differences in means across treatment arms *and* test for statistical significance of these differences. Do gender and computer skills look balanced across race groups?\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2\n",
    "\n",
    "Do a similar tabulation for education (`education`). Education is a categorical variable coded as follows:\n",
    "\n",
    "- 0: Education not reported\n",
    "- 1: High school dropout\n",
    "- 2: High school graduate\n",
    "- 3: Some college\n",
    "- 4: College graduate or higher\n",
    "\n",
    "Because these are categorical, you shouldn't just calculate and compare means—you should compare share or count of observations with each value (e.g., a chi-squared contingency table). You may also find the `pd.crosstab` function useful.\n",
    "\n",
    "Does education look balanced across racial groups?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3\n",
    "\n",
    "What do you make of the overall results on resume characteristics? Why do we care about whether these variables look similar across the race groups? And if they didn't look similar, would that be a threat to internal or external validity?\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimating Effect of Race"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4\n",
    "\n",
    "The variable of interest in the data set is the variable `call`, which indicates a call back for an interview. Perform a two-sample t-test comparing applicants with black sounding names and white sounding names.\n",
    "\n",
    "Interpret your results—in both percentage terms *and* in terms of percentage points, what is the effect of having a Black-sounding name (as opposed to a White-sounding name) on your resume?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 5\n",
    "\n",
    "Now, use a linear probability model (regression!) to estimate the differential likelihood of being called back by applicant race (i.e. the racial discrimination by employers). \n",
    "\n",
    "Since we have a limited dependent variable, be sure to use [heteroskedastic robust standard errors.](https://www.statsmodels.org/stable/generated/statsmodels.regression.linear_model.OLSResults.get_robustcov_results.html) Personally, I prefer the `HC3` implementation, as it tends to do better with smaller samples than other implementations.\n",
    "\n",
    "Interpret these results—what is the *effect* of having a Black-sounding name (as opposed to a White-sounding name) on your resume in terms of the likelihood you'll be called back?\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 6\n",
    "\n",
    "Now let's see if we can improve our estimates by adding in other variables as controls. Add in `education`, `yearsexp`, `female`, and `computerskills`—be sure to treat education as a categorical variable!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimating Heterogeneous Effects"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 7\n",
    "\n",
    "What we've been estimating up until this point are the *average* effects. Now let's look for evidence of *heterogeneous treatment effects*—effects that are different for different types of people in our data. \n",
    "\n",
    "Is there more or less racial discrimination among applicants who do *not* have a college degree? What is the difference in both percentage terms and in percentage points? Is the difference statistically significant?\n",
    "\n",
    "Please still include `education`, `yearsexp`, `female`, and `computerskills` as controls.\n",
    "\n",
    "*(Hint: use an interaction term)*"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 8\n",
    "\n",
    "Now let's compare men and women—is the penalty for having a Black-sounding name greater for Black men or Black women?\n",
    "\n",
    "Again, please still include `education`, `yearsexp`, `female`, and `computerskills` as controls."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 9\n",
    "\n",
    "Calculate and/or lookup the following online:\n",
    "\n",
    "- What is the share of applicants in our dataset with college degrees?\n",
    "- What share of Black adult Americans have college degrees (i.e. have completed a bachelors degree)?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 10\n",
    "\n",
    "Bearing in mind your answers to Exercise 7 and to Exercise 9, how do you think the Average Treatment Effect you estimated in Exercise 6 might generalize to the experience of the average Black American (i.e., how do you think the ATE for the average Black American would compare to the ATE estimated from this experiment)?\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 11\n",
    "\n",
    "What does your answer to Exercise 10 imply about the study's *internal* validity?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 12\n",
    "\n",
    "What does your answer to Exercise 10 imply about the study's *external* validity?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What Did We Just Measure?\n",
    "\n",
    "It's worth pausing for a moment to think about exactly what we've measured in this experiment. Was it the effect of race on hiring? Or the difference in the experience of the average White job applicant from the average Black job applicant?\n",
    "\n",
    "Well... no. What we have measured in this experiment is **just** the effect of having a Black-sounding name (as opposed to a White-sounding name) on your resume on the likelihood of getting a followup call from someone hiring in Boston or Chicago given identical resumes. In that sense, what we've measured is a small *piece* of the difference in the experience of Black and White Americans when seeking employment. As anyone looking for a job knows, getting a call-back is obviously a crucial step in getting a job, so this difference—even if it's just one part of the overall difference—is remarkable."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  },
  "vscode": {
   "interpreter": {
    "hash": "718fed28bf9f8c7851519acf2fb923cd655120b36de3b67253eeb0428bd33d2d"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}