Of instrumental variables and sample definition

Welcome back to the world of econometrics debates! I hope you enjoyed the first installment. Round 2 pits Heckman and Urzúa against Imbens, and is about instrumental variables and estimates the local average treatment effect (LATE).

First, a little bit of background about instrumental variables (I’ll get to the debate below). Economists often use instrumental variables (IV) to estimate causal relationships.  The problem with simple regressions is that they yield biased estimates when factors that are supposed to affect a particular outcome depend on that outcome. I’ll use an example to help explain this.

One of the most famous examples of an instrumental variable-based analysis is from Joshua Angrist’s 1989 work.  Angrist wants to measure the impact of military service on future earnings. Analyzing the difference in earnings between veterans and non-veterans does not give the causal impact of military service on earnings for at least two reasons. First, there is an endogeneity problem: a young person’s decision to enroll in the military is likely affected by his expectation of future earnings. Second, there is a selection problem: veterans have special characteristics, observed and unobserved, that affect their decision to enroll in their military, and the same characteristics could influence their earnings. Observable characteristics can be controlled for in a regression, but not unobservable ones (e.g. motivation).

Angrist took advantage of the Vietnam draft lottery to overcome these problems. Since the lottery was random, the number that young men drew was unrelated to any of their characteristics. One idea to eliminate the selection problem is to compare the earnings of those with lottery numbers that give them a low probability of actually being called to service (“good” numbers) to the earnings of those with numbers that give them a higher probability of serving (“bad” numbers). The problem is that we are not really interested in the impact of having a good lottery number, but in the impact of service itself, which is not randomly assigned (there is not a 1:1 correspondence between getting a bad lottery number and military service: some people will always serve regardless of the lottery, some will never serve, and some who got a bad number will avoid being drafted). Angrist therefore uses the lottery assignment as an instrument for service. The draft lottery possesses the two qualities of an instrument: it is correlated with serving in the military, and it is uncorrelated with future earnings (because it is random).1

There is one important element to highlight. Imbens and Angrist showed in 1994 that this methodology only applies to a sub-group of the sample used in the analysis: those induced to participate in the intervention by the random assignment (in our example, those who served because of their lottery number, and would not have served if their number had been “good”). Because the treatment effect that is measured is not an average for the entire sample, it is called a local average treatment effect (LATE). It is the average treatment effect for those whose outcomes were affected by the random assignment.

This nuance is very important. In most cases, samples are drawn to be representative of the larger population of interest. Findings are, therefore, generalizable to that population. For example, taking a random sample of young men in the 1970s would allow researchers to measure the average impact of military service for that cohort. The IV approach, however, only provides an estimate of service for the compliers. From a policy point of view, this estimate might or might not be of interest.

This approach is also applied when studying the impact of financial services. Take the example of a financial literacy training program.  Rather than randomly assign applicants for the literacy program to receive the training or not, the idea is to leave the training open to whoever wants to participate, but randomly decide what areas of the city are subject to a marketing campaign about the financial literacy program. This design is called an “encouragement design” because the encouragement to participate is randomized, not the actual participation. An instrumental variable approach is then used to recover the impact of the training itself. The randomly-assigned marketing campaign mirrors Angrist’s draft lottery, and the literacy training mirrors actual service in the military.

Debate

Now, on to the Imbens vs. Heckman-Urzúa debate itself. Heckman and Urzúa highlight some limitations of the instrumental variable approach. An important limitation is the incidence of LATE: instrumental variable approaches may not identify what really interests economists or policymakers. Think of the impact of military service on earnings for a specific segment of the population that complies with the draft lottery assignment, versus the overall average impact of military service on earnings. From a policy point of view, is the former as interesting as the latter? (I don’t want to actually answer that question here, but the case for both ‘yes’ and ‘no’ can be made – this is economics, after all.) In addition, Heckman and Urzúa show how simple IV approaches can provide biased estimates of impact when the intervention is not a binary choice (e.g. military service or not) but an ordered choice (e.g. high-school graduation vs. GED vs. high-school drop-out). They propose the local instrumental variable (LIV) approach, which relies on structural approaches (fancier econometrics, for short) to model these types of situations.

I don’t think Imbens would contest the validity of the methods developed by Heckman and Urzúa. But his point is that IV approaches do have a lot of merit when used properly, namely their ability to estimate an impact cleanly – better a reliable and clear local average effect than nothing. More elaborate techniques can be used when necessary, but they do not take away the strengths of the IV approach.

So, what does all of this mean for understanding access to finance? Much of the empirical work associated with the Financial Access Initiative involves randomized control trials(RCTs). RCTs can be seen—in a formal sense—as akin to an instrumental variables method.  In that sense, it’s easy to apply Heckman’s criticism to RCT’s as well—that they only give estimates relevant to the particular experimental context, and generalizing isn’t always appropriate.  On the other hand, since researchers have a hand in designing the experiments, they at least have a clear sense of the populations at issue and the exact nature of the “treatment”.  That’s a big plus over common contexts with instrumental variables—where it’s often hard to know who’s actually affected by the instrument and how. Without knowing that, it’s hard to know exactly what’s being estimated.


Note:
1. For the mechanics of IV estimation, see econometric textbooks such as Wooldridge’s or Greene’s, or Angrist and Pischke’s excellent new book, “Mostly Harmless Econometrics” (Princeton, 2009).