SERC Symposium: Speaker Bio and Abstract

Ashesh Rambachan, Assistant Professor of Economics, MIT

Abstract

Identifying Prediction Mistakes in Observational Data

Decision makers, such as doctors, judges, and managers, make consequential choices based on predictions of unknown outcomes. Do these decision makers make systematic prediction mistakes based on the available information?

If so, in what ways are their predictions systematically biased? In this paper, I characterize behavioral and econometric assumptions under which systematic prediction mistakes can be identified in empirical settings such as hiring, medical diagnosis, and pretrial release. I derive a statistical test for whether the decision maker makes systematic prediction mistakes under these assumptions and provide methods for estimating the ways in which the decision maker’s predictions are systematically biased. As an illustration, I analyze the pretrial release decisions of judges in New York City, estimating that at least 20% of judges make systematic prediction mistakes about failure to appear risk given defendant characteristics. Motivated by this behavioral analysis, I estimate the effects of replacing judges with algorithmic decision rules and find that automating decisions where systematic prediction mistakes occur weakly dominates the status quo.

Bio

Ashesh Rambachan is currently a post-doctoral researcher at Microsoft Research New England. He will be joining MIT’s economics department as an assistant professor in July. His research interests are primarily in theoretical and applied econometrics with a focus on causal inference and data-driven decision-making.