SOURCES OF SELECTION BIAS IN EVALUATING SOCIAL PROGRAMS - AN INTERPRETATION OF CONVENTIONAL MEASURES AND EVIDENCE ON THE EFFECTIVENESS OF MATCHING AS A PROGRAM-EVALUATION METHOD

This paper decomposes the conventional measure of selection bias in ob servational studies into three components. The first two components ar e due to differences in the distributions of characteristics between p articipant and nonparticipant (comparison) group members: the first ar ises from differences in the supports, and the second from differences in densities over the region of common support. The third component a rises from selection bias precisely defined. Using data from a recent social experiment, we find that the component due to selection bias, p recisely defined, is smaller than the first two components. However, s election bias still represents a substantial fraction of the experimen tal impact estimate. The empirical performance of matching methods of program evaluation is also examined, We find that matching based on th e propensity score eliminates some but not all of the measured selecti on bias, with the remaining bias still a substantial fraction of the e stimated impact. We find that the support of the distribution of prope nsity scores for the comparison group is typically only a small portio n of the support for the participant group. For values outside the com mon support, it is impossible to reliably estimate the effect of progr am participation using matching methods. If the impact of participatio n depends on the propensity score, as we find in our data, the failure of the common support condition severely limits matching compared wit h random assignment as an evaluation estimator.