Ecometrics: Toward a science of assessing ecological settings, with application to the systematic social observation of neighborhoods

This paper considers the quantitative assessment of ecological settings suc h as neighborhoods and schools. Available administrative data typically pro vide useful but limited information on such settings. We demonstrate how mo re complete information can be reliably obtained from surveys and observati onal studies. Survey-based assessments are constructed by aggregating over multiple item responses of multiple informants within each setting. Item an d rater inconsistency produce uncertainty about the setting being assessed, with definite implications for research design. Observation-based assessme nts also have a multilevel error structure. The paper describes measures co nstructed from interviews, direct observations, and videotapes of Chicago n eighborhoods and illustrates an "ecometric" analysis-a study of bias and ra ndom error in neighborhood assessments. Using the observation data as an il lustrative example, we present a three-level hierarchical statistical model that identifies sources of error in aggregating across items within face-b locks and in aggregating across face-blocks to larger geographic units such as census tracts. Convergent and divergent validity are evaluated by study ing associations between the observational measures and theoretically relat ed measures obtained from the U.S. Census, and a citywide survey of neighbo rhood residents.