Assessing the impact of pediatric-based developmental services on infants,families, and clinicians: Challenges to evaluating the healthy steps program

Background. Begun in 1996, the Healthy Steps for Young Children Program (HS ) is a new model of pediatric practice that incorporates child development specialists and enhanced developmental services for families of young child ren. HS is for all families, not just those at high-risk. It is expected to strengthen parents' knowledge, attitudes, and behaviors in ways that promo te child health and development, and in turn, to lead to improved child out comes, such as improved language development, increased utilization of well child care, and decreased problem behaviors, hospitalizations, and injurie s. The HS evaluation is designed to assess whether HS is successful in achi eving the desired outcomes, measure the program's costs, and determine the relation of the program's costs to its outcomes. Objective. This article is the first report of the HS evaluation. It descri bes the evaluation design and characteristics of the HS sites and sample fo r the evaluation. Methods. The evaluation is following a cohort of children from birth to age 3 at 15 evaluation sites across the country. The sites represent a range o f organizational practice settings that include group practices, hospital-b ased clinics, and health maintenance organization pediatric clinics. The ev aluation design relies on 2 comparison strategies. At 6 randomization desig n sites, 400 children were randomized to the intervention or control group. At 9 quasi-experimental design sites, a comparison location with a similar organizational setting and patient profile has been selected and up to 200 children are being followed at each of these sites. At each site, 2 developmental specialists (or their full-time equivalents) work as a team with 4 to 8 pediatricians and pediatric nurse practitioners. The specialist conducts office visits (jointly or sequentially with the pe diatric clinician) and home visits, assesses children's developmental progr ess, provides referrals and follow-up to resources in the community, organi zes and conducts parent discussion groups, coordinates early reading activi ties, and maintains a telephone information line for questions about child development and behavior. The evaluation relies on many data sources including self-administered prov ider surveys, key informant interviews, forms completed by parents at offic e visits, telephone interviews with parents, medical record reviews, data f rom each site on program costs and health services use, and an ongoing log of family contacts maintained by each developmental specialist. Analyses for this article are based on enrollment data for the Healthy Step s sample and national data on 1997 US live births. The chi(2) goodness-of-f it test was used to evaluate whether the distribution of selected demograph ic variables, insurance, and infant's birth weight for the Healthy Steps sa mple was similar to the distributions for US births in 1997. In addition, c omparisons were made between intervention and comparison families at the ra ndomization and quasi-experimental evaluation sites. The chi(2) test of ind ependence was used to evaluate differences in variables across groups. Results. Throughout a 26-month period, 5565 children enrolled in the evalua tion, 2963 (53.2%) children in the intervention group and 2602 (46.8%) in t he comparison group. More than 10% of mothers in the Healthy Steps sample a re teenagers; 18% have 11 years of education or less; 27% have completed co llege; 18% are black or African-American; slightly >20% are of Hispanic ori gin; 36% are single; and close to one-third used Medicaid for their prenata l care. Approximately 7% of infants were low birth weight. When compared with national birth data for the United States as a whole, th e Healthy Steps sample seems similarly diverse. However, with the exception of maternal age, the distribution of variables was significantly different from the distribution for US births. There are no differences between intervention and comparison families at ra ndomization sites on any of the maternal characteristics, insurance status, or infant's birth weight. However, there are a number of differences betwe en the intervention and the comparison groups at the quasi-experimental sit es and between the randomization and quasi-experimental sites. At quasi-exp erimental sites, mothers in the comparison group were more likely than were mothers in the intervention group to be 29 years old or younger, to have f ewer years of education, to be black or African-American, to report Hispani c origin, and to be single. Mothers at the quasi-experimental sites were sl ightly older than were mothers at randomization sites. They also were more likely to be married, to have <11 years of education (but also to be colleg e graduates), to be of Hispanic origin, and to report private insurance or self-pay rather than Medicaid as their source of payment for prenatal care. Conclusion. The Healthy Steps sample is economically and ethnically diverse , reflecting the diversity of the nation as a whole. There seem to be diffe rences between randomization and quasi-experimental sites as well as betwee n intervention and comparison groups at the quasi-experimental design sites . As important, randomization at the sites that selected to use this approa ch seems to have been effective in equalizing the characteristics of famili es in the intervention and comparison groups. The differences between the intervention and comparison families at the qua si-experimental sites indicate the need to take account of these difference s in analyzing program effects. In addition, to account for within-site cor relation of outcomes, between-site variability of the effects of HS, and be tween-provider type variability of the effects of HS, random effects models will be used in the data analyses. These models are also referred to as hi erarchical linear models or multilevel models. The HS evaluation is carefully designed to address the complexities of a pr ogram with multiple objectives, multiple components, and a wide range of ex pectations. The evaluation will provide information for practicing clinicia ns on the effectiveness of HS in improving care for families. It will provi de policy makers with empirical evidence to inform the national debate on w hether pediatric practices are an appropriate venue for helping parents pro mote their children's development. In addition, it will assess whether pote ntial outcomes such as increased satisfaction with care and decreased hospi talizations make HS a valuable intervention.