THE RELIABILITY OF OBSERVATIONS OF YOUNG CHILDRENS ENGAGEMENT - AN APPLICATION OF GENERALIZABILITY THEORY

The reliability of measurement of how young children spend their time has traditionally been computed in terms of interobserver exact agreem ent. This study sought to apply generalizability theory to the measure ment of engagement Forty-seven young children, 15 of whom had disabili ties, were observed four times in their child care setting. Types and levels of engagement were coded by three raters. Using ANOVA procedure -s for determining the relative contribution of different sources of e rror, a fully crossed (with subjects) two-facet (sessions, raters) gen eralizability (G) study was employed. The nine outcome measures were f our types and five levels of engagement Results showed that raters acc ounted for less than 2% of the variance in the error of the scores, wh ile sessions accounted for most of the variance other than between sub ject variance. The outcome measures proved to vary in their reliabilit y robustness. The G study was followed by a decision (D) study to dete rmine the levels of the facets that would be required to achieve a gen eralizability coefficient of .80. The number of sessions could be real istically increased to achieve this aim, but the increase in ''dependa bility'' achieved with the addition of each rater was minimal. Conclus ions are drawn about the importance of assessing more than one source of error, raters, in observational research overall, molecular methods were determined to be relatively unstable for measuring the molar con struct of engagement, but certain engagement outcomes were stable acro ss sessions and raters.