STILL WEIGHTING FOR THE RIGHT CRITERIA TO VALIDATE STUDENT-EVALUATIONS OF TEACHING IN THE IDEA SYSTEM

This study clarifies issues raised in a series of studies about the In structional Development and Effectiveness Assessment (IDEA) system of students' evaluations of teaching effectiveness (SETEs) and pursues re search stimulated by these studies. On the basis of a paradigm propose d by D. P. Hoyt (1969), W. E. Cashin, R. G. Downey, and G. R. Sixbury (1994) used agreement between student ratings of progress on 10 course objectives and teacher ratings of the importance of each objective to validate the ratings. The major new contribution of the present inves tigation is to evaluate the construct validity of progress ratings (di mensionality and relation to importance), the central feature of the I DEA system. Much of the modest agreement between progress and importan ce ratings was explained by discipline, and the extent of agreement wa s nearly unrelated to teaching effectiveness. Two factors underlying t he 10 progress ratings (Subject/Professional Mastery and Personal Deve lopment) were found, but the correlation between the factors was large (.757). This apparent lack of discrimination among the progress ratin gs may explain in part why various weighted and unweighted averages of the progress ratings performed so similarly. Studies validating IDEA ratings in relation to traditional criteria in SETE research (e.g., st udent learning, teacher self-evaluations, evaluations by former studen ts, interventions designed to improve teaching) are needed.