USING SUBJECT-MATTER EXPERTS TO ASSESS CONTENT REPRESENTATION - AN MDS ANALYSIS

Demonstration of content domain representation is of central importanc e in test validation. An expanded version of the method of content eva luation proposed by Sireci & Geisinger (1992) was evaluated with respe ct to a national licensure examination and a nationally standardized s ocial studies achievement test. Two groups of 15 subject-matter expert s (SMEs) rated the similarity of all item pairs comprising a test, and then rated the relevance of the items to the content domains listed i n the test blueprints. The similarity ratings were analyzed using mult idimensional scaling (MDS); the item relevance ratings were analyzed u sing procedures proposed by Hambleton (1984) and Aiken (1980). The SME s' perceptions of the underlying content structures of the tests emerg ed in the MDS solutions. All dimensions were germane to the content do mains measured by the tests. Some of these dimensions were consistent with the content structure specified in the test blueprint, others wer e not. Correlation and regression analyses of the MDs item coordinates and item relevance ratings indicated that using both item similarity and item relevance data provided greater information of content repres entation than did using either approach alone. The implications of the procedure for test validity are discussed and suggestions for future research are provided.