Sg. Sireci et Kf. Geisinger, USING SUBJECT-MATTER EXPERTS TO ASSESS CONTENT REPRESENTATION - AN MDS ANALYSIS, Applied psychological measurement, 19(3), 1995, pp. 241-255
Demonstration of content domain representation is of central importanc
e in test validation. An expanded version of the method of content eva
luation proposed by Sireci & Geisinger (1992) was evaluated with respe
ct to a national licensure examination and a nationally standardized s
ocial studies achievement test. Two groups of 15 subject-matter expert
s (SMEs) rated the similarity of all item pairs comprising a test, and
then rated the relevance of the items to the content domains listed i
n the test blueprints. The similarity ratings were analyzed using mult
idimensional scaling (MDS); the item relevance ratings were analyzed u
sing procedures proposed by Hambleton (1984) and Aiken (1980). The SME
s' perceptions of the underlying content structures of the tests emerg
ed in the MDS solutions. All dimensions were germane to the content do
mains measured by the tests. Some of these dimensions were consistent
with the content structure specified in the test blueprint, others wer
e not. Correlation and regression analyses of the MDs item coordinates
and item relevance ratings indicated that using both item similarity
and item relevance data provided greater information of content repres
entation than did using either approach alone. The implications of the
procedure for test validity are discussed and suggestions for future
research are provided.