A visual exploration of the orderliness of TREC relevance judgments

TREC topic specification statements 1-50 are converted to a similarity matr ix, scaled, and plotted. Two close topics and two distant topics are select ed from within the topic visual field. Subsequent scaling and visualization of documents associated with the close topics reveals a strong mixing of d ocuments from both topic sets. Scaling and visualization of documents assoc iated with the distant topics reveals a bifurcated distribution of document s from both topic sets. Relevant documents in both cases present near the c enter of both visualizations. Scaling and visualization of documents by mul tidimensional scaling using a maximum likelihood estimation method is shown to accurately model token similarity relationships among topic specificati on statements. The implications of these findings for prior critical argume nts regarding IR test collections generally, and TREC specifically, by othe r scholars is examined.