Item selection counts: A comparison of empirical key and rational scale validities in theory-based and non-theory-based item pools

Little explicit attention has been given to the impact of item pools on the validities and cross-validities of different background data scoring appro aches. This study tests the idea that pools of items theoretically related to the performance of interest will outperform pools of items with no hypot hesized relationship with the criterion. Validities and cross-validities of rational scales and empirical keys created from theory- and non-theory-bas ed item pools were compared for 3 criteria. When size of the item pools was held constant, theory-based empirical keys (correlational and vertical per cent) and rational scales showed larger validities and cross-validities tha n non-theory-based empirical keys (correlational and vertical percent) and showed minimal shrinkage in cross-validities. Even when item pool for the n on-theory-based keys was expanded to include all items in the instrument, t he theory-based keys showed comparable or slightly better validities and cr oss-validities for 2 of the 3 criteria, including college GPA, which was se parated from the predictors by 4 years.