Assessment of resident knowledge: Subjective assessment versus performanceon the ACR in-training examination

Rationale and Objectives. The authors assessed the ability of faculty and r esidents to predict the ranked performance of residents on the American Col lege of Radiology (ACR) In-Training Examination. Materials and Methods. Radiology faculty at Penn State Geisinger Health Sys tem (PSGHS), the Medical College of Virginia (MCV), and the University of V irginia (WA) and residents at PSGHS and MCV ranked the expected performance s of residents taking the 1997 ACR In-Training Examination. Surveyed facult y and residents were blinded to the actual performances on the examination. Forty-nine residents took the examination (21 at PSGHS, 22 at MCV, six at UVA), and 37 faculty members (11 at PSGHS, Il at MCV, 15 at UVA) participat ed in the study. Correlation analysis was performed to assess the agreement between the subjective and actual ranking of residents in each residency c lass. Results. Faculty were moderately accurate in the overall ranking of residen t performances (r = 0.34). High levels of concordance for ranking individua l residents correlated with accuracy in only certain cases. Differences in agreement and accuracy of the respondents existed between PSGHS and MCV (P =.0001 and .0014, respectively). The concordance of respondents increased s ignificantly from the 1st- to the 2nd-year class at MCV (P=.0002), whereas accuracy increased significantly between these classes for the PSGHS (P =.0 42). Conclusion. Faculty are only moderately successful in ranking resident perf ormances on the ACR In-Training Examination, and a high level of agreement is not necessarily indicative of increased accuracy. The concordance accura cy of subjective rankings differ among residency programs and classes.