The results of the first Critical Assessment of Fully Automated Structure P
rediction (CAFASP-1) are presented. The objective was to evaluate the succe
ss rates of fully automatic web servers for fold recognition which are avai
lable to the community This study was based on the targets used in the thir
d meeting on the Critical Assessment of Techniques for Protein Structure Pr
ediction (CASP-3). However, unlike CASP-3, the study was not a blind trial,
as it was held after the structures of the targets were known. The aim was
to assess the performance of methods without the user intervention that se
veral groups used in their CASP-3 submissions. Although it is clear that "h
uman plus machine" predictions are superior to automated ones, this CAFASP-
1 experiment is extremely valuable for users of our methods; it provides an
indication of the performance of the methods alone, and not of the "human
plus machine" performance assessed in CASP, This information may aid users
in choosing which programs they wish to use and in evaluating the reliabili
ty of the programs when applied to their specific prediction targets. In ad
dition, evaluation of fully automated methods is particularly important to
assess their applicability at genomic scales.
For each target, groups submitted the top-ranking folds generated from thei
r servers. In CAFASP-1 we concentrated on fold-recognition web servers only
and evaluated only recognition of the correct fold, and not, as in CASP-3,
alignment accuracy. Although some performance differences appeared within
each of the four target categories used here, overall, no single server has
proved markedly superior to the others. The results showed that current fu
lly automated fold recognition servers can often identify remote similariti
es when pairwise sequence search methods fail. Nevertheless, in only a few
cases outside the family-level targets has the score of the top-ranking fol
d been significant enough to allow for a confident fully automated predicti
on. Because the goals, rules, and procedures of CAFASP-1 were different fro
m those used at CASP-3, the results reported here are not comparable with t
hose reported in CASP-3. Nevertheless, it is clear that current automated f
old recognition methods can not yet compete with "human-expert plus machine
" predictions.
Finally, CAFASP-1 has been useful in identifying the requirements for a fut
ure blind trial of automated served-based protein structure prediction. (C)
1999 Wiley-Liss, Inc.