LiveBench-1: Continuous benchmarking of protein structure prediction servers

We present a novel, continuous approach aimed at the large-scale assessment of the performance of available fold-recognition servers. Six popular serv ers were investigated: PDB-Blast, FFAS, T98-lib, Gen-THREADER, 3D-PSSM, and INBGU. The assessment was conducted using as prediction targets a large nu mber of selected protein structures released from October 1999 to April 200 0. A target was selected if its sequence showed no significant similarity t o any of the proteins previously available in the structural database. Over all, the servers were able to produce structurally similar models for one-h alf of the targets, but significantly accurate sequence-structure alignment s were produced for only one-third of the targets. We further classified th e targets into two sets: easy and hard. We found that all servers were able to find the correct answer for the vast majority of the easy targets if a structurally similar fold was present in the server's fold libraries. Howev er, among the hard targets--where standard methods such as PSI-BLAST fail-- the most sensitive fold-recognition servers were able to produce similar mo dels for only 40% of the cases, half of which had a significantly accurate sequence-structure alignment. Among the hard targets, the presence of updat ed libraries appeared to be less critical for the ranking. An "ideally comb ined consensus" prediction, where the results of all servers are considered , would increase the percentage of correct assignments by 50%. Each server had a number of cases with a correct assignment, where the assignments of a ll the other servers were wrong. This emphasizes the benefits of considerin g more than one server in difficult prediction tasks. The LiveBench program (http://BioInfo.PL/LiveBench) is being continued, and all interested devel opers are cordially invited to join.