Evaluation of new online automated embolic signal detection algorithm, including comparison with panel of international experts

Background and Purpose-The clinical application of Doppler detection of cir culating cerebral emboli will depend on a reliable automated system of embo lic signal detection; such a system is not currently available. Previous st udies have shown that frequency filtering increases the ratio of embolic si gnal to background signal intensity and that the incorporation of such an a pproach into an offline automated detection system markedly improved perfor mance. In this study, we evaluated an online version of the system. In a si ngle-center study, we compared its performance with that of a human expert on data from 2 clinical situations, carotid stenosis and the period immedia tely after carotid endarterectomy. Because the human expert is currently th e "gold standard" for embolic signal detection, we also compared the perfor mance of the system with an international panel of human experts in a multi center study. Methods-In the single-center evaluation, the performance of the software wa s tested against that of a human expert on 20 hours of data from 21 patient s with carotid stenosis and 18 hours of data from 9 patients that was recor ded after carotid endarterectomy. For the multicenter evaluation, a separat e a-hour data set, recorded from 5 patients after carotid endarterectomy, w as analyzed by 6 different human experts using the same equipment and by th e software. Agreement was assessed by determining the probability of agreem ent. Results-In the 20 hours of carotid stenosis data, there were 140 embolic si gnals with an intensity of greater than or equal to 7 dB. With the software set at a confidence threshold of 60%, a sensitivity of 85.7% and a specifi city of 89.9% for detection of embolic signals were obtained. At higher con fidence thresholds, a specificity >95% could be obtained, but this was at t he expense of a lower sensitivity. In the Is hours of post-carotid endarter ectomy data, there were 411 embolic signals of greater than or equal to 7-d B intensity. When the same confidence threshold was used, a sensitivity of 95.4% and a specificity of 97.5% were obtained. In the multicenter evaluati on, a total of 127 events were recorded as embolic signals by at least 1 ce nter. The total number of embolic signals detected by the 6 different cente rs was 84, 93, 108, 92, 63, and 78. The software set at a confidence thresh old of 60% detected 90 events as embolic signals. The mean probability of a greement, including all human experts and the software, was 0.83, and this was higher than that for 2 human experts and lower than that for 4 human ex perts. The mean values for the 6 human observers were averaged to give P=0. 84, which was similar to that of the software. Conclusions-By using the frequency specificity of the intensity increase oc curring with embolic signals, we have developed an automated detection syst em with a much improved sensitivity. Its performance was equal to that of s ome human experts and only slightly below the mean performance of a panel o f human experts.