A knowledge-based approach for duplicate elimination in data cleaning

Existing duplicate elimination methods for data cleaning work on the basis of computing the degree of similarity between nearby records in a sorted da tabase. High recall can be achieved by accepting records with low degrees o f similarity as duplicates, at the cost of lower precision. High precision can be achieved analogously at the cost of lower recall. This is the recall -precision dilemma. We develop a generic knowledge-based framework for effe ctive data cleaning that can implement any existing data cleaning strategie s and more. We propose a new method for computing transitive closure under uncertainty for dealing with the merging of groups of inexact duplicate rec ords and explain why small changes to window sizes has little effect on the results of the sorted neighborhood method. Experimental study with two rea l-world datasets show that this approach can accurately identify duplicates and anomalies with high recall and precision, thus effectively resolving t he recall-precision dilemma. (C) 2001 Published by Elsevier Science Ltd.