Data integration using similarity joins and a word-based information representation language

The integration of distributed, heterogeneous databases, such as those avai lable on the World Wide Web, poses many problems. Here we. consider the pro blem of integrating data from sources that lack common object identifiers. G solution to this problem is proposed for databases that contain informal, natural-language "names" for objects; most Web-based databases satisfy thi s requirement, since they usually present their information to the end-user through a veneer of text. We describe WHIRL, a "soft" database management system which supports "similarity joins," based on certain robust, general- purpose similarity metrics for text. This enables fragments of text (e.g., informal names of objects) to be used as keys. WHIRL includes textual objec ts as a built-in type, similarity reasoning as a built-in predicate, and an swers every query with a list of answer substitutions that are ranked accor ding to an overall score. Experiments show that WHIRL is much faster than n aive inference methods, even for short queries, and efficient on typical qu eries to real-world databases with tens of thousands of tuples. Inferences made by WHIRL are also surprisingly accurate, equaling the accuracy of hand -coded normalization routines on one benchmark, problem, and outperforming exact matching with a plausible global domain on a second.