Highly efficient algorithms for structural clustering of large websites

In this paper, we present a highly scalable algorithm for structurally clustering webpages for extraction. We show that, using only the URLs of the webpages and simple content features, it is possible to cluster webpages effectively and efficiently. At the heart of our techniques is a principled framework, based on the principles of information theory, that allows us to effectively leverage the URLs, and combine them with content and structural properties. Using an extensive evaluation over several large full websites, we demonstrate the effectiveness of our techniques, at a scale unattainable by previous techniques. Copyright © 2011 by the Association for Computing Machinery, Inc. (ACM).

Duke Scholars

Author Ashwinkumar Venkatanaga Machanavajjhala Computer Science