Web Mining and Social Networking: Techniques and ... - tud.ttu.ee

4Web Content MiningIn recent years the growth of the World Wide Web exceeded all expectations. Today thereare several billions of HTML documents, pictures and other multimedia files available viaInternet and the number is still rising. But considering the impressive variety of the Web,retrieving interesting contents has become a very difficult task. Web Content Mining uses theideas and principles of data mining and knowledge discovery to screen more specific data.The use of the Web as a provider of information is unfortunately more complex than workingwith static databases. Because of its very dynamic nature and its vast number of documents,there is a need for new solutions that are not depending on accessing the complete data onthe outset. Another important aspect is the presentation of query results. Due to its enormoussize, a Web query can retrieve thousands of resulting Web pages. Thus meaningful methodsfor presenting these large results are necessary to help a user to select the most interestingcontent. In this chapter we will discuss several basic topics of Web document representation,Web search, short text processing, topic extraction and Web opinion mining.4.1 Vector Space ModelThe representation of a set of documents as vectors in a common vector space is known as thevector space model and is fundamental to a host of information retrieval operations rangingfrom scoring documents on a query, document classification and document clustering. We firstdevelop the notion of a document vector that captures the relative importance of the terms ina document.Towards this end, we assign to each term in a document a weight for that term, that dependson the number of occurrences of the term in the document. We would like to computea score between a query term and a document based on the weight of t in d. The simplestapproach is to assign the weight to be equal to the number of occurrences of term t in documentd. This weighting scheme is referred to as term frequency and is denoted tf t,d , with thesubscripts denoting the term and the document in order.Raw term frequency as above suffers from a critical problem: all terms are consideredequally important when it comes to assessing relevancy on a query. In fact certain terms havelittle or no discriminating power in determining relevance. For instance, a collection of documentson the auto industry is likely to have the term auto in almost every document. To thisend, we introduce a mechanism for attenuating the effect of terms that occur too often in theG. Xu et al., Web Mining and Social Networking,DOI 10.1007/978-1-4419-7735-9_4, © Springer Science+Business Media, LLC 2011