Web Mining and Social Networking: Techniques and ... - tud.ttu.ee

2.7 Information Retrieval Performance Evaluation Metrics 23|{retrieved documents}| ∩ |{relevant documents}|recall =|{relevant documents}|(2.8)In binary classification, recall is called sensitivity. So it can be considered as the howmany percentages of relevant documents are correctly retrieved by the query.F-measureThe traditional F-measure or balanced F-score is defined by taking both of precisionand recall into account:2 · precision · recallF = (2.9)precision + recallThis is also sometimes known as the F 1 measure, because recall and precision areevenly weighted. An ideal search mechanism or IR system requires finding as manyrelevant documents from all relevant existing in the repository as possible, but sometimesit is not easy to achieve the optimal results simultaneously. Thus F-measure (orsometimes called F-score) gives an overall performance indicator of the system.Mean Average Precision (MAP)Precision and recall are single-value metrics based on the whole list of documentsreturned by the system. For systems that return a ranked sequence of documents, it isdesirable to also consider the order in which the returned documents are presented.Average precision reflects ranking relevant documents higher. It is the average ofprecisions computed at the point of each of the relevant documents in the rankedsequence:MAP = ∑N n=1 P@(n) · rel (n)(2.10)|relevant documents|where n is the rank, N the number retrieved, rel(n) a binary function indicating therelevance of page at the given rank n, and P@(n) precision at a given cut-off rank|relevant retrieved documents at or below rank n|that defined: P(n)=nDiscounted Cumulative Gain (DCG)In the cases of using a graded relevance scale of documents rather than a binaryvalue of relevance in a search engine result set, the above metrics could not effectivelymeasure the performance required. To deal with this, in particular, anotherevaluation quantity i.e. Discounted Cumulative Gain (DCG) is proposed to measurethe usefulness, or gain, of a document based on its position in the result list. The gainis accumulated cumulatively from the top of the result list to the bottom with the gainof each result discounted at lower ranks.The premise of DCG is that highly relevant documents should ideally appearhigher in a search result list to achieve the bigger accumulated gain. Otherwise it