Theory of Locality Sensitive Hashing - SNAP - Stanford University

More documents

Recommendations

Info

Goal: Given a large number (N in the millions or billions) of text documents, find pairs that are “near duplicates” Application: Detect mirror and approximate mirror sites/pages: Don’t want to show both in a web search Problems: Many small pieces of one doc can appear out of order in another Too many docs to compare all pairs Docs are so large and so many that they cannot fit in main memory 1/20/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 2
1. Shingling: Convert documents to large sets of items 2. Minhashing: Convert large sets into short signatures, while preserving similarity 3. Locality-sensitive hashing: Focus on pairs of signatures likely to be from similar documents 1/20/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 3
Page 1: CS246: Mining Massive Datasets Jure
Page 5 and 6: A k-shingle (or k-gram) is a sequen
Page 7 and 8: ands Hash cols of signature matrix
Page 9 and 10: Remember: b bands, r rows/band Col
Page 11 and 12: Docu- ment The set of strings of le
Page 13 and 14: For min-hash signatures, we got a m
Page 15 and 16: High probability; at least p 1 Pr[h
Page 17 and 18: Claim: H is a (1/3, 2/3, 2/3, 1/3)-
Page 19 and 20: Given family H, construct family H
Page 21 and 22: AND makes all probs. shrink, but by
Page 23 and 24: Picking r and b to get the best 50
Page 25 and 26: p 1-(1-p 4 ) 4 .2 .0064 .3 .0320 .4
Page 27 and 28: p (1-(1-p) 4 ) 4 .1 .0140 .2 .1215
Page 29 and 30: Pick any two distances x < y Start
Page 31 and 32: For cosine distance, d(A, B) = θ =
Page 33 and 34: Look in the plane of x and y. Hyper
Page 35 and 36: Expensive to pick a random vector i
Page 37 and 38: Bucket width a Points at distance d
Page 39 and 40: If points are distance d < a/2, pro
Page 41: But as long as x < y, the probabili

Theory of Locality Sensitive Hashing - SNAP - Stanford University

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?