# Theory of Locality Sensitive Hashing - SNAP - Stanford University

Theory of Locality Sensitive Hashing - SNAP - Stanford University

CS246: Mining Massive Datasets

Jure Leskovec, Stanford University

http://cs246.stanford.edu

Goal: Given a large number (N in the millions or

billions) of text documents, find pairs that are

“near duplicates”

Application:

Detect mirror and approximate mirror sites/pages:

Don’t want to show both in a web search

Problems:

Many small pieces of one doc can appear out of order

in another

Too many docs to compare all pairs

Docs are so large and so many that they cannot fit in

main memory

1/20/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 2

1. Shingling: Convert documents to large sets

of items

2. Minhashing: Convert large sets into short

signatures, while preserving similarity

3. Locality-sensitive hashing: Focus on pairs of

signatures likely to be from similar

documents

1/20/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 3

Docu-

ment

The set

of strings

of length k

that appear

in the doc-

ument

Signatures :

short integer

vectors that

represent the

sets, and

reflect their

similarity

1/20/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets

Locality-

sensitive

Hashing

Candidate

pairs :

those pairs

of signatures

that we need

to test for

similarity.

4

A k-shingle (or k-gram) is a sequence of k

tokens that appears in the document

Example: k=2; D 1= abcab

Set of 2-shingles: C 1 = S(D 1) = {ab, bc, ca}

Represent a doc by the set of hash values of

its k-shingles

A natural document similarity measure is then

the Jaccard similarity:

Sim(D 1, D 2) = |C 1∩C 2|/|C 1∪C 2|

1/20/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 5

Prob. h π(C 1) = h π(C 2) is the same as Sim(D 1, D 2):

Pr[h π(C 1) = h π(C 2)] = Sim(D 1, D 2)

Permutation π

1

3

7

6

2

5

4

4

2

1

3

6

7

5

3

4

7

6

1

2

5

1

1

0

0

0

1

1

Input matrix

0

0

1

1

1

0

0

1

0

0

0

0

1

1

0

1

1

1

1

0

0

Signature matrix M

1/20/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 6

2

1

2

1

2

1

4

1

2

1

2

1

ands

Hash cols of signature matrix M:

Similar columns likely hash to same bucket

r rows

Divide matrix M into b bands of r rows

Candidate column pairs are those that hash

to the same bucket for ≥ 1 band

Buckets

Matrix M

Prob. of sharing

a bucket

Sim. threshold s

1/20/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 7

2

1

2

1

2

1

4

1

2

1

2

1

The S-curve is where the “magic” happens

Probability of sharing

a bucket

Remember:

Probability of

equal hash-values

= similarity

Similarity s of two sets

This is what 1 band gives you

Pr[h π(C 1) = h π(C 2)] = sim(D 1, D 2)

No chance

if s < t

Probability

= 1 if s > t

t

This is what we want!

How?

By picking r and s!

1/20/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets 8

Remember:

b bands, r rows/band

Columns C 1 and C 2 have

similarity s

Pick any band (r rows)

Prob. that all rows in band equal = s r

Prob. of sharing

a bucket

Prob. that some row in band unequal = 1 - s r

Prob. that no band identical = (1 - s r ) b

Prob. that at least 1 band identical =

1 - (1 - s r ) b

Similarity s

1/20/2012 Jure Leskovec, Stanford C246: Mining Massive Datasets

9

Prob. sharing a bucket

Prob. sharing a bucket

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

r=1..10, b=1

0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

r=1, b=1..10

0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Similarity

0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 10

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

r=5, b=1..50

r=10, b=1..50

0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Similarity

1 - (1 - s r ) b

Docu-

ment

The set

of strings

of length k

that appear

in the doc-

ument

Signatures :

short integer

vectors that

represent the

sets, and

reflect their

similarity

Locality-

sensitive

Hashing

Candidate

pairs:

those pairs

of signatures

that we need

to test for

similarity.

We have used LSH to find similar documents

In reality, columns in large sparse matrices with

high Jaccard similarity

e.g., customer/item purchase histories

Can we use LSH for other distance measures?

e.g., Euclidean distances, Cosine distance

Let’s generalize what we’ve learned!

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

12

For min-hash signatures, we got a min-hash

function for each permutation of rows

An example of a family of hash functions:

A “hash function” is any function that takes two

elements and says whether or not they are “equal”

Shorthand: h(x) = h(y) means “h says x and y are equal”

A family of hash functions is any set of hash functions

A set of related hash functions generated by some mechanism

We should be able to efficiently pick a hash function at

random from such a family

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

13

Suppose we have a space S of points with

a distance measure d

A family H of hash functions is said to be

(d 1,d 2,p 1,p 2)-sensitive if for any x and y in S :

1. If d(x,y) < d 1, then the probability over all h ∈ H,

that h(x) = h(y) is at least p 1

2. If d(x,y) > d 2, then the probability over all h ∈ H,

that h(x) = h(y) is at most p 2

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

14

High

probability;

at least p 1

Pr[h(x) = h(y)]

p 1

p 2

d 1

d(x,y)

d 2

Low

probability;

at most p 2

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

15

Let:

S = sets,

d = Jaccard distance,

H is family of minhash functions for all

permutations of rows

Then for any hash function h ∈ H:

Pr[h(x)=h(y)] = 1-d(x,y)

Simply restates theorem about min-hashing in

terms of distances rather than similarities

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

16

Claim: H is a (1/3, 2/3, 2/3, 1/3)-sensitive family

for S and d.

If distance < 1/3

(so similarity > 2/3)

Then probability

that min-hash values

agree is > 2/3

For Jaccard similarity, minhashing gives us a

(d 1,d 2,(1-d 1),(1-d 2))-sensitive family for any d 1

Can we reproduce the “S-curve”

effect we saw before for any LS

family?

Prob. of sharing

a bucket

The “bands” technique we learned

for signature matrices carries over

to this more general setting

Two constructions:

AND construction like “rows in a band”

OR construction like “many bands”

Similarity s

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

18

Given family H, construct family H’ consisting

of r functions from H

For h = [h 1,…,h r] in H’,

h(x)=h(y) if and only if h i(x)=h i(y) for all i

Theorem: If H is (d 1,d 2,p 1,p 2)-sensitive,

then H’ is (d 1,d 2,(p 1) r ,(p 2) r )-sensitive

Proof: Use the fact that h i ’s are independent

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

19

Given family H, construct family H’ consisting of

b functions from H

For h = [h 1,…,h b] in H’,

h(x)=h(y) if and only if h i(x)=h i(y) for at least 1 i

Theorem: If H is (d 1,d 2,p 1,p 2)-sensitive,

then H’ is (d 1,d 2,1-(1-p 1) b ,1-(1-p 2) b )-sensitive

Proof: Use the fact that h i ’s are independent

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

20

AND makes all probs. shrink, but by choosing r

correctly, we can make the lower prob. approach

0 while the higher does not

OR makes all probs. grow, but by choosing b

correctly, we can make the upper prob. approach

1 while the lower does not

Prob. sharing a bucket

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

AND

r=1..10, b=1

0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Prob. sharing a bucket

0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 21

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

OR

r=1, b=1..10

Similarity of a pair of items Similarity of a pair of items

1-(1-s r ) b

-way AND followed by b-way OR construction

Exactly what we did with min-hashing

If bands match in all r values hash to same bucket

Cols that are hashed into at least 1 common bucket

Candidate

Take points x and y s.t. Pr[h(x) = h(y)] = p

H will make (x,y) a candidate pair with prob. p

Construction makes (x,y) a candidate pair with

probability 1-(1-p r ) b

The S-Curve!

Example: Take H and construct H’ by the AND

construction with r = 4. Then, from H’, construct

H’’ by the OR construction with b = 4

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

22

50 hash-functions (r=5, b=10)

Prob. sharing a bucket

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Similarity

Blue area: False Negative rate

Green area: False Positive rate

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 23

50 hash-functions (b*r=50)

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

r=2, b=25

r=5, b=10

r=10, b=5

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 24

p 1-(1-p 4 ) 4

.2 .0064

.3 .0320

.4 .0985

.5 .2275

.6 .4260

.7 .6666

.8 .8785

.9 .9860

Example: Transforms a

(.2,.8,.8,.2)-sensitive

family into a

(.2,.8,.8785,.0064)-

sensitive family.

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

25

Apply a b-way OR construction followed by an

r-way AND construction

Transforms probability p into (1-(1-p) b ) r .

The same S-curve, mirrored horizontally and

vertically

Example: Take H and construct H’ by the OR

construction with b = 4. Then, from H’,

construct H’’ by the AND construction

with r = 4.

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

26

p (1-(1-p) 4 ) 4

.1 .0140

.2 .1215

.3 .3334

.4 .5740

.5 .7725

.6 .9015

.7 .9680

.8 .9936

Example: Transforms a

(.2,.8,.8,.2)-sensitive

family into a

(.2,.8,.9936,.1215)-

sensitive family.

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 27

Example: Apply the (4,4) OR-AND

construction followed by the (4,4) AND-OR

construction

Transforms a (0.2,0.8,0.8,.02)-sensitive family

into a (0.2,0.8,0.9999996,0.0008715)sensitive

family

Note this family uses 256 (=4*4*4*4) of the

original hash functions

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

28

Pick any two distances x < y

Apply constructions to produce (x, y, p, q)sensitive

family, where p is almost 1 and q is

almost 0

The closer to 0 and 1 we get, the more hash

functions must be used

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

29

LSH methods for other distance metrics:

Points

Cosine: Random Hyperplanes

Euclidean: Project on lines

Signatures: short

integer signatures that

reflect their similarity

Design a (x,y,p,q)-sensitive

family of hash functions

(for that particular

distance metric)

Locality-

sensitive

Hashing

Candidate pairs :

those pairs of

signatures that

we need to test

for similarity.

Amplify the family

using AND and OR

constructions

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 30

For cosine distance,

d(A, B) = θ = arccos(A⋅B/ǁAǁǁBǁ)

there is a technique called

Random Hyperplanes

Technique similar to minhashing

Random Hyperplanes is a (d 1,d 2,(1-d 1/180),(1d

2/180))-sensitive family for any d 1 and d 2

Reminder: (d 1,d 2,p 1,p 2)-sensitive

1. If d(x,y) < d 1, then prob. that h(x) = h(y) is at least p 1

2. If d(x,y) > d 2, then prob. that h(x) = h(y) is at most p 2

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

31

Pick a random vector v, which determines a

hash function h v with two buckets

h v(x) = +1 if v⋅x > 0; = -1 if v⋅x < 0

LS-family H = set of all functions derived

from any vector

Claim: For points x and y,

Pr[h(x) = h(y)] = 1 – d(x,y)/180

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

32

Look in the

plane of x

and y.

Hyperplane

normal to v

h(x) = h(y)

v

θ

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 33

x

y

Prob[Red case]

= θ/180

v

Hyperplane

normal to v

h(x) ≠ h(y)

Pick some number of random vectors, and

hash your data for each vector

The result is a signature (sketch) of +1’s

and –1’s for each data point

Can be used for LSH like the minhash

signatures for Jaccard distance

Amplified using AND and OR constructions

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

34

Expensive to pick a random vector in M

dimensions for large M

M random numbers

A more efficient approach

It suffices to consider only vectors v

consisting of +1 and –1 components

Why is this more efficient?

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

35

Simple idea: Hash functions correspond to lines

Partition the line into buckets of size a

Hash each point to the bucket containing its

projection onto the line

Nearby points are always close; distant points

are rarely in same bucket

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

36

Bucket

width a

Points at

distance d

If d

If d >> a, θ must

be close to 90 o

for there to be

any chance points

go to the same

bucket.

Bucket

width a

Points at

distance d

θ

d cos θ

Randomly

chosen

line

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

38

If points are distance d < a/2, prob. they are in same

bucket ≥ 1- d/a = ½

If points are distance > 2a apart, then they can be in

the same bucket only if d cos θ ≤ a

cos θ ≤ ½

60 < θ < 90

i.e., at most 1/3 probability.

Yields a (a/2, 2a, 1/2, 1/3)-sensitive family of hash

functions for any a

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

39

For previous distance measures, we could

any x < y, and drive p and q to 1 and 0 by

AND/OR constructions

Here, we seem to need y > 4x

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

40

But as long as x < y, the probability of points

at distance x falling in the same bucket is

greater than the probability of points at

distance y doing so

Thus, the hash family formed by projecting

onto lines is an (x, y, p, q)-sensitive family for

some p > q

Then, amplify by AND/OR constructions

1/19/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets

41

More magazines by this user
Similar magazines