Compound Words Query Parser - ApacheCon

Compound Words 

Query Parser 

Mikhail Khludnev

Agenda 

● eCommerce Search is Special 

● Compound Words Query Parser 

● Staged Search 

● Match Spotting

Part I 

eCommerce Search is Special

Boolean 

Retrieval 

(+ -) 

Vector 

Space 

Model 

http://upload.wikimedia. 

org/wikipedia/commons/f/ff/Vector_space_ 

model.jpg

Does she search this way? 

+"michael kors" type:handbag 

http://tustinhairsalon.com/wpcontent/uploads/2011/02/Blonde_Hair_stylist_s 

tylists_blondes_tustin_santa_ana_orange_cou

That's how she searches! 

michael kors hand bag 

http://tustinhairsalon.com/wpcontent/uploads/2011/02/Blonde_Hair_stylist_s 

tylists_blondes_tustin_santa_ana_orange_cou

Plain Text Documents 

Lorem ipsum dolor sit amet, 

Else Jeans Skinny Jeans, consectetur adipisicing elit, sed 

Colored Denim Dark Green- do eiusmod tempor incididunt 

Wash 

ut labore et dolore magna 

In a dark green wash perfect aliqua. Ut enim ad minim 

for fall, these Else Jeans skinny veniam, quis nostrud 

jeans hit the colored-denim exercitation ullamco laboris nisi 

trend right on the mark! Lyocell ut aliquip ex ea commodo 

cotton rayon polyester Lycra consequat. Duis aute irure 

Machine washable Imported dolor in reprehenderit in 

Low rise: approx. 8 inches voluptate velit esse cillum 

Skinny fit Skinny leg Zipper fly dolore eu fugiat nulla pariatur. 

with button closure 5-pocket Excepteur sint occaecat 

style Dark green wash, colored cupidatat non proident, sunt in 

denim Waistband with belt culpa qui officia deserunt mollit 

loops Inseam: approx. 30-1/2 anim id est laborum. 

inches

Plain Text Documents 

approx velit 

Lorem ipsum dolor sit amet, 

Else Jeans Skinny Jeans, consectetur adipisicing elit, sed 

Colored Denim Dark Green- do eiusmod tempor incididunt 

Wash 

ut labore et dolore magna 

In a dark green wash perfect aliqua. Ut enim ad minim 

for fall, these Else Jeans skinny veniam, quis nostrud 

jeans hit the colored-denim exercitation ullamco laboris nisi 

trend right on the mark! Lyocell ut aliquip ex ea commodo 

cotton rayon polyester Lycra consequat. Duis aute irure 

Machine washable Imported dolor in reprehenderit in 

Low rise: approx. 8 inches voluptate velit esse cillum 

Skinny fit Skinny leg Zipper fly dolore eu fugiat nulla pariatur. 

with button closure 5-pocket Excepteur sint occaecat 

style Dark green wash, colored cupidatat non proident, sunt in 

denim Waistband with belt culpa qui officia deserunt mollit 

loops Inseam: approx. 30-1/2 anim id est laborum. 

inches

Product Catalog Document 

BRAND: 

GENDER: 

TYPE: 

FIT: 

OCCASION: 

LEG: 

WEIGHT: 

COLOR: 

Calvin Klein 

Women's 

Jeans 

At Waist 

Casual 

Classic Straight 

Super Skinny 

Light Wash

Polysemy Problem

Polysemy Problem 

COLOR:"Pink" TYPE:"Sweater" 

BRAND: "Tomas Pink" TYPE:"Sweater" 

BRAND: "Pink Lotus" TYPE:"Sweater"

http://www.shopstyle.com/browse/sweaters? 

fts=pink+sweater&fl=b7&fl=b28462&fl=b1830&fl=b15382&fl=b12130&fl=b737

http://www.shopstyle.com/browse/sweaters? 

fts=pink+sweater&fl=b7&fl=b28462&fl=b1830&fl=b15382&fl=b12130&fl=b737

pink jeans by google 

https://www.google.com/search?hl=ru&tbm=shop&q=pink+sweater&oq=pink+sweater&gs_l=products-cc.3.. 

0l7j0i5l3.4658.7244.0.7699.12.11.0.1.1.0.298.1153.5j4j1.10.0...0.0...1ac.1. 

denzbOMlNDg#q=pink+sweater&hl=ru&tbm=shop&ei=oDmIUPXgAoqC4gSylYGACg&start=20&sa=N&num=20&

pink jeans by google 

https://www.google.com/search?hl=ru&tbm=shop&q=pink+sweater&oq=pink+sweater&gs_l=products-cc.3.. 

0l7j0i5l3.4658.7244.0.7699.12.11.0.1.1.0.298.1153.5j4j1.10.0...0.0...1ac.1. 

denzbOMlNDg#q=pink+sweater&hl=ru&tbm=shop&ei=oDmIUPXgAoqC4gSylYGACg&start=20&sa=N&num=20&

ight pink jeans

ight pink jeans

Polysemy Problem No.2 

sweater 

BRAND:"Pink Rose" TYPE:"Sweater" 

COLOR:"Pink","Rose" TYPE:"Sweater"

pink rose sweater 

pink rose jeans 

problem

pink rose sweater 

precise pink rose jeans

Ranking Can't Help

Ranking Can't Help

Precision 

is a MUST!

Think About the User 

http://tustinhairsalon.com/wpcontent/uploads/2011/02/Blonde_Hair_stylist_stylists_blondes_tustin_ 

santa_ana_orange_county.jpg

http://www.amazon.com/Silver-Jeans/e/2206256011

Wrap-Up 

● Product Descriptions Differs to Plain Text 

● No Query Syntax 

● Polysemy Problem 

● Ranking Can't Help When Precision is Low 

eCommerce Search is Special!

Part II 

Compound Words Query Parser

Don't tokenize at index time

lack and decker toaster oven 

"black" 

"black and" 

"black and decker" 

... "and" 

"and decker" 

"and decker toaster" 

... "decker" 

"decker toaster" 

"decker toaster oven" 

"toaster oven"


"black" 

"black and" 


... "and" 

"and decker" 


... "decker" 



..."toaster oven" 

ID: 

123 

BRAND: 

TYPE: 

COLOR: 

"black and 

decker" 

"toaster 

oven" 

white


"black" 

"black and" 


... "and" 

"and decker" 


... "decker" 

"toaster" 


... "oven" 

ID: 

456 

BRAND: 

TYPE: 

COLOR: 

"black and 

decker" 

"oven" 

white


"black" 

"black and" 


... "and" 

"and decker" 


... "decker" 



..."toaster oven" 

ID: 

789 

BRAND: 

TYPE: 

COLOR: 

"decker 

toaster" 

"toaster 

oven" 

black

fee people sweater

Part III 

Staged Search

Expected

Actual 

Expected

Actual 

Expected 

× OK 

× 

False 

Positive 

True 

Positive 

False 

Negative

Actual 

Expected 

Precision = 

Actual ∩ Expected 

Actual

Actual 

Expected 

Recall = 

Actual ∩ Expected 

Expected

100% Recall 

Actual 

Expected

Tuning Trade-Off 

Actual 

Expected


Actual 

Expected


Actual 

Expected


Actual 

Expected

defaultOperator 

AND 

OR 

numFound=100 

numFound=1000

defaultSearchField / queryFields 

BRAND 

BRAND 

TYPE 

STYLE 

text 

numFound=100 

numFound=1000

Stemming & Synonyms 

text 

text_stemmed 

numFound=100 

numFound=1000

ORDer by DESCending Precision 

BRAND:(Silver AND Jeans) 

numFound=100 

BRAND:(Silver OR Jeans) 

BRAND_stem:(Silver OR Jean) 

OR 

text_stem:(Silver OR Jean) 

numFound=1000


numFound>0 


numFound>0 


OR 

text_stem:(Silver OR Jean)

Searching Complexity 

O(n log p) 

num found 

page size 

It Depends

Stages Ratios 

Compounds - Exact Match 

Compounds - Omit Words 

50% 

30% 

Text Match 

20%

Think About The User 

http://tustinhairsalon.com/wpcontent/uploads/2011/02/Blonde_Hair_stylist_stylists_blondes_tus

Part IV 

Match Spotting


numFound>0 


numFound>0 


OR 

text_stem:(Silver OR Jean)

Facet Counts

Explain Info 

15.110098 = (MATCH) sum of: 

11.197777 = (MATCH) weight(BRAND:Free People in 28807), product 

of: 

0.81810623 = queryWeight(BRAND:Free People), product of: 

9.678479 = idf(docFreq=232, maxDocs=1368899) 

0.08452839 = queryNorm 

13.687436 = (MATCH) fieldWeight(BRAND:Free People in 28807), product of: 

1.4142135 = tf(termFreq(BRAND:Free People)=2) 


1.0 = fieldNorm(field=BRAND, doc=28807) 

3.912321 = (MATCH) weight(PRODUCT_TYPE:SWEATER in 

28807), product of: 

0.5750671 = queryWeight(PRODUCT_TYPE:SWEATER), product of: 


0.08452839 = queryNorm 

6.8032427 = (MATCH) fieldWeight(PRODUCT_TYPE:SWEATER in 28807), 

product of: 

1.0 = tf(termFreq(PRODUCT_TYPE:SWEATER)=1) 


1.0 = fieldNorm(field=PRODUCT_TYPE, doc=28807)

Explain 

Info 

Facet 

Counts 

Match 

Spotting

found 190 

BRAND: "Silver Jeans" - (156) 

COLOR: "Silver" TYPE: "Jeans" - (34)

http://www.amazon.com/Silver-Jeans/e/2206256011

Wrap-Up 

● eCommerce Search is Special 

● Compound Words Query Parser 

● Staged Search 

● Match Spotting 

http://goo.gl/2OHk2

Compound Words Query Parser - ApacheCon

Create successful ePaper yourself

Delete template?

Save as template?