Query optimization in relational databases

Query optimization in relational databases 

Java C (speed-up factor 2-5) 

C Assembler (speed-up factor 2-5) , but 

SQL optimized SQL (speed-up by 1-100 and more) - Why? 

Chapter 2: Storage structures and indices 

- database buffer 

- storage structures 

- indices 

Chapter 3: Query optimization 

- logical query optimization 

- physical query optimization 

- query execution plans and cost estimation 

Databases and Information Systems 1 - WS 2005 / 06 - Prof. Dr. Böttcher - Query optimization in RDBMS / 1

Selectivity of queries 

significant for size of intermediate result 

selectivity of the selection with condition B : 

selectivity ( S B (R) ) = | S B (R) | / | R | 

selectivity of the Join with condition B : 

selectivity ( R |X| B S ) = | R |X| B S | / ( | R | * | S | ) 

estimated (e.g. based on samples, histograms) 


Goal of logical query optimization 

SQL queries 

select A1,...,An from R1,..., Rm where B 

correspond to algebra expression 

P(A1,...,An) S B ( R1 x ... x Rm ) 

- very large intermediate results 

==> task: 

obtain the same result with 

smaller intermediate results 

e.g. move selection and projection inside expressions 

as far as possible 


Logical query optimization– example 

select S.Vorname, S.Name 

from Student S, Hört H, Kurs K 

where K.titel = 'Datenbanken' and K.kursnr = H.kursnr and S.mnr=H.mnr ; 

200 

200 

500.000.000 

10.000 

P S.Vorname, S.Name 

| 

S K.titel = 'Datenbanken' and K.kursnr = H.kursnr and S.mnr=H.mnr 

| 

X 500.000.000.000 

/ \ 

X K 1.000 

/ \ 

S H assumptions: 

50.000 

10000 students, each taking 5 courses on avarage 

1000 courses, 2 of which are entitled ‘Datenbanken‘ 

and taken by 100 students each 



A possible optimization : 

200 P S.Vorname, S.Name 

| 

200 |X| 

K.kursnr = H.kursnr 

/ \ 2 

50.000 

|X| S K.titel = 'Datenbanken' 

S.mnr=H.mnr \ 

/ \ \ 

S H K 1.000 

10.000 

50.000 

assumptions: 






A better optimization : 

200 

10.000 

50.000 

P S.Vorname, S.Name 

| 

|X| 200 

S.mnr=H.mnr 

/ \ 200 

S |X| 


/ \ 

H 

1.000 

2 

S K.titel = 'Datenbanken' 

| 

K 

assumptions: 





Rules of logical query optimization 

Union, intersection, cartesian product and join 

are commutative and associative . 

R1 U R2 = R2 U R1 

R1 ∩ R2 = R2 ∩ R1 

R1 X R2 = R2 X R1 

R1 |X| B R2 = R2 |X| B R1 

( R1 U R2 ) U R3 = R1 U ( R2 U R3 ) 

( R1 ∩ R2 ) ∩ R3 = R1 ∩ ( R2 ∩ R3 ) 

( R1 X R2 ) X R3 = R1 X ( R2 X R3 ) 

( R1 |X| B R2 ) |X| B R3 = R1 |X| B ( R2 |X| B R3 ) 



selections can be splitted and their order can be switched : 

S B1 and B2 (R) = S B1 (S B2 (R)) = S B2 (S B1 (R)) 

push selections inside union, difference and intersection: 

S B ( R1 U R2 ) = S B ( R1 ) U S B ( R2 ) 

S B ( R1 - R2 ) = S B ( R1 ) - S B ( R2 ) 

S B ( R1 ∩ R2 ) = S B ( R1 ) ∩ S B ( R2 ) 



push selection inside a join, i.e. to a join argument, 

S B ( R1 |X| B2 R2 ) = S B ( R1 ) |X| B2 R2 

if B only uses attributes of R1 

push selection inside an argument of a cartesian product 

S B ( R1 X R2 ) = S B ( R1 ) X R2 

if B only uses attributes of R1 

if this is impossible for both R1 and R2, 

i.e., B uses attributes of R1 and of R2 : 

substitute selection applied to cartesian product with join 

S B ( R1 X R2 ) = R1 |X| B R2 



order of projection and selection can be switched, 

if the projection yields all attributes needed for the selection condition : 

S B ( P A1,...,Am ( R1 ) ) = P A1,...,Am ( S B ( R1 ) ) 

if B only uses attributes of A1,...,Am. 

push projection inside union 

P A1,...,Am ( R1 U R2 ) = P A1,...,Am ( R1 ) U P A1,...,Am ( R2 ) 

push projection into the join, i.e. apply it a join argument, 

if the join attributes are contained in the projection 

P A1,...,Am 

( R1 |X| B R2 ) = P A1,...,Am 

( ( P A1,...,Am,AB1,...,ABn 

( R1 ) ) |X| B 

R2 ) 

where AB1,...,ABn are the attributes of R1 needed to check the join condition B. 

projections can be combined and inserted additionally 

P A1,...,Am ( R1 ) = P A1,...,Am ( P A1,...,Am,AB1,...,ABn ( R1 ) ) 


Logical query optimization - steps 

SQL query represented as a logical query tree 

apply the following optimizations to this query tree 

• split and push down selections 

• combine selections and cartesian products to joins. 

• determine join sequence with smallest intermediate result. 

• where possible push down and insert projections 


Finding common sub-expressions 

SQL query represented as a logical query tree 

S1 = S2 ? 

op 

op 

S1 ⊆ S2 ? 

reuse ! 

subtree 

S1 

subtree 

S2 

recompute 


Physical query optimization 

goal : minimize number of pages loaded from disk 

because : 1 disc access costs about as much as 100.000 to 1000.000 

main storage operations 

translate query tree into equivalent iterator tree : 

iterator P S.Vorname, S.Name 

| 

iterator |X| 

S.mnr=H.mnr 

/ \ 

iterator-Rel(S) 

iterator |X| 


1 iterator per algebra operator and 

1 iterator per relation 

/ \ 

iterator-Rel(H) iterator S K.titel = 'Datenbanken' 

| 

iterator-Rel(K) 


Iterator concept 

abstract data type iterator: 

Open( ) 

Next( ) 

Close( ) 

Size( ) 

Cost( ) 

initialize the iterator 

return / hand over next tuple 

close iterator, release ressources 

estimates the size of the result 

on the basis of the estimation for the input variables 

estimates the costs of the result 

on the basis of the estimation for the input variables 


Classes of iterators 

We try to load R and S as rarely as possible 

for all operations op(R) and ( R op S ) : 

• 1-pass iterators: to load R (and S) only once is sufficient 

• nested-loop iterators: read R once and S several times 

• multi-pass iterators: 

read R (and often also S) several times 

• sort-based iterators: based on sorting input 

• index-based iterators: construct an index 

• hash-based iterators: construct a hash table 


1-pass iterators for unary operators 

for operations op( R ) 

with op ∈ { load, store, P, S, removeDup, group, sort, hash } 

we try to get along with one pass, 

i.e. to load the whole relation only once into main memory 

for load, store, P, and S , 

this always works 

for other operations only, 

if R fits into main memory 

This is somewhat simplified 

because if necessary there 

still must be space for a 

search structure and space 

for collecting the output. 


Iterator for loading data base relation 

implementation of operations 

Open( ) 

Next( ) 

Close( ) 

Size( ) 

Cost( ) 

open relation, load first page into main memory 

return / hand over next tuple 

close relation, release main memory ressources 

return size of relation 

return number of blocks, 

which have to be loaded from disk to main memory 

Same implementation for 

intermediate results displaced to disk 


1-pass iterator for projection (P) 

implementation of the operations for P(R) 

concatenated with previous iterator (which produced R) 

Open( ) call R.Open( ) 

result of the input Iterator (R) is in main memory 

and will be used further 

Next( ) 

return / hand over projection of result tuple of R.Next() 

Close( ) call R.Close( ). 

Size( ) 

return R.Size( ) * space reduction factor of projection 

Cost( ) 0 , 

because everything is done in main memory 


1-pass iterator for standard selection 

implementation of the selection operations S B (R) 

concatenated with previous iterator (that generates R) 

Open( ) call R.Open( ) 

result of the input iterator (R) is in main memory 

and will be used further 

Next( ) tuple = R.Next() ; 

while ( tuple≠null and not B(tuple) ) { tuple = R.Next(); } 

return tuple ; 

Close( ) call R.Close( ) 

Size( ) 

Cost( ) 

return R.Size( ) * selectivity of S B 

0 for standard selection 

however R must be in main memory already 


1-pass iterator for bag-union 

computation of R ∪bag S : 

1. transfer each tuple of R to Output 

R 

Output 

2. transfer each tuple of S to Output 

S 

Output 

cost: 0 , if R and S are already in main memory 

size: return R.size( ) + S.size( ) 

space requirements : 2 pages, 1 for (R and S) and 1 for Output 


1-pass iterators for binary operators 

for operations ( R op S ) 

with op ∈ { ∪set, ∩set, -set, ∩bag, -bag, X, |X| } 

and S fits into main memory 

construct search structure for S 

read R (and S) only once 

R 

Output 

search 

tree 

S 

which tuples shall be transferred 

from where to Output ? 




with op ∈ { ∪set, ∩set, -set, ∩bag, -bag, X, |X| } 


construct a search structure for S 

read R (and S) only once 

R 

Output 

search 

tree 

S 

∩set : transfer tuple t from R to Output, 

if t is found in S . 

∪set : 1. transfer tupel t from R to Output, 

if t is not found in S . 

2. transfer tuples of S to Output 

in case of ∩bag, count number of identical tuples in S 


Exercise: 1-pass iterators for 

binary operators R-S and S-R 

assumption: S fits into main memory , but R does not fit 

How can we implement 

1. R – set S , 

2. S – set R , 

3. R – bag S and 

4. S – bag R 

as 1-pass iterators, i.e., in such a way that 

a search structur for S in main memory 

+ 1 input page for R 

+ 1 output page for Output 

is sufficient? 

R 

Output 

search 

tree 

S 




with op ∈ { ∪set, ∩set, -set, ∪bag, ∩bag, -bag, X, |X| } 


construct a search structure for S 

and read R (and S) only once 

search 

tree 

S 

R 

Output 

R- set S : transfer tuple t of R to Output 

if t is not found in S. 

S- set R : 1. delete tuple t of R from S. 

2. transfer remaining tuples of S to Output 

in case of -bag, count number of identical tuples in tree S 


Iterator for 1-pass "nested-loop join", 

if one relation fits into main memory 

implementation of the join operation R |X| B S with a nested-loop join 

if the smaller relation (say S) fits into main memory , and 

one main memory page is left for loading blocks of R , and 

one main memory page is left for collecting the results in Output : 

load S into main memory 

while R has blocks not yet loaded into main memory 

{ load the next page Ri of R into main memory 

} 

real implementation is more complex 

as Next() returns single tuples 

for each tuple r in Ri 

for each tuple s in S 

if ( r |X| B s ) collect in output page(r,s) 


Nested-loop iterators 

implementation of operations 

R op S by nested-loop iterators 

read one (usually the larger) relation R once 

and the other (usually the smaller) relation S multiple times, 

sometimes back and forth (in zig-zag mode) 


Naive iterator for nested-loop join 

if no relation fits into main memory 

implementation of the operation 

the naive implementation 

R |X| B S 

reads R only once 

for each tuple r in R 

for each tuple s in S 

if ( r |X| B s ) collect in output page (r,s) 



produces too many page faults 

because the whole relation S is read once per tuple of R 


Iterator for nested-loop join 


implementation of the operation 

R |X| B S 

divide main memory into k pages for S and m-k pages for R 

K = page sequence containing the next k pages of S 

M = page sequence containing the next m-k pages of R 

for each M in R 

reads R only once 

for each K in S 

for each tuple r in M 

for each tuple s in K 

if ( r |X| B s ) collect in output page(r,s) 



produces fewer page faults 

because the whole relation S is read only once per sequence M of R 


Zig-zag iterator for nested-loop join 


1. 

3. 

5. 

blocks of S 

2. 

4. 

for each page Ri in R 

while not all of S has been read 

read a sequence K of k pages of S in zig-zag mode 

for each tuple r in Ri 

for each tuple s in K 

if ( r |X| B s ) collect in output (r,s) 

k 

k pages of S 

produces even fewer page faults because 

pages "at both ends of S" are loaded less frequently 

Ri 

Output 


Merge-sort 

Example: 3-way merge-sort with 4 main memory pages. 

for every sequence F of 4 blocks of R do 

{ load F into the database buffer in main memory ; 

Quicksort ( database buffer ) ; 

write sorted database buffer back to disk ; 

} 

// disk contains sorted sequences of the length 4 blocks 

while the disk contains several sorted sequences 

{ merge (up to) 3 sorted sequences 

into a single longer sorted sequence 

} 

Ordered output sequence 

1 output page 

3 input 

pages 


Sort-based unary iterators 

1. sort R ( use e.g n-way merge-sort ) 

2. perform intended operation, 

e.g. duplicate elimination 

e.g. grouping 

(this is possible with a single scan of the sorted relation) 


Other sort-based iterators 

sort R or S or (R and S) 

sort-merge join 

sort R and S according to join attributes 

merge run: 

choose all pairs of tuples with identical join attribute values 

( white board ) 

sort-based operations for ∪ set , ∩ , - 

sort R and S , 

count number of identical tuples in merge run 


Index-based iterators 

construct an index for R or for S or for both R and S 

in case of a primary index: 

1. sort R according to index 

2. construct B + -tree 

in case of a secondary index: 

1. extract secondary key values from primary data 

2. collect pairs ( secondary key , reference to data block ) 

in a B + -tree 


Iterator for index-based selection 

implementation of operations for S B (R) , e.g. for B=(key>66) 

Open( ) 

Next( ) 

Close( ) 

Size( ) 

Cost( ) 

step down the index (e.g. B + -tree) to the first page 

containing a record that fulfills the selection condition 

use index to access next tuple 

(may be on a different page) 

stop it when index key values become too high 

call R.Close( ) and close the index 

return R.Size( ) * selectivity of S B 

limited by 

number of result tuples 

+ number of index leaf nodes * selectivity of S B 

+ index depth 


Two iterators for index join 

1. given an index on R using the join attributes of R, 

we sequentially scan S 

for each tuple s of S 

use index of R to search join partner r of R 

if join (r,s) collect in output (r,s) 

2. given an index on R using the join attributes of R 

and an index on S using the join attributes of S : 

use the index on R and the index on S for a merge-run 

(as within sort-merge join) 


Hash-based iterators 

build hash-table for R or for S or both for R and S 

Goal: 

partition large relations using hash-functions, 

until certain partitions fit into main memory 

After partitioning: 

unary operators (e.g. duplicate elimination) 

are executed on main memory partitions 

binary operators are applied to partitions that correspond to each other 

thereby: 

use other iterators 


Hash-join R |X| a=b S 

build (evtl. repeatedly) hash-table for (partitions of) R and S 

Split( R, S ) 

{ choose new hash function h for R.a , and use h for S.b too 

use h to split R and S into different buckets R1,…,Rn and S1,…,Sn 

} 

for each pair of corresponding buckets (Ri,Si): 

if (at least one bucket (Ri or Si) fits into main memory) 

nested-loop-join( Ri, Si ) ; 

else Split( Ri, Si ) 

Example on whiteboard: Men |X| age=age Women 


Hash-join R |X| a=b S 

Why do we partition repeatedly with (different) hash functions ? 

• partitioning everything at once 

could require too many partitions 

(nearly empty buckets need to much space) 

• degree of partitioning (fan-out) 

is limited by number of available main memory pages 

that can be reserved for this operation 


Summary: iterator classes 

for each operation (op R) and (R op S) try to load R and S 

as rarely as possible. 

• for single-pass iterators, 

a single load of R (and S) is sufficient 

• nested-loop iterators read R once and S multiple times 

• multi-pass iterators read R (and often also S) multiple 

times 

• sort-based iterators use sorting 

• index-based iterators construct an index 

• hash-based iterators construct a hash-table

Query optimization in relational databases

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?