PLMCE2014-Innodb-Architecture-And-Performance-Optimization

InnoDB Architecture, and
Performance Op7miza7on
Peter Zaitsev, Ovais Tariq
Percona Live, Santa Clara,CA
April 1,2014
1
www.percona.com

Architecture and Performance
• Advanced Performance Op?miza?on requires
transparency/X-­‐ray Vision
• Impossible without understanding system
architecture
• Focus on Conceptual Aspects
– Exact Checksum algorithm InnoDB uses is not
important
– What maSers
• How fast is that algorithm ?
• How checksums are checked/updated
2
www.percona.com

Aspects of Architecture
• General Architecture
• Storage Files Layout
• Threads Architecture
• Indexes
• Buffer Pool
• Page Cleaner Thread
• Adap?ve Hash Index
• Change Buffer
3
www.percona.com

Aspects of Architecture 2
• Doublewrite Buffer
• Redo Logging
• Recovery
• Mul? Versioning
• Locking and Latching
• BLOB Storage
• Compression
• Miscellaneous
4
www.percona.com

InnoDB Versions
• Below MySQL 5.5
– Ancient History. Upgrade Recommended
• MySQL 5.5
– Scalability further improved
• MySQL 5.6 (GA)
– Further improvements in Scalability
– Full Text Search, fast checksums etc.
– Focus of today
• MySQL 5.7
– Future
5
www.percona.com

XtraDB
• Follows MySQL/InnoDB Versions
• Included in Percona Server and MariaDB
– No more available as separate plugin
• Includes all InnoDB features and
improvements plus many more
– hSp://goo.gl/bnq5sI
• Percona Server 5.6 is latest GA version
6
www.percona.com

Overview of General InnoDB Architecture
GENERAL ARCHITECTURE
7
www.percona.com

General Architecture
• Tradi?onal OLTP Engine
– “Emulates Oracle Architecture”
• Implemented using MySQL Storage engine API
• Row Based Storage. Row Locking. MVCC
• Data Stored in Tablespaces
• Log of changes stored in circular log files
• Data, Indexes Pages cached in “Buffer Pool”
8
www.percona.com

Physical Structure of InnoDB Tablespace and Logs
STORAGE FILES LAYOUT
9
www.percona.com

InnoDB Tablespaces
• All data stored in Tablespaces
– Changes to these databases stored in Circular Logs
– Changes has to be reflected in tablespace before log
record is overwriSen
• Single tablespace or mul?ple tablespaces
– innodb_file_per_table=1
– Mul?ple tablespace support is enabled by default
• System informa?on is always in main tablespace
– Main tablespace can consist of many files
• They are concatenated
10
www.percona.com

Tablespace Format
• Collec?on of Segments
– Segment is like a “file”
• Segment is number of extents
– Typically 64 of 16K page sizes
– Smaller extents for very small objects
• First Tablespace page contains header
– Tablespace size
– Tablespace id
11
www.percona.com

Types of Segments
• Each table is Set of Indexes
– InnoDB has “index organized tables”
• Each index has
– Leaf node segment
– Non Leaf node segment
• Special Segments
– Rollback Segment(s)
– Insert buffer, etc.
12
www.percona.com

InnoDB On-­‐Disk Layout
13
www.percona.com

InnoDB Space Alloca7on
• Small Segments (less than 32 pages) – by page
• Large Segments
– Extent at the ?me (to avoid fragmenta?on)
• Free pages recycled within same segment
• All pages in extent must be free before it is used
in different segment of same tablespace
– innodb_file_per_table=1 -­‐ free space can be used by
same table only
• InnoDB never shrinks its tablespaces
14
www.percona.com

InnoDB Page Structure
15
www.percona.com

Storage Tuning Parameters
• innodb_file_per_table
– Store each table in its own file/tablespace
• innodb_autoextend_increment
– Extend all tablespaces in this increment
– Extension of tablespaces can happen concurrently
• innodb_page_size
– May be set to 4K, 8K or 16K
– Must be set before the database is created
16
www.percona.com

Using File per Table
• Typically more convenient
• Reclaim space from dropped table
• OPTIMIZE TABLE mytable;
– reduce file size aler data was deleted
• Store different tables/databases on different drives
• Backup/Restore tables one by one
• Support for compression
• Will use more space with many tables
• Longer unclean restart ?me with many tables
17
www.percona.com

Performance and InnoDB File Per
Table
• Performance is Similar in majority of cases
• Very large number of tables is a problem
– Can use a mix of tables in their own tablespaces
and tables in shared tablespace
• Can help with i-­‐node level locking on some
filesystems
– EXT3
18
www.percona.com

Drop Table with
innodb_file_per_table
• Dropping tablespace can get expensive
– Even empty one!
• DROP TABLE; TRUNCATE TABLE; some ALTER
• Nega?ve Scalability
– More expensive with larger buffer pool size
• Significant improvements in MySQL 5.6
• More work in MySQL 5.7
– Especially for temporary tables
19
www.percona.com

Dealing with Runaway tablespace
• Main Tablespace does not shrink
– Consider sepng max size
–
innodb_data_file_path=ibdata1:10M:autoextend:
max:10G
• Dump and Restore
• Use Transportable Tablespaces feature
20
www.percona.com

Transportable Tablespaces
• Copy tables from one loca?on to another
• FLUSH TABLES FOR EXPORT
• Copy .frm, .ibd, .cfg file to a backup loca?on
• UNLOCK TABLES
• Import the tablespace at a different loca?on
– But make sure that the table defini?on is the same on
the second loca?on
• Or use Percona XtraBackup hSp://goo.gl/MVGfIS
21
www.percona.com

Separate Undo Tablespace
• Store undo tablespace in separate set of files
– innodb_undo_directory
• Path where external undo tablespace are stored
• Should be SSD backed for best performance
– innodb_undo_tablespaces
• How many undo tablespaces to create
• Must be set at database crea?on ?me and cannot be
changed
• The tablespace size grows and never shrinks
• Beware of running long running transac?ons
22
www.percona.com

Separate Undo Tablespace (cont.)
• Store undo tablespace in separate set of files
– innodb_undo_logs
• earlier innodb_rollback_segments
23
www.percona.com

Page Checksums
• Protec?on from corrupted data
– Bad hardware, OS Bugs, InnoDB Bugs
– Are not completely replaced by filesystem checksums
• Checked when page is Read to Buffer Pool
• Updated when page is flushed to disk
• Can be significant overhead
– Especially for very fast storage
• Can be disabled by innodb_checksums=0
24
www.percona.com

Page Checksums (cont.)
• For Fast Storage you might use faster
checksums
– innodb_checksum_algorithm=crc32
25
www.percona.com

What threads are there and what they do
THREADS ARCHITECTURE
26
www.percona.com

General Thread Architecture
• Using MySQL Threads for execu?on
– Normally thread per connec?on
• Transac?on executed mainly by such thread
– LiSle benefit from Mul?-­‐Core for single query
• innodb_thread_concurrency can be used to limit
number of execu?ng threads
– This limit is number of threads in kernel
• Including threads doing Disk IO or storing data in TMP Table.
– Limits concurrency but reduces conten?on
27
www.percona.com

General Thread Architecture (cont.)
• innodb_thread_sleep_delay can be used to
control thread queuing
– This limit controls for how long threads sleep
before joining the queue of wai?ng threads
– Only used if innodb_thread_concurrency > 0
– innodb_adap7ve_max_sleep_delay
• Makes thread sleep delay more adap?ve
• InnoDB can adap?vely tune
innodb_thread_sleep_delay, based on this
28
www.percona.com

Helper Threads
• Main Thread
– Schedules background ac?vi?es
• Change buffer merges, possible purge, etc.
• IO Threads
– Read -­‐ mul?ple threads used for read ahead
• innodb_read_io_threads
– Write -­‐ mul?ple threads used for background writes
• innodb_write_io_threads
– Insert Buffer thread used for change buffer merges
29
www.percona.com

Helper Threads (cont.)
• Page Cleaner Thread
• Purge thread(s)
• Deadlock detec?on thread
• And others
30
www.percona.com

ThreadPool
• Helpful for handling very large number of
connec?ons
– 10K+ connec?ons are possible
• Available in MySQL Enterprise, Percona Server,
MariaDB
• Implementa?ons are a bit different
31
www.percona.com

How Indexes are Implemented in InnoDB
INDEXES
32
www.percona.com

Everything is the Index
• InnoDB tables are “Index Organized”
– PRIMARY KEY contains data instead of data pointer
• Hidden PRIMARY KEY is used if not defined (6b)
– Hidden PRIMARY KEY is a global conten?on point
• Data is “Clustered” by PRIMARY KEY
– Data with close PK value is stored close to each other
– Clustering is within page ONLY
• InnoDB system tables INNODB_SYS_TABLES and
INNODB_SYS_INDEXES expose informa?on
33
www.percona.com

INNODB_SYS_TABLES Example
• Gives good overview informa?on
mysql (information_schema) > select * from INNODB_SYS_TABLES limit 10;
+----------+----------------------------+------+--------+-------+-------------+------------+---------------+
| TABLE_ID | NAME | FLAG | N_COLS | SPACE | FILE_FORMAT | ROW_FORMAT | ZIP_PAGE_SIZE |
+----------+----------------------------+------+--------+-------+-------------+------------+---------------+
| 14 | SYS_DATAFILES | 0 | 5 | 0 | Antelope | Redundant | 0 |
| 11 | SYS_FOREIGN | 0 | 7 | 0 | Antelope | Redundant | 0 |
| 12 | SYS_FOREIGN_COLS | 0 | 7 | 0 | Antelope | Redundant | 0 |
| 13 | SYS_TABLESPACES | 0 | 6 | 0 | Antelope | Redundant | 0 |
| 16 | mysql/innodb_index_stats | 1 | 11 | 2 | Antelope | Compact | 0 |
| 15 | mysql/innodb_table_stats | 1 | 9 | 1 | Antelope | Compact | 0 |
| 18 | mysql/slave_master_info | 1 | 26 | 4 | Antelope | Compact | 0 |
| 17 | mysql/slave_relay_log_info | 1 | 11 | 3 | Antelope | Compact | 0 |
| 19 | mysql/slave_worker_info | 1 | 15 | 5 | Antelope | Compact | 0 |
+----------+----------------------------+------+--------+-------+-------------+------------+---------------+
9 rows in set (0.00 sec)
34
www.percona.com

INNODB_SYS_INDEXES Example
• Gives good overview informa?on
mysql (information_schema) > select * from INNODB_SYS_INDEXES where table_id in (15,
16);
+----------+---------+----------+------+----------+---------+-------+
| INDEX_ID | NAME | TABLE_ID | TYPE | N_FIELDS | PAGE_NO | SPACE |
+----------+---------+----------+------+----------+---------+-------+
| 17 | PRIMARY | 15 | 3 | 2 | 3 | 1 |
| 18 | PRIMARY | 16 | 3 | 4 | 3 | 2 |
+----------+---------+----------+------+----------+---------+-------+
2 rows in set (0.00 sec)
35
www.percona.com

Index Structure
• Secondary Indexes refer to rows by Primary Key
– No update when row is moved to different page
• Long Primary Keys are expensive
– Increase size of all indexes
• Random Primary Key Inserts are expensive
– Cause page splits; Fragmenta?on
– Make page space u?liza?on low
• Auto-­‐Increment keys are olen beSer than
ar?ficial keys, UUIDs, SHA1 etc.
36
www.percona.com

More on Clustered Index
• PRIMARY KEY lookups are the most efficient
– Secondary key lookup is essen?ally 2 key lookups
• Op?mized with Adap?ve Hash Index
• PRIMARY KEY ranges are very efficient
– Build Schema keeping it in mind
– (user_id,message_id) may be beSer than (message_id)
• Changing PRIMARY KEY is expensive
– Effec?vely removing row and adding new one
– Effects all secondary indexes
• Sequen?al Inserts give compact, least fragmented
storage
37
www.percona.com

More on Indexes
• There is no Prefix Index compressions
– Index can be 10x larger than for MyISAM table
– InnoDB has page compression. Not the same thing.
• Indexes contain transac?on informa?on = fat
– Allow to see row visibility = index covering queries
• Fast Index Crea?on
– Secondary indexes built by sor?ng
– Good page fill factor and no fragmenta?on
– innodb_sort_buffer_size
38
www.percona.com

Fragmenta7on
• Inter-­‐row fragmenta?on
– The row itself is fragmented
– Happens in MyISAM but NOT in InnoDB
• Intra-­‐row fragmenta?on
– Sequen?al scan of rows is not sequen?al
– Happens in InnoDB, outside of page boundary
• Empty Space Fragmenta?on
– A lot of empty space can be lel between rows
39
www.percona.com

Fragmenta7on (cont.)
• Defragmenta?on can be done by OPTIMIZE TABLE
– OPTIMIZE TABLE mytable;
• Cannot be done online in MySQL
• Can be done online using pt-­‐online-­‐schema-­‐change
hSp://goo.gl/pKBV2R
pt-online-schema-change --alter
"ENGINE=InnoDB" D=sakila,t=actor
40
www.percona.com

Structure and working of InnoDB in-­‐memory storage area
BUFFER POOL
41
www.percona.com

What is the Buffer Pool?
• In-­‐memory storage area that caches data and
indexes
• The unit of data storage is pages
• All data access requests first go to the buffer pool
• Data not found is loaded from disk and copied to
it
• In-­‐memory data access is very fast as compared
to disk IO
• Data modifica?ons are also cached and then
synced to disk in the background
42
www.percona.com

Important Structures
• LRU List
– List of pages in the buffer pool ordered by last access
?me
– Uses a modified LRU algorithm
– Actually maintains two lists
• List of young block
• List of old blocks
– Addi?onal list when dealing with compressed tables
• unzip_LRU list -­‐ maintains a list of uncompressed pages for
compressed tables
43
www.percona.com

Important Structures (cont.)
• Flush List
– List of dirty pages in the buffer pool
– Dirty pages are pages that contain modified rows
and that need to be synced to data files on disk
– Pages are moved from flush list to LRU list when
changes have been synced to disk
– Flush list size is capped by the redo log size (among
other limits)
44
www.percona.com

Configuring the Buffer Pool
• innodb_buffer_pool_size is most important
op?on
– Use all your memory nor commiSed to anything else
– Keep overhead into account (~5%)
– Never let Buffer Pool Swapping to happen
– Up to 80-­‐90% of memory on InnoDB only Systems
– innodb_buffer_pool_instances=N
• Set to 4-­‐16 depending number of cores to get beSer
scalability
• Default is 8 instances
• Percona Server does well with few instances
45
www.percona.com

Buffer Pool Page Replacement
• InnoDB uses LRU for page replacement
– With Midpoint Inser?on
• innodb_old_blocks_pct, innodb_old_blocks_7me
– Offers Scan resistance from large full table scans
– innodb_old_blocks_7me defaults to 1000ms now
meaning pages read in by scans age out faster
• Page Cleaner thread maintains clean pages in free
list
• Evic?on is done from the list of old blocks
46
www.percona.com

Buffer Pool Page Replacement (cont.)
• Compressed / uncompressed pages evic?on
– If size of unzip_LRU list

Buffer Pool Page Flushing
• Different flushing flavors
– Background flushing
• Adap?ve, async, innodb_max_dirty_pages_pct, LRU
and idle flushing
– Foreground Sync flushing
• Done by query thread that detects the condi?on
• All other query threads are halted
48
www.percona.com

Neighbor Page Flushing
• InnoDB looks for next and previous dirty pages
and flushes them as well to keep IO more
bulky
• Controlled by innodb_flush_neighbors
• This is a good op?miza?on for HDD
• Such behavior may be a poor choice
– Especially for SSD which does not have random IO
penalty
– Disable it with innodb_flush_neighbors=0
49
www.percona.com

Buffer Pool Dump and Restore
• Ability to dump Buffer Pool contents allowing
for warm cache on restarts
• Automa?cally dump and reload on shutdown
and restart
– innodb_buffer_pool_dump_at_shutdown
– innodb_buffer_pool_load_at_startup
• On-­‐demand dump and reload
– innodb_buffer_pool_dump_now
– innodb_buffer_pool_load_now
50
www.percona.com

What’s in the Buffer Pool?
• Informa?on about Buffer Pool pages and
sta?s?cs is available in I_S tables
– INNODB_BUFFER_PAGE_LRU
– INNODB_BUFFER_PAGE
– INNODB_BUFFER_POOL_STATS
mysql > select POOL_ID, POOL_SIZE, FREE_BUFFERS, DATABASE_PAGES, OLD_DATABASE_PAGES, MODIFIED_DATABASE_PAGES from
INNODB_BUFFER_POOL_STATS;
+---------+-----------+--------------+----------------+--------------------+-------------------------+
| POOL_ID | POOL_SIZE | FREE_BUFFERS | DATABASE_PAGES | OLD_DATABASE_PAGES | MODIFIED_DATABASE_PAGES |
+---------+-----------+--------------+----------------+--------------------+-------------------------+
| 0 | 8191 | 8043 | 148 | 0 | 0 |
+---------+-----------+--------------+----------------+--------------------+-------------------------+
1 row in set (0.00 sec)
51
www.percona.com

What’s in the Buffer Pool? (cont.)
• InnoDB Status also gives high level picture of
the buffer pool
– Percona Server exposes more informa?on
– SHOW ENGINE INNODB STATUS\G
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 137363456; in additional pool allocated 0
Dictionary memory allocated 43128
Buffer pool size 8191
Free buffers 8043
Database pages 148
Old database pages 0
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 148, created 0, written 1
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 148, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
52
www.percona.com

Data Dic7onary
• Holds informa?on about InnoDB Tables
– Sta?s?cs; Auto-­‐increment value, System informa?on
– Can be 4KB to 10KB per table
• Can consume a lot of memory with huge number
of tables
– Think hundreds of thousands
• Data dic?onary LRU
– Controlled automa?cally by table_defini7on_cache
• If you have many tables increasing it is important
53
www.percona.com

Buffer Pool Improvements in Percona
Server
• Lot of work in Percona Server to improve
buffer pool scalability
• Different mutex for different jobs reduce
global conten?on
– LRU list mutex
– Flush list mutex
– Free list mutex
– and others .. hSp://goo.gl/Ln1Z9a
54
www.percona.com

How does it func?on and what work does it do?
PAGE CLEANER THREAD
55
www.percona.com

Page Cleaner Thread
• Handles all types of background flushing
– Flushes pages from end of LRU list
– Flushes pages from flush list
• Wakes up once per second
– Skips sleeping if server is idle
• Server will do merges and flushes faster when it is
idle
• In some cases flushing can be done from user
threads
– Percona Server allows you to disable this
56
www.percona.com

LRU List Flushing
• Can happen in workloads with data sets larger
than memory
• Flushes dirty pages from the end of the LRU
list
• Moves clean pages at the tail of the LRU to the
free list
• Page cleaner does LRU flushing in all cases
with some excep?ons
57
www.percona.com

LRU List Flushing (cont.)
• innodb_lru_scan_depth=N
– How deeply to examine tail for dirty pages
– User thread may scan up to this depth as well if no
page available in free list
– Important to tune to prevent synchronous flushes
58
www.percona.com

Synchronous LRU List Flushing
• Synchronous flush from user thread if it cannot
find a clean page in the free list
• Flushes a single page from user thread in case
of synchronous flush
• Typically when page cleaner thread is not able
to keep up with page dirtying rate
59
www.percona.com

Flushing Overview
• Hard Problem to Balance
– Flush too much now
• Compete with IO which must happen
• Loose possibility of op?miza?on
– Delay too much
• Might have to do too much IO in the future
• Poten?ally cause “stalls” or performance dips
• Lots of con?nuing work
• Too Many tuning op?ons to play with
60
www.percona.com

Flush List Flushing / Adap7ve Flushing
• Flushing to advance “earliest modify LSN”
– To free log space so it can be reduced
• Mostly done from Page Cleaner from the end
of the flush list
– Approximately done once per second
• Most of writes typically happen this way
• Number of pages to flush per cycle depends on
the load
61
www.percona.com

Some Variables
• innodb_io_capacity
• innodb_io_capacity_max
• innodb_max_dirty_pages_pct
• innodb_max_dirty_pages_pct_lwm
• innodb_flushing_avg_loops
62
www.percona.com

Synchronous Flush List Flushing
• Synchronous flushing outside of Page Cleaner
in cases when sync point is reached
• Synchronous flushing is done from user thread
which detects the condi?on
• Every other user thread is blocked to prevent
increase in checkpoint age
• Causes system-­‐wide stalls
63
www.percona.com

Percona Server Improvements
• Focus on BeSer default behavior with no
tuning needed
– Increase Priority of Page Cleaner Thread
– Reduce flushing from user level threads
– Make algorithm more adap?ve to changes in
workload
– BeSer balancing on processing mul?ple buffer
pools
64
www.percona.com

Some Performance Gains
65
www.percona.com

What does this structure do?
ADAPTIVE HASH INDEX
66
www.percona.com

What is Adap7ve Hash Index?
• On-­‐demand hash index built automa?cally by
InnoDB
• Built on top of exis?ng B+TREE Indexes to
speed up lookups
– Both PRIMARY and Secondary indexes
• Can be built for full index and prefixes
• Par?al Index
– Only built for index values which are accessed
olen
67
www.percona.com

What is Adap7ve Hash Index? (cont.)
• Adap?ve Hash Index lookup
Search the
adaptive hash
extension_number
1
4
2 6
8
Find a pointer directly to
12 the leaf node of the
clustered index.
10 14
3 5 7 9 11 13 15
4 0xACDC
12 0xACDC
12 0xACDC
2 0xACDC
6 0xACDC 10 0xACDC 14 0xACDC
1 ..
3 .. 5 .. 7 .. 9 ..
11 ..
13 ..
15 ..
68
www.percona.com

Tuning Adap7ve Hash Index
• No tuning op?on available
– Can either be enabled or disabled
– Enabled by default
• Can be disabled for performance reasons
– innodb_adap7ve_hash_index
– Improves concurrency but reduces performance
• Can be Par??oned in Percona Server
– innodb_adap7ve_hash_index_par77ons=8
69
www.percona.com

What is change buffering in InnoDB?
CHANGE BUFFER
70
www.percona.com

What is Change Buffer?
• Buffer INSERT, UPDATE and PURGE opera?ons
– DELETE is covered as a special UPDATE
• Designed to speed up modifica?ons into
secondary Indexes when index page not in buffer
pool
– Secondary index updates are expensive
– Reported up to 15 ?mes IO reduc?on for some cases
• Works for Non-­‐Unique Secondary Indexes only
• If leaf index page is not in buffer pool
– Store a note the page should be updated in memory
71
www.percona.com

What is Change Buffer ? (cont.)
• If page containing buffered entries is read
from disk they are merged transparently
• InnoDB performs gradual change buffer
merges in background
– Change buffer merges may con?nue across
restarts
– InnoDB slow shutdown forces complete change
buffer merge on shutdown
• innodb_fast_shutdown=0
72
www.percona.com

Tuning Change Buffer
• Can take up to ¼ the size of the buffer pool
– Size of the change buffer can be controlled via
innodb_change_buffer_max_size
• Background merge speed may not be enough
– Tune innodb_io_capacity
• You can disable change buffering
– innodb_change_buffering=none
– Can be good for SSDs
– Can also be good if dataset fits in memory
73
www.percona.com

Change Buffering Overhead
• Full change buffer is useless and wastes
memory
• Delayed change buffering may cause
slowdown
– Too many merges need to happen on page read
• Aler restart merge speed may slow down
– Finding index entries to merge requires random IO
74
www.percona.com

How is this structure used in InnoDB
DOUBLEWRITE BUFFER
75
www.percona.com

What is Doublewrite Buffer?
• InnoDB redo logs requires consistent pages for
recovery
• Page write may only be par?ally completed
– Upda?ng part of 16K and leaving the rest
• Doublewrite Buffer is short term page level log
• All page writes go to doublewrite buffer first:
– Write pages to doublewrite buffer; Sync
– Write Pages to their original loca?ons; Sync
– Pages contain tablespace_id+page_id
76
www.percona.com

How DoubleWrite helps recovery
• On crash recovery pages in this buffer are
compared to their original loca?on
– At least one of them should be conistent
77
www.percona.com

What is Doublewrite Buffer? (cont.)
• Doublewrite Buffer usage:
Sync to double
write area.
Buffer
Buffer Pool
Sync individual
pages to correct
destinations.
Tablespace
78
www.percona.com

Tuning Doublewrite Buffer
• No par?cular tuning op?ons available
– Either enable it or disable it
– innodb_doublewrite
• Not much overhead on tradi?onal disks because
writes are sequen?al
• Much more overhead on SSDs
– Also reduces their lifespan
• Makes sense to disable it on atomic writes
guarantee
79
www.percona.com

Tuning Doublewrite Buffer (cont.)
• Atomic writes guarantees
– ZFS filesystem
– DirectFS filesystem on Fusion-­‐io devices
• Percona Server has support
• hSp://goo.gl/ylpDK1
80
www.percona.com

What is it and how does it work?
REDO LOGGING
81
www.percona.com

InnoDB Log Files
• Set of log files (ib_logfile?)
– 2 log files by default. Effec?vely concatenated
• Log Header
– Stores informa?on about last checkpoint
• Log is NOT organized in pages, but records
– Records aligned 512 bytes, matching disk sector
• Log record format “physiological”
– Stores Page# and opera?on to do on it
• Only REDO opera?ons are stored in logs.
82
www.percona.com

More on Log Files
• The log files are used to perform recovery in case
of an unclean shutdown
– Larger log files -­‐ Longer recovery
• There is virtually no limit on the size of the log
files
– Well the combined size of the log files cannot exceed
~512GB
• Percona Server allows different log file block size
– innodb_log_block_size
• Default value is 512 bytes and good in most general cases
• 4096 can be beSer for SSDs
83
www.percona.com

More on Log Files (cont.)
• Percona Server adds
– innodb_log_checksum_algorithm=crc32
• Reduce CPU load for heavy writes to log file
• If you're using compressed pages full pages
can be logged to log file.
84
www.percona.com

More on Log Files (cont.)
• InnoDB Logs usage
01010
Log Files
UPDATE City SET
name = 'Morgansville'
WHERE name = 'Brisbane'
AND CountryCode='AUS'
Buffer Pool
Tablespace
85
www.percona.com

InnoDB Log IO
• Log are opened in buffered mode
– Even with innodb_flush_method=O_DIRECT
– Percona Server use O_DIRECT for logs as well
• innodb_flush_method=ALL_O_DIRECT
• innodb_log_block_size may need to be adjusted for EXT4
filesystem for ALL_O_DIRECT to work
• Flushed by fsync() -­‐ default or O_SYNC
• Logs which fit in cache may improve performance
– Small transac?ons and
innodb_flush_log_at_trx_commit=1 or 2
86
www.percona.com

Resizing Log Files
• Log files are automa?cally resized
– If innodb_log_file_size is changed between
restarts
– Log files are automa?cally resized to match the
new desired size
• Automa?c resizing does not work with
versions < 5.6
87
www.percona.com

Tuning Parameters
• Total log file size is calculated as
– innodb_log_file_size * innodb_log_files_in_group
• innodb_log_file_size
– A general principle is to have these big enough to
hold one hour worth of changes
– Extremely important to tune this for write
performance
• innodb_log_files_in_group
– No good reason to change
88
www.percona.com

Log Buffer
• Redo log records are first wriSen to a log
buffer in memory
• innodb_flush_log_at_7meout=N
– Log buffer is synced to disk every N seconds
irrespec?vely anyhow
– Once per second by default
89
www.percona.com

Log Buffer (cont.)
• innodb_log_buffer_size
– Values 32-­‐256MB typically make sense
– Larger values are good to reduce conten?on
– May need to be larger if using large BLOBs
– See amount of data wriSen to the logs
– Log buffer covering 10sec is good enough
90
www.percona.com

Log Buffer (cont.)
• innodb_flush_log_at_trx_commit
Log Files
Buffer Pool
Log Buffer OS Cache RAID Cache
Physical Storage
1 fsync every commit
2
fsync once per innodb_flush_log_at_7meout
second
0
fsync once per innodb_flush_log_at_7meout
second
91
www.percona.com

How InnoDB recovers from crash?
RECOVERY
92
www.percona.com

Recovery Stages
• Physical Recovery
– Recover par?ally wriSen pages from doublewrite
buffer
• Redo Recovery
– Redo all the changes stored in transac?onal logs
• Undo Recovery
– Rollback uncommiSed transac?ons
93
www.percona.com

Redo Recovery Stage
• Foreground
– Server is not started un?l it is complete
• Larger Logs = Longer recovery ?me
– Though row sizes, database size, workload also maSer
• Scan Log files
– Buffer modifica?ons on per page basics
– Apply modifica?ons to data file
• LSN stored in the page tells if change needs to be
applied
94
www.percona.com

Tuning Redo Recovery
• Redo log size maSers
– large logs longer recovery
– tune innodb_log_file_size accordingly
• innodb_max_dirty_pages_pct
– Fewer the dirty pages, faster the recovery
• innodb_buffer_pool_size
– Larger buffer pool means faster IO recovery
95
www.percona.com

Undo Recovery Stage
• Is done in the background
– Performed aler MySQL is started
• Speed depends on transac?on length
– Very large UPDATE, INSERT... SELECT is problem.
• Is NOT a problem with ALTER TABLE
– Commits every 10000 rows to avoid this problem
– Unless it is Par??oned table
• Faster with larger innodb_log_file_size
• Be careful killing MySQL with runaway update
queries.
96
www.percona.com

Implementa?on of Mul? Versioning
MULTI VERSIONING
97
www.percona.com

Mul7 Versioning Overview
• InnoDB maintains old row versions of modified
rows
• Read Transac?on can read old version of row,
while it is modified
– Prevents the need to do locking reads
• Locking reads can s?ll be performed with
SELECT FOR UPDATE and LOCK IN SHARE
MODE Modifiers
98
www.percona.com

Updates and Locking Reads
• Updates bypass Mul? Versioning
– You can only modify row which currently exists
• Locking Read bypass mul?-­‐versioning
– Result from SELECT vs. SELECT .. LOCK IN SHARE
MODE will be different
• Locking Reads are slower
– Because they have to set locks
– Can be 2x+ slower!
– SELECT FOR UPDATE has larger overhead
99
www.percona.com

Mul7 Versioning Implementa7on
• The most recent row version is stored in the page
– Even before it is commiSed
• Previous row versions stored in undo space
• The number of versions stored is not limited
– Can cause system/undo tablespace size to explode
• Access to old versions require going through
linked list
– Long transac?ons with many concurrent updates can
impact performance.
100
www.percona.com

Mul7 Versioning Internals
• Each row in the database has
– DB_TRX_ID (6b) – Transac?on inserted/updated row
– DB_ROLL_PTR (7b) -­‐ Pointer to previous version
• Dele?on handled as Special Update
• DB_TRX_ID + list of currently running transac?ons
is used to check which version is visible
• Insert and Update Undo Segments
– Inserts history can be discarded when transac?on
commits.
– Update history is used for MVCC implementa?on
101
www.percona.com

Mul7 Versioning Performance
• Only changed columns stored in the undo
segment
• Short rows are faster to update
– Separate table to store counters olen make sense
• Beware of long transac?ons
– Especially containing many updates
• “Rows Read” can be misleading
– Single row may correspond to scanning thousand
of versions/index entries
102
www.percona.com

Mul7 Versioning Indexes
• Indexes contain pointers to all versions
– Index key 5 will point to all rows which were 5 in
the past
• Indexes contain TRX_ID
– Easy to check entry is visible
– Can use “Covering Indexes”
• Many old versions is performance problem
– Slow down accesses
– Will leave many “holes” in pages when purged
103
www.percona.com

Dealing with Runaway Transac7ons
• Monitor SHOW ENGINE INNODB STATUS for
transac?ons ACTIVE for long ?me
– Looking for large InnoDB History List Length is also
good idea
• Innodb_history_list_length status variable
• Percona Server has feature to kill idle transac?ons
– Idle transac?ons are ones that are open but inac?ve
– innodb_kill_idle_transac7on hSp://goo.gl/poBUVi
104
www.percona.com

Transac7on Isola7on Modes
• SERIALIZABLE
– Locking reads. Bypass mul? versioning
• REPEATABLE-­‐READ (default)
– Read commiSed data at it was on start of transac?on
• READ-­‐COMMITTED
– Read commiSed data as it was at start of statement
• READ-­‐UNCOMMITTED
– Read uncommiSed data as it is changing live
105
www.percona.com

Purging the Garbage
• Old row and index entries need to be removed
– When they are not needed for any ac?ve transac?on
• REPEATABLE READ
– Need to be able to read everything at transac?on start
• READ-­‐COMMITTED
– Need to read everything at statement start
• Purging can be done by the main InnoDB thread
or by dedicated thread(s)
106
www.percona.com

Purging the Garbage (cont.)
• Purging is done by default by a single thread
– Can have between 0 -­‐ 32 purge threads
– A minimum of 1 purge thread is recommended
– 4 is probably good for write intensive workloads
– More than 1 thread may decrease throughput for IO
bound workload
• innodb_purge_batch_size
– Default is 300 which is good in most cases
– Keep this to a reasonably high value because purging
many rollback segments is expensive
107
www.percona.com

Purging the Garbage (cont.)
• Purge Thread(s) may be unable to keep up
with intensive updates
– InnoDB “History List Length” will grow high
– innodb_max_purge_lag slows updates down
• A “History List Length” threshold that when crossed,
triggers DML(s) to be throSled
– innodb_max_purge_lag_delay
• Controls maximum delay in milliseconds which may be
added to a DML
108
www.percona.com

How row-­‐level locking and internal locks work?
LOCKING AND LATCHING
109
www.percona.com

InnoDB Locking Basics
• Row level locking
– No lock escala?on
• Pessimis?c Locking Strategy
• Row Level lock wait ?meout
– innodb_lock_wait_7meout
• Tradi?onal “S” and “X” locks
• Inten?on locks on tables “IS” “IX”
– Restric?ng table opera?ons
110
www.percona.com

InnoDB Locking Basics (cont.)
• Locks on Rows AND Index Records
• Automa?c deadlock detec?on
– Non-­‐recursive
– innodb_print_all_deadlocks
• Prints all deadlocks to the error log
– For some deadlocks you may get
innodb_lock_wait_7meout error
111
www.percona.com

Gap Locking
• InnoDB not only locks rows but also gap between them
• Needed for Consistent Reads
– Also used by update statements
– READ-­‐COMMITTED + RBR can reduce those needs
• InnoDB has no phantom rows even in Consistent Reads
• Gap locking olen cause complex deadlock situa?ons
• “Infinum”, “Supremum” records define bounds of data
stored on the page
• Only record lock is needed for PK Update
112
www.percona.com

Types of Locks in InnoDB
• Next-­‐Key-­‐Lock
– Lock Key and gap before the key
• Gap-­‐Lock
– Lock just the gap before the key
• Record-­‐Only-­‐Lock
– Lock record only
• Insert inten?on gap locks
– Held when wai?ng to insert into the gap
113
www.percona.com

Advanced Gap Locking Stuff
• Gaps can change on row dele?on
– Actually when Purge thread removes record
• Leaving conflic?ng Gap locks held
• Gap Locks are “purely inhibi?ve”
– Only block inser?on.
– Holding lock does not allow inser?on. Must also wait
for conflic?ng locks to be released
• “supremum” record can have lock, “infinum”
can't
• You rarely need it in prac?ce
114
www.percona.com

InnoDB Lock Storage
• InnoDB locks storage is preSy compact
– This is why there is no lock escala?on!
• Lock space needed depends on lock loca?on
– Locking sparse rows is more expensive
• Each Page having locks gets bitmap allocated
for it
– Bitmap holds lock informa?on for all records on
the page
• Locks typically take 3-­‐8 bits per locked row
115
www.percona.com

Auto Increment Locks
• Tradi?onal vs. Consecu?ve Lock Mode
• Tradi?onal lock mode means taking table lock
• Consecu?ve lock mode means a lightweight
mutex is used instead
• innodb_autoinc_lock_mode – set lock
behavior
– “1” -­‐ Does not hold table lock for simple Inserts
– “2” -­‐ Does not hold lock in any case.
• Only works with Row level replica?on
116
www.percona.com

InnoDB Latching
• InnoDB implements its own mutex and RW-­‐Locks
– For transparency not only Performance
• Latching stats shown in SHOW INNODB STATUS
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐
SEMAPHORES
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐
OS WAIT ARRAY INFO: reserva?on count 13569
OS WAIT ARRAY INFO: signal count 11421
-­‐-­‐Thread 1152170336 has waited at ./../include/buf0buf.ic line 630 for 0.00 seconds the semaphore:
Mutex at 0x2a957858b8 created file buf0buf.c line 517, lock var 0
-­‐-­‐Thread 1147709792 has waited at ./../include/buf0buf.ic line 630 for 0.00 seconds the semaphore:
Mutex at 0x2a957858b8 created file buf0buf.c line 517, lock var 0
Mutex spin waits 5018, rounds 70800, OS waits 2033
RW-­‐shared spins 2736, rounds 84289, OS waits 2791
RW-­‐excl spins 904, rounds 111941, OS waits 3607
Spin rounds per wait: 14.11 mutex, 30.81 RW-­‐shared, 123.83 RW-­‐excl
117
www.percona.com

Latching Performance
• Has been improving over the years
– Great improvements in MySQL 5.6 & XtraDB
– S?ll hot spots remain
• innodb_thread_concurrency
– Limi?ng concurrency can reduce conten?on
– Introduces conten?on on its own
• innodb_sync_spin_loops
– Trade Spinning for context switching
– Typically limited produc?on impact
118
www.percona.com

Current Hot Spots
• log_mutex
– Wri?ng data to the log buffer
• Index-­‐>lock
– Lock held for dura?on of low level index modifica?on
– Can be serious hot spot for heavy write workloads
– Improvements in MySQL 5.7 (not a full solu?on)
• Adap?ve Hash Latch
– Global latch to protect access to Adap?ve Hash Index
119
www.percona.com

Current Hot Spots (cont.)
• Adap?ve Hash Latch
– Introduces conten?on in heavy read-­‐write
workload
– innodb_adap7ve_hash_index=0
• Can slow things down but also reduces conten?on
– Percona Server tackles this by par??oning the
global hash index
• innodb_adap7ve_hash_index_par77ons=N
• Helps when workload is spread across mul?ple tables
120
www.percona.com

Where is my conten7on
• Performance Schema is best way to look
– Check out ps_helper
+---------------------------------------------+---------+--------+
| event_name |nsecs_per| seconds|
+---------------------------------------------+---------+--------+
| wait/synch/rwlock/innodb/index_tree_rw_lock | 19543.3 | 3456.1 |
| wait/synch/mutex/innodb/log_sys_mutex | 2071.3 | 385.8 |
| wait/synch/rwlock/innodb/hash_table_locks | 165.7 | 184.5 |
| wait/synch/mutex/innodb/fil_system_mutex | 328.3 | 113.6 |
| wait/synch/mutex/innodb/redo_rseg_mutex | 1766.4 | 84.9 |
| wait/synch/rwlock/sql/MDL_lock::rwlock | 430.2 | 73.9 |
| wait/synch/mutex/innodb/buf_pool_mutex | 264.5 | 72.7 |
| wait/synch/rwlock/innodb/fil_space_latch | 27216.1 | 53.8 |
| wait/synch/mutex/sql/THD::LOCK_query_plan | 167.0 | 50.7 |
| wait/synch/mutex/innodb/trx_sys_mutex | 394.9 | 41.5 |
+---------------------------------------------+---------+--------+
121
www.percona.com

How are BLOBs stored in InnoDB
BLOB STORAGE
122
www.percona.com

Handling Blobs
• Blobs are handled by InnoDB in a special way
– And differently by different file formats
• Small blobs
– Whole row fits in ~8000 bytes stored on the page
• Large Blobs
– Can be stored full on external pages (Barracuda)
– Can be stored par?ally on external page
• First 768 bytes are stored on the page (Antelope)
• InnoDB will NOT read external blobs unless they
are needed by the query
123
www.percona.com

Blobs in a Separate Table
• It Depends :)
• No need to store large blobs in separate table as
they are already stored outside of the row
• Storing medium size blobs (which fit on the page)
in the separate table makes sense
• Only split blobs to separate table if they are
accessed infrequently
• TODO: Would love to be able to specify BLOBs to
be stored off the page
124
www.percona.com

InnoDB BLOB != MySQL BLOB
• MySQL Has limit of 65535 bytes per row
excluding BLOB and TEXT column
– This limit applies to VARCHAR columns
• InnoDB limit is only 8000 (half a page)
– So long VARCHAR fields may be stored as a BLOB
inside InnoDB
mysql> create table ai(c varchar(40000), d varchar(40000));
ERROR 1118 (42000): Row size too large.
The maximum row size for the used table type, not counting BLOBs, is 65535.
You have to change some columns to TEXT or BLOBs
125
www.percona.com

BLOB Page Alloca7on
• Each BLOB Stored in separate segment
– Normal alloca?on rules apply. By page when by extent
– One large BLOB is faster than several medium ones
– Many BLOBs can cause extreme waste
• 500 byte blobs will require full 16K page if it does not fit with
complete row
• External BLOB is NOT updated in place
– InnoDB always creates a new version
• Large VARCHAR/TEXT are handled similarly to
BLOB
126
www.percona.com

How is compression implemented in InnoDB?
COMPRESSION
127
www.percona.com

Compression Basics
• Requires “Barracuda” and innodb_file_per_table=1
• Per Page compression (mostly)
• Uses zlib for compression
– innodb_compression_level sets the zlib compression level
• Default is 6 and max is 9
• ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8;
– Es?mate how well the data will compress
– Compressed page size of 8KB works well in most cases
– Choosing too small a value can cause more page splits
leading to subop?mal performance
128
www.percona.com

Compression Basics (cont.)
• Per page update log to avoid recompression
• Both Compressed and Uncompressed page can be
stored in Buffer Pool
– Uncompressed pages would be evicted from the
Buffer Pool first
• Compression failure can be expensive
– Causes InnoDB to split the page and compress each
one separately
– InnoDB can automa?cally add padding to prevent
compression failure
129
www.percona.com

Compression Considera7ons
• Controlling padding to avoid compression failure
– innodb_compression_failure_threshold_pct
• Determines when to start adding padding to avoid
compression failure
– innodb_compression_pad_pct_max
• Max percentage of space that can be reserved in page for
whitespace padding
• KEY_BLOCK_SIZE=16
– Only compress externally stored BLOBs
– Can reduce size without overhead
130
www.percona.com

Compression Considera7ons (cont.)
• May need to provision more memory for the
buffer pool to avoid frequent uncompression
• May not be too good for a workload that is
already CPU bound
• Page size is too small for good compression
• Have to “Guess” how data will compress to set
the compressed block size
131
www.percona.com

Compression Considera7ons (cont.)
• Compression sepng is Per table
– Though some indexes compress beSer than others
• It may be more suitable to compress within
the applica?on before wri?ng to the database
– Can avoid overhead where some part of data
compresses well and some doesn’t
– Can help distribute the CPU consump?on across
applica?on servers
132
www.percona.com

Compression Sta7s7cs
• Per-­‐index compression sta?s?cs
– innodb_cmp_per_index_enabled
• Enables I_S table INNODB_CMP_PER_INDEX
• Can get expensive
• Compression info exposed via I_S tables
– Exposed via INNODB_CMP% tables
– INNODB_CMP, INNODB_CMP_PER_INDEX,
INNODB_CMPMEM, etc.
133
www.percona.com

Alterna7ves to Innodb Compression
• Compress on Applica?on
• Use Filesystem/Storage which support
compression
• FusionIO – Hardware assisted compression
– Inform Flash of “holes” which do not need to be
kept
• TokuDB Storage Engine
– Offers much higher compression rate
134
www.percona.com

Miscellaneous features and topics
MISCELLANEOUS
135
www.percona.com

Foreign Keys
• Implemented on InnoDB level
• Requires indexes on both tables
– Can be very expensive some?mes
• Checks happen when row is modified
– No delayed checks ?ll transac?on commit
• Foreign Keys introduce addi?onal locking
overhead
– Many tricky deadlock situa?ons are foreign key
related
136
www.percona.com

Direct IO Opera7on
• Default IO mode for InnoDB data is Buffered
• Good
– Faster flushing when no write cache
– Faster warm up on restart
– Reduce problems with inode locking on EXT3
• Bad
– Lost of effec?ve cache memory due to double buffering
– Increased tendency to swap due to IO pressure
• innodb_flush_method=O_DIRECT
– O_DIRECT_NO_FSYNC on some filesystems
– ALL_O_DIRECT on Percona Server
137
www.percona.com

Direct IO Opera7on (cont.)
• innodb_flush_method=O_DIRECT
01010 0101
Buffered
Log Files
OS
Cache
Direct DMA write
Buffer Pool
Tablespace
138
www.percona.com

Direct IO Opera7on (cont.)
• innodb_flush_method=ALL_O_DIRECT
01010 0101
Direct DMA write
Log Files
OS
Cache
Direct DMA write
Buffer Pool
Tablespace
139
www.percona.com

READ-­‐ONLY Mode
• innodb_read_only=1
• Make all InnoDB tables and opera?ons read-­‐only
– Allows InnoDB to be operated from read-­‐only media
– Allows more than one MySQL instances to share the
same dataset
• Following must be done for this to work
– InnoDB slow shutdown
– InnoDB started with change buffering disabled
140
www.percona.com

Persistent Op7mizer Stats
• Persistent stats are enabled by default
• Enable query execu?on plans to be more stable
• Stats recalculated on-­‐demand
• Collect stats via ANALYZE/OPTIMIZE TABLE
• Make sure you collect stats aler ALTER TABLE
• Stats can be copied from master to slave
– Stats tables: mysql.innodb_index_stats,
mysql.innodb_table_stats
141
www.percona.com

Unlocking Features in XtraBackup
• Some important XtraBackup features are only
available when used with XtraDB
• Fast Incremental Backups
– A bitmap based fast incremental backup strategy
– Relies on “Changed Page Tracking” feature in
XtraDB -­‐ hSp://goo.gl/xhjLRc
• Archive Log Backups
– Keep history of modifica?ons in redo log archives
– Use with XtraBackup -­‐ hSp://goo.gl/W3bm8F
142
www.percona.com

Innodb Full Text Search
• Available since MySQL 5.6
• Is NOT 100% same as with MyISAM
• Small/Medium scale implementa?ons
– Larger ones use Solr, Sphinx, Elas?cSearch
• Implemented as several “hidden” Innodb tables
– Would require separate tutorial to cover in details
• Loading a lot of data:
– Load the data and when add FTS index
143
www.percona.com

The Future
SOME HIGHLIGHTS ON MYSQL 5.7
144
www.percona.com

More Online
• Online Index Rename
• Online VARCHAR length increase
• Online OPTIMIZE TABLE
145
www.percona.com

Temporary Tables
• Faster Create/Drop of Temporary Tables
• Can create temporary tables tablespace in
separate path
– Innodb_temp_data_path
• Undo logs op?mized to be used for rollback
only
– No redo logging for temporary tables
146
www.percona.com

General Performance
• Automa?c detec?on of read only transac?ons
• Internal Btree handling op?mized for lock
conten?on
• Mul?ple Cleaner Threads supported
147
www.percona.com

Thank you !
pz@percona.com
148
www.percona.com