Lustre SMP scalability and data I/O performance on Bullx ... - EOFS

Compute PartitionsWhat is it ?divide multi-CPU server into several processing partitionspartition-local <strong>and</strong> per-partition <strong>data</strong> allocators<strong>Lustre</strong> threads can be bound to compute partitionsConfiguration parameterscpu_npartitions : # of CPU partitions options libcfs cpu_npartitions=2cpu_pattern : CPU partitions pattern options libcfs cpu_pattern="0[0-7] 1[8-15]" options libcfs cpu_pattern="N 0[0] 1[2]"server0 1 2 3 4 5 6 7socket 0 socket 18 9 10 11socket 212 13 14 15socket 3©Bull 20127

Partitioned LNet <strong>and</strong> LNDWhat is it ?LND thread pool for each compute partitionLNet per-partition <strong>data</strong>reduce lock contention©Bull 20129

Partitioned ptlrpc serviceWhat is it ?ptlrpc service thread pool for each partitionrequest-queue <strong>and</strong> wait-queue for each partitionto be completed with ptlrpc requests posting (lustre client)©Bull 201212

Partitioned ptlrpc serviceWhat is it ?ptlrpc service thread pool for each partitionrequest-queue <strong>and</strong> wait-queue for each partitionto be completed with ptlrpc requests posting (lustre client)Configuration parametersCPU partitions threads should run on– MDS threads : mds_num_cpts, mds_rdpg_num_cpts, mds_attr_num_cpts– OSS threads : oss_cpts, oss_io_cpts– ldlm threads : ldlm_cpts©Bull 201213

Data I/O Performance©Bull 201214

Data I/O <strong>performance</strong>Goalevaluate the effect on <strong>data</strong> I/O <strong>performance</strong>measure the improvement on bullx Supernodes– large <strong>SMP</strong> node (up to 128 cores), highly NUMIOA– follow up of NUMIOA presentations at LUG 2010 <strong>and</strong> LUG 2011ib3ib2BCSib1BCSib0 socket BCS 12 socket 13socket 8 socket 9BCSsocket 4 socket 50 1 2 3 socket 14 socket 15socketsocket1014 5 6 7socket 11QPIsocket 6 socket 7socket 2 socket 3©Bull 201216

Test Cluster Architecturekernel 2.6.32OFED 1.5.48 x 2-socketClientsbullx R424F3ClientClientClientClientClientClientClientClientIB networkFDR2 x IB2 x IBlustre 2.2.93two o2ib lustre networksOSSOSS8 x FC88 x FC8DiskArray16-socketClientbullx S6010-4Large <strong>SMP</strong>Client2 x IB QDR2-socket OSSsbullx R423E330 OSTsDDN SFA10k©Bull 201217

<strong>Lustre</strong> Configurationsnocpt compute partitions disabledauto compute partitions automatic setupcpt2 two compute partitionscpt2lcl two compute partitions, network interfaces bound to local CPTcpt2rmt two compute partitions, network interfaces bound to remote CPTlustre21CLIENTOSSfc0 fc1 fc2 fc3socket 0 socket 1QPIsocket 0 socket 1QPIib0 ib0:0o2ib0 o2ib1ib0o2ib0ib1o2ib1©Bull 201218

<strong>Lustre</strong> Configurationsnocpt compute partitions disabledauto compute partitions automatic setupcpt2 two compute partitionscpt2lcl two compute partitions, network interfaces bound to local CPTcpt2rmt two compute partitions, network interfaces bound to remote CPTlustre21CPT0CLIENTCPT0OSSfc0 fc1 fc2 fc3socket 0 socket 1QPIsocket 0 socket 1QPIib0 ib0:0o2ib0 o2ib1ib0o2ib0ib1o2ib1©Bull 201219

<strong>Lustre</strong> Configurationsnocpt compute partitions disabledauto compute partitions automatic setupcpt2 two compute partitionscpt2lcl two compute partitions, network interfaces bound to local CPTcpt2rmt two compute partitions, network interfaces bound to remote CPTlustre21CLIENTOSSfc0 fc1 fc2 fc3socket 0 socket 1QPICPT0 CPT1 CPT2 CPT3socket 0 socket 1QPICPT0 CPT1 CPT2 CPT3ib0 ib0:0o2ib0 o2ib1ib0o2ib0ib1o2ib1©Bull 201220

<strong>Lustre</strong> Configurationsnocpt compute partitions disabledauto compute partitions automatic setupcpt2 two compute partitionscpt2lcl two compute partitions, network interfaces bound to local CPTcpt2rmt two compute partitions, network interfaces bound to remote CPTlustre21CPT0CLIENTCPT1socket 0 socket 1QPICPT0OSSfc0 fc1 fc2 fc3CPT1socket 0 socket 1QPIib0 ib0:0o2ib0 o2ib1ib0o2ib0ib1o2ib1©Bull 201221

<strong>Lustre</strong> Configurationsnocpt compute partitions disabledauto compute partitions automatic setupcpt2 two compute partitionscpt2lcl two compute partitions, network interfaces bound to local CPTcpt2rmt two compute partitions, network interfaces bound to remote CPTlustre21CLIENTOSSfc0 fc1 fc2 fc3socket 0 socket 1QPIsocket 0 socket 1QPICPT0CPT1CPT0CPT1ib0 ib0:0o2ib0 o2ib1ib0o2ib0ib1o2ib1©Bull 201222

<strong>Lustre</strong> Configurationsnocpt compute partitions disabledauto compute partitions automatic setupcpt2 two compute partitionscpt2lcl two compute partitions, network interfaces bound to local CPTcpt2rmt two compute partitions, network interfaces bound to remote CPTlustre21BCSib1BCSib0 socket 12 socket 13BCSsocket 8 socket 9BCSsocket 4 socket 5socket 14 socket 15socket 0socketsocket101socket 11QPIsocket 6 socket 7socket 2 socket 3Large <strong>SMP</strong> CLIENT©Bull 201223

<strong>Lustre</strong> Configurationsnocpt compute partitions disabledauto compute partitions automatic setupcpt2 two compute partitionscpt2lcl two compute partitions, network interfaces bound to local CPTcpt2rmt two compute partitions, network interfaces bound to remote CPTlustre21BCSib1BCSib0 socket 12 socket 13BCSsocket 8 socket 9BCSsocket 4 socket 5socket 14 socket 15socket 0socketsocket101socket 11QPIsocket 6 socket 7socket 2 socket 3CPT0Large <strong>SMP</strong> CLIENT©Bull 201224

<strong>Lustre</strong> Configurationsnocpt compute partitions disabledauto compute partitions automatic setupcpt2 two compute partitionscpt2lcl two compute partitions, network interfaces bound to local CPTcpt2rmt two compute partitions, network interfaces bound to remote CPTlustre21CPT0BCSib1CPT6 BCS CPT7ib0 socket 12 socket 13CPT4 BCS CPT5socket 8 socket 9CPT2 BCS CPT3socket 4 socket 5CPT1 socket 14 socket 15socket 0socketsocket101socket 11QPIsocket 6 socket 7socket 2 socket 3Large <strong>SMP</strong> CLIENT©Bull 201225

<strong>Lustre</strong> Configurationsnocpt compute partitions disabledauto compute partitions automatic setupcpt2 two compute partitionscpt2lcl two compute partitions, network interfaces bound to local CPTcpt2rmt two compute partitions, network interfaces bound to remote CPTlustre21BCSib1BCSib0 socket 12 socket 13BCSsocket 8 socket 9BCSCPT1socket 4 socket 5socket 14CPT0socket 15socket 0socketsocket101socket 11QPIsocket 6 socket 7socket 2 socket 3Large <strong>SMP</strong> CLIENT©Bull 201226

<strong>Lustre</strong> Configurationsnocpt compute partitions disabledauto compute partitions automatic setupcpt2 two compute partitionscpt2lcl two compute partitions, network interfaces bound to local CPTcpt2rmt two compute partitions, network interfaces bound to remote CPTlustre21BCSib1BCSib0 socket 12 socket 13BCSsocket 8 socket 9BCSCPT1socket 4 socket 5socket 14CPT0socket 15socket 0socketsocket101socket 11QPIsocket 6 socket 7socket 2 socket 3Large <strong>SMP</strong> CLIENT©Bull 201227

Data I/O <strong>performance</strong> measurementsLNet<strong>Lustre</strong> <strong>data</strong> I/OObdfilterClientClientClientClientClientClientClientClientIB networkFDR2 x IB2 x IBOSSOSS8 x FC88 x FC8DiskArrayLarge <strong>SMP</strong>Client2 x IB QDR©Bull 201228

LNet <strong>data</strong> b<strong>and</strong>widthsingle client2-socket client : 5% improvement vs. lustre21 or no CPTlarge <strong>SMP</strong> client : 80% improvementLNet Selftests - <strong>data</strong> b<strong>and</strong>widthsingle client R424F3 (1 IB FDR)LNet Selftests - <strong>data</strong> b<strong>and</strong>widthsingle client S6010-4 (2 IB QDR)60006000Throughput (MB/s)50004000300020001000nocptautocpt2cpt2lclcpt2rmtlustre21Throughput (MB/s)50004000300020001000nocptautocpt2cpt2lclcpt2rmtlustre210w riteread0w riteread©Bull 201229

Obdfilter b<strong>and</strong>widthtest process placement according to NUMA node of each OSTenhanced version of obdfilter-surveyno NUMIOA effect observedThroughput (MB/s)120001000080006000400020000Throughput (MB/s)Obdfilter1 obj/OST - write12000100008000600040002000Obdfilter1 obj/OST - writeThroughput (MB/s)1200010000001 2 4 81 16 2 32 4 64 8 16 1 2 32 4 64 8 16 32 648000600040002000Obdfilter1 obj/OST - readthreads/OSTthreads/OSTthreads/OSTlustre21, no binding lustre21, lcl binding lustre21, rmt bindingauto, no binding auto, lcl binding auto, rmt binding©Bull 201230

File System <strong>data</strong> I/O B<strong>and</strong>widthsingle clientfar below LNet b<strong>and</strong>width:<strong>performance</strong> is limited by other <strong>Lustre</strong> client componentsLU-744 Single client's <strong>performance</strong> degradation on 2.1IOR single client - file per processR424F3 - 16 processesIOR single client - file per processS6010-4 - 30 processes30003000Throughput (MB/s)2500200015001000500nocptautocpt2cpt2lclcpt2rmtlustre21Throughput (MB/s)2500200015001000500nocptautocpt2cpt2lclcpt2rmtlustre210w riteread0w riteread©Bull 201231

File System <strong>data</strong> I/O B<strong>and</strong>widthmultiple clientsOSSs have a low NUMIOA effect (obdfilter results)at least no regression8000IOR multiple clients - file per process7 R424F3 clients - 15 procs per clientThroughput (MB/s)7000600050004000300020001000nocptautocpt2cpt2lclcpt2rmtlustre210w riteread©Bull 201232

Conclusion<strong>SMP</strong> <strong>scalability</strong> <strong>and</strong> affinity featureefficient APIs to run <strong>Lustre</strong> on <strong>SMP</strong> <strong>and</strong> NUMIOA machinesconfiguration is manual– "dynamic LNet configuration" projectData I/O <strong>performance</strong> resultsat LNet level: visible improvementat filesystem level:– at least no regression– masked by client limitations©Bull 201233

©Bull 201234

Hardware / Software configurationOSS2 bullx R423E3 - Intel S<strong>and</strong>yBridgeE5-2660, 2.20GHz, 16 CPU cores, 32GBMemory, 2 x IB FDR adapters, 4 x biportFC8 adaptersDDN SFA10k - 16 x FC8, 300 x 940GBSATA disks, 30 x RAID6 pools30 OSTs - 30 <strong>data</strong> + 30 journal LUNsClients8 bullx R424F3 - Intel S<strong>and</strong>yBridgeE5­2660, 2.20GHz, 16 CPU cores, 32GBMemory, IB FDR adaptersNetworkInfiniB<strong>and</strong> FDRtwo o2ib lustre networksOSTs restricted to one of thelustre networkSoftware Stackbullxlinux 6.2 (based on rhel 6.2)OFED 1.5.4lustre 2.2.931 bullx S6010-4 - Intel NehalemEXX7550, 2.00GHz, 128 CPU cores, 256GBMemory, 2 x IB QDR adapters©Bull 201235

Test parametersLNet Selftestsbrw read/write, o2ib0+o2ib1, size=1M, concurrency=64, check=noneObdfilter-surveyIORcase=disk, size=10GB, recordsize=1Mnobjlo=1, nobjhi=64, thrlo=1, thrhi=256verify=0numactl=none/local/remotenumactl --physcpubind=xxx --localalloc lctl test_brw ...stripe count=1, stripe size=1Mapi=POSIX, filePerProc=1, fsync=1, transferSize=1MBsingle R424F3 client: blockSize=4GB, numTasks=16single S6010-4 client: blockSize=2GB, numTasks=30multiple R424F3 clients: blockSize=4GB, numTasks=15*#clients©Bull 201236

<strong>Lustre</strong> <strong>SMP</strong> Node Affinity - MDS resultsfrom Liang Zhen's presentation - Aug, 28 2012LNet Selftestslustre 2.3 ping is 900% (600%) of lustre 2.2 with Portal-RR OFF (ON)lustre 2.3 4K-BRW is 500%-700% of lustre2.2mdtestlustre 2.3 open-create <strong>performance</strong> is 350%-400% of lustre 2.2lustre 2.3 unlink <strong>performance</strong> is 150%-300% of lustre 2.2lustre 2.3 stat <strong>performance</strong> is 200%-400% of lustre 2.2©Bull 201237