One-sided vs. Two-sided Communication Paradigms on Relaxed ...

F U T U R E T E C H N O L O G I E S G R O U P<strong>One</strong>-<strong>sided</strong> <strong>vs</strong>. <strong>Two</strong>-<strong>sided</strong><strong>Communication</strong> <strong>Paradigms</strong> onRelaxed Ordering InterconnectsKhaled IbrahimPaul Hargrove, Costin Iancu, Kathy YelickLAWRENCE BERKELEY NATIONAL LABORATORY

OverviewF U T U R E T E C H N O L O G I E S G R O U P Application performance of one-<strong>sided</strong> <strong>vs</strong>. two-<strong>sided</strong> on Cray XE06(Gemini Interconnect). Brief Overview of Cray software stack and hardware on Hopper. Support of Relaxation Performance comparison of communication primitives using strictand relaxed ordering. Single-<strong>sided</strong> <strong>vs</strong>. two-<strong>sided</strong> communication paradigms interactionwith relaxed ordering.LAWRENCE BERKELEY NATIONAL LABORATORY

MotivationF U T U R E T E C H N O L O G I E S G R O U PPercentage UPC over MPI speedup250%200%40%30%20%10%0%-10%64 procs256 procsep ft is lu mg sp bt Harmonic meanAlso, why Gasnet is not matching vendor performance?Why single-<strong>sided</strong> performs better than two-<strong>sided</strong> MPI?LAWRENCE BERKELEY NATIONAL LABORATORY

Cray Software StackF U T U R E T E C H N O L O G I E S G R O U P uGNI Multiplecommunication protocols RDMA (Block TransferEngine (BTE)).• Optimized for largemessages FMA• Optimized for smallmessages DMAPP communicationprotocol High-level protocol forsingle-<strong>sided</strong>communication support collectivesIOCTLGNIKernel LevelGNI (kGNI)GNI CoreFortran, CMPICH2DIRECTSHMEMHardware (Gemini/Arie)UPCDMAPPGeneric Hardware Abstraction Layer (GHAL)libpgasDIRECTLAWRENCE BERKELEY NATIONAL LABORATORY

Hopper HardwareF U T U R E T E C H N O L O G I E S G R O U PHopper Node(Cray XE 6)2 twelve-core AMD 'MagnyCours'2.1-GHz processors per node.Four DDR3 1333-MHz memorychannels per twelve-core'MagnyCours' processorHopper NodeCore 0DDR3 Core 1Core 2Core 3DDR3 Core 4L3Cache6MBHT3L3Cache6MBCore 0Core 1Core 2Core 3Core 4DDR3DDR3201.6 Gflops/nodeL1: 64 KB, L2 caches::512KBCore 5Core 0HT3HT3HT3Core 5Core 06-MB L3 cache shared between 6cores.DDR3Core 1Core 2HT3Core 1Core 2DDR3DDR3Core 3Core 4L3Cache6MBL3Cache6MBCore 3Core 4DDR3Core 5Core 5HT3HT3HT3HT3NIC 0NetLinkBlockNIC 13D torus48 port YARC Router...LAWRENCE BERKELEY NATIONAL LABORATORY

Relaxation of Memory TransfersF U T U R E T E C H N O L O G I E S G R O U P Hypertransport transactions terms Posted (writes without ack) Non-Posted (reads or writes with Ack) Relaxation on Gemini Strict (no reodering) Default (non-posted get pass posted writes) Relaxed (all non-posted pass posted writes) Relaxation affects both in-node memory transactions and remotememory transactions Interconnect Relaxation: How to? Dynamic Routing Multiple virtual channels Why? Performance: easier way to improve throughput (BW), than to improvelatency (wire delay) Resilience and fault tolerance.LAWRENCE BERKELEY NATIONAL LABORATORY

MicrobenchmarkF U T U R E T E C H N O L O G I E S G R O U PNode 0&1UPCT 1UPCT 2...UPCT mNode 2&3UPCT 1+mUPCT 2+m...UPCT 2m Objective: DialectsCompare the performance of lowlevelAPIs <strong>vs</strong>. high level runtimes.uGNI (FMA & BTE)DMAPPBerkeley UPCCray UPCOSU MBW (MPI) Ranks: m Window: w number of outstandingnon-blocking operations. Testbed: defaultbidirectional48 communication pairsMultiple outstanding messages(Window size)• Default 1LAWRENCE BERKELEY NATIONAL LABORATORY

DMAPP <strong>vs</strong>. uGNI (FMA and BTE) - strictF U T U R E T E C H N O L O G I E S G R O U PDMAPP strict FMA strict BTE strict1600010000Bandwidth (MB/s)10001008 16 32 64 128 256 512 1K 2K 4K 8K 16KMsg Size2K4K8K16K32K64K128K256K512K1M2M4M1200080004000Bandwidth (MB/s)LAWRENCE BERKELEY NATIONAL LABORATORY

DMAPP <strong>vs</strong>. uGNI (FMA and BTE)- RelaxedF U T U R E T E C H N O L O G I E S G R O U PDMAPP relaxed FMA relaxed BTE relaxed1600010000Bandwidth (MB/s)10001008 16 32 64 128 256 512 1K 2K 4K 8K 16KMsg Size2K4K8K16K32K64K128K256K512KLAWRENCE BERKELEY NATIONAL LABORATORY1M2M4MDMAPP can hide complexity of uGNI (FMA and BTE)1200080004000Bandwidth (MB/s)

Strict Ordering and ConcurrencyF U T U R E T E C H N O L O G I E S G R O U PConcurrency (proc/node): 1 2 4 8 16 241000012000Bandwidth (MB/s)100010010FMA transport8 16 32 64 128 256 512 1K 2K 4K 8K 16KMsg SizeBTE transport2K4K8K16K32K64K128K256K512K1M2MLAWRENCE BERKELEY NATIONAL LABORATORY4M80004000Bandwidth (MB/s)

Relaxed Ordering and ConcurrencyF U T U R E T E C H N O L O G I E S G R O U PConcurrency (proc/node): 1 2 4 8 16 2410000Bandwidth (MB/s)10001001200080004000Bandwidth (MB/s)10FMA transport8 16 32 64 128 256 512 1K 2K 4K 8K 16K2K4K8KMsg SizeBTE transport16K32K64K128K256K512K1M2M4MLAWRENCE BERKELEY NATIONAL LABORATORY

Low-Level API SummaryF U T U R E T E C H N O L O G I E S G R O U P DMAPP hides switchingcomplexity between FMA and BTE It also provides collectives Relaxed ordering improvesperformance (expected). Strict ordering hurts performanceMOSTLY under high concurrency. Registration is an expensivememory operations that better beremoved from critical path ofexecution. Performance Difference between<strong>One</strong>-<strong>sided</strong> and <strong>Two</strong> <strong>sided</strong> is NOTdue to different Performance oflow-level APIs.IOCTLGNIKernel LevelGNI (kGNI)GNI CoreFortran, CMPICH2DIRECTSHMEMHardware (Gemini/Arie)UPCDMAPPGeneric Hardware Abstraction Layer (GHAL)libpgasDIRECTLAWRENCE BERKELEY NATIONAL LABORATORY

<strong>Communication</strong> <strong>Paradigms</strong>F U T U R E T E C H N O L O G I E S G R O U P<strong>One</strong>-<strong>sided</strong> UPC<strong>Two</strong>-<strong>sided</strong> MPIP0P1P0P1LocalpromotedLocalLocal sharedLocal sharedpromotedput/get operationssend/recv operationsLAWRENCE BERKELEY NATIONAL LABORATORY

<strong>Communication</strong> Paradigm TerritoryF U T U R E T E C H N O L O G I E S G R O U PTraditional Shared Memory - CoherentCaches <strong>One</strong>-<strong>sided</strong> load/store Default to relaxed ordering• Strict for synchronization Message Passing - nocoherence Send/recv matching Strict matchingLAWRENCE BERKELEY NATIONAL LABORATORY

<strong>Communication</strong> Paradigm TerritoryF U T U R E T E C H N O L O G I E S G R O U PTraditional Shared Memory - CoherentCaches <strong>One</strong>-<strong>sided</strong> load/store Default to relaxed ordering• Strict for synchronizationPGAS Shared Memory - CoherentCaches <strong>One</strong>-<strong>sided</strong> load/store Default to relaxed ordering• Strict for synchronization Message Passing - nocoherence between nodes Load/store matching Strict matching Partition Global Address space <strong>One</strong>-<strong>sided</strong> put/get strict ordering if blocking• Relaxed if non-blocking.LAWRENCE BERKELEY NATIONAL LABORATORY

<strong>Communication</strong> Paradigm TerritoryF U T U R E T E C H N O L O G I E S G R O U PTraditional Shared Memory - CoherentCaches <strong>One</strong>-<strong>sided</strong> load/store Default to relaxed ordering• Strict for synchronizationMPI+MPI Shared Memory - CoherentCaches Send/recv matching Shared memory bypass Message Passing - nocoherence between nodes Send/recv matching Strict matching Message passing Send/recv Matching Matching strict, progress is notLAWRENCE BERKELEY NATIONAL LABORATORY

Relaxed Ordering and<strong>Communication</strong> ParadigmF U T U R E T E C H N O L O G I E S G R O U P Requirement to service multiple outstanding requests Minimal remote-end involvement Unambiguous destination HPC runtime expectation from Device Driver APIs point-to-point ordering• Node-to-node ordering• Rank-to-rank ordering (multi-core makes it different from node-to-node) Or, relaxed ordering, with multiple completion semantic (local andglobal)LAWRENCE BERKELEY NATIONAL LABORATORY

Unambiguous Destination(Registration)F U T U R E T E C H N O L O G I E S G R O U PRegistration (Resolving remote ambiguity):Prevents memory swappingAllows offloading communication to NICAllows out-of-order progress of many transfers1000Expensive (hopefully not frequent)Limited and shared resourcesLocalP0LocalP1P0P1u sec10010a) Registration100010010sharedsharedpromotedpromoted12K4K8K16K32K64K128K256K512KMemory segment size1M2M4M8M1put/get UPCsend/recv MPILAWRENCE BERKELEY NATIONAL LABORATORY

UPC Shared Memory Ordering Challenge-ExampleF U T U R E T E C H N O L O G I E S G R O U P Cray definition of “Global Completion” GASnet Heisenbug!P0:01: Get (m(A 0,0 ) m(A 1,0 ))02: m(A 0,0 )m(A 0,0 )+103: Put(m(A 0,0 )m(A 1,0 ))04: m(A 0,1 )P 0HT3P 1P1:11: Wait until m(A 0,1 )12: Get (m(A 0,2 ) m(A 1,0 ))13: Print m(A 0,2 )A 0,0A 0,1A 0,2...GeminiNICGlobally address MemoryMemory Address: A node,offsetAll initialized to zeros3D torus010312GeminiNICHT3A 1,0A 1,1A 1,2...P 2P 3LAWRENCE BERKELEY NATIONAL LABORATORY

MPI OrderingF U T U R E T E C H N O L O G I E S G R O U P Non-overtake rule (determinism) If a sender sends two messages (Message 1 and Message 2) in successionto the same destination, and both match the same receive, the receiveoperation will receive Message 1 before Message 2. If a receiver posts two receives (Receive 1 and Receive 2), in succession,and both are looking for the same message, Receive 1 will receive themessage before Receive 2.Send: S order source rankdest rankS 2 0 S 1 0 S 0 0P 0P 1HT3GeminiNICReceive: R order source rankGeminiNIC3D torusHT3R 0 ANY S 1 2 R 2 ANYGemini P 2HT3NICP 3R 0 ANY R 1 ANYP 4P 6LAWRENCE BERKELEY NATIONAL LABORATORY

Cray MPI Method to ExploitGemini Relaxed OrderingF U T U R E T E C H N O L O G I E S G R O U PTypical eager <strong>vs</strong>. rendezvousCray’s<strong>Two</strong> eager modes<strong>Two</strong> rendezvous modeSwitching is controlled based on Message sizeRegistration cache is used(may not be able to tell which memory will be used for communication)E0 E1 R0 R10 512 1K 2K 4K 8K 16K 32K 64K 128K 256K 512K 1MB 4MBMPICH_GNI_MAX_VSHORT_MSG_SIZEMPICH_GNI_MAX_EAGER_MSG_SIZEMPICH_GNI_NDREG_MAXSIZELAWRENCE BERKELEY NATIONAL LABORATORY

MPI Eager 0F U T U R E T E C H N O L O G I E S G R O U PSender(PE0)1. GNI SMSG send(header+data)SMSGMailboxesPEnReceiver(PE1)PE4PE02CompletionQueue3.copyMax size changewith the number of ranks.LAWRENCE BERKELEY NATIONAL LABORATORY

MPI Eager 1F U T U R E T E C H N O L O G I E S G R O U PApplication spaceSender(PE0)1. copy2. GNI SMSG send(header)Pre-allocated(registered)Runtime buffersSMSGMailboxesPE4PE04. RDMA FMAGET5. GNI SMSG (Recv Done)3PEnCQReceiver(PE1)6. copyLAWRENCE BERKELEY NATIONAL LABORATORY

MPI Rendezvous 0F U T U R E T E C H N O L O G I E S G R O U PSender(PE0)2. GNI SMSG send(header)5. RDMA BTEGETSMSGMailboxesPE4PE03PEnCQReceiver(PE1)4. Registerreceive data(or lookup incache)1. Registersend data(or lookup incache)6. GNI SMSG (Recv Done)LAWRENCE BERKELEY NATIONAL LABORATORY

MPI Rendezvous 1F U T U R E T E C H N O L O G I E S G R O U PSender(PE0)1. GNI SMSG send(header)4. GNI SMSG send(CTS)SMSGMailboxesPE4PE0PEnCQReceiver(PE1)3. Registerreceive data(in chunks)25. pipelined GNI RDMA BTE PUTs4. Registersend data(in chunks)6. GNI SMSG Send (done)LAWRENCE BERKELEY NATIONAL LABORATORY

<strong>Two</strong>-<strong>sided</strong> Summary of ChallengesF U T U R E T E C H N O L O G I E S G R O U P Remote involvement Rank-to-rank strict ordering can cause• Node-to-node strict ordering by hardware• How to handle message cancellation?! Message size ambiguity• int MPI_Irecv(buf, count,datatype, source, tag, …)– count reflects buffer size NOT msg size.– Receiver cannot tell what protocol will be used before matching• Probing obstruct relaxed ordering progress. Remote readiness for transfer All memory space default to local Registration is on-demand• Variability of performance based on hit/miss in registration cache.LAWRENCE BERKELEY NATIONAL LABORATORY

1648Bwindow size1648BF U T U R E T E C H N O L O G I E S G R O U P25.0%11 2 4 8 16 248KB16412.5%25.0%1 111 1 2 2 4 4 8 816 162424 1 2 48KB 16 Experiment: Vary number of messages per rank.81.3%43.8%4 Vary the53.1% 71.9% number of ranks 4per node.434.4%93.2%81.3%43.8%453.1% 71.9%4LAWRENCE BERKELEY NATIONAL 79.5% 34.4%LABORATORYwindow sizeMPI default25.0%MPI <strong>vs</strong>. UPC 8B Transactions1648B MPI buffers UPC shared (no BTE) dest8B 8Bwindow size16168KB8KB16window size12.5%25.0%25.0%50.0%87.5%100%87.5%75.0%62.5%50.0% 4111 2 Concurrency4 8 16 24 1 2 4 8 16 248KBMPI default164168KBMPI buffers (no BTE)37.5%25.0%12.5%0.00%100%90.6%81.3%71.9%62.5%453.1%43.8%34.4%25.0%16UPC non-8B100%87.5%75.0%62.5%50.0%37.5%25.0%12.5%0.00%16 8KB100% 86.4%90.6%81.3%79.5%71.9%62.5%53.1%65.9%43.8%25.0%

window sizewindow sizewindow size87.5%62.5%25.0%62.5%25.0%425.0%25.0%44450.0% 4 50.0%12.5%12.5%37.5%50.0%37.5%25.0%25.0%MPI <strong>vs</strong>. UPC 8KB Transactions25.0%12.5%12.5%0.00%0.00%F U T U R E T E C H N O L O G I E S G R O U P111111 1 2 2 4 4 8 8 16 16 24 1241 2 2 4 4 8 8 16 1616 24 2424 1 2window siz8KB8KB16 16window size481.3% 81.3%53.1% 53.1% 71.9% 71.9%81.3% 81.3% 4 43.8% 443.8%34.4% 34.4%93.2%43.8% 43.8%1 11 11 1 2 2 4 4 8 8 16 16 24 1241 2 2 4 4 8 8 16 161624 24124 1 216window size44256 256 KB KB164MPI Defaultwindow sizeApplication developer perspective: 16 161616window size4window size8KB168KB16479.5%256 256 KBKBWhat level of concurrency should I use?What is the implication for intra-node communications?LAWRENCE BERKELEY NATIONAL LABORATORY62.5% 62.5%UPC shared dest71.9% 71.9%96.8%8KB16100% 100%90.6% 90.6%86.4%81.3% 81.3%71.9% 71.9%62.5% 62.5%4 79.5%53.1% 53.1%43.8% 43.8%34.4% 34.4%25.0% 25.0% 65.9%16256 KB100% 100%90.6% 90.6%81.3% 81.3%71.9% 71.9%62.5%462.5%53.1% 53.1%43.8% 43.8%34.4% 34.4%

window sizwindowwindow sizewindow size71.9%425.0%450.0%481.3% 4 43.8% 412.5%71.9%62.5% 496.8%37.5%53.1%25.0%43.8%62.5%12.5%MPI <strong>vs</strong>. UPC 2MB Transactions34.4%0.00%25.0%11F U T U R E T E C H1N 1O L O G I E S G R O U P11 1224488161624 124 1 22 444 888 161616 242424 1 21642 8KB16 MBMPI Default81.3%453.1% 71.9%71.9%window siz16window size4216MB28KBMB16UPC shared dest2 MB100% 100%90.6% 90.6%81.3% 81.3%43.8%71.9% 71.9%4 34.4% 34.4% 53.1%96.8%71.9% 62.5% 4 62.5%53.1% 53.1%96.8%43.8% 43.8%34.4% 34.4%25.0% 25.0%11112 4 8 16 24 1 112 4 8 8 16 16 241 2 4 8 16 24 1 2 4 8 16 24 241 2ConcurrencyConcurrencyConcurrency256 KB256 KB161616100%90.6%81.3%window size481.3% 4 43.8%71.9%LAWRENCE BERKELEY NATIONAL LABORATORY62.5%71.9%62.5%53.1%43.8%34.4%

<strong>Two</strong>-<strong>sided</strong> MPI <strong>vs</strong>. <strong>One</strong>-<strong>sided</strong> UPCF U T U R E T E C H N O L O G I E S G R O U PUPC shared UPC non-shared MPI100% window size = 1Difference is NOT due:UPC <strong>vs</strong>. MPIRuntime optimizationRegistration overheador, CopyingDifference is due to:language semanticand ability to exploit relaxedordering.Percentage of peak bandwidth90%80%70%60%50%40%30%20%10%0%1 proc4 procs24 procs8 B1 proc4 procs24 procs2MBLAWRENCE BERKELEY NATIONAL LABORATORY

ConclusionF U T U R E T E C H N O L O G I E S G R O U P <strong>One</strong>-<strong>sided</strong> communication exploits relaxed ordering with ease Communicate-able memory is easier to identify• annotated shared - can be registered upfront. Relaxed <strong>vs</strong>. strict ordering is explicitly specified• by programmer (or programming language). (Default to relaxed) Large percentage of the peak performance for different communicationpatterns <strong>Two</strong>-<strong>sided</strong> faces the following challenges Strict matching between send and receives (large startup overhead) Locality notion (everything default to local)• Expensive to prepare buffer for communication (registration). Receiver ambiguity about transactions (probing or over allocation) Performance may need high node concurrency (problematic to in-nodecommunication).LAWRENCE BERKELEY NATIONAL LABORATORY