Using MKL, the Intel Math Kernel Library (v11.0.0.079) - ICHEC

ICHECTECHNICALREPORT€€Alin M Elena & Buket B. GursoyICHEC Computational Scientists− ∂ 2 u∂x − ∂ 2 u+ qu = f (x, y),q = const ≥ 02 2∂yGeneral form of the Discrete Fourier TransformUsing MKL :: Intel Math Kernel Library2D Helmholtz Equationn d −1 n 2 −1 n 1 −1⎛⎛ d ⎞⎞z k1 ,k 2 ,...,k d= σ × ∑...∑ ∑W j1 , j 2 ,..., j dexp⎜⎜δi2π∑j lk l/n l ⎟⎟⎝⎝⎠⎠j d = 0j 2 = 0j 1 = 0Intel MKL (Math Kernel Library) provides mathematical routines forscientific and engineering applications which Intel have optimised andcustomised for performance. The library provides sequential, threaded anddistributed versions of routines. The library targets three architectures IA-32(32bit machines), Intel 64 (AMD64/EM64T machines) and IA64 (Itaniumprocessor family).l=1IntroductionIntel MKL is available on Stokes. To use the lastinstalled version just load the intel-mkl module. This willconfigure your environment to use MKL for Intel 64architecture as appropriate for Stokes. At the time of writingthe default version is 11.0.0.079.$ module load intel-mklMKL provides routines in the following areas:•BLAS•Sparse BLAS•LAPACK•PBLAS•ScaLAPACK•Sparse Solver routines•Vector Mathematical Library Functions•Statistical Functions•Fourier Transform Functions•Partial Differential Equations Support•Nonlinear Optimisation Problem Solvers•Support Functions•BLACS Routines•Data Fitting FunctionsMKL supports for C/C++ and Fortran, however oneshould note that not all the functions are available directlyfor both C/C++ and Fortran. In these cases mixed languagetechniques should be used. Intel MKL offers Fortran 90/95and up interfaces for some routines, BLAS and LAPACK.LinkingUsing MKLIntroduction 1Linking 1Interface layer 2Threading layer 2Computational layer 2Run-time library layer 3Linking model 4Static & Dynamic 5Examples 6-8Intel MKL employs a layered model for linking to moreeasily account for differences in, threading, computation,run-time libraries (RTL) on different operating systems. Thelayers are:•Interface Layer•Threading Layer•Computational Layer•Run-Time Library Layer1

•Interface LayerThis layer provides matching between the compiled code of an application and the threading/computationalcomponents of the library. This layer:•provides an LP64 interface.•access to MKL ILP64.•deals with the way in which different compilers return function values.•a way to map single precision names to double precision names in applications that employ ILP64programming model (e.g. Cray-style naming).Interface Layer Librarieslibmkl_intel_lp64 libmkl_intel_ilp64 libmkl_gf_lp64 libmkl_gf_ilp64 libmkl_intel_sp2dp•Threading LayerThis layer helps the threaded MKL to co-operate with compiler level threading. This also provides thesequential version layer.Threading Layer Librarieslibmkl_intel_thread libmkl_gnu_thread libmkl_pgi_thread libmkl_sequential•Computational LayerThis is the heart of MKL and has only one variant for any processor/operating system family, Thecomputational layer accommodates multiple architectures through identification of the architecture orarchitectural feature and chooses the appropriate binary code at execution. Intel MKL may be thought of asthe large computational layer that is unaffected by different computational environments. Then, as it has noRTL requirements, RTLs refer not to the computational layer but to one of the layers above it: the Interfacelayer or Threading layer. The most likely case is matching the threading layer with the RTL layer.Computational Layer Librarieslibmkl_avx libmkl_core libmkl_def libmkl_mc/mc3 libmkl_scalapack_lp64/ilp64libmkl_vml_def/p4n/mc/mc2/mc3/avx/cmptlibmkl_cdft_core2

•Run-Time Library LayerThis layer has run-time library support functions. For example, libiomp and libguide are run-time librariesproviding threading support for the OpenMP threading in Intel MKL. In addition to the Intel compiler, itprovides support for one more threading compiler on Linux OS (GNU). Note that when using the libiompyou should also link against the POSIX threads library by appending -lpthread.In addition to the libraries provided through the layered model you have the Fortran 90/95 interfaces andcluster components. Each of them fits in the computational or RTL layer.Run-Time Layer Librarieslibiomp5Fortran 90/95 Interfaceslibmkl_lapack95_lp64libmkl_blas95_lp64libmkl_lapack95_ilp64libmkl_blas95_ilp64Cluster Componentslibmkl_blacs_intelmpi_lp64libmkl_blacs_openmpi_lp64libmkl_blacs_sgimpt_lp64libmkl_scalapack_lp64libmkl_blacs_lp64libmkl_blacs_intelmpi_ilp64libmkl_blacs_openmpi_ilp64libmkl_blacs_sgimpt_ilp64libmkl_scalapack_ilp64libmkl_blacs_ilp64libmkl_cdft_coreFFT Interfaceslibfftw2x_cdft_DOUBLE/SINGLE libfftw2xc_intel/intel_sp libfftw2xf_intel/intel_splibfftw3x_cdft/cdft_sp libfftw3xc_intel/intel_sp libfftw3xf_intel/intel_spLP64and ILP64LP64 stands for long and pointer as 64 bit types and ILP64 stands for int, long and pointer as 64 bittypes. To operate on large data arrays (of more than 231-1 elements), you need to select the ILP64interface, where integers are 64-bit; otherwise, use the default, LP64, interface, where integers are 32-bit.3

Static/Dynamic LinkingIntel MKL supports both linking models static ordynamic. Each of them has their pros and cons. Staticlinking resolves all symbolic references at link time. Thebehaviour of statically built executables is predictable,because there are no run-time dependencies. The maindisadvantage is that having to relink new versions of thelibrary to your application may be error-prone and timeconsuming,because you have to relink the entireapplication. Moreover, static linking results in largeexecutables and uses memory less efficiently. If severalexecutables are linked with the same library, each of themmust load it into memory independently. This matters mostfor executables having data sizes that are small andcomparable with the size of the executable.Dynamic linking postpones the resolution of someundefined symbolic references until run time. Dynamicallybuilt executables contain those symbols along with a list oflibraries that provide definitions of the symbols. When theexecutable is loaded, the final linking is done before theapplication runs. If several dynamically built executablesreference the same library, it is loaded into memory onlyonce and the executables share it, thereby saving memory.Dynamic linking enables you to separately update thelibraries on which applications depend and does notrequire relinking the applications. The developmentadvantages of dynamic linking are achieved at some costto performance, because every unresolved symbol has tobe looked up in a dedicated table and resolved at runtime.Linking SyntaxDynamic Case -L$MKLPATH -I$MKLINCLUDE[-lmkl_blas{95_ilp64|95_lp64}][-lmkl_lapack{95_ilp64|95_lp64}][ ]-lmkl_{intel_ilp64|intel_lp64|intel_sp2dp|gf_ilp64|gf_lp64}-lmkl_{intel_thread|gnu_thread|pgi_thread|sequential}-lmkl_core[-liomp5] [-lpthread] [-lm] [-ldl]Static Case -L$MKLPATH -I$MKLINCLUDE[$MKLPATH/libmkl_blas{95_ilp64|95_lp64}.a][$MKLPATH/libmkl_lapack{95_ilp64|95_lp64}.a]-Wl,--start-group[ ]$MKLPATH/libmkl_{intel_ilp64|intel_lp64|intel_sp2dp|gf_ilp64|gf_lp64}.a$MKLPATH/libmkl_{intel_thread|gnu_thread|pgi_thread|sequential}.a$MKLPATH/libmkl_core.a-Wl,--end-group[-liomp5] [-lpthread] [-lm] [-ldl]Notes: {a,b,c} only one of the libraries should be chosen, [a] optional. On stokes and stoney$MKLPATH=$MKLROOT/lib/intel64$MKLINCLUDE=$MKLROOT/include5

Linking examplesOne should note the dummy libraries mkl_intel, mkl_gf, mkl_scalapack, are just pointing to the _lp64 libraries.Further details about the libraries contained by MKL can be found in the “Intel MKL User Guide”:http://registrationcenter.intel.com/irc_nas/2690/mkl_userguide_lnx.pdfOn stokes and/or stoney you will need to load the following modules intel-cc and/or intel-fc, intel-mkl and mvapich2-intel for the distributed version. The $MKLROOT variable is defined when the MKL module is loaded and will correspond toa given version of the MKL e.g. it may be set to: /ichec/packages/intel/composerxe_mkl/2013.0.079/mkl.Sequential and Threaded:intel_lp64, sequential, this example will link all the domain specific functions available in MKL which do not need anyextra libraries or interfaces, e.g. BLAS, FFT, VML.Dynamic-L$MKLROOT/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthreadStatic-L$MKLROOT/lib/intel64 -Wl,--start-group$MKLROOT/lib/intel64/libmkl_intel_lp64.a $MKLROOT/lib/intel64/libmkl_sequential.a$MKLROOT/lib/intel64/libmkl_core.a -Wl,--end-group -lpthreadintel_lp64, intel_threads, iomp5Dynamic-L$MKLROOT/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthreadStatic$MKLROOT/lib/intel64 -Wl,--start-group$MKLROOT/lib/intel64/libmkl_intel_lp64.a $MKLROOT/lib/intel64/libmkl_intel_thread.a$MKLROOT/lib/intel64/libmkl_core.a -Wl,--end-group -liomp5 -lpthreadLapack, intel_lp64, intel_threads, iomp5Dynamic-L$MKLROOT/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lmStatic-L$MKLROOT/lib/intel64 -Wl,--start-group$MKLROOT/lib/intel64/libmkl_intel_lp64.a $MKLROOT/lib/intel64/libmkl_intel_thread.a$MKLROOT/lib/intel64/libmkl_core.a -Wl,--end-group -liomp5 -lpthread -lmLapack Fortran 90/95 interface, intel_lp64, intel_threads, iomp5Dynamic-L$MKLROOT/lib/intel64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core-liomp5 -lpthread -lmStatic-L$MKLROOT/lib/intel64 $MKLROOT/lib/intel64/libmkl_lapack_lp64.a -Wl,--start-group $MKLROOT/lib/intel64/libmkl_intel_lp64.a $MKLROOT/lib/intel64/libmkl_intel_thread.a $MKLROOT/lib/intel64/libmkl_core.a6

Parallel Direct Sparse Solver, intel_lp64, intel_threads, iomp5Dynamic-L$MKLROOT/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lmStatic-L$MKLROOT/lib/intel64 -Wl,--start-group$MKLROOT/lib/intel64/libmkl_intel_thread.a $MKLROOT/lib/intel64/libmkl_core.a -Wl,--endgroup-liomp5 -lpthread -lmcblas,intel_lp64, intel_threads, iomp5Dynamic-L$MKLROOT/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lmStatic-L$MKLROOT/lib/intel64 -Wl,--start-group $MKLROOT/lib/intel64/libmkl_intel_lp64.a$MKLROOT/lib/intel64/libmkl_intel_thread.a $MKLROOT/lib/intel64/libmkl_core.a -Wl,--endgroup-liomp5 -lpthread -lm±Touse single precision fftw interfaces add the suffix _sp to the fftw interface name. Do not forget toadd the preprocessor option -DFFTW_ENABLE_FLOAT at compile time and include the headers from$MKLROOT/include/fftwfftw2xf interface, intel_lp64, intel_threads, iomp5-L$MKLROOT/lib/intel64 _I$MKLROOT/include/fftw $MKLROOT/lib/intel64/libfftw2xf_intel.a-Wl,--start-group $MKLROOT/lib/intel64/libmkl_intel_lp64.a $MKLROOT/lib/intel64/libmkl_intel_thread.a $MKLROOT/lib/intel64/libmkl_core.a -Wl,--end-group -liomp5 -lpthread -lmfftw2xc interface, intel_lp64, intel_threads, iomp5-L$MKLROOT/lib/intel64 -I$MKLROOT/include/fftw $MKLROOT/lib/intel64/libfftw2xc_intel.a-Wl,--start-group $MKLROOT/lib/intel64/libmkl_intel_lp64.a$MKLROOT/lib/intel64/libmkl_intel_thread.a $MKLROOT/lib/intel64/libmkl_core.a-Wl,--end-group -liomp5 -lpthread -lmfftw3xf interface, intel_lp64, intel_threads, iomp-L$MKLROOT/lib/intel64 -I$MKLROOT/include/fftw $MKLROOT/lib/intel64/libfftw3xf_intel.a-Wl,--start-group $MKLROOT/lib/intel64/libmkl_intel_lp64.a$MKLROOT/lib/intel64/libmkl_intel_thread.a $MKLROOT/lib/intel64/libmkl_core.a-Wl,--end-group -liomp5 -lpthread -lm7