A Compiler for Parallel Exeuction of Numerical Python Programs on ...

Chapter 1IntroductionIn recent years, processor designers have hit a wall in increasing the per<strong>for</strong>mance <strong>of</strong> a singleprocessor core. Processors have adopted multicore designs fitting multiple cores on a singledie or a single package. Quad-core CPUs are already common on the desktop and hexa-coreprocessors are available <strong>for</strong> servers. Another new development in the hardware field has beenthe emergence <strong>of</strong> programmable graphics processing units (GPUs). GPUs have evolved froma fixed function pipeline to a highly programmable pipeline. Modern GPUs are a massivelyparallel architecture containing hundreds <strong>of</strong> programmable ALUs that can be utilized <strong>for</strong>tasks such as numerical computation. A programmable GPU which can be utilized <strong>for</strong>non-graphic related tasks is termed a general-purpose GPU (GPGPU). GPGPUs are lessflexible than CPUs but <strong>of</strong>fer much greater floating-point capability. GPGPUs <strong>of</strong>fering up toa teraflop <strong>of</strong> floating-point per<strong>for</strong>mance are already available. This per<strong>for</strong>mance is an order<strong>of</strong> magnitude better than current x86 processors.The per<strong>for</strong>mance gains <strong>of</strong>fered by multicores and GPGPUs comes at the cost <strong>of</strong> increasedprogrammning ef<strong>for</strong>t. To utilize multicores, a programmer is required to write parallel code.Many abstractions, such as threads and parallel loops, are available <strong>for</strong> writing parallel programs.For numerical programs, expressing parallelism through parallel <strong>for</strong>-loops (availablein APIs such as OpenMP) is a common choice <strong>for</strong> multicores as well as <strong>for</strong> symmetric multiprocessorarchitectures. Programming <strong>for</strong> GPGPUs requires the programmer to rewritemuch <strong>of</strong> her code to utilize a GPU specific API. GPGPUs usually have a completely separateaddress space and there<strong>for</strong>e the programmer has to copy data from the system RAM to theGPGPU address space.On the s<strong>of</strong>tware side, languages such as Matlab and <strong>Python</strong> are becoming more popular<strong>for</strong> scientific computing. <strong>Python</strong> in particular is starting to attract a lot <strong>of</strong> attention in thenumerical-programming community. <strong>Python</strong> is a general-purpose object-oriented languagewith very clean syntax. An extension library <strong>for</strong> <strong>Python</strong>, called NumPy, <strong>of</strong>fers very flexiblemulti-dimensional array abstractions and is particularly useful <strong>for</strong> scientific programming.However, while <strong>Python</strong> <strong>of</strong>fers great flexibility to the programmer, per<strong>for</strong>mance is usually1