A Compiler for Parallel Exeuction of Numerical Python Programs on ...

access pattern analysis and GPU code generation is done by a JIT compiler jit4GPU andun<strong>Python</strong> only plays a supporting role in GPU code generation.4.2.1 Motivation <strong>for</strong> the JIT compilerOne <strong>of</strong> the original goals in the project to compile <strong>Python</strong> was to extend un<strong>Python</strong> togenerate GPU code. However, after some ef<strong>for</strong>t, it became clear that un<strong>Python</strong> does nothave sufficient in<strong>for</strong>mation to per<strong>for</strong>m the analysis required <strong>for</strong> GPU code generation. Forexample, the compiler does not know the data layout and the aliasing <strong>of</strong> NumPy arraysthat are parameters to a function being compiled. NumPy arrays are generally views <strong>of</strong>an underlying data array and are more like pointers than true arrays from the perspective<strong>of</strong> compiler analysis. The memory layout <strong>of</strong> a NumPy array is determined by the strides<strong>of</strong> the NumPy array and these strides are dynamic quantities unknown to un<strong>Python</strong>. Twoapparently distinct NumPy arrays passed into a function may be two different views <strong>of</strong>the same piece <strong>of</strong> memory. A somewhat similar problem arises in C functions that havepointer parameters. Typically C compilers per<strong>for</strong>m a global analysis and can customize thefunction differently <strong>for</strong> different calling contexts. However, such an analysis is not applicable<strong>for</strong> un<strong>Python</strong>. Un<strong>Python</strong> is not a whole-program compiler and there<strong>for</strong>e cannot do globalanalysis, such as finding the calling context <strong>of</strong> all functions.The objective <strong>of</strong> un<strong>Python</strong> is to compile <strong>Python</strong> libraries that can be used by variousapplications. The idea is that un<strong>Python</strong> will only see a small portion <strong>of</strong> the applicationbecause only a small portion <strong>of</strong> the application is per<strong>for</strong>mance critical. <strong>Python</strong> programmerscannot be expected to impose the typing restrictions required by un<strong>Python</strong> on theirentire program. Even if a whole-program compiler were implemented <strong>for</strong> <strong>Python</strong>, <strong>Python</strong>applications <strong>of</strong>ten contain bindings to C libraries and those libraries are opaque to a <strong>Python</strong>compiler unless it is also interfaced with a full C compiler. The complexity <strong>of</strong> such a systemcan be prohibitive to implement. Thus, effectively, un<strong>Python</strong> cannot do any globalanalysis. In absence <strong>of</strong> a global analysis, one other possible solution <strong>for</strong> generating codefrom un<strong>Python</strong> was to generate different versions <strong>of</strong> the loop <strong>for</strong> different cases <strong>of</strong> layouts<strong>of</strong> NumPy arrays accessed within a loop. But NumPy arrays are very general structuresand the number <strong>of</strong> possible loop versions can become intractable as the number <strong>of</strong> NumPyarrays increases.A simpler solution to the problem <strong>of</strong> per<strong>for</strong>ming analysis <strong>for</strong> GPU code generation is tocompile a parallel loop to GPU code just be<strong>for</strong>e the loop is to be executed. At this stage,jit4GPU can query all the NumPy arrays to determine their data layouts and also knowsthe value <strong>of</strong> loop-invariant numeric constants. For example, loop bounds may appear tobe unknown symbolic constants to un<strong>Python</strong> but those constants are known to jit4gpu.Jit4GPU operates on a typed AST <strong>of</strong> the loop to be compiled and per<strong>for</strong>ms several analysis30