Is Parallel Programming Hard, And, If So, What Can You Do About It?

1.3. ALTERNATIVES TO PARALLEL PROGRAMMING 5PerformanceProductivityApplicationMiddleware (e.g., DBMS)System LibrariesOperating System KernelFirmwareHardwareGeneralityUser 1User 3Special−PurposeEnv Productivefor User 1HW /AbsGeneral−PurposeEnvironmentSpecial−Purpose EnvironmentProductive for User 3User 2Special−PurposeEnvironmentProductive for User 2User 4Special−PurposeEnvironmentProductive for User 4Figure 1.3: Software Layers and Performance, Productivity,and Generalityshown in Figure 1.3, which shows how productivitybecomes increasingly important at the upper layersof the system stack, while performance and generalitybecome increasingly important at the lowerlayers of the system stack. The huge developmentcosts incurred near the bottom of the stack must bespreadoverequallyhugenumbersofusersontheonehand (hence the importance of generality), and performancelost near the bottom of the stack cannoteasily be recovered further up the stack. Near thetop of the stack, there might be very few users fora given specific application, in which case productivityconcerns are paramount. This explains thetendency towards “bloatware” further up the stack:extra hardware is often cheaper than would be theextra developers. This book is intended primarilyfor developers working near the bottom of the stack,where performance and generality are paramountconcerns.It is important to note that a tradeoff betweenproductivity and generality has existed for centuriesin many fields. For but one example, a nailgun isfar more productive than is a hammer, but in contrastto the nailgun, a hammer can be used for manythings besides driving nails. It should therefore beabsolutely no surprise to see similar tradeoffs appearin the field of parallel computing. This tradeoffis shown schematically in Figure 1.4. Here, Users 1,2, 3, and 4 have specific jobs that they need thecomputer to help them with. The most productivepossible language or environment for a given user isone that simply does that user’s job, without requiringany programming, configuration, or other setup.Quick Quiz 1.10: This is a ridiculously unachievableideal!!! Why not focus on something thatis achievable in practice?Figure 1.4: Tradeoff Between Productivity and GeneralityUnfortunately, asystemthatdoesthejobrequiredby user 1 is unlikely to do user 2’s job. In otherwords, the most productive languages and environmentsare domain-specific, and thus by definitionlacking generality.Another option is to tailor a given programminglanguageorenvironmenttothehardwaresystem(forexample, low-level languages such as assembly, C,C++, or Java) or to some abstraction (for example,Haskell, Prolog, or Snobol), as is shown by the circularregion near the center of Figure 1.4. These languagescan be considered to be general in the sensethat they are equally ill-suited to the jobs requiredby users 1, 2, 3, and 4. In other words, their generalityis purchased at the expense of decreased productivitywhen compared to domain-specific languagesand environments.With the three often-conflicting parallelprogramminggoals of performance, productivity,and generality in mind, it is now time to look intoavoiding these conflicts by considering alternativesto parallel programming.1.3 Alternatives to ParallelProgrammingIn order to properly consider alternatives to parallelprogramming, you must first have thought throughwhat you expect the parallelism to do for you. Asseen in Section 1.2, the primary goals of parallel programmingare performance, productivity, and generality.Although historically most parallel developersmight be most concerned with the first goal, one ad-