Is Parallel Programming Hard, And, If So, What Can You Do About It?

1.2. PARALLEL PROGRAMMING GOALS 3CPU Clock Frequency / MIPS1000010001001010.119751980198519901995Year2000200520102015Figure 1.1: MIPS/Clock-Frequency Trend for IntelCPUsceased to provide its traditional performance benefits,as can be seen in Figure 1.1. 1 This meansthat writing single-threaded code and simply waitinga years or two for the CPUs to catch up mayno longer be an option. Given the recent trends onthe part of all major manufacturers towards multicore/multithreadedsystems, parallelism is the wayto go for those wanting the avail themselves of thefull performance of their systems.Even so, the first goal is performance rather thanscalability, especially given that the easiest way toattain linear scalability is to reduce the performanceof each CPU [Tor01]. Given a four-CPU system,which would you prefer? A program that provides100 transactions per second on a single CPU, butdoes not scale at all? Or a program that provides 10transactions per second on a single CPU, but scalesperfectly? The first program seems like a better bet,though the answer might change if you happened tobe one of the lucky few with access to a 32-CPUsystem.That said, just because you have multiple CPUs isnot necessarily in and of itself a reason to use themall, especially given the recent decreases in price of1 ThisplotshowsclockfrequenciesfornewerCPUstheoreticallycapable of retiring one or more instructions per clock,and MIPS for older CPUs requiring multiple clocks to executeeven the simplest instruction. The reason for taking thisapproach is that the newer CPUs’ ability to retire multipleinstructions per clock is typically limited by memory-systemperformance.multi-CPU systems. The key point to understandis that parallel programming is primarily a performanceoptimization, and, as such, it is one potentialoptimization of many. If your program is fastenoughascurrentlywritten, thereisnoreasontooptimize,either by parallelizing it or by applying anyof a number of potential sequential optimizations. 2By the same token, if you are looking to apply parallelismas an optimization to a sequential program,then you will need to compare parallel algorithmsto the best sequential algorithms. This may requiresome care, as far too many publications ignore thesequential case when analyzing the performance ofparallel algorithms.1.2.2 ProductivityQuick Quiz 1.8: Why all this prattling on aboutnon-technical issues??? And not just any nontechnicalissue, but productivity of all things???Who cares???Productivity has been becoming increasingly importantthrough the decades. To see this, considerthat early computers cost millions of dollars at atime when engineering salaries were a few thousanddollars a year. If dedicating a team of ten engineersto such a machine would improve its performanceby 10%, their salaries would be repaid many timesover.One such machine was the CSIRAC, the oldeststill-intact stored-program computer, put in operationin1949[Mus04,Mel06].Giventhatthemachinehad but 768 words of RAM, it is safe to say that theproductivity issues that arise in large-scale softwareprojects were not an issue for this machine. Becausethis machine was built before the transistor era, itwas constructed of 2,000 vacuum tubes, ran with aclock frequency of 1KHz, consumed 30KW of power,and weighed more than three metric tons.It would be difficult to purchase a machine withthis little compute power roughly sixty years later(2008), with the closest equivalents being 8-bit embeddedmicroprocessors exemplified by the venerableZ80 [Wik08]. This CPU had 8,500 transistors,andcanstillbepurchasedin2008forlessthan$2USper unit in 1,000-unit quantities. In stark contrasttotheCSIRAC,software-developmentcostsareanythingbut insignificant for the Z80.TheCSIRACandtheZ80aretwopointsinalongtermtrend, as can be seen in Figure 1.2. This figureplots an approximation to computational power per2 Of course, if you are a hobbyist whose primary interest iswriting parallel software, that is more than reason enough toparallelize whatever software you are interested in.