Is Parallel Programming Hard, And, If So, What Can You Do About It?

F.1. CHAPTER 1: INTRODUCTION 273of users can easily justify substantial developereffort, as the cost of this development can bespread over the tens of millions of users. Notethat this includes things like kernels and systemlibraries.3. If the low-cost parallel machine is controllingthe operation of a valuable piece of equipment,then the cost of this piece of equipment mighteasily justify substantial developer effort.4. Ifthesoftwareforthelow-costparallelproducesan extremely valuable result (e.g., mineral exploration),then the valuable result might againjustify substantial developer cost.5. Safety-critical systems protect lives, which canclearly justify very large developer effort.6. Hobbyists and researchers might seek knowledge,experience, fun, or glory rather than meremoney.So it is not the case that the decreasing cost of hardwarerenders software worthless, but rather that it isno longer possible to “hide” the cost of software developmentwithin the cost of the hardware, at leastnot unless there are extremely large quantities ofhardware.Quick Quiz 1.10:This is a ridiculously unachievable ideal!!! Why notfocusonsomethingthatisachievableinpractice?Answer:This is eminently achievable. The cellphone is acomputer that can be used to make phone calls andto send and receive text messages with little or noprogramming or configuration on the part of theend user.This might seem to be a trivial example at firstglance, but if you consider it carefully you will seethat it is both simple and profound. When we arewilling to sacrifice generality, we can achieve trulyastounding increases in productivity. Those whocling to generality will therefore fail to set the productivitybar high enough to succeed in productionenvironments.Quick Quiz 1.11:What other bottlenecks might prevent additionalCPUs from providing additional performance?Answer:There are any number of potential bottlenecks:1. Main memory. If a single thread consumes allavailable memory, additional threads will simplypage themselves silly.2. Cache. If a single thread’s cache footprintcompletely fills any shared CPU cache(s), thenadding more threads will simply thrash the affectedcaches.3. Memory bandwidth. If a single thread consumesall available memory bandwidth, additionalthreads will simply result in additionalqueuing on the system interconnect.4. I/O bandwidth. If a single thread is I/O bound,adding more threads will simply result in themall waiting in line for the affected I/O resource.Specific hardware systems may have any numberof additional bottlenecks.Quick Quiz 1.12:What besides CPU cache capacity might requirelimiting the number of concurrent threads?Answer:There are any number of potential limits on thenumber of threads:1. Main memory. Each thread consumes somememory (for its stack if nothing else), so thatexcessive numbers of threads can exhaust memory,resulting in excessive paging or memoryallocationfailures.2. I/O bandwidth. If each thread initiates a givenamount of mass-storage I/O or networking traffic,excessive numbers of threads can result inexcessive I/O queuing delays, again degradingperformance. Some networking protocols maybe subject to timeouts or other failures if therearesomanythreadsthatnetworkingeventscannotbe responded to in a timely fashion.3. Synchronization overhead. For many sychronizationprotocols,excessivenumbersofthreadscan result in excessive spinning, blocking, orrollbacks, thus degrading performance.Specific applications and platforms may have anynumber of additional limiting factors.