Algorithms and Data Structures for External Memory

More documents

Recommendations

Info

5.3 Prefetching, Caching, and Applications to Sorting 43 Hon et al. [198] have developed competitive and optimal methods for prefetching blocks in parallel I/O systems. We focus in the remainder of this section on prefetching with knowledge of future read requests. We use the general framework of Hutchinson et al. [202], who demonstrate a powerful duality between prefetch scheduling and the following problem “in the reverse direction,” which we call output scheduling: We are given a sequence Σ ′ = 〈b ′ 1,b ′ 2,...,b ′ N〉 of blocks that an application program produces (or “writes”) in internal memory. A mapping disk :Σ ′ →{1,2,...,D} specifies the desired final disk location for each block; that is, the target location for block b ′ i is disk disk(b′ i ). The goal of output scheduling is to construct the shortest schedule of parallel I/O output operations so that each block b ′ i is output to its proper target location disk disk(b′ i ). (Note that “write” refers to the handover of the block from the application to the scheduling algorithm, whereas “output” refers to the subsequent I/O operation that moves the block onto disk.) We use the m blocks of internal memory as output buffers to queue the blocks that have been written but not yet output to the disks. If the blocks in Σ ′ are distinct, we call the output scheduling problem write-once scheduling. If some blocks are repeated in Σ ′ , we call the problem write-many scheduling. If we are able to keep a block in an output buffer long enough until it is written again by the application program, then we need to output the block at most once rather than twice. The output scheduling problem is generally easy to solve optimally. In Sections 5.3.2–5.3.4, we exploit a duality [202] of the output scheduling problems of write-once scheduling, write-many scheduling, and distribution in order to derive optimal algorithms for the dual prefetch problems of read-once scheduling, read-many scheduling, and merging. Figure 5.4 illustrates the duality and information flow for prefetch and output scheduling.
44 External Sorting and Related Problems Prefetch buffers / Output buffers Stream of blocks are read in Σ order 1 2 3 4 5 6 D = 6 Disks written in ΣR Stream of blocks are order Up to m prefetched or queued blocks D = 6 Disk numbers Internal Memory Correspondence between output step in greedy write-once schedule and prefetching step in lazy read-once schedule Fig. 5.4 Duality between prefetch scheduling and output scheduling. The prefetch scheduling problem for sequence Σ proceeds from bottom to top. Blocks are input from the disks, stored in the prefetch buffers, and ultimately read by the application program in the order specified by the sequence Σ. The output scheduling problem for the reverse sequence Σ R proceeds from top to bottom. Blocks are written by the application program in the order specified by Σ R , queued in the output buffers, and ultimately output to the disks. The hatched blocks illustrate how the blocks of disk 2 might be distributed. 5.3.1 Greedy Read-Once Scheduling Before we discuss an optimum prefetching algorithm for read-once scheduling, we shall first look at the following natural approach adopted by SRM [68, 72] in Section 5.2.1, which unfortunately does not achieve the optimum schedule length. It uses a greedy approach: Suppose that blocks b1, b2, ..., bi of the sequence Σ have already been read in prior steps and are thus removed from the prefetch buffers. The current step consists of reading the next blocks of Σ that are already in the prefetch buffers. That is, suppose blocks bi+1, bi+2, ..., bj are in the prefetch buffers, but bj+1 is still on a disk. Then blocks bi+1, bi+2, ..., bj are read and removed from the prefetch buffers. The second part of the current step involves input from the disks. For each of the D disks, consider its highest priority block not yet input,
Page 1 and 2:
TCSv2n4.qxd 4/24/2008 11:56 AM Page
Page 4 and 5:
Algorithms and Data Structures for
Page 6 and 7:
Foundations and Trends R○ in Theo
Page 8 and 9:
Foundations and Trends R○ in Theo
Page 10 and 11: Preface I first became fascinated a
Page 12: Preface xi I would like to thank my
Page 15 and 16: xiv Contents 6 Lower Bounds on I/O
Page 18 and 19: 1 Introduction The world is drownin
Page 20 and 21: However, not all memory references
Page 22 and 23: 1.1 Overview 5 Our general goal is
Page 24: Table 1.1 Paradigms for I/O efficie
Page 27 and 28: 10 Parallel Disk Model (PDM) spindl
Page 29 and 30: 12 Parallel Disk Model (PDM) D = nu
Page 31 and 32: 14 Parallel Disk Model (PDM) The pr
Page 33 and 34: 16 Parallel Disk Model (PDM) 2.3 Re
Page 35 and 36: 18 Parallel Disk Model (PDM) archit
Page 38 and 39: 3 Fundamental I/O Operations and Bo
Page 40: As Table 3.1 indicates, the multipl
Page 43 and 44: 26 Exploiting Locality and Load Bal
Page 45 and 46: 28 Exploiting Locality and Load Bal
Page 47 and 48: 30 External Sorting and Related Pro
Page 59: 42 External Sorting and Related Pro
Page 74 and 75: 6 Lower Bounds on I/O In this chapt
Page 76 and 77: 6.1 Permuting 59 in backwards order
Page 78 and 79: 6.2 Lower Bounds for Sorting and Ot
Page 80: 6.2 Lower Bounds for Sorting and Ot
Page 83 and 84: 66 Matrix and Grid Computations the
Page 86 and 87: 8 Batched Problems in Computational
Page 88 and 89: 8.1 Distribution Sweep 71 Goodrich
Page 90 and 91: 8.1 Distribution Sweep 73 the curre
Page 92 and 93: Time (seconds) Time (seconds) 7000
Page 94 and 95: 9 Batched Problems on Graphs The pr
Page 96 and 97: Table 9.2 Best known I/O bounds for
Page 98 and 99: 9.2 Special Cases 81 that contains
Page 100 and 101: 10 External Hashing for Online Dict
Page 102 and 103: 2 000 000 001 010 011 100 101 110 1
Page 104 and 105: 10.2 Directoryless Methods 87 a tot
Page 106 and 107: 11 Multiway Tree Data Structures In
Page 108 and 109: 11.1 B-trees and Variants 91 “sha
Page 110 and 111:
11.3 Parent Pointers and Level-Bala
Page 112 and 113:
11.4 Buffer Trees 95 I/Os by follow
Page 114:
11.4 Buffer Trees 97 between update
Page 117 and 118:
100 Spatial Data Structures and Ran
Page 119 and 120:
Page 121 and 122:
Page 123 and 124:
Page 125 and 126:
Page 127 and 128:
Page 129 and 130:
Page 131 and 132:
Page 133 and 134:
Page 136 and 137:
13 Dynamic and Kinetic Data Structu
Page 138 and 139:
13.2 Continuously Moving Items 121
Page 140 and 141:
14 String Processing In this chapte
Page 142 and 143:
5 a b a 3 c a 0 b a 4 b 6 6 a b a b
Page 144 and 145:
14.3 Suffix Trees and Suffix Arrays
Page 146 and 147:
15 Compressed Data Structures The f
Page 148 and 149:
15.1 Data Representations and Compr
Page 150 and 151:
15.2 External Memory Compressed Dat
Page 152 and 153:
Page 154:
Page 157 and 158:
140 Dynamic Memory Allocation For s
Page 159 and 160:
142 External Memory Programming Env
Page 161 and 162:
144 External Memory Programming Env
Page 163 and 164:
146 Conclusions intriguing connecti
Page 165 and 166:
148 Notations and Acronyms lowercas
Page 167 and 168:
150 Notations and Acronyms � �
Page 169 and 170:
152 References Proceedings of the A
Page 171 and 172:
154 References [41] L. Arge, K. H.
Page 173 and 174:
156 References Computer Science, pp
Page 175 and 176:
158 References ACM Conference on In
Page 177 and 178:
160 References [132] F. K. H. A. De
Page 179 and 180:
162 References [165] R. W. Floyd,
Page 181 and 182:
164 References [195] M. R. Henzinge
Page 183 and 184:
166 References [228] K. Küspert,
Page 185 and 186:
168 References [261] D. R. Morrison
Page 187 and 188:
170 References [293] J. T. Robinson
Page 189 and 190:
172 References [325] “TerraServer
Page 191:
174 References [356] O. Wolfson, P.
show all

Algorithms and Data Structures for External Memory

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?