Algorithms and Data Structures for External Memory

34 External Sorting and Related Problems
on the average, which corresponds to an even distribution. However,
if the load factor b/d is 1, the largest bin contains (lnd)/lnlnd balls
on the average, whereas any individual bin contains an average of only
one ball [341]. 1 Intuitively, the blocks in a bucket act as balls and
the disks act as bins. In our case, the parameters correspond to b =
Ω(dlogd), which suggests that the blocks in the bucket should be evenly
distributed among the disks.
By further analogy to the occupancy problem, if the number of
blocks per bucket is not Ω(D logD), then the technique breaks down
and the distribution of each bucket among the disks tends to be uneven,
causing a bottleneck for I/O operations. For these smaller values of N,
Vitter and Shriver use their second partitioning technique: The file is
streamed through internal memory in one pass, one memoryload at a
time. Each memoryload is independently and randomly permuted and
output back to the disks in the new order. In a second pass, the file
is input one memoryload at a time in a “diagonally striped” manner.
Vitter and Shriver show that with very high probability each individual
“diagonal stripe” contributes about the same number of items to each
bucket, so the blocks of the buckets in each memoryload can be assigned
to the disks in a balanced round robin manner using an optimal number
of I/Os.
DeWitt et al. [140] present a randomized distribution sort algorithm
in a similar model to handle the case when sorting can be done in
two passes. They use a sampling technique to find the partitioning
elements and route the items in each bucket to a particular processor.
The buckets are sorted individually in the second pass.
An even better way to do distribution sort, and deterministically
at that, is the BalanceSort method developed by Nodine and Vitter
[273]. During the partitioning process, the algorithm keeps track
of how evenly each bucket has been distributed so far among the disks.
It maintains an invariant that guarantees good distribution across the
disks for each bucket. For each bucket 1 ≤ b ≤ S and disk 1 ≤ d ≤ D,
let numb be the total number of items in bucket b processed so far
during the partitioning and let numb(d) be the number of those items
1 We use the notation lnd to denote the natural (base e) logarithm loge d.