Algorithms and Data Structures for External Memory

17
External Memory Programming Environments
There are three basic approaches to supporting development of
I/O-efficient code, which we call access-oriented, array-oriented, and
framework-oriented.
Access-oriented systems preserve the programmer abstraction of
explicitly requesting data transfer. They often extend the I/O interface
to include data type specifications and collective specification of multiple
transfers, sometimes involving the memories of multiple processing
nodes. Examples of access-oriented systems include the UNIX file system
at the lowest level, higher-level parallel file systems such as Whiptail
[316], Vesta [116], PIOUS [262], and the High Performance Storage
System [351], and I/O libraries MPI-IO [115] and LEDA-SM [124].
Array-oriented systems access data stored in external memory primarily
by means of compiler-recognized data types (typically arrays)
and operations on those data types. The external computation is
directly specified via iterative loops or explicitly data-parallel operations,
and the system manages the explicit I/O transfers. Arrayoriented
systems are effective for scientific computations that make
regular strides through arrays of data and can deliver high-performance
parallel I/O in applications such as computational fluid dynamics,
molecular dynamics, and weapon system design and simulation.
141