GPFS: Administration and Programming Reference - IRA Home

gpfsDataShipStart_t Structure
Name
gpfsDataShipStart_t – Initiates data shipping mode.
Library
GPFS Library (libgpfs.a for AIX, libgpfs.so for Linux)
Structure
typedef struct
{
int structLen;
int structType;
int numInstances;
int reserved;
} gpfsDataShipStart_t;
Description
The gpfsDataShipStart_t structure initiates data shipping mode.
Once all participating threads have issued this directive for a file, GPFS enters a mode where it logically
partitions the blocks of the file among a group of agent nodes. The agents are those nodes on which one
or more threads have issued the GPFS_DATA_SHIP_START directive. Each thread that has issued a
GPFS_DATA_SHIP_START directive and the associated agent nodes are referred to as the data shipping
collective.
In data shipping mode:
v All file accesses result in GPFS messages to the appropriate agents to read or write the requested
data.
v The GPFS_DATA_SHIP_START directive is a blocking collective operation. That is, every thread that
intends to access the file through data shipping must issue the GPFS_DATA_SHIP_START directive
with the same value of numInstances. These threads all block within their gpfs_fcntl() subroutines
until all numInstances threads have issued the GPFS_DATA_SHIP_START directive.
v The number of threads that issue the GPFS_DATA_SHIP_START directive does not have to be the
same on all nodes. However, each thread must use a different file handle. The default agent mapping
can be overridden using the GPFS_DATA_SHIP_MAP directive.
v Applications that perform a fine-grained write, sharing across several nodes, should benefit most from
data shipping. The reason for this is that the granularity of GPFS cache consistency is an entire file
block, which rarely matches the record size of applications. Without using data shipping, when several
nodes simultaneously write into the same block of a file, even non-overlapping parts of the block, GPFS
serially grants, and then releases, permission to write into the block to each node in turn. Each
permission change requires dirty cached data on the relinquishing node to be flushed to disk, yielding
poor performance. Data shipping avoids this overhead by moving data to the node that already has
permission to write into the block rather than migrating access permission to the node trying to write the
data.
However, since most data accesses are remote in data shipping mode, clients do not benefit from
caching as much in data shipping mode as they would if data shipping mode were not in effect. The
cost to send a message to another instance of GPFS to fetch or write data is much higher than the cost
of accessing that data through the local GPFS buffer cache. Thus, whether or not a particular
application benefits from data shipping is highly dependent on its access pattern and its degree of block
sharing.
v Another case where data shipping can help performance is when multiple nodes must all append data
to the current end of the file. If all of the participating threads open their instances with the O_APPEND
Chapter 9. GPFS programming interfaces 349