Lustre 1.6 Operations Manual

ME/MD Queue Length
The ptllnd uses a single portal set by the portal module parameter (default value of
9) for both message and bulk buffers. Message buffers are always attached with
PTL_INS_AFTER and match anything sent with "message" matchbits. Bulk buffers
are always attached with PTL_INS_BEFORE and match only specific matchbits for
that particular bulk transfer.
This scheme assumes that the majority of ME / MDs posted are for "message"
buffers, and that the overhead of searching through the preceding "bulk" buffers is
acceptable. Since the number of "bulk" buffers posted at any time is also dependent
on the bulk transfer breakpoint set by max_msg_size, this seems like an issue
worth measuring at scale.
TX Descriptors
The ptllnd has a pool of so-called "tx descriptors", which it uses not only for
outgoing messages, but also to hold state for bulk transfers requested by incoming
messages. This pool should scale with the total number of peers.
To enable the building of the Portals LND (ptllnd.ko) configure with this option:
./configure --with-portals=
Variable
ntx
(256)
concurrent_peers
(1152)
peer_hash_table_size
(101)
cksum
(0)
timeout
(50)
portal
(9)
rxb_npages
(64 * #cpus)
Description
Total number of messaging descriptors.
Maximum number of concurrent peers. Peers that attempt to
connect beyond the maximum are not allowed.
Number of hash table slots for the peers. This number should scale
with concurrent_peers. The size of the peer hash table is set by the
module parameter peer_hash_table_size which defaults to a value
of 101. This number should be prime to ensure the peer hash table
is populated evenly. It is advisable to increase this value to 1001 for
~10000 peers.
Set to non-zero to enable message (not RDMA) checksums for
outgoing packets. Incoming packets are always check-summed if
necessary, independent of this value.
Amount of time (in seconds) that a request can linger in a peersactive
queue before the peer is considered dead.
Portal ID to use for the ptllnd traffic.
Number of pages in an RX buffer.
31-16 Lustre 1.6 Operations Manual • September 2008