Lustre 1.6 Operations Manual

Information on the Socket LND (socklnd) protocol
Lustre layers the socket LND (socklnd) protocol above TCP/IP. The first message
sent on the TCP/IP bytestream is HELLO, which is used to negotiate connection
attributes. The protocol version is determined by looking at the first 4+4 bytes of the
hello message, which contain a magic number and the protocol version
In KSOCK_PROTO_V1, the hello message is an lnet_hdr_t of type
LNET_MSG_HELLO, with the dest_nid (Destination Server/Machine) replaced by
net_magicversion_t. This is followed by 'payload_length' bytes of IP addresses (each
4 bytes) which list the interfaces that the sending socklnd owns. The whole message
is sent in little-endian (LE) byte order. There is no socklnd level V1 protocol after the
initial HELLO meaning everything that follows is unencapsulated LNET messages.
In KSOCK_PROTO_V2, the hello message is a ksock_hello_msg_t. The whole
message is sent in byte order of sender and the bytesex of 'kshm_magic' is used on
arrival to determine if the receiver needs to flip. From then on, every message is a
ksock_msg_t also sent in the byte order by sender. This either encapsulates an LNET
message (ksm_type == KSOCK_MSG_LNET) or is a NOOP. Every message includes
zero-copy request and ACK cookies in every message so that a zero-copy sender can
determine when the source buffer can be released without resorting to a kernel
patch. The NOOP is provided for delivering a zero-copy ACK when there is no
LNET message to back it on.
Note that socklnd may connect to its peers via a "bundle" of sockets - one for
bidirectional "ping-pong" data and the other two for unidirectional bulk data.
However the message protocol on every socket is as described earlier.
D-28 Lustre 1.6 Operations Manual • September 2008