The Design and Implementation of the Anykernel and Rump Kernels

143
We use shared memory provided by a memory mapped file as the network bus.
The filename acts as the bus handle — all network interfaces configured to be on
the same bus use the same filename. The shmif driver (manual page shmif.4 at
A–47) in the rump kernel accesses the bus. Each driver instance accesses one bus,
so it is possible to connect a rump kernel to multiple different busses by configuring
multiple drivers. Since all nodes have full access to bus contents, the approach does
not protect against malicious nodes. As our main use case is testing, this lack of
protection is not an issue. Also, the creation of a file on the host is not an issue,
since testing is commonly carried out in a working directory which is removed after
a test case has finished executing.
The shmif driver memory maps the file and uses it as a ring buffer. The header
contains pointers to the first and last packet and a generation number, along with
bus locking information. The bus lock is a spinlock based on cross-process shared
memory. A downside to this approach is that if a rump kernel crashes while holding
the bus lock, the whole bus will halt. However, we have never encountered this
situation, so we do not consider it a serious flaw.
Sending a packet requires locking the bus and copying the contents of the packet to
the buffer. Receiving packets is based on the ability of the kqueue interface to be
able to deliver notifications to processes when a file changes. The interface driver
gets a kqueue notification, it locks the bus and analyzes the bus header. If there are
new packets for the interface on question, the driver passes them up to the IP layer.
An additional benefit of using a file is that there is always one ringbuffer’s worth of
traffic available in a postmortem situation. The shmif_dumpbus tool (manual page
shmif dumpbus.1 at A–10) can be used to convert a busfile into the pcap format
which the tcpdump tool understands. This conversion allows running a post-mortem
tcpdump on a rump kernel’s network packet trace.