The Design and Implementation of the Anykernel and Rump Kernels

86
The fastpath is taken in cases where the same thread schedules the rump kernel
consecutively without any other thread running on the virtual CPU in between.
The fastpath not only applies to the entry point, but also to relinquishing and
rescheduling a CPU during a blocking hypercall. The implementation uses atomic
operations to minimize the need for memory barriers which are required by full
locks.
Next, we offer a verbose explanation of the scheduling algorithm.
1. Use atomic compare-and-swap (CAS) to check if we were the previous thread
to be associated with the CPU. If that is the case, we have locked the CPU
and the scheduling fastpath was successful.
2. The slow path does a full mutex lock to synchronize against another thread
releasing the CPU. In addition to enabling a race-free sleeping wait, using a
lock makes sure the cache of the physical CPU the thread is running on is
up-to-date.
3. Mark the CPU as wanted with an atomic swap. We examine the return value
and if we notice the CPU was no longer busy at that point, try to mark it
busy with atomic CAS. If the CAS succeeds, we have successfully scheduled
the CPU. We proceed to release the lock we took in step 2. If the CAS did
not succeed, check if we want to migrate the lwp to another CPU.
4. In case the target CPU was busy and we did not choose to migrate to another
CPU, wait for the CPU to be released. After we have woken up, loop and
recheck if the CPU is available now. We must do a full check to prevent races
against a third thread which also wanted to use the CPU.