Performance Analysis and Optimization of the Hurricane File System ...

CHAPTER 5. MICROBENCHMARKS – READ OPERATION 44
5.5.1 Unoptimized HFS
This experiment was based on the configuration of Section 5.2, except that regular 4 MB files were used
rather than extent-based 12 MB files. There were two reasons for this selection. First, the file size was
chosen so the block index occupied exactly 1 meta-data cache entry, making it simple to trace cache entries.
Second, initial trials using 12 MB regular files were very time consuming. Using 4 MB files was adequate to
show the performance characteristics.
The results, shown in Figure 5.20, indicate extremely poor performance for more than 2 processors,
especially when compared with results from the extent-based file experiments in Section 5.2, despite reading
only 1/3 as much data (4 MB vs 12 MB). The obvious bottlenecks identified by code inspection were (1) the
hash function of the block cache system, (2) the limited number of hash lists (only 4) in the block cache
system, and (3) the global lock used in the block cache system. The original hash function resulted in all
processors sharing a single hash list. The reason for having only a limited number of hash lists in the original
HFS design has not been determined. The global lock is used to protect a variety of data structures from
conflicting concurrent accesses.
Outline of Optimizations
We subsequently applied a number of optimizations. These include (1) padded hash list headers, cache
entries, and other critical data structures, (2) modified hash functions, (3) larger hash tables, (4) modified
free list, and (5) use of fine grain locks. The fully optimized results are shown in Figure 5.21. With the
optimizations, good scalability was achieved. Sections 5.5.2 to 5.5.9 describe the investigation process that
led to these results. A modified block cache hash function or an increased number of block cache hash lists
were ineffective, as shown in Sections 5.5.2 and 5.5.3, respectively. Surprisingly, implementing fine grain
locks in Section 5.5.5 along with the padding of critical data structures in Section 5.5.6 did not result in
scalability either. As demonstrated in Section 5.5.7, the first critical bottleneck was the block cache free list,
followed by the global block cache lock, as shown in Section 5.5.8. Finally, Section 5.5.9 describes the fully
optimized configuration in detail.
Many of the experiments are described in part to demonstrate how non-trivial it is to find bottlenecks in
large, complex, parallel system software.
5.5.2 Improved Hash Function
This experiment examined the impact of modifying the hash function of the block cache system. The
configuration is similar to Section 5.5.1 except that the hash function was modified to better distribute the
use of hash lists, as shown in Table 5.4. The hash table size used by the block cache remained unmodified at