Red Hat Enterprise Linux 5 Administration Unleashed

446CHAPTER 21Monitoring and Tuning the KernelSetting nmi_watchdog to 1 enables I/O APIC. To enable local APIC instead, set its value to2. To verify that NMI watchdog and profiling are enabled, make sure the /proc/profilefile exists.How does NMI watchdog work? A lockup is defined as the processor not executing thelocal timer interrupt more than 5 seconds from the last timer interrupt. If a lockupoccurs, the NMI handler generates a kernel oops, writes debug messages, and kills theprocess causing the lockup. If the lockup is so bad an NMI interrupt can’t be issued or thekernel can’t write debug messages, NMI watchdog cannot work.Local APIC works with the cycles unhalted processor event, meaning that it can only betriggered to write debug messages if the processor is not idle. If a system lockup occurswhile the processor is idle, watchdog will not be triggered. On the other hand, the I/OAPIC works with events outside the processor, but its frequency is higher, which cancause more impact on the overall performance of the system.The /proc/profile file is not written in a human-readable format. The readprofileutility must be used to read the data. The System.map file for the currently running kernelmust be specified with the -m option such as the following:readprofile -m /boot/System-map-`uname -r`TIPThe output can be quite lengthy. To redirect the output to a file, append > filename.txtto the end of the command as follows:readprofile -m /boot/System-map-`uname -r` > filename.txtIf no other options are passed to the command, the output is in three columns: thenumber of clock ticks, the name of C function in the kernel where those click ticksoccurred, and the normalized load of the procedure as a ratio of the number of ticks tothe procedure length. Table 21.5 describes the available command-line options.TABLE 21.5 Command-Line Options for readprofileCommand-Line Option Description-m Provide the location of the System.map file for the running kernel.-i Output the profiling step used by the kernel. Use with -t to onlyprint the number.-a List symbols in map file.-b List the individual histogram-bin counts.-r Reset the profiling buffer /proc/profile. Because only root canwrite to the file, this option can only be executed as root.-M Set the frequency at which the kernel sends profiling interrupts tothe processor. Frequency should be set as a multiplier of thesystem clock frequency, which is in Hertz. Resets the buffer fileas well. Only executable by root. Not available on all processors.