IBM AIX Continuous Availability Features - IBM Redbooks

eases the burden placed on system administrators and IT support staff, but also enables
rapid and precise collection of problem data.
On the software side, serviceability is the ability to diagnose and correct or recover from an
error when it occurs. The most significant serviceability capabilities and enablers in AIX are
referred to as the software service aids. The primary software service aids are error logging,
system dump, and tracing.
With the advent of next generation UNIX servers from IBM, many hardware reliability-,
availability-, and serviceability-related issues such as memory error detection, LPARs,
hardware sensors and so on have been implemented. These features are supported with the
relevant software in AIX. These abilities continue to establish AIX as the best UNIX operating
system.
1.7 First Failure Data Capture
IBM has implemented a server design that builds in thousands of hardware error checker
stations that capture and help to identify error conditions within the server. The IBM System p
p5-595 server, for example, includes almost 80,000 checkers to help capture and identify
error conditions. These are stored in over 29,000 Fault Isolation Register bits. Each of these
checkers is viewed as a “diagnostic probe” into the server, and, when coupled with extensive
diagnostic firmware routines, allows quick and accurate assessment of hardware error
conditions at run-time.
Integrated hardware error detection and fault isolation is a key component of the System p
and System i platform design strategy. It is for this reason that in 1997, IBM introduced First
Failure Data Capture (FFDC) for IBM POWER servers. FFDC plays a critical role in
delivering servers that can self-diagnose and self-heal. The system effectively traps hardware
errors at system run time.
FFDC is a technique which ensures that when a fault is detected in a system (through error
checkers or other types of detection methods), the root cause of the fault will be captured
without the need to recreate the problem or run any sort of extended tracing or diagnostics
program. For the vast majority of faults, an effective FFDC design means that the root cause
can also be detected automatically without servicer intervention. The pertinent error data
related to the fault is captured and saved for further analysis.
In hardware, FFDC data is collected in fault isolation registers based on the first event that
had occurred. FFDC check stations are carefully positioned within the server logic and data
paths to ensure that potential errors can be quickly identified and accurately tracked to an
individual Field Replaceable Unit (FRU).
This proactive diagnostic strategy is a significant improvement over less accurate “reboot and
diagnose” service approaches. Using projections based on IBM internal tracking information,
it is possible to predict that high impact outages would occur two to three times more
frequently without an FFDC capability.
In fact, without some type of pervasive method for problem diagnosis, even simple problems
that occur intermittently can cause serious and prolonged outages. By using this proactive
diagnostic approach, IBM no longer has to rely on an intermittent “reboot and retry” error
detection strategy, but instead knows with some certainty which part is having problems.
This architecture is also the basis for IBM predictive failure analysis, because the Service
Processor can now count and log intermittent component errors and can deallocate or take
other corrective actions when an error threshold is reached.
Chapter 1. Introduction 7