National Energy Research Scientific Computing Center

Innovations 25
Operational Excellence
Thruk: A New Fault-Monitoring Web Interface
Cori Phase I installation in the
new Wang Hall computer room.
Image: Kelly Owen, Lawrence
Berkeley National Laboratory
The move to Wang Hall prompted NERSC’s Operational Technology Group (OTG)—the team that
ensures site reliability of NERSC systems, storage, facility environment and ESnet—to implement
several new fault-monitoring efficiencies. Foremost among them is Thruk, an enhancement to
OTG’s nagios fault-monitoring software that uses the Livestatus API to combine all instances of
systems and storage into a single view.
Implementing Thruk provided OTG with an efficient method for correlating various alerts and
determining if an issue affects systems throughout Wang Hall or just in certain parts of the facility.
The most important advantage of this new tool is that it allows the OTG to monitor multiple aspects
of systems in a single view instead of multiple tabs or multiple url interfaces. This has decreased the
possibility of missing critical alerts. Other advantages include:
• A single location to view alerts to multiple systems
• A configuration interface that is flexible, allowing the OTG to easily add and remove systems
• The ability to send multiple commands at once without waiting for a response
• Multiple filtering schemes