National Energy Research Scientific Computing Center

26 NERSC ANNUAL REPORT 2015
Screen shot of the new
Thruk interface.
Thruk has also been instrumental in supporting the OTG’s efforts to improve workflow automation.
In 2015 OTG staff implemented an automated workflow for timely reporting of user issues that uses
program event handlers to manage multiple instances of issues and multiple action items. For
example, alerts that called for a staff person to open a new ticket and assign it to a group can now
be automated based on a filtering mechanism. The new interface saves the group’s workflow
approximately 17 hours per week, in addition to tickets now being opened accurately on a more
consistent basis. In the past, the language and the information included on the ticket depended on
which staff member opened the ticket.
Another advantage of the automation process involves opening tickets with vendors as a result of an
alert. In the past, a staff person would either e-mail or call the vendor call center to report a problem.
The phone call could take anywhere from 15 minutes to 45 minutes, depending on the vendor, and
the time it took for the vendor tech to respond could be anywhere from one to four hours. In
contrast, the automated process sends an e-mail to the vendor call center, which usually puts a
priority on clearing out their inbox of requests. Thus a ticket sent to the vendor can be opened within
15 minutes, providing detailed information such as diagnostic logs as an attachment to the e-mail.
This has resulted in improving vendor tech response times to under one hour. In addition, with the
appropriate information provided with each alert, vendors spend less time on diagnosis and more
time on problem resolution.
The event handlers can also create and assign work tickets to staff. For example, an alert that says a
drive has failed and needs replacement can activate an event handler that opens a ticket and assigns it
to a staff person who can replace that drive, work with vendors for parts under warranty and allow the
staff to bring the node back into production sooner. One of the areas that allowed this process to be
automated involves centralizing the location of the warrantied parts with their purchase date, serial
number, vendor information and warranty expiration date. With the verification process handled
automatically, it now takes only seconds to decide whether to exchange or acquire a part. This has
resulted in compute nodes coming back to production in a shorter amount of time—in some instances
as little as four to eight hours, where previously it could sometimes take twice that long.