National Energy Research Scientific Computing Center

Innovations 27
Environmental Data Collection Projects
With the move to Wang Hall, NERSC is also taking the opportunity to collect environmental data
from areas of the machine room where we had not previously done so at the Oakland Scientific Facility.
We’ve installed sensors in different areas of the computational floor such as underneath the racks and
even on the ceiling. We felt that getting a wide range of data on the environment will assist us in
correlating between events that could affect the center and jobs running on the computing resources.
For example, we are collecting data from sensors at the substation, through the different levels of
PDUs available in the building down to the node level if possible, including the UPS/generator setup.
The data is collected from the subpanels and passed to the central data collection system via a Power
Over Ethernet (PoE) network setup, since many of these switches and panels are located far away from
the traditional network infrastructure. In one case, the substation might not include normal 110 volt
power for the sensors and hardware, which is why PoE is an easy way to get both networking and
power to these remote locations. We plan to eventually install more than 3,000 sensors, with each
sensor collecting 10 data points every second.
Because Wang Hall is so dependent on the environment, we can leverage all the environmental data
to determine optimal times to run computation-intensive jobs or small jobs or potentially predict how
the environment affects computation on a hot summer day.
Centralizing Sensor Data Collection
In addition to environmental sensors, NERSC's move to Wang Hall also prompted opportunities for
the OTG to install other types of sensors in areas where this had not been done previously. These
include temperature sensors from all parts of the computer floor and from all levels (floor to ceiling),
power sensors from the substations to the PDUs in the rack, water flow and temperature sensors for
the building and large systems and host and application measurement from all running systems. The
data collected from these sensors gives us more ways to correlate causes to issues than was previously
possible. We believe that this data should be able to answer questions such as power consumption of a
job, power efficiency of systems for processing jobs or how we can leverage a schedule based on size of
job to time of day or even the current weather.
However, the additional sensors generate a large amount of data that needs to be stored, queried,
analyzed and displayed, and our existing infrastructure needed to be updated to meet this challenge.
Thus, during 2015 the OTG created a novel approach to collecting system and environmental data
and implemented a centralized infrastructure based on the open source ELK (Elasticsearch, Logstash
and Kibana) stack. Elasticsearch is a scalable framework that allows for fast search queries of data;
Logstash is a tool for managing events and parsing, storing and collecting logs; and Kibana is a data
visualization plugin that we use on top of the indexed content collected by Elasticsearch.
Using the ELK stack allows us to store, analyze and query the sensor data, going beyond system
administration and adding the ability to extend the analysis to improve the operational efficiency of an