Electricity for Free? The Dirty Underbelly of SCADA and Smart Meters

sorryElectricity for Free?The Dirty Underbelly of SCADAand Smart MetersJonathan Pollet, CISSP, CAP, PCIPJoe Cummins, PCIPSeptember 2010

Table of ContentsIntroduction ............................................................................................................................3 Power Generation, Transmission, and Distribution................................................................4 Power Generation ............................................................................................................4 The Generation Process .................................................................................................................................4 Typical Generation System Diagram ..............................................................................................................5 Power Transmission.........................................................................................................6 The Transmission Process..............................................................................................................................6 Typical Transmission System Diagram...........................................................................................................7 Power Transmission.........................................................................................................8 The Distribution Process .................................................................................................................................8 Typical Distribution System Diagram ..............................................................................................................9 Methodology Used for SCADA Assessments ......................................................................10 Red Tiger Security’s Assessment Approach for SCADA ...............................................10 ISA S99 Model for Security Levels.................................................................................12 SCADA Vulnerability Statistics.............................................................................................13 We Don’t Need No Stinking SCADA 0-Days..................................................................13 Where area of the SCADA System typically has the most Vulnerabilities? ...................14 SCADA, how can I own thee, let me count the ways.....................................................16 AMR and Smart Meter Vulnerabilities..................................................................................18 Summary Remarks ..............................................................................................................19 Appendix A - Test Results with Common PLC used in Power Generation..........................20 PING Testing..................................................................................................................20 Appendix B - Test Results with Typical Power Meter used in AMR systems ......................23 PING test........................................................................................................................23 Appendix C - Some Smart Meter Vendors Send Username/Password in the Clear ...........24 pg [ 2 ]

IntroductionSCADA Systems control the generation, transmission, and distribution of electric power, andSmart Meters are now being installed to measure and report on the usage of power. Whilethese systems have in the past been mostly isolated systems, with little if no connectivity toexternal networks, there are many business and consumer issuing driving both of thesetechnologies to being opened to external networks and the Internet.Over the past 10 years, we have performed over 100 security assessments on SCADA(Supervisory Control and Data Acquisition Systems), EMS (Energy Management Systems),DCS (Distributed Control Systems), AMI (Automated Metering Infrastructure), and Smart Gridsystems. We have compiled very interesting statistics regarding where the vulnerabilities inthese systems are typically found, and how these vulnerabilities can be exploited.The purpose of this paper is not to disclose any specific exploits that will allow you to stealpower from your neighbors, but we can give away enough meat in this paper to exposecommon vulnerabilities at the device, protocol, application, host, and network layers.After performing hundreds of security assessments of systems that play a vital role in theproduction, transmission, and distribution of electric power, we now more than ever aware ofthe vulnerabilities that lie within these systems. The knowledge that we gained from theseassessments out in the operational world has been combined with real incidence responseexperiences and the research that our team has done in this particular field of study.We have seen PLCs (Programmable Logic Controllers), RTUs (Remote Terminal Units), aswell as computing systems running HMI (Human Machine Interface) applications crash and failwhen under high network traffic or malware. We have also participated in first responder’steams to cyber incidents that have caused immediate downtime and system failure for controlsystems that were compromised from worms, viruses, and malware. In several cases it was theEnterprise IT network traffic, heavy IT backup processes, or 3 rd party network interconnectionsthat were the cause of the SCADA incident.The intent of this paper is to first provide a quick definition of how SCADA and Smart Meteringsystems are used in the generation, transmission, and distribution of power. Next, we willprovide a summary of the 6-layer approach that we use to conduct vulnerability assessments ofthese systems. Lastly, we will summarize the vulnerabilities that we have found in thesemission-critical systems to raise awareness to the need for increased hardening and security ofthese systems.pg [ 3 ]

Power Generation, Transmission, and DistributionUnlike other utility commodities like water or natural gas, electricity cannot be stored. Energycan also neither be created nor destroyed, only transformed into different types of energy.Electric Power Systems are typically grouped into three components. Power Generation coversthe process of transforming various types of energy into electricity. Transmissions Systems stepup the electricity to a higher voltage then transport and route the electricity over long distancesfor delivery to local markets. Distribution systems handle the process of stepping downelectricity to proper delivery levels and distributing it to the final consumers of the power. Thediagram below is a good example depicting these three components.* graphic courtesy of NERCPower GenerationThe Generation ProcessPower generation is the process of converting non-electrical energy into electricity, and is thefirst step in the process of delivering power to consumers. Electricity is most typically generatedat a power plant by electromechanical generators, which involve a spinning turbine to convertresident or stored energy into electricity.Although there are many operational differences in the various forms of power generation fromsources such as coal, natural gas, nuclear fission, flowing water, wind, and geothermal, allpower generation operations share similar system components, including how the SCADA andcontrol systems are connected back to Enterprise IT networks, the Internet, and other outsidenetworks. Because of all of the generation systems are typically in one geographical location,and typically have little outside TCP/IP connections, of the three parts to the electric powerindustry, power generation tends to have less vulnerabilities than transmission and distributionsystems. The diagram on the next page depicts a typical Power Generation network diagram,from the Internet down through the various Enterprise IT layers, to the physical equipment thatproduce electricity.pg [ 4 ]

Typical Generation System Diagrampg [ 5 ]

Power TransmissionThe Transmission ProcessElectric power transmission is the bulk transfer of electric power to the consumer. A powertransmission network connects power plants, which are responsible for the generation ofelectricity, to substations. The substations in turn distribute electricity to industrial, commercial,or residential consumers.A power transmission network is also referred to as a “grid”. Multiple redundant power linesbetween points on the network are in place so that the power can be routed from any powerplant to any load center through a variety of transmission routes based on the available pathand cost of power.The systems that reliably monitor and control the transport of electricity from power generationplants to the distribution load centers are called Energy Management Systems (EMS). Thesesystems reside on the industrial control system or SCADA network. The EMS systems aretypically only responsible for control of the power from after it is generated, up until it reachesthe regional substations, where the DMS (Distribution Management Systems) then take over tomanage the distribution and sale of power to individual customers.EMS systems span large geographic areas, often rely on 3 rd party telecommunicationsproviders, and have connections to other business applications for some of the followingfunctions: commodities marketing, weather monitoring, outage management systems (OMS),geographic information systems (GIS), protective relaying systems, and additional engineeringapplications for line efficiency calculations. For these reasons, EMS systems must be closelystudied to insure that all components of the system are properly secured from external andinternal threat vectors.The diagram on the next page depicts a typical Power Transmission network diagram from theInternet, down through the various Enterprise IT layers, to the control room assets that monitorand control the flow of electricity. This network extends from the primary and backup controlcenters out to numerous regional substations that locally step down the voltage for localdistribution.pg [ 6 ]

Typical Transmission System Diagrampg [ 7 ]

Power TransmissionThe Distribution ProcessAs stated before, electric energy cannot be stored, so the amount produced through the electricgeneration process, and transmitted through the EMS (Energy Management Systems), must beimmediately transported to a load where it can be consumed. Distribution Management Systems(DMS) utilize similar SCADA technologies as power generation and transmission systems.Distribution systems have the ability to remotely disconnect or shed large sections of load ifrequired to protect the stability of the grid.Distribution systems also have the requirement to monitor and bill off of the usage of the power.Prior to the movement to digitize power meters, the customer meter was an analog meter thatwas read manually by a meter reader technician. Over the past ten years, many powercompanies have invested in AMR (automatic meter reading) projects that either read the meterby rolling a truck through the neighborhood to locally electronically collect usage data wirelessly,or by remotely reading the meters through a variety of communication methods that includeWiMax, Cellular, Broadband-over-Power, Power Line Carrier, or dual-use circuits shared by theDMS system.At the end of 2009, the Obama Administration released billions of dollars under the federalstimulus act to drive technological innovation in the areas of clean technologies and smart-gridnetworks. The goal was to inject investments into the electric grid to break down the legacybarriers between generation, transmission, and distribution systems to eventually create a“Smart Grid” that can be used by power producers, transmitters, consumers, and researchers tofacilitate a more robust and flexible power grid.The first step in creating a “Smart Grid” involves converting the old analog electric meters intodigital “Smart Meters” that can be polled remotely. This new injection of money into the electricindustry fueled an entire industry of AMR and “Smart Meter” companies that manufacture digitalmeters than utilize a variety of communication methods for remote meter management andmeter reading. These AMR systems are typically interconnected with the DistributionManagement Systems, Enterprise IT systems, and other 3 rd party networks.The network diagram on the next page depicts a typical Power Distribution System, andcommon connection points with Enterprise IT systems, other outside networks, and AutomatedMeter Reading infrastructures.pg [ 8 ]

Typical Distribution System Diagrampg [ 9 ]

Methodology Used for SCADA AssessmentsRed Tiger Security’s Assessment Approach for SCADAOver the past nine years of conducting security assessments of SCADA systems andperforming research in the field of SCADA security, we have found that unless you use adefense-in-depth approach of assessing all components of the system, then small vulnerabilitiescan be used as threat vectors into the system.Since we have been involved in the development and bench marking of several control systemssecurity regulations and standards, we developed our assessment methodology in compliancewith NERC CIP, ISA S99, and NIST 800-53 requirements for SCADA and Process ControlSystems. We start with a review of the physical controls in place that restrict physical access tothe systems used for monitoring and control. Then we use a “Follow-the-Wire” approach toanalyze all of the digital components in the path from outside connections, through the networkinfrastructure, computing systems, applications, protocols, and lastly the field devices.The diagram below shows how our six layer approach maps to the physical control systemsnetwork infrastructure, hardware, applications, protocols and field equipment.pg [ 10 ]

Our assessment approach was designed to ensure that each protective layer helps secure thelayer(s) below it. For example, the physical controls should protect access to the networkinfrastructure, which should restrict access over the network to the DMZ and host systems downin the control room environment. Since the field devices do more than just monitor the system,and are used for control functions, these are the most vulnerable and have the biggest impact tothe reliability and availability of the system. That is why SCADA field devices must have themost protective controls in place.This is the same methodology and approach that we use for assessments of DCS (distributivecontrol systems), EMS (energy management systems), DMS (distribution managementsystems), AMR (automated meter reading) systems, and other mission-critical systems thathave similar 24x7 uptime requirements and system components.pg [ 11 ]

ISA S99 Model for Security LevelsThe International Society of Automation (ISA) has drafted a standard for the security of SCADAand process control systems entitled the ISA S99 standard. This standard outlines specificsecurity levels for each functional area of the system, then advocates the use of zones andconduits to separate, isolate, or provide security controls to challenge connecting from onesecurity zone to another.If we take a power generation system, like the typical one that we diagramed before, and thenput all of the various functional components into the ISA model, we would find the most publicsystems such as Internet or Internet –facing systems at the top of the diagram at level 5. ThenCorporate IT networks, DMZ networks for systems that need access from the SCADA networks,Supervisory HMI systems, Field Devices (controllers), and finally instrumentation and sensorslast at level 0.pg [ 12 ]

SCADA Vulnerability StatisticsHaving a standard model for the various security levels in a SCADA, DCS, EMS, DMS, or AMRsystem is an important first step in classifying SCADA vulnerabilities. Once the model becamepart of the ISA S99 standard, we began to log SCADA vulnerabilities as to where in this modelthey were discovered.After conducting over 100 assessments of SCADA and various types of control systems, ourteam eventually logged over 38,000 security findings and vulnerabilities from theseassessments. In 2009, we were under contract from DHS to comb through these vulnerabilitiesand determine if trends in the data could help expose typical areas of vulnerability withinSCADA systems, and the type of vulnerabilities most commonly found.We Don’t Need No Stinking SCADA 0-DaysWhen we set out to start capturing statistics on our vulnerability assessments, we wanted toknow if these vulnerabilities that we are finding were already out in the public domain, and whenthese vulnerabilities were disclosed. We started entering the vulnerability disclosure date to thedatabase, just to see if any interesting trends would develop. Some of the vulnerabilities that wefound were not previously disclosed, and in some cases, the disclosure date was unknown, sowe threw those records out. In a sample set of over 38,000 vulnerability records, we found thatthe average number of days between when the vulnerability was disclosed publically and whenour team discovered the vulnerability was 331 days.In some of the worst cases, we found vulnerabilities that had been disclosed over 1100 daysbefore we found it, meaning that these mission-critical SCADA systems were vulnerable toa known exploit for over 3 years before we found the problem. It is well known that controlsystems are more difficult to patch than Enterprise IT systems, and the statistics in the dataproves that we still have a patch management issue with critical infrastructure, especially whenthe patch has been available for on average of one year or more, and these systems are stillpopping up with old vulnerabilities. Which raises a serious issue, how many more criticalinfrastructure systems are working today like ticking time bombs with known vulnerabilities andexploits out in the wild that can take them down?pg [ 13 ]

Almost half of the total vulnerabilities were found in the DMZ between the Enterprise IT andSCADA systems. Often we find that SCADA system owners struggle with which group in theircompany has the ownership and responsibility for maintaining the systems in this part of thenetwork.It is usually very clear that the Enterprise IT systems are under the responsibility of the IT orMIS group, and the operational networks at the Supervisory HMI LAN and below are usually theresponsibility of Operations or Engineering groups, but who maintains the systems in themiddle?Typically, the Operational DMZ networks are setup to share data with the Enterprise ITapplications, and then left functioning for years without anyone maintaining them. Many of thesecontain embedded SMTP servers, database servers, web servers, and system components thatare at risk to malware and network attacks.The Operational DMZ network is the first stepping-stone from the Enterprise IT network, and isthe most common threat vector for attacks against SCADA systems. Once access is madethrough the Enterprise IT network, then finding this Operational DMZ is a simple process.Almost all SCADA vendors use a design that places Data Historians, Web Servers, ReportingSystems, and other back-end servers in an area that is both accessible from the SCADAnetworks as well as the Enterprise IT networks.Engineering and management personnel require access to near real-time and trended data fromthe SCADA systems to analyze the performance and efficiency of the SCADA systems.Marketing, Trading, and Business Modeling functions also need constant feeds of data from theSCADA systems to update systems that must interact with government, regulatory, businesspartners, customers, or other third parties.Now we have found the “Perfect Storm” whereby the most connected area of the SCADAsystem also contains the most vulnerabilities, and is often overlooked by systemadministrators.The next part of our research dealt with peeling back the next layer of the onion to classify thesevulnerabilities by the type of exploit that would take advantage of these vulnerabilities.pg [ 15 ]

SCADA, how can I own thee, let me count the waysThe Operational DMZ network is the last line of defense before any traffic hits the SCADA andIndustrial Process Control systems, and in many cases, the servers, workstations, andapplications in this middle area are all authorized and trusted by the SCADA systems. Bydissecting the vulnerabilities in this level of the network, we can determine how thevulnerabilities at this level in the architecture can be exploited.Our technicians and analysts had to first filter the SCADA vulnerability dataset for only thosefound in the Operational DMZ (Security Level 3), and then assigned a classification to eachvulnerability based on the type of exploit that could be used to take advantage of thevulnerability. The classification dataset naturally fell into 16 different categories, and the resultsare disclosed in the table and chart below:Exploit Type % CountArbitrary Code Execution 3.52% 658Arbitrary File Access 3.03% 567Arbitrary File Overwrite 1.23% 230Authentication 1.08% 201Configuration 16.08% 3004Cross Site Scripting 15.04% 2809Denial Of Service 12.11% 2263Disclosure 8.07% 1508File Inclusion 0.99% 185Injection 0.80% 149Input 1.80% 336Overflow 11.45% 2139Privilege Escalation 3.50% 654Remote Code Execution 0.48% 90Remote File Inclusion 10.19% 1903SQL 10.64% 1987100.00% 18683pg [ 16 ]

The underlining systems that control and monitor the generation, transmission, and distributionof electric power are utilizing similar computer networking components and architectures asEnterprise IT networks, yet they do not receive the same level of security maintenance orlifecycle planning.These systems often are at least a year out of patch cycle, typically do not have any loggingenabled, and rarely utilize any monitoring defense techniques like IDS, network, or host eventmonitoring. This does not even touch the topics relating to the security of the field devices,which are also very fragile and can be made to crash with simple PING commands. SeeAppendix A for detailed results from the testing we did with an Ethernet-connected PLC.Now that we have exposed the dirty underbelly of the SCADA and Control Systems that governthe generation, transmission, and distribution of power, it is often a wonder why we haven’t seenmore incidents and security events with these systems. With that in mind, we next turn to thesystems that are responsible for the tracking and billing for the usage of power to answer thequestion: Electricity for Free?pg [ 17 ]

Appendix A - Test Results with Common PLC used in PowerGenerationPING TestingRed Tiger Security conducted a PING test against a well known PLC vendor that uses TCP/IPfor communications with Power Generation Systems. The PING testing was ramped up fromsmaller packet size to larger packet size, until the maximum packet size was reached. Typicallythis test is ran by sending simultaneous PING packets at the following sizes to the target IPaddress:• 60 byte• 600 byte• 6,000 byte• 60,000 byteThe actual commands used to create and send the PING tests are provided in the resultsbelow, as well as the impact the command had on the performance of the PLC.ping -f -s 60 (target IP Address)PING 11.128.66.170 (11.128.66.170): 60 data bytes...................................................................................................................................................................................................................................................................................................................................................................................................................................................................--- 11.128.66.170 ping statistics ---601 packets transmitted, 150 packets received, 75% packet loss(The device lost communications and was unreachable on the network. The device was notreachable by SCADA scans, but came back up in a few seconds after the attack was over.)ping -f -s 600 (target IP Address)PING 11.128.66.170 (11.128.66.170): 600 data bytes.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................--- 11.128.66.170 ping statistics ---497 packets transmitted, 32 packets received, 93% packet loss(The device lost communications and was unreachable on the network. The device was notreachable by SCADA scans, but came back up in a few seconds after the attack was over.)pg [ 20 ]

PING 11.128.66.170 (11.128.66.170): 6000 data bytes....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................--- 11.128.66.170 ping statistics ---518 packets transmitted, 0 packets received, 100% packet loss(The device lost communications and was unreachable on the network. The device was notreachable by SCADA scans, and it went from a RUN to a FAULTED state. All configurationwas lost, and we had to recycle power, then transfer the configuration back to the deviceover a serial connection to restore its operation.)PING 11.128.66.170 (11.128.66.170): 60000 data bytes...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................--- 11.128.66.170 ping statistics ---819 packets transmitted, 0 packets received, 100% packet loss(The device lost communications and was unreachable on the network. The device was notreachable by SCADA scans, and it went from a RUN to a FAULTED state. All configurationwas lost, and we had to recycle power, then transfer the configuration back to the deviceover a serial connection to restore its operation.)pg [ 21 ]

We used the programming software to monitor the health of the PLC while using the PING testscenarios. The screen shot below shows to status of the PLC when it failed under a simplePING test.pg [ 22 ]

Appendix B - Test Results with Typical Power Meter used inAMR systemsPING testRed Tiger Security conducted a similar PING test against a well-known power meter that usesTCP/IP for transmitting meter data. The team also tested the software application that waspoling the meter. The PING testing will be ramped up from smaller packet size to larger packetsize, until the maximum packet size is reached. Typically this test is ran by sending 1000simultaneous PING packets at the following sizes to the target IP address:• 60 byte• 600 byte• 6,000 byte• 60,000 byteThe actual commands used to create and send the PING tests are provided in the table below,as well as the results that the test had on the meter.command type Result test had on Meter Operationping -f 137.20.5.86 -s 60 Ping Flood with 60 bytesize payloadThis test crashed the METER. After the attack wasturned off, it took about 3 minutes for the METERto recover on its own.ping -f 137.20.5.86 -s 600ping -f 137.20.5.86 -s 6000Ping Flood with 600 bytesize payloadPing Flood with 6,000 bytesize payloadThis test crashed the METER. After the attack wasturned off, it took about 3 minutes for the METERto recover on its own.This test crashed the METER. After the attack wasturned off, the METER never recovered on its own.The METER had to be rebooted, and then theconfiguration had to be reloaded into theMETER through a serial cable.ping -f 137.20.5.86 -s 60000Ping Flood with 60,000 bytesize payloadThis test crashed the METER. After the attack wasturned off, the METER never recovered on its own.The METER had to be rebooted, and then theconfiguration had to be reloaded into theMETER through a serial cable.pg [ 23 ]

Appendix C - Some Smart Meter Vendors SendUsername/Password in the ClearIn some of our other testing of metering systems that use Ethernet protocols, we have foundthat many of these protocols send the usernames and passwords to log into the meters in theclear over the network. A simple network sniffing software like Wireshark can pickup thisinformation as shown in the figures below.pg [ 24 ]