8 | <strong>Texas</strong> WET May 2009WET Tech TalkWastewater Treatment SystemsSCADA Disaster Recovery Center(DRC)By Yehuda Morag, Principal Technologist,CH2M HILL and Hoss Forouzan,Wastewater Operations, City <strong>of</strong> HoustonIntroductionSCADA systems monitoring and controlling the City<strong>Water</strong> and Wastewater systems are expected to functionwithout any disruptions, however several planned(preventive maintenance) or unplanned (short termpower outage, equipment failure) system shutdownsmight occur periodically, and in most cases thosesystem’s operations are restored in a short time.While many SCADA system vulnerabilities can bereduced or eliminated using the proper engineeringdesign, technical, and operational procedures, somesevere disruptions caused by natural disasters, terroristattack, or large accident (fire, toxic gas evaporation,train derailment, etc) require the City to prepare aneffective contingency plan. The execution <strong>of</strong> this planincludes SCADA Disaster Recovery Centers (DRC)/ Emergency Operation Centers (EOC), testing andexercising, training, and auditing and updating thecontingency plan and policy.This paper will discuss the steps taken to design,construct, and maintain the Wastewater SCADADisaster Recovery Center to respond to future severesystem disruptions caused by natural disasters, terroristattack, or large accidents, and the lessons learnedfollowing the past years Atlantic hurricane season.Several steps were taken to accomplish this task:Develop SCADA Emergency Plan PolicyWastewater SCADA contingency planning includedthe tasks to identify threats and vulnerabilities thatmight shut down the SCADA system and prevent theCity from operating the wastewater system securely andeffectively. Those threats were classified as:l Natural – Hurricane, tornado, flood, and fire.l Human – Operator error, implant <strong>of</strong> maliciouscode (computer virus), sabotage, and terroristattacks.l <strong>Environment</strong>al – Power failure, SCADAsystem failure (hardware or s<strong>of</strong>tware), andtelecommunication network failure.The City felt that the current SCADA system designand the procedures in place can respond to theenvironmental listed threats and most <strong>of</strong> the humanerrors; however the City was very concerned with thenatural disasters affects and the ramification <strong>of</strong> anyterrorist attack on the central SCADA system.Various types <strong>of</strong> contingency plans were studied;however the COOP (Continuity <strong>of</strong> Operation Plan)was adapted, customized for the City’s wastewateroperations, and developed to match the currentlyoperating SCADA system.The COOP was selected, as it focuses on restoringthe wastewater SCADA essential functions at alternatesites and performs the necessary functions for a certainperiod <strong>of</strong> time before returning to normal operations.The COOP was also chosen because it addresses the CityCenter <strong>of</strong> Operations model requirements on one hand,and is developed and executed independently from theBCP (Business Continuity Plan) that applies to generalCity services and the IT system on the other. Becausethe COOP emphasizes the recovery <strong>of</strong> the SCADAoperational capability at an alternate site, the planintentionally does not include the City’s IT operations.In addition, minor disruptions such as short term powerfailure or local communication failure that do notrequire relocation to an alternate site were taken intoconsideration, and therefore were not addressed.In accordance with PDD-63, Critical InfrastructureProtection, COOP plans for systems that are critical tosupporting the nation’s infrastructure are to be in placeby May 2003.Once it was established that the DRC design wouldfollow the COOP standards, the System DevelopmentLife Cycle (SDLC) was examined to reduce the overallcontingency planning costs, to enhance contingencycapabilities, and to reduce the impacts to systemoperations when the contingency plan is implemented.l Initiation Phase. During the initiation phase,DRC system requirements were identifiedand matched to the City wastewater SCADAoperational processes. The DRC system requiresvery high availability therefore redundant,real-time mirroring at an alternate site andContinued on page 10
The Key to OptimizingCloth Media FiltrationTechnology ::The legacy <strong>of</strong> Aqua-Aerobic ® cloth media began with the introduction <strong>of</strong> thefirst cloth media filter system to the wastewater treatment industry in 1991.Our extensive cloth research and development efforts over the past 16 yearshave resulted in the OptiFiber ® family <strong>of</strong> cloth media.Unequaled knowledge <strong>of</strong> cloth construction and expertise in its properapplication are the reasons why Aqua-Aerobic ® is chosen more <strong>of</strong>ten overalternative filtration technologies. Aqua-Aerobic ® is dedicated to leading theindustry in developing advanced cloth media solutions to meet the waterand wastewater treatment challenges <strong>of</strong> today and tomorrow.OptiFiber ® Cloth Media is AvailableExclusively with Aqua-Aerobic ® Cloth Media Filters :::: Advantages• Designed for specific applications• Tested and proven for optimumprocess performance• Engineered for quality and durability• Less operator attention andmaintenance required• Meets virtually any tertiaryfiltration requirement:: Applications• Municipal and Industrial- Tertiary filtration- Reuse / Recycle- Phosphorus removal- New plants and retr<strong>of</strong>itsFor More Information Contact:AquaDisk ®Aqua MiniDisk ®AquaDiamond ®AquaDrum ®The original diskfilter with over 600units installed.All the features <strong>of</strong>the AquaDisk ® , butfor lower flowapplications.Ideal for travelingbridge filterretr<strong>of</strong>its.Treats lower flowswith a lowerhydraulic pr<strong>of</strong>ile.Houston: 281.351.8501Austin: 512.347.7676Dallas: 817.446.9500Tulsa: 918.299.8555www.hartwellenvironmental.comAQUA-AEROBIC SYSTEMS, INC. • 6306 N. Alpine Rd. • Rockford, IL 61111 • PH 815.654.2501 • FX 815.654.2508 • www.aqua-aerobic.com/OptiFiber.asp