Change of Leadership - Water Environment Association of Texas

More documents

Recommendations

Info

10 | Texas WET May 2009WET Tech Talk Continued from page 8fail-over capabilities were built into the systemdesign. During this phase, the new DRC systemwas also evaluated against other existing andplanned SCADA and communications systems todetermine its appropriate switch over and recoveryprocedures.• l Development/Acquisition Phase. During thedesign phase significant emphasis was given to theredundancy and robustness of the DRC systemarchitecture to optimize reliability, maintainability,and availability during the operation/maintenancephase. By incorporating those factors into theearly stages of the DRC design, implementationcosts were reduced and issues relating to futureplanned system upgrades were dealt with as well.Continuous data replication and mirroring wasplanned to take place to ensure that the DRCwas ready to take over when needed. DRC datacommunication system reliability and availabilitywere one of the major concerns during thedesign phase, as the City wastewater SCADAcommunications components, services and pathshad to be carefully examined. SCADA power supplysystems (regular feed and UPS) had to be reviewedand appropriately sized for load balancing.l Implementation Phase. The City WastewaterSCADA DRC system implementation was plannedto be accomplished in two steps. The initialimplementation phase included the installation ofthe DRC with new and upgraded communicationcomponents but utilized mostly spare CitySCADA equipment currently installed at variouswastewater treatment plants. By doing so, theinitial cost of the DRC was reduced substantiallywhen the target of fully tested and operationalDRC were achieved. The second and final DRCimplementation phase included the replacement ofthe outdated wastewater SCADA system hardwareand software, and the addition of networksecurity devices and software. Test proceduresand forms were developed to ensure that the DRCcontingency plan technical features and recoveryprocedures are fully functional and respond tothe City requirements. Once the DRC system wastested and approved for operation, the developedprocedures were documented and distributed tothe dedicated DRC team.l Operation/Maintenance Phase. Duringthe operational phase, the City DRC team,administrators, and managers are required tomaintain training and awareness of the DRCplan procedures. The SCADA team exercises andperiodically tests the system to ensure that thesystem functions per the DRC procedures. It isalso the DRC team’s responsibility to update theprocedures and DRC documentation to reflectchanges based on hardware or software changesbut also on lessons learned.l Disposal Phase. As the City DRC projectwas carried out in two consecutive steps,considerations were given to the process of retiringthe currently installed computer system and theinstallation of the system replacing it.Conduct Business Impact AnalysisThe BIA objective is to verify the City wastewaterSCADA system components with the critical servicesthat they provide, and based on that information,to determine the impact and consequences of thedisruption of that system in case of a component failure.The results from the BIA analysis were then incorporatedinto the development of the COOP and the DRC designand implementation.The City SCADA system is very complex. The systemmonitors and controls a large number of wastewaterfacilities dispersed over a large geographic area, withnumerous components, interfaces, and processes. Thefirst step taken to evaluate the SCADA system was todetermine the critical functions performed by the systemand to identify the specific system resources required toperform them.The City and Engineer’s DRC team had identifiedand coordinated with the City and AT&T personnel thesystem dependency on various communications links,and external support in case of disruption Continued and the on need page 12to switch over to the DRC. This coordination suppliedthe DRC design team with the needed information tocharacterize the full range of support provided by thesystem, including security, managerial, technical, andoperational requirements.While performing the Business Impact Analysis,the DRC team followed the contingency plan policyrequiring the City wastewater SCADA system to berecovered immediately (within 15 minutes, but notmore than 8 hours in case of a major catastrophe). Bydocumenting and reviewing the recovery strategies, theDRC design team could make well informed, tailoreddecisions regarding contingency resource allocations andexpenditures, saving time, effort, and costs. Based on theBIA, it was defined that:l The DRC system is to have all the functionalityof the existing SCADA system, excluding thebackup redundancy, and to have at the minimum,necessary hardware, software and communicationequipment to access the current WWTPs and fieldsites.l The DRC system is to be designed based on thecurrent SCADA system, including the necessaryhardware, software and configuration for a fullyfunctional and operational system. To save the Citycost, it was decided that several of the currentlyinstalled but not used servers be utilized for theDRC, following a hardware upgrade. Databasetags and graphic screens were to be identical toboth sites, when new LCD wide display screens will
May 2009 Texas WET | 11be installed at the DRC to complete mirroring ofthe SCADA control system.l It is anticipated that during normal operation,database values collected by the system willbe exported to the DRC via the existing Citycommunication system, and will be stored at thelocal DRC servers as well for future use.l In the case of a SCADA system failure, the switchoverto the DRC is to be performed manually byAT&T or by the dedicated City DRC designatedand authorized personnel utilizing the suppliedNetwork Management Console / Workstationwith the appropriate software, for communicationsystem configuration, testing and switch-overcapabilities.l Per the developed BIA it was also defined that theswitchover is not to be considered “hot transfer”however AT&T is guaranteeing that such transferwill take place within 15 minutes but not morethan 1 to 8 hours in a worst case scenario.l The SCADA communication and WAN/LANhardware components were also upgraded perthe BIA, in a manner to make the communicationsystem compatible with the DRC-AT&Tcommunication system.Develop Recovery Strategy and Operation ProceduresRecovery strategies provide the required means torestore the SCADA operations quickly and effectivelyfollowing a service disruption, in the allowable outagetimes identified in the BIA. Several alternatives wereconsidered when developing the City wastewater SCADAstrategy, including cost, allowable outage time, andrequired systems’ security, but initially without theintegration with the larger, City- level contingency plans.The selected DRC strategy addressed the potentialimpacts identified in the BIA and therefore wasintegrated into the system architecture during the designand implementation phases of the system life cycle.The DRC design included a combination of methodsto provide wastewater SCADA monitoring and controlrecovery capabilities over a full spectrum of incidents.One of the major tasks was to select the offsite DRCfacility, where the following criteria were considered:l Geographic Area: Distance from the City SCADAcenter mainly trying to avoid the probability ofthe DRC site being affected by the same disaster(flood, terrorist attack, long term power orcommunication outage).l Accessibility: Length of time necessary to have theDRC operating team access this facility, have thecommunications switched over to DRC, and havethe DRC facility fully operational.l Security: Security capabilities of the designatedDRC facility and employee confidentiality, whichmust meet the data’s sensitivity and securityrequirements.l Environment: Structural and environmentalconditions of the DRC facility (i.e., temperature,humidity, fire prevention, and power managementcontrols).l Cost: Design, construction, and operation andmaintenance costs to have the disaster responseand recovery services.Searching for the appropriate site to support the DRCsystem operations as defined in the plan and followingthe BIA, several site types were studied during the DRCdesign:l Cold Sites typically consist of a facility withadequate space and infrastructure (electricpower, telecommunications connections, andenvironmental controls) to support the SCADAsystem. The space may have raised floors and otherattributes suited for computer operations. This sitedoes not contain SCADA equipment and usuallydoes not contain office automation equipment,such as telephones, facsimile machines, or copiers.Should the cold site alternative be selected, the Cityhas to provide and install the necessary SCADAequipment and telecommunications capabilities.l Warm Sites are partially equipped office spacesthat contain some or all of the system hardware,software, telecommunications, and power sources.The warm site is maintained in an operationalstatus ready to receive the relocated SCADA DRCTexas Ad:Layout 1 12/9/08 1:37 PM Page 1Continued on page 12
Page 1 and 2: Official Publication of the Water E
Page 3 and 4: May 2009 Texas WET | 3Leadership fo
Page 5 and 6: May 2009 Texas WET | 5WET Tech Talk
Page 7 and 8: May 2009 Texas WET | 7storage basin
Page 9: The Key to OptimizingCloth Media Fi
Page 13 and 14: May 2009 Texas WET | 13planned to b
Page 15 and 16: March 2009 Texas WET | 15Student an
Page 18 and 19: 18 | Texas WET March 2009Highlights
Page 20 and 21: 2 0 | Texas WET May 2009Utility Man
Page 22 and 23: 22 | Texas WET May 20092009 Operati
Page 24: 24 | Texas WET May 2009WEF and WEAT
Page 27 and 28: May 2009 Texas WET | 27WEAT Outstan
Page 29: May 2009 Texas WET | 29If you would
Page 32 and 33: 3 2 | Texas WET May 2009Water for P
Page 34 and 35: 34 | Texas WET May 2009Water Enviro
Page 36: ExpandingYour PossibilitiesHydromat

Change of Leadership - Water Environment Association of Texas

Create successful ePaper yourself

Delete template?

Save as template?