Change of Leadership - Water Environment Association of Texas

More documents

Recommendations

Info

12 | Texas WET May 2009WET Tech Talk Continued from page 11system. The site may need to be prepared beforereceiving the system and recovery personnel. Inmany cases, a warm site may serve as a normaloperational facility for another system or function,and in the event of contingency plan activation,the normal activities are displaced temporarily toaccommodate the disrupted system.l Hot Sites are office spaces appropriately sized tosupport the SCADA DRC system requirements andconfigured with the necessary system hardware,supporting infrastructure, and support personnel.Hot sites are typically staffed 24 hours a day, 7days a week. Hot site personnel begin to preparefor the system switchover as soon as they arenotified that the contingency plan has beenactivated.l Mobile Sites are self-contained, transportableshells custom-fitted with specifictelecommunications and SCADA equipmentnecessary to meet the DRC system requirements.Usually the time required to configure themobile site can be extensive, and without priorcoordination, the time to deliver the mobile sitemay exceed the DRC system’s allowable outagetime.l Mirrored Sites are fully redundant facilitieswith full, real-time information mirroring.Mirrored sites are identical to the primary sitein all technical respects. These sites provide thehighest degree of availability because the data isprocessed and stored at the primary and alternatesite simultaneously. These sites typically aredesigned, built, operated, and maintained by theorganization.In analyzing the above options, it became obviousthat the mirrored site was the most expensive choice, butensured a virtual 100 percent availability. Cold sites werethe least expensive to maintain; however, they requiresubstantial time to transport and install the necessaryDRC equipment. Partially equipped sites, such as warmsites, fall in the middle of the spectrum. In many cases,mobile sites may be delivered to the desired locationwithin 24 hours. However, the time necessary forinstallation can increase this response time.The City DRC team selection was for the fixed-sitelocation, taking into consideration that it is operationalwith City employees 24/7 and the time to transport thededicated DRC personnel there is minimal. In addition,the selected fixed site is located in a geographic area thatis unlikely to be negatively affected by the same disasterevent (e.g., weather-related impacts or power gridfailure) as the SCADA center. As sites were evaluated, theCity and the Engineer team reviewed that the system’ssecurity, management, operational, and technicalcontrols were compatible with the required plan andresponded to the BIA.However following the devastating Atlantic stormsKatrina and Rita, it became obvious that only one DRCsite might not respond to the City wastewater SCADAemergency plan policy as traffic became a significantgridlock with many people trying to evacuate, and theDRC managers and operators simply could not commuteto the dedicated DRC site and operate the system.It then became clear that to effectively operate thewastewater SCADA system more sites would be needed.The Business Impact Analysis was reviewed andrevised to include additional EOC sites geographicallydispersed in various areas, accessible to the DRC team.To keep the cost low for those EOC sites, it was decidedto have several City services commonly share the DRCequipment and operations.DRC Testing, Personnel Training and Plan ExerciseThe DRC intensive testing plan, which was a criticalelement to ensure that the system is ready to operateper the design and the policy set forth by the City, wascarried out initially by the contractor installing the DRCsystem and periodically by the DRC team. The thoroughtesting enabled the DRC technical and operationaldeficiencies to be identified, addressed and corrected.The performed tests also assisted in evaluating the abilityof the recovery staff to implement the plan quickly andeffectively. The following areas were addressed during thetests:l System switchover to the DRC with alternatemethods (by the DRC team and AT&T)l Coordination among DRC team membersl SCADA system performance following the DRCswitchoverl Notification proceduresl Restoration of normal operations after testingPrior to DRC system delivery to the City, the contractorwas required to perform the following tests:l Failure mode and backup procedures includingpower failure, AUTO restart, and disk backup andreload.l Dual Computer Operation: Processor transfermodes, peripheral switching, and communicationsswitching.l Message logging and alarm handling.l Communication with field interface units.l Data acquisition.l Human-Machine Interface: Database and displayconfiguration and use of all types of displays.l Data collection and data retrieval.l Report Generation: Creation of a typical reportand production of specified reports.l Operational Readiness Test.l Performance Acceptance Test.l Reliability Acceptance Test.Training for the City DRC team with the contingencyplan responsibilities came to complement testing.Training took place during system construction and is
May 2009 Texas WET | 13planned to be provided at least annually to ensure thatthe DRC operations are able to execute their respectiveDRC procedures without the aid of actual documentsor the assistance of the DRC management team. This isan important goal to achieve to ensure that the team isready to operate the DRC even if documentation is notavailable due to the extent of the disaster. DRC personnelare to be trained as follows:l Classroom Exercises. Walk through the procedureswithout actual DRC switch over and operationsoccurring. Classroom exercises are the most basicand least costly of the two types of exercisesand should be conducted before performing thefunctional exercises.l Functional Exercises. Functional exercises requirethe event to be simulated and the DRC switchoverand operation to take place. The functionalexercise is to be coordinated with the City EOCmanagement, with the AT&T team and thewastewater SCADA operations. This exerciseincludes the actual switchover to the DRC site,thorough communication testing and SCADAsystem recovery following the successful testing.DRC Plan MaintenanceTo keep the DRC fully functional and to maintain itsreadiness, the plan procedures and policies must be kept.However, as the City SCADA systems undergo frequentchanges because of technology upgrades, or new internalor external policies, the DRC operational plan is beingreviewed and updated periodically as part of the Citychange management process. Certain elements arerequired to be taken into consideration:l Operational requirementsl Security requirementsl Technical proceduresl Hardware, software, and other equipment (types,specifications, and amount)l Names and contact information of DRC teammembersl Names and contact information of AT&Tl Vital records (electronic and hardcopy)A copy of the DRC procedures is kept on locationhowever additional copies are stored at the DRC teamsites and with the backup media. Storing a copy of theplan at the alternate site ensures its availability andgood condition in the event local plan copies cannot beaccessed because of the disaster.Changes made to the DRC plan, strategies, andpolicies are coordinated through the City DRC planningcoordinator, who then communicates the changes to theDRC team members as necessary.The DRC coordinator also evaluates the supportinginformation to ensure that the information is current andcontinues to meet system requirements adequately. Thisinformation includes the following:l DRC team contactsl Hardware and software requirements and licensesl System network communicationsl Security requirementsl Recovery strategyl Contingency policiesl Training and awareness materialsl Testing scope, and required testing scheduleConclusionIn this age of highly computerized control andmonitoring systems, much of the information received isavailable only via computer and digital network systems.Regularly backing up the information stored on theSCADA system computers is a very important step thatmight protect the loss of information due to computerfailure. However the SCADA system itself could bedestroyed or damaged due to flood, earthquake, terrorattack, or other natural disaster or man-made problem.Furthermore those actions might prevent access to theSCADA monitoring and control center and will preventthem from supplying the required wastewater services toCity residents.The City wastewater SCADA DRC systemimplementation project, which added a fully functionalbackup system to the wastewater SCADA monitoringand control system have supplied the city with theContinued on page 21
Page 1 and 2: Official Publication of the Water E
Page 3 and 4: May 2009 Texas WET | 3Leadership fo
Page 5 and 6: May 2009 Texas WET | 5WET Tech Talk
Page 7 and 8: May 2009 Texas WET | 7storage basin
Page 9 and 10: The Key to OptimizingCloth Media Fi
Page 11: May 2009 Texas WET | 11be installed
Page 15 and 16: March 2009 Texas WET | 15Student an
Page 18 and 19: 18 | Texas WET March 2009Highlights
Page 20 and 21: 2 0 | Texas WET May 2009Utility Man
Page 22 and 23: 22 | Texas WET May 20092009 Operati
Page 24: 24 | Texas WET May 2009WEF and WEAT
Page 27 and 28: May 2009 Texas WET | 27WEAT Outstan
Page 29: May 2009 Texas WET | 29If you would
Page 32 and 33: 3 2 | Texas WET May 2009Water for P
Page 34 and 35: 34 | Texas WET May 2009Water Enviro
Page 36: ExpandingYour PossibilitiesHydromat

Change of Leadership - Water Environment Association of Texas

Create successful ePaper yourself

Delete template?

Save as template?