PBS Business Continuity & Disaster Recovery

PBS Business Continuity 

& 

Disaster Recovery 

April 2011

PBS BCP & Disaster Recovery 

• PBS Distribution Services & Workflow 

• BCP / DR Initiatives 

• Recovery Solutions 

• Satellite / Transponder Recovery 

• PBS DRS Facility 

• “Do-it-yourself” DR plan template 

• Q&A 

2


Stefan Petrat 

VP, Distribution Operations & Engineering 

Thomas Crowe 

Sr. Director, Technology 

James Cutright 

Sr. Project Manager Interconnection 

Cindy Jackson 

Director, Operations Strategic Planning & BCP 

3

PBS Distribution Services 

• PBS has been providing signals to the member stations from the PBS 

Technology Center in Springfield Virginia since 1978. 

• The PBS Technology Center is comprised of Network Operations (NOC) 

and Technical Maintenance (TMC) services. 

• PBS’s Technology Center currently originates and uplinks 11 digital video 

services, including two 24-hour a day HD feeds and two HD “soft” feeds. 

These digital services are transmitted to three different satellites over both 

Ku and C-Band satellite transponders. 

• The PBS Tech Center is located 12 miles southwest of downtown 

Washington DC near Springfield, Virginia on five acres of land owned by 

PBS. Two buildings on campus; 

• SOC-1 used for NOC & SOC operations 

• SOC-2 used for TMC, Lab and administration. 

4

PBS 

Technology Center 

SOC-2 

SOC-1 Building 

SOC-1 

SOC-2 Building 

5


Distribution Services & Workflow 

6


• Real-time Services 

• NPS services: HD01, HD02 

• Soft services: HD03, HD04, SD01 

• VMe, Create, World: SD02, SD03, SD04 

• Tape feeds, Ingest & Live Turn-around 

• Content QC 

• Satellite Access Coordination 

• HD04, SD05, SD06, SD07 

• Disaster Recovery / Business Continuity 

• Engineering & Lab 

• Non Real-time 

• Supporting development & testing 

• Facilities support 

7


PBS Workflow - Overview 

Telstar 5 

Transponder 25V (KU) 

8

PBS Distribution – Ku Band 

AMC-21 Tx22 

Contract End: 09/2016 

Bandwidth: 36Mhz 

Modulation: DVB-S2 8PSK 

MCPC – MPEG2 

Fully Protected 

Services 

1. HD01 

2. HD02 

AMC-21 Tx23 


Bandwidth: various 

Modulation: DVB-S QPSK 

SCPC – MPEG2 

Transponder Protected 

Services 

1. HD04 (18Mhz) 

2. SD05 (6Mhz) 

3. SD06 (6Mhz) 

4. SD07 (6Mhz) 

AMC-21 Tx24 






Services 

1. HD03 

2. SD01 

3. SD02 (Create) 

4. SD03 (VMe) 

5. SD04 (World) 

AMC-21 

125W Ku Band 

Vendor: SES World Skies 

Satellite EOL: 10/2022 

AMC-21 Tx20 

Contract End: 09/2013 & 03/2014 


Modulation: various 

SCPC 


Services 

1. Montana PBS (6Mhz) 

2. Oklahoma PBS (10.75Mhz) 

AMC-21 Tx21 




SCPC 


Services 

1. NRT 

9


Business Continuity 

& 

Disaster Recovery 

Initiatives 

10


• “Evergreen” emergency playback unit installed at Nebraska 

• 48 hours of content available on a video server for instant access & availability 

• Replaced older DVD system and updated program content 

• Status: Completed 

• Completion date: June 2010 

• Initial Response Team (IRT) Drill 

• Scenario based “table top” exercise with PBS senior executives 


• Completion date: August 2010, another drill targeted for mid 2011 

• Satellite Phone Test 

• Annual verification of backup communication system 


• Completion date: August 2010 

11


• NOC / SOC Evacuation Drills 

• Level 1 … “Remote control” of Tech Center from Crystal City via MOC 

• Status: Completed & Ongoing 

• Completion date: June 2010 & September 2010 

• Level 2 … Program origination and satellite distribution from Crystal City via 

MOC 

• Status: Completed & Ongoing 

• Completion date: November 2010 & February 2011 

• Satellite Disaster Recovery Plan update 

• Update, confirm and initiate recovery solution for satellite and/or transponder 

failures 


• Completion date: December 2010 

12


• BCP Software / Crisis Communication update 

• “Recovery Planner” will text or email alert staff of issues and desired actions 

• Company-wide testing has taken place to confirm the system functions as expected 

• “Canned” messages have been loaded into the application 

• Cindy worked with Anne & Jan in Communications to approve messages for various emergencies 

• Personnel information update (e.g. mobile phone, home phone, contact details 

etc.) is in process on a department by department basis 

• T&O and Interactive were done first since they represent a third of the company. Remainder of company completed; All 

Staff email from COO with instructions and to reflect importance of task. 


• Completion date: February 2011 

• IT Recovery Solution Project 

• Review critical business systems & department priorities and needs 

• Determine next steps and cost for multi-year capx project starting in FY2012 

• Status: In progress 

• Completion date: May 2011 

13


Recovery Scenarios 

14


“Normal operations” 

Telstar 5 


Member 

Station 

15


“Evacuation of Tech Center” 

Telstar 5 


Member 

Station 

16


“Loss of Tech Center” 

Telstar 5 


Member 

Station 

17


“Loss of Satellite or Transponder” 

Telstar 5 Telstar 5 

Transponder 25V Transponder (KU) 25V (KU) 

18


“Loss of PBS HQ – Crystal City” 

Telstar 5 


Member 

Station 

19


“Overview of Future Operations” 

Telstar 5 


Member 

Station 

Operational 

est. CY2012 / 13 

(capx FY2012 / 13) 

Operational 

Q3/Q4-CY2011 

(capx FY2011) 

20


Satellite / Transponder 

Recovery 

21


Transponder failure 

“ … loss of a single or multiple transponders on a satellite …” 

Satellite failure 

“ … loss of at least 13 or more transponders on a satellite…” 

22


AMC-21 Tx22 



Modulation: DVB-S2 8PSK 



Services 

1. HD01 

2. HD02 

AMC-21 Tx23 




SCPC – MPEG2 


Services 

1. HD04 (18Mhz) 

2. SD05 (6Mhz) 

3. SD06 (6Mhz) 

4. SD07 (6Mhz) 

AMC-21 Tx24 






Services 

1. HD03 

2. SD01 

3. SD02 (Create) 

4. SD03 (VMe) 

5. SD04 (World) 

AMC-21 

125W Ku Band 



AMC-21 Tx20 

Contract End: 09/2013 & 03/2014 


Modulation: various 

SCPC 


Services 

1. Montana PBS (6Mhz) 

2. Oklahoma PBS (10.75Mhz) 

AMC-21 Tx21 




SCPC 


Services 

1. NRT 

23

AMC-21 - 125 o W 

Transponder Failure Recovery 

Rank 

4 

AMC-21 

(Tx20 V) 

Montana / Oklahoma 

AMC-21 

Spare Transponder 

(same polarity) 

5 

(Tx21 H) 

NRT 



1 

(Tx22 V) 

HD01 & HD02 



3 

(Tx23 H) 

HD04, SD05, SD06, SD07 



2 

(Tx24 V) 

HD03, SD01, SD02, SD03, SD04 



For one or more transponder failures, spare components on the space craft are first used. 

If spares are not available, PBS has the right to pre-empt another service from their transponder. 

Currently there are 8 spare transponders on AMC-21 

24

AMC-21 - 125 o W 

Satellite Failure Recovery 

Rank 

1 

2 

AMC-21 

(Tx22 V) 

HD01 & HD02 

(Tx24 V) 

HD03, SD01, SD02, SD03, SD04 

AMC-1 

(Tx18V) 

HD01 (HD Packaged) 

SD01, SD02, SD03, SD04 

3 

(Tx23 H) 

HD04, SD05, SD06, SD07 

(Tx16V) 

HD03 (HD Soft) 

SD05, SD06, SD07, 

HD04 

Full details of this recovery scenario are available on PBS Connect 

Connect > Engineer Compute > Distribution Operations Engineering > Disaster Recover Plan 

25


AMC-1 Tx20 

103W C Band 








Services 

1. PBS C-Band HD & SD 

26

AMC-1 - 103 o W 

Transponder Failure Recovery 

Rank 

6 

AMC-1 

(Tx20 H) 

PBS HD& SD Digital Feed 

AMC-1 



27

AMC-1 - 103 o W 

Satellite Failure Recovery 

Rank 

AMC-1 

SES-1 

6 (Tx20 H) 

(Tx16 H) 



28


Uplink 

Freq 

Downlink 

Freq UL Mod Sym Rate 

Satellite Band Xpder Arc C/Ku (Mhz) L Band (Mhz) BW (Mhz) Pol Type FEC MS/s Services 

AMC-21 Ku 20 125 W 14404.00 1354.00 6 V 

DVB-S 

QPSK 

AMC-21 Ku 20 125 W 14412.50 1362.50 10.75 V DVB-S2 

8PSK 

AMC-21 Ku 21 125 W 14420.00 1370.00 36 H DVB-S2 

8PSK 

DVB-S 

AMC-21 Ku 22 125 W 14440.00 1390.00 36 V 

QPSK 

18 

AMC-21 Ku 23 125 W 14450.00 1400.00 18 H DVB-S 

QPSK 

AMC-21 Ku 23 125 W 14463.00 1413.00 6 H DVB-S 

QPSK 

AMC-21 Ku 23 125 W 14469.00 1419.00 6 H DVB-S 

QPSK 

AMC-21 Ku 23 125 W 14475.00 1425.00 6 H DVB-S 

QPSK 

AMC-21 Ku 24 125 W 14480.00 1430.00 36 V 

36 

DVB-S 

QPSK 

3/4 4.4444 Montana PBS 

3/4 8.916 Oklahoma (OETA) U/L 

UNK UNK NRT Test 

3/4 30.000 MCPC - HD01 & HD02 

3/4 14.028732 HD04 

3/4 4.444 SD05 SCPC 

3/4 4.444 SD06 SCPC 

3/4 4.444 SD07 SCPC 

SES-1 C 16 101 W 6245.00 1130.00 36.00 V Analog N/A N/A Not used 

3/4 30.000 MCPC - HD03, SD01, SD02, SD03 & SD04 

AMC-1 C 20 103 W 6316.00 1059.00 18 H DVB-S 

QPSK 

3/4 14.028732 PBS C-Band HD & SD (Consumer use) 

29


Network Operations Center 

Disaster Recovery Site (DRS) 

30



Purpose: 

… ‘The purpose of disaster recovery/business resumption project is to assure 

continuity of Network and Non-Realtime distribution operations needed to 

support critical system functions. The business resumption plan should aim at 

achieving a systematic and orderly resumption of all operational services. The 

plan should provide for a hot standby functionality between sites as well as 

restoring full service due to catastrophic failure as soon as possible. …’ 

31



Business Decision: 

… ‘Given the “PBS Satellite & Broadcast Operations Disaster Recovery 

Planning Guide”, and the “Integrated Public Alert and Warning System 

(IPAWS)” that mandates a geographically separated facility, the decision to 

build and operate a DRS site seems to have been embraced. Also, as can be 

gleaned from the two papers, PBS has a business responsibility to its member 

stations and also the Federal Government in its IPAWS commitment.’ … 

32

Project Scope: 



… ‘One of the major challenges facing the Public Broadcasting Service (PBS) 

today is preparing for the continuity of the system business following a possible 

catastrophic event that disables the operation of the Network Operations 

Center (NOC) facility. 

Whether referred to as “disaster recovery” or “business continuity”, such a 

contingency plan must be flexible enough to accommodate both short and longterm 

outages of the primary transmission site. At the same time, plans to 

survive an unlikely catastrophic event must also make sense within the 

economics of an ongoing business.’ …. 

33


Project Scope: 

• Business considerations 


- What Do We Stand to Lose 

- How closely must the backup site replicate our primary site 

- What types of recovery scenarios are to be covered 

• Logistical 

- Staffing (UNL-TV, PBS) 

- Remote Operation (MOC, NOC) 

• Operational requirements 

- File Content Delivery & Sizing of Content Requirements 

- Control Data & Transfer Solution Component 

• Technologies 

34


Technology Solutions: 

1. Conventional apparatus 

2. IT/Software solution 


Objectives for the DRS focused on: 

• Lower capital cost as compared to NOC 

• Simplicity of operation, including system compatibility of content files with NRT 

• Lower operational cost 

• Lower annual support costs 

Side benefit: 

• Confirmation of new technology for future upgrade of NOC site 

35


36



DRS Equipment Center (DEC) … 

is a full function technical equipment center 

contained within the UNL-TV Nebraska facility. All 

equipment needed to support the DRS operation is 

contained within the DEC. The DEC operates on its 

own power (commercial & emergency generator) 

with full Uninterruptable Power Supply (UPS). 

Redundant HVAC units provide the necessary 

cooling and humidity control for the equipment and 

systems housed within the DEC. The equipment 

center is protected by a staged fire alarm system 

insuring the safety of systems and personnel. 

Within the DEC are equipment racks, work stations, 

tape storage shelves, cable trays, electrical 

distribution, grounding connections and access to 

UNL-TV’s cable trays which connect to their 

technical equipment center and satellite antenna 

farm. 

37



PBS DRS Remote Operations position 1 & 2 

(PDRO1 & PDRO2) … 

Is a pair of remote operations positions located at 

the PBS NOC (PDRO1) in Alexandria, Va. and the 

PBS MOC (PDRO2) in Arlington, Va. Both 

positions will have full remote control of the DRS 

facility. PDRO1 will operate the DRS facility on a 

daily basis with PDRO2 being available in the event 

the NOC is disabled. In the event of an actual 

disaster involving the NOC, PDRO2 at the MOC will 

assume daily operations of the DRS Facility 

38



DRS Network Operations Center (DNOC) … 

Is a full function operations facility located within the 

UNL-TV Transmission Center. It is the local 

operations, status, monitoring, alarm & control 

position in Lincoln, Ne. It will be the operations 

position during scheduled training operations, 

monthly satellite tests, and PBS Network 

Operations Center (NOC) maintenance or in the 

event of an actual disaster and the PBS NOC is 

disabled (either long term or short term). 

Laptop w/ Broadband & VPN Access … 

Is a full function operations workstation with full 

access to all applications and monitoring within the 

DRS. In a worse case scenario the DRS may be 

operated in this manner. 

39



Terrestrial Interconnection Sub-system (TIS) … 

Means dedicated and internet based connectivity 

between the Media Operations Center (MOC) in 

Arlington, Va., the Network Operations Center 

(NOC) in Alexandria, Va. and University of 

Nebraska – Lincoln Television Department (UNL- 

TV) in Lincoln, Ne 

40



Ku Band Satellite Uplink Transmission Subsystem 

(KSUTS) … 

Means an RF transmission system, housed within 

the UNL-TV satellite equipment shelter located in 

the “backyard” dish farm, consisting of DVB-S/S2 

QPSK/8PSK modulation sub-systems, IF-RF upconversion 

sub-systems, high power amplifier subsystems, 

automatic fault protection sub-systems, 

signal switching sub-systems, remote 

monitoring/control/alarm sub-systems, 

interconnected to satellite uplink antennas 

(including redundant antenna). We have exclusive 

use of this uplink chain and antenna, there will 

never be a conflict with UNL-TV? 

41



C Band Satellite Uplink Transmission 

Sub-system (CSUTS) … 

Means an existing UNL-TV RF 

transmission system (fiber STL from UNL- 

TV Studio Building), contained within the 

UNL-TV satellite equipment shelter located 

near the Lincoln Municipal Airport (LNK) 

(including redundant antenna). 

42



Real Time (RT) Program Origination Subsystem 

(POS) … 

Means a platform of equipment and software 

including but not limited to a traffic system, 

integrated ingest & play to air systems, signal 

processing sub-systems, SDI & IT switches & 

routers and IT management tools enabling the 

delivery of RT program streams to Public 

Television from UNL-TV Ne. 

Cache-Archive Sub-system (CAS) … 

Means a disk based cache & LTO tape 

based archive sub-system. 

43



Non-Real Time (NRT) 

File Transfer System (FTS) … 


enabling the delivery of NRT program files to Public 

Television Stations from UNL-TV Ne 

44



Satellite Encoding/Mux Sub-system (SEMS) … 

Means a master control system that is designed 

to encode and multiplex multiple channels of SD 

and HD video into transport streams for 

distribution to Public Television Stations from 

UNL-TV Ne 

Local & remote Status, Monitoring, Alarm & 

Control sub-system (SMAC) … 


enabling the control and monitoring, both locally 

and remotely, of the systems and services 

originating from or being distributed by the DRS 

facility 

45



The following conditions are examples of DRS activation scenarios: 

NOC Diversity: In order to maintain PBS programs and services to PTV 

Stations the DRS will provide satellite uplink services whenever the NOC 

uplink is disabled by 1) sun transients, 2) rain-fades, 3) blizzard conditions, or 

any other naturally occurring phenomena. 

NOC Maintenance: In order to maintain PBS programs and services to PTV 

Stations the DRS will provide program origination and uplink services 

whenever the NOC facility is out of service for maintenance functions 

(scheduled event). 

NOC Emergency: In order to maintain PBS programs and services to PTV 

Stations the DRS will provide program origination and uplink services 

whenever the NOC facility is temporarily (short term) out of service due to an 

emergency condition. 

46



The following conditions are examples of DRS activation scenarios: 

NOC Disaster: In order to maintain PBS programs and services to PTV 

Stations the DRS will provide program origination and uplink services if the 

NOC facility is rendered inoperable due to a catastrophic event. 

MOC/NOC Disaster: In order to maintain PBS programs and services to PTV 

Stations the DRS will provide content acquisition and management, program 

origination and uplink services if both the MOC and NOC facilities are 

rendered inoperable due to a catastrophic event. 

47

Original Concept: 


• Limited services (2 to 3 channels) 

• Legacy technology (automation, video servers, branding, …) 

• Parallel operation (in sync with NOC) 

• Restoration of supported services in < 15 mins 

• Remote control from PBS (NOC only) 

• Media based file delivery (LTO tape) 

• AS03 file format for distribution 

• Limited connectivity for: 

• Urgent file delivery only (limited by bandwidth) 

• Remote control 


Past & Present: A Project History 

• Status and monitoring 

• System integrated into existing PTV facilities 

48

Site Selection: 


• RFP written and issued to PTV 

• PTV Stations responded 

• Responses review/evaluated 

• Site selected (UNL-TV, Lincoln, Ne) 

Technology Changes: 

• IT based Ingest & Play to Air systems emerged 

• Integrated systems on the market (multi-vendor) 

• Asset management included in integrated solutions 

• SD & HD capability 

• Client/Server architecture (Directly Supports Workstation Model) 

• Web based services 



49


Technology Changes (Cont.): 

• Transcoding/Re-wrapping on IT server platforms comes of age 

• Lower initial investment as well as reduced operating costs 

New Concept for DRS: 

• Support the full bouquet of NOC services 

• HD01, HD02, HD03, HD04, SD01, SD02, SD03, SD04, SD05, SD06 

• Use DRS Uplink as Diversity Site 

• Rainfade, Sun Transient,… 

• Use IT technologies where applicable 

• S/W versus H/W solutions 



• Play to Air, Ingest, Cache, Convert… 

• Use commodity versus proprietary H/W (PBS purchases H/W) 

• IT server platforms (multi CPU, multi-core, local cache…) 

• Commodity video cards (multi-vendor) 

• IT network switches, routers, VPN, … 

50


New Concept for DRS: 

• Client/Server architecture 

• Stream Monitoring Solution back to PBS (Multiviews, Desktops, …) 

• Workstation based operations (Desktop/Laptop) 

• DRS Lab at PBS (support S/W & H/W testing prior to implementation) 

Program Origination Subsystem (POS) Vendor Shoot-out: 

• DRS Design Document created 

• “Proof of Concept” Lab @ PBS (concepts tested and demonstrated) 

• RFB issued to vendors 

• Multi-vendors responded 



• Three vendors participated in the Shoot-out 

• Two weeks allocated per vendor (1 wk setup, 1 wk test/eval) 

51


Program Origination Subsystem (POS) Vendor Shoot-out: 

• Demo/Evaluation conducted in Lab @ PBS, Alexandria 

• Results evaluated and scored 

• Vendors evaluated based on: 

• Scored performance, 

• Compliance to DRS model (design specification), 

• and business parameters (D&B, assets, …) 

• Vendor selected for DRS 

• GmbH, Munich, Germany 

Cache/Archive Subsystem (CAS): 



• Centralized Disk Cache & Tape Archive Solution 

• Common to RT & NRT Systems 

• Commodity H/W based solutions 

52


Cache/Archive Subsystem (CAS): 



• High Availability and Reliability Required 

• No standard off the shelf solution 

• Develop Solution with Vendor to support DRS 

• Test/Eval Proposed Solution in PBS Lab 

PBS DRS Connectivity (TIS): 

• Determine Bandwidth Requirements 

• Connectivity between MOC and DRS (File Delivery, Status/Mon/Alarm/Ctl) 

• Connectivity between NOC and DRS (Transport Streams, Status/Mon/Alarm/Ctl) 

• Evaluate, Test, and Demonstrate Available Solutions 

• Determine Best Quality Solution for DRS (BW vs $$$) 

53




More new Technology for DRS: 

• IP based Status, Monitoring, Alarming (TS over IP) 

• Research Available Applications 

• Implement S/W Solutions where possible 

• Commodity H/W where applicable 

• Exception based monitoring/alarming 

• Decode SPTS and MPTS within Mon/Alarm Engines 

• Full support for Closed Captions & Annc Data 

• DVB Encode/Multiplex 

• Use TSoIP for Signal Transport 

• Use Ethernet (cabling, switches, vlans, …) to: 

• Interconnection (Encoder/Mux/Modulator) 

• Provide Full Redundancy 

• Support TSoIP Status/Monitoring/Alarming 

• Support Full Control & Configuration 

• Encode using S/W where viable 

• Multiplex using S/W where viable 

54


Status, Monitoring, Alarming and Control: 

• Local Monitoring, Alarming & Control 

• Remote Monitoring, Alarming & Control 

• Duplicate User Experience at PBS (MOC, NOC) 

• Support Baseband, ASI & TSoIP formats 

• Exception Based 

• Display HD/SD, Alarm on Faults 

• Display Audio bouquet (16 chs), Alarm on Faults 

• Display Closed Captions, Alarm on Faults (608/708) 

• Display Annc. Data (AFD, V-Chip, …), Alarm on Faults 

• Fault Tolerant Solution 



55


MPEG4 Requirements: 



• System must be MPEG4 Ready 

• Work with PBS MPEG4 Team to Eval/Demo/Test Systems 

RF System for DRS: 

• Design Signal System in Cooperation with UNL-TV 

• Use Existing Regional Uplink Infrastructure in Design 

• Integrate Additional Equipment & Systems to Support DRS 

• Design Control System Using Existing UNL-TV Technologies 

• Design for Joint Use of Uplink for DRS & Regional Services 

• Activate Diversity Site functionality as soon as possible 

56


Where we are at this time: 



• The Agreement with UNL-TV is in place 

• The General Contractor has been hired 

• DRS Equipment Center (DEC) construction underway 

• The Program Origination (POS) Vendor is under Contract with PBS 

• Pre-build of the POS Subsystem Racks at PBS is underway 

• Connectivity Solutions are being Tested/Eval’d at this time 

• RF Design in Support of Diversity Site Functionality is underway 

• We are in development for the Redundant Cache/Archive Subsystem (working w/ 

Vendors) 

• Working with the MPEG4 Team, Test/Eval/Demo Vendor Solutions 

57

Project Goals: 




• Diversity Site Capability no later than end of Q2, beginning of Q3 

• Power Up Full DRS no later than Q3 of 2011 

• Commence Staff Training (PBS & UNL-TV) in Q3, 2011 

• Parallel Operations no later than end of Q3, 2011 

• First Satellite Testing and Demonstration, during Q4, 2011 

• DRS Facility transferred to PBS/UNL-TV Operations, end of Q4 

• Project wrap-up and documentation completed by end of Q4, 2011 

58

Project Partners: 


• NGIS Project, The Grand Pobah!, Jerry Butler, Alexandria, Va 

• PBS Engineering & Operations, Stefan Petrat, Alexandria, Va 

• UNL-TV, Stacey Decker, Lincoln, Nebraska 

• HA Design Group, Consulting, Willy Halla, Springfield, Va 

• Caleca Consulting, MPEG4, Ed Caleca, Washington, DC 

• Burt Hill Architects, Scott Jones, Washington, DC 

• Cinegy GmbH, POS, Jan & Daniella Weigner, Munich, Germany 

• ABC Electric, Inc., General Contractor, Brad Wittstruck, Lincoln, Ne 

• JT Packard, UPS/GenSet, Dani Maher & Sam Shannon, Verona, WI 

• More to Come!!! 


Thanks to All of You !!! 

59


“DIY” 

Disaster Recovery Template 

60


“Do-it-Yourself” DR Template 

•Organization of Document 

– Emergency Management Plan 

– Disaster Recovery Plan 

– “How-To” Document 

•Foundational Document 

– Building block for a complete business continuity plan 

•Published in PBS Connect (May 2010) 

– Engineer / Compute Technical Reference Library 

61


• Emergency Management is pre-determined actions that are 

performed immediately following emergency responders to an 

incident and before recovery. It is designed to create a “state of 

readiness” for an organization to function and begin a strategic, 

orderly recovery, avoiding confusion and reducing exposure to error. 

It is also known as Crisis Management. 

• Emergency Management Plan 

– Crisis Manager role and responsibilities 

– Emergency Teams roles and actions to perform 

• Initial Response Team 

• Emergency Management Team 

• Logistics Support Team 

• Emergency Response Team 


EMERGENCY MANAGEMENT PLAN 

62

Team Activation During a Crisis 

I N C I D E N T 

Activate Emergency Procedures 

Building Evacuation (ERT). Security 

Notified. Civil Authorities Notified. 

Activate Initial Response Team 

Damage Assessment. Disaster Declaration. Alert 

Critical Vendors. Establish Emergency Mgmt Ctr 

Activate Emergency Mgmt Team 

Recovery 

Recommendations. 

Required Resources. 

Manage Business Unit 

Activate Business Unit Recovery Teams 

Activate Logistics Support Team 

Recovery of respective 

business areas 

Support IRT and EMT for recovery of the organizational 

infrastructure. 

63


“Do-it yourself” DR Template 

64



•Disaster Recovery Plan 

– Disaster Scenarios 

– Critical Functions 

– Checklists 

– Appendices 

65



Disaster Recovery Plan 

DISASTER SCENARIOS 

– Fire/Explosion 

– Loss of Tower or Transmitter 

– Loss of Satellite 

– Loss of STL 

– Satellite Uplink Failure 

– Loss of PBS downlink or NGIS 

server 

– Loss of electrical power 

– Loss of HVAC 

– Internal Hazardous Material 

– External Hazardous Material 

– Weather – Advance Notice 

– Weather – No Notice 

– Loss of Transportation 

– Loss of Telecom 

– IT Disaster 

– Workplace Violence 

– Flooding 

– Civil Unrest 

– Loss of Multiple Key Personnel 

66




CRITICAL FUNCTIONS 

– Employee Safety 

– Executive Decision Making 

– Program Production 

– News Production 

– Program Origination 

– Program Distribution 

– Program Distribution 

– Over-the-Air Transmission 

– PBS Downlinking / Storage 

– Internal and External Communications 

– Feeds to cable, DirecTV, Dish, etc. 

67


• One for each Scenario 

• Indicates how to recover all critical functions 

• Less than 2 pages in length 



CHECKLISTS 

• Gives all pertinent information but not all details 

– Refers to Appendices with more details if necessary 

• Split into two timeframes: 1 st 72 hours and afterwards 

• Some scenarios use the same checklist 

68




APPENDICES 

– Evacuation Plan 

– Emergency Communications 

– Emergency Operations Centers 

– Shelter-in-Place 

– Television Ops Restoration 

– News Production Restoration 

– Studio Production Restoration 

– Long-term Evacuation 

– Electrical load reduction guidelines 

– Vendor List 

– IT Restoration 

69


How-To Document 

PLAN DEVELOPMENT PROCESS 

• Process is described in the 

How-To Document 

• How-To Document also 

includes: 

– Risk Assessment Guidelines 

– “Best Practices” information 

from other station plans 

– Typical Recovery Solutions 


70

Questions? 

71

PBS Business Continuity & Disaster Recovery

Create successful ePaper yourself

Delete template?

Save as template?