ITS Unplanned Outage Process - Information Technology Services

ITS Unplanned Outage Escalation ProcessJanuary 2013What to do if there is an Unplanned Outage♦♦♦Since unplanned outages require quick attention, all that is initially required is an immediate notification to sc.updatereporting the outage. Once the unplanned outage is resolved and service restored, a more detailed follow up is required.All unplanned outages are considered incidents and a ticket in IT Request that will record the impact and status update of theoutage. If the reporter does not create a ticket via “report an outage”, then the ITS Support Center will create one.All subsequent communication should occur within the IT Request ticket, including outage status updates and resolutiononce service has been restored. Make sure that sc.update is always included in the Watch List.What to do:1) Unplanned Outage – A service or system outage that wasn’t planned in advance that lasts more than 1 minute.The following needs to occur: Submit the outage information either by:• IT Request: http://itrequest.ucsc.edu (preferred)Login to IT Request and under Self-Service left navigation, click on "Report an Outage" and fill out the outage form.• Or Email: sc.update@ucsc.edu• As a last resort, call 459-JOBS (5627) if you can’t access the network.Include this information in your notice to sc.update:• Who: Technician(s) working on the outage and person to contact for status.• What: Summary of outage: Include client facing service name and description of outage.• Status: Is the outage still being worked on or is it resolved?• When: Date and start time of outage plus estimated time of recovery.• Clients: Who and how many are impacted? Campus business impact if known.• Communication: What communication steps have you taken and do you need assistance?• Status update: When will you send your next status update?Once the outage is reported to sc.update, the *ITS Support Center will create an outage ticket to collect all further updates.If the outage occurs after regular business hours (5PM-8AM) or on weekends, the reporter will need to create the ITR ticketincluding sc.update in the ticket copy field. This is recommended for all outages and mandatory for a Service Disruption. Ifyou use the Unplanned Outage web form, it will automatically notify sc.update and create an ITR ticket. You will still need toemail sc.update the ticket number and place sc.update in the ticket’s copy field.2) Service Disruption – If an unplanned outage is on an essential or necessary service (Network, Email, Calendar, Wireless, Cell, FIS,AIS, WWW, Phones, eCommons, Shibboleth, IDM-LDAP, etc.) or high impact system (AFS, VM Clusters, etc.) that affects a divisionor the campus, and is down and unavailable and/or disrupts campus business processes for more than 2 hours, the outage isconsidered a Service Disruption and the following needs to occur:• Designate an Incident Coordinator (frequently the lead technician or their manager).• Reply back to the ITR ticket email or update the ticket directly in IT Request with the outage status and note that it is now anofficial Service Disruption.• Be prepared to be on a Service Resolution Team (aka Mini-ITS DOC).• Know the outage escalation responsibilities associated with your ITS role (see page 2).Between 6:30AM – 8PM (any day, including weekends) contact your director/manager and/or one of the peoplebelow for help with ITS wide communication: Doug Hartline, Janine Roeth, Marian Sherrin, Lisa Bono, PeterMcMillan, or Andrea Hesse. DCO Operators (459-2714) have escalation phone numbers if needed.3) IT Disaster – If an unplanned outage is an essential campus service or system that is down and unavailable for more than 5 days,or an outage of a system that prevents instruction (when classes are in session) for more than 24 hours, the outage is considered an ITDisaster. The ITS Divisional Operations Center (ITS DOC) will be activated.Outage process questions: contact Marian Sherrin at msherrin@ucsc.edu / 2-7365Communication questions: contact Lisa Bono at lbono@ucsc.edu / 9-1583