Nagios-Surfer - A Quality Reporting Tool for Nagios

FunetThe Finnish NREN80 connectedorganizations375000 end usersOperated by CSC –IT Center forScience Ltd2

IntroductionFunet uses Nagios extensively formonitoring.– network– servers– servicesTwo Nagios monitoring servers– Over 900 monitored hosts– Over 10000 monitored servicesNagios suits us well.3

Challenges:Huge number of servicesPossibility for configuration errorsWe needed an administrative overview ofthe used monitoring configuration.– What is monitored and how?– Who receives notifications and when?– What differences are there?4

Challenges:Nagios availability reporting is slowA large number of services means that theNagios event log files become large– 1.5 GB of event log per month– 18 GB of event log per yearNagios avail.cgi reads through event logseach time an availability report isrequested.– Generating an availability report may takeseveral minutes.5

Challenges:Finding the biggest availabilityissuesHow do we find the biggest issues from10000 services?– How to take different service availabilityrequirements into account?If service availability is low, what hascaused it?How to inform service administrators aboutavailability issues?– Short breaks may not have triggered Nagios e-mail notifications.6

Introducing Nagios-SurferDeveloped at Funet to help with theaforementioned issues in Nagios reporting.Provides:– Administrative reports about Nagiosconfiguration– Availability reports– Event logs– Tools for categorizing service breaks andinforming administrators about unexplaineddowntime.7

Overview8

Nagios-Surfer features:Nagios configuration overviewAutomatically generated for all hosts,services, contacts, and groups.Reports information about– Service checks– NotificationsReports differences between themonitoring configuration of hosts orservices in the same group.9

Nagios-Surfer features:Availability ReportsAutomatically generates availability reportsof all hosts, services, contacts and groups.Availability reports are pregenerated.– No need to wait.Availability numbers are reported permonth.– Also availability excluding scheduled downtimeis reported.10

Nagios-Surfer features:Event logsNagios-Surfer generates monthly event logsummaries of all hosts and services.– Redundant information, such as duplicate andsubsequent OK lines are removed.Each break contains a link to a form whichcan be used to examine and modify thecategorization and description of thebreak.Event logs can be accessed easily throughthe availability reports.11

Nagios-Surfer features:Break categorizationNagios-Surfer can send break clarificationrequests to administrators by e-mail.Administrators can categorize anddescribe breaks. The information is savedto Nagios-Surfer database for later use.If a break is categorized as scheduleddowntime, the change will be reflected inthe availability reports.– If a break happens during Nagios scheduleddowntime, the break is automaticallycategorized as scheduled downtime. 12

Example: Configuration overview13

Example: Availability report14

Example: Event log15

Example: Break information16

A tool for quality assuranceWe have internal quality assuranceprocesses that oversee that services meetthe set reliability requirements.Nagios-Surfer allows a quality assuranceteam to use better data.– Service administrators investigate new servicebreaks and save the information to Nagios-Surfer.– When the causes are known, a qualityassurance process can concentrate on themost relevant issues.17

Archiving information about breaksNagios-Surfer provides a central place forinformation about service disruptions.– Nagios-Surfer does not forget what happened.Makes it easier to notice patterns.– Investigating old issues becomes easier too,as the breaks of possible servicedependencies are visible.– Works also across organizational boundaries.There is no need to send e-mail and wait for ananswer.18

Providing availability reports to enduserorganizationsAn organization connected to Funet will beable to see the availability history of allused services at a glance.– IP connections– Light paths– … and more?Availability data is provided by Nagios-Surfer.Work in progress19

Other uses for Nagios-SurfercomponentsThe Nagios configuration API of Nagios-Surfer is used in other tools developed atFunet.– Scheduling Nagios downtime according topredefined templates.Server X is rebooted – affects also services Y and Z.The breaks in all affected services are documentedautomatically in Nagios-Surfer.– Combining several Nagios service groups intoone large service group.20

Technical detailsObject-oriented Perl– ~15k lines, ~360kB of codeSQLite– For saving information about breaks andscheduled downtime– For caching Nagios configuration dataReports use XHTML, CSS and someJavaScript.21

Questions?For more information:teemu.kiviniemi@csc.fi22