Zenoss Labs Documentation - Read the Docs

Zenoss Labs DocumentationReleaseZenoss LabsJune 18, 2014

Contents1 Contents 21.1 ZenPack Development Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 ZenPack Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 641.3 ZenPack Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 782 Documentation Formats 893 Contribution 904 Contact 91i

Zenoss Labs Documentation, ReleaseHerein lies documentation created by Zenoss Labs. This will have to do until a better home can be found.Contents 1

CHAPTER 1Contents1.1 ZenPack Development GuideWelcome to the ZenPack Development Guide. The purpose of this guide is to walk you through the creation of aZenPack. We will be focusing on solving specific problems rather than attempting to provide an exhaustive list of thecapabilities of a ZenPack.Contents:1.1.1 PrerequisitesTo follow the steps in this guide you will need to have access to the following:• A Linux server with Zenoss installed on it. This should not be a Zenoss server you care about. We will breakthings. You can download Zenoss from the Zenoss download site.• An SSH client to connect to your Zenoss server. PuTTY works well for Windows, ssh from the command lineworks well for Mac and Linux.• This guide.This guide will provide full examples to create a working ZenPack. However, the examples and the explanations forthem will be much easier to understand with at least basic skills in the following areas:• Linux: Ability to move around the file system, manage files and run commands.• Programming or scripting with Python experience being a plus.• XML syntax.• SNMP.Finally, it is expected that you are familiar with Zenoss from a configuration perspective.1.1.2 Development EnvironmentThe process of developing a ZenPack can be made much faster and less frustrating by starting with a good developmentenvironment. The following development environment setup is only a set of recommendations. If you already havetools or techniques that you’re more comfortable with or you consider superior, you should use them.2

Zenoss Labs Documentation, ReleaseInstalling ZenossTo best develop and deploy ZenPacks into a production environment, I recommend having three Zenoss systems:development, staging and production.Development Zenoss SystemYour development Zenoss system should not be shared with others, and should be as minimal an installation as possible.This will allow for problems to be diagnosed, fixed and tested as quickly as possible.I recommend installing Zenoss from the single zenoss-x.y.z RPM on Red Hat Enterprise Linux or CentOS. TheZenoss installation instructions will ordinarily have you install a zenoss-core-zenpacks RPM, and a zenoss- enterprisezenpacksRPM for commercial Zenoss customers. To keep the system as small as possible, these should not be installedinto your ZenPack development system.Staging Zenoss SystemThe staging Zenoss system should be shared by all ZenPack developers. After a ZenPack has been successfully testedin the development system should it be installed into the staging environment.This system should mimic the production system as closely as possible. It should be installed in the same way as theproduction system, and should have the same base set of ZenPacks installed. The purpose of the staging system is todo final integration testing before ZenPacks are installed into the production system.Production Zenoss SystemNew ZenPacks and updates to existing ZenPacks should only be deployed into the production system after they’vebeen successfully tested in the development and staging systems.Running a Minimal ZenossOften times ZenPack development is done within virtual machines, or spare hardware that doesn’t have the sameresources as a production Zenoss system. Additionally, during development you will need to restart Zenoss far morefrequently than in a production setup. For these reasons you will want to run as little of Zenoss as necessary.After installing your Zenoss development system, run the following commands to reduce your Zenoss deployment tothe minimum typical processes:su - zenosszenoss stopcat > $ZENHOME/etc/daemons.txt

Zenoss Labs Documentation, ReleaseNote: For the commercial version of Zenoss you will need to add zencatalogservice to the top of daemons.txt.See the following notes for more information on what these commands are doing.1. The su - zenoss command is an important one that you’ll be using very frequently. This command switchesfrom the root user to the zenoss user. The hyphen (-) creates what’s called a login shell. This means that thezenoss user’s full environment will be loaded. This full environment is necessary to run any Zenoss commands.2. The zenoss stop command stops all Zenoss processes.3. The cat > ... that ends with EOF on a blank line writes those specific daemon names into the$ZENHOME/etc/daemons.txt file. This file contains the names of any daemons in addition to the defaultthat should be started, stopped and otherwise managed by the zenoss master control script.4. The touch $ZENHOME/etc/DAEMONS_TXT_ONLY command will cause the zenoss master controlscript to only manage daemons listed in the aforementioned daemons.txt file.5. The zenoss start command will only start daemons listed in daemons.txt.Running Zenoss in the ForegroundTo take your development and debugging environment to the next level, I recommend running most Zenoss processesin the foreground with full debug logging enabled. This allows you to easily see exactly what is happening, andprovides a very quick way, CTRL-C, to stop and start individual processes. Perhaps the most useful benefit is that youcan use the Python debugger to set breakpoints without having them hang background daemons.The zeneventserver process is the only process that I recommend running as a daemon while developing Zen-Packs. This is because there are no changes you can make while developing a ZenPack that require restartingzeneventserver. There’s also no common reason you’d want to see its debugging output.You can open multiple SSH sessions to your Zenoss server, or use the GNU screen or tmux terminal multiplexers tosimultaneously run Zenoss processes in the foreground.Open five (5) sessions as the zenoss user on your zenoss server using any of the aforementioned methods. Thesessions should run the following commands respectively:1. zopectl fg2. zeneventd run -v103. zenhub run -v10 --workers=04. zenjobs run -v10 --cycle5. nothing: miscellaneous shell work is done hereThe -v10 option enables DEBUG level logging for the process. The --workers=0 option to zenhub is importantto prevent zenhub from spawning other worker process to do work. If work was performed in a worker process youmay not have foreground visibility to it. The --cycle option to zenjobs is necessary to prevent it from executing allcurrent pending jobs then exiting.Installing ZenPacksZenPacks can be installed either from a packaged .egg file, or from a source directory. The only situations in which Irecommend installing from a packaged egg is for the ZenPacks that ship with Zenoss and are automatically installedfrom their .egg file, and when the source is not available.There are some important reasons why installing in development mode from a source directory is preferable. Theyinclude:1.1. ZenPack Development Guide 4

Zenoss Labs Documentation, Release• The running ZenPack code can be a checkout from version control. This makes it easier to audit ZenPack codefor changes.• ZenPacks can be upgraded in-place. Depending on the changes, this can often allow for less Zenoss daemonsneeding to be restarted after upgrading a ZenPack.When a new ZenPack is created in the user interface, it is created in development mode with the source directorylocated in $ZENHOME/ZenPacks/. To install or upgrade an existing ZenPack from it’s source directory, the --linkoption is used as follows:zenpack --link --install $ZENHOME/ZenPacks/ZenPacks.namespace.ZenPackName1.1.3 Background InformationThe section contains high-level background concepts and other information that will make the subsequent sectionseasier to understand. I recommend that you read and understand everything here.Device vs. DeviceComponentUnderstanding the difference between a Zenoss Device and DeviceComponent (often referred to as simply component)is useful for the Zenoss user, but critical to the ZenPack developer. The Zenoss object model has these two primarytypes of objects that can be modeled and monitored.Device Devices are what you see on the Zenoss Infrastructure view in the web interface. If you see it in the Infrastructureview, it’s a device. If you don’t, it’s not. A device has an id attribute that makes it unique in the system.It will have configuration properties associated with it either directly, or acquired from the device class withinwhich it is contained. It will also have a manageIp attribute that Zenoss uses for modeling and monitoring.Devices are contained within a single device class such as /Server/Linux or /Network/Ping. Devices and deviceclasses have a lot in common. They both have configuration properties (a.k.a zProperties) and can containmonitoring templates.Configuration properties and monitoring templates are defined in a hierarchical fashion with the most specificcopy being used. Take the following example of the zCommandUsername configuration property.Where zCommandUsername is set:• / = “”• /Server = “root”• /Server/Linux/AWS = “ec2-user”• server1 in /Server/Linux/AWS device class = “joeuser”In this case, the zCommandUsername used for server1 would be “joeuser” because it is most specific to server1.If zCommandUsername has not been set directly on the device, “ec2-user” would be used instead.Monitoring templates work the same way. If a device has the Device monitoring template bound to it, it willfirst check to see if there’s a local copy of that monitoring template contained within the device. If there isn’t,the device class path will be walked back to / until a monitoring template named Device is found.DeviceComponent Device components are what you see if you drill-down into a device in the web interface, thenchoose one of the component types from the left navigation pane. Each row in the grid in the top-right pane is acomponent. Examples include things like network interface, file systems, disks and processes. These are thingsthat have many instances per device.Device components, commonly just called “components”, don’t have their own configuration properties. Theyonly acquire configuration properties through the device that contains them. They do not have a manageIpbecause they’re typically managed through the same IP address and protocol(s) as the device that contains them.1.1. ZenPack Development Guide 5

Zenoss Labs Documentation, ReleaseTemplate BindingHow monitoring templates get bound is different for devices and device classes than it is for components.A device or device class can easily have many monitoring templates bound to it. This is controlled by selecting BindTemplates from the gear menu of a device or device class. You will be presented with an available list of templates thatcould be bound on the left, and the list of templates that are currently bound on the right. You will only see monitoringtemplates that are appropriate to be bound directly to devices.Note: Using the Bind Templates dialog is really just setting the zDeviceTemplates configuration property in a morefriendly way. You can directly modify zDeviceTemplates to achieve the same result.Relationship TypesThe Zenoss model uses relationships between objects in the database in much the same way that a relational databasedoes. Here are examples of some of the relationships working behind the scenes in a standard Zenoss installation.• DeviceClass has many Device• Device has one OperatingSystem• OperatingSystem has one ProductClass• OperatingSystem has many FileSystem• OperatingSystem has many IpInterface• IpInterface has many IpAddress• IpNetwork has many IpAddressZenPacks can contribute new types of objects and new relationships to the system.Packs.zenoss.RabbitMQ adds the following object types and relationships.• Device has many RabbitMQNode• RabbitMQNode has many RabitMQVHost• RabbitMQVHost has many RabbitMQExchange• RabbitMQVHost has many RabbitMQueueThere are some important details you must understand to create new relationships.• There are containing and non-containing relationships.For example, Zen-The idea of containment comes from Zenoss’ database being a tree of objects. Every object in the database mustbe attached in some way to another persistent object, and only one other persistent object.In the case of a containing relationship you’re saying that this is the relationship that attaches the member objectsto the object model and therefore is how they get persisted. Removing an object from its containing relationshipis the same thing as deleting the object from the database entirely. Any non-containing relationships it mighthave will be cleaned up.Of the above example standard relationships, the following are implemented as containing relationships.– DeviceClass has many Device– Device has one OperatingSystem– OperatingSystem has many FileSystem– OperatingSystem has many IpInterface1.1. ZenPack Development Guide 6

Zenoss Labs Documentation, Release– IpNetwork has many IpAddressOn the other hand, a non-containing relationship has nothing to do with persistence. Non-containing relationshipsare used to create logical linkages from one object to another.Of the above example standard relationships, the following are implemented as non-containing relationships.– OperatingSystem has one ProductClass– IpInterface has many IpAddress• All relationships must be bi-directional and explicitly defined on both sides of the relationship.In the RabbitMQNode has many RabitMQVHost example above this means that RabbitMQNode must declarethat it has a ToManyCont relationship to RabbitMQVHost, and RabbitMQVHost must declare that it has a ToOnerelationship to RabbitMQNode.Relationship ClassesRelationships between object types (Python classes) are objects themselves. You use the following classes from theProducts.ZenRelations.RelSchema module to define them.• ToMany• ToManyCont• ToOne• ToOneContOnly the following pairs of relationships are valid because of the requirement that relationships be defined bidirectionally.• ToMany ToOne (non-containing many-to-one)• ToMany ToMany (non-containing many-to-many)• ToManyCont ToOne (containing many-to-one)• ToOneCont ToOne (containing one-to-one)1.1.4 Creating a ZenPackThe actual creation of a new ZenPack is very simple, and done through the Zenoss web interface. However, there someconsiderations related to the ZenPack’s name and version that should be considered at creation time. Once a ZenPackis in use by you and potentially others renaming or changing the version scheme can be more painful.Follow these steps to create a new ZenPack:1. Login to the Zenoss web interface as a user with the global Manager role.2. Navigate to the Advanced section.3. Choose ZenPacks from the left navigation.4. Choose Create a ZenPack from the gear menu next to Loaded ZenPacks.5. Name the ZenPack according to the Naming a ZenPack documentation.6. Click OK.7. Set Version according to the ZenPack Versioning documentation.8. Set Author to Your Name .1.1. ZenPack Development Guide 7

Zenoss Labs Documentation, Release9. Set Dependencies according to the ZenPack Dependencies documentation.10. Click Save.After setting the version, author and dependencies for your ZenPack you should export it so the changes get written tothe ZenPack’s source on the file system.1. Choose Export ZenPack from the gear menu in the bottom-left corner of the screen from the ZenPack’s detailpage.2. Leave Export to $ZENHOME/exports selected then click OK.Naming a ZenPackZenPacks exist within the ZenPacks namespace. Zenoss requires at least one additional level of namespace beforethe name of the ZenPack to guard against different Zenoss developers creating ZenPacks by the same name. See thefollowing ZenPack name as an example:ZenPacks.zenoss.Memcached• ZenPacks is mandatory.• zenoss is your definable namespace. zenoss is reserved for Zenoss, Inc. If you’re creating a ZenPack that youaim to publish as open source into the Zenoss community as the standard ZenPack for performing some function,the community namespace is recommended. Otherwise your company name or username should be used.• Memcached is the name of the ZenPack and should describe what the ZenPack is for in the shortest way possible.The most common functionality to provide with a ZenPack is to monitoring something. For this reason a namelike Memcached is acceptable over MemcachedMonitor.It’s also possible to use additional namespaces. It will mainly just create more typing for the developer, so it isn’t oftenused. An example of why you might want to do this is if you want to create many ZenPacks around the same targetsystem. A fictitious example of this would be AWS (Amazon Web Services) monitoring.• ZenPacks.community.AWS.EC2• ZenPacks.community.AWS.S3• ZenPacks.community.AWS.RDSZenPack VersioningZenPacks should be thought of as standalone software packages and versioned as such. A three part version numberis recommended. For example, X.Y.Z (major.*minor*.*patch*).Follow these simple rules to maintain a proper version of your ZenPack:• 0.7.0: First version you’re playing with that others should still be very frightened of. (a.k.a alpha)• 0.9.0: First version that you feel is consumable by others. (a.k.a. beta)• 1.0.0: First stable version that has been thoroughly tested. Ideally by people other than the original developer,and in environments other than that of the original author.For versions prior to 1.0.0, any bug fixes, enhancements or API breakages should result in the patch number beingincremented.For versions subsequent to 1.0.0, the following rules should be followed:• Bug fixes result in the patch number being incremented.• New features that are backward-compatible result in the minor number being incremented and the patch numberbeing reset to 0 even if bug fixes are also included.1.1. ZenPack Development Guide 8

Zenoss Labs Documentation, Release• New features are not backward-compatible result in the major number being incremented and the minor andpatch numbers being reset to 0 even if backwards-compatible features or bug fixes are also included.I recommend Semantic Versioning for a reference on good software versioning practices.ZenPack DependenciesYou can specify dependencies for your ZenPack. There are two major dependency types. Dependency on the supportedversion of the Zenoss platform, and dependencies on other ZenPacks.For ZenPacks installed in developer mode, you can view and edit dependencies by going to Advanced -> ZenPacks ->Your ZenPack in the web interface. Under the Dependencies section you’ll have a row for Zenoss and a row for everyother ZenPack that’s installed in the system.If you don’t plan to test your ZenPack on prior versions of Zenoss, I recommend always setting the Versions(s) field forthe Zenoss row to be >=CURRENT_ZENOSS_MINOR_VERSION where CURRENT_ZENOSS_MINOR_VERSIONwould be something like 3.2 or 4.2. This will prevent your ZenPack from being installed on earlier versions of Zenoss.If your ZenPack depends on functionality provided by other ZenPacks, you can choose to simply put a check mark inthe Required field for each of those ZenPacks and not specify version. This only enforces that some version of thatZenPack must be installed. To create a version-specific dependency, you would enter a >=VERSION or ==VERSIONin the Versions(s) field for each ZenPack dependency.For more information on what comparisons can be used in the Version(s) field you can reference the Python RequirementsParsing documentation.1.1.5 Monitoring an SNMP-Enabled DeviceThe following sections will describe a common approach to monitoring an SNMP- enabled device. We’ll start withthe basics that can be done without writing a line of code, and then move on to more sophisticated capabilities.For purposes of this guide we’ll be building a ZenPack to support a NetBotz environmental sensor device. This devicehas a variety of sensors that monitor temperature, humidity, dew point, audio levels and air flow.Exercises:SNMP ToolsTo configure Zenoss to monitor a device using SNMP, it is necessary to understand a bit about SNMP and the specificcapabilities of your device. This section will walk you through using Net-SNMP tools to learn about SNMP and yourdevice.The netsnmp-utils package is a prerequisite to installing Zenoss so you already have the SNMP tools you needinstalled on your Zenoss server.Using SNMPosterWhen developing a ZenPack to monitor an SNMP-enabled device it can often be useful to simulate the device’s SNMPagent. There are many tools out there that can be used to do this. Some commercial and some free. Out of the freetools I recommend SNMPoster mainly because it can easily be run on your Zenoss development system, and usessnmpwalk output as its input format. This makes it easy to grab and use data from real devices.Use the following instructions to setup SNMPoster to simulate the NetBotz device used throughout this guide. Thesesteps should all be run as the root user.1. Install SNMPoster according to the instructions on its site.1.1. ZenPack Development Guide 9

Zenoss Labs Documentation, Release2. Download and configure the NetBotz agent.mkdir -p /etc/snmposter/agentscd /etc/snmposter/agentswget --no-check-certificate https://github.com/cluther/snmposter/raw/master/agents/NetBotz.snmpwcat > /etc/snmposter/agents.csv

Zenoss Labs Documentation, Release.1.3.6.1.2.1.1.4.0 = STRING: unknown.1.3.6.1.2.1.1.5.0 = STRING: Netbotz01.1.3.6.1.2.1.1.6.0 = STRING: Z1 Rack02 NetBotz01While this data is mostly less valuable than the decoded version above, it’s more useful for a single reason. We cantake that .1.3.6.1.4.1.5528.100.20.10.2006 value and search the Internet for it. It’s best to remove theleading . and search for 1.3.6.1.4.1.5528.100.20.10.2006 instead.This should lead you to the NETBOTZV2-MIB which will contain the decoding information we need to learn moreabout this device. Download NETBOTZV2-MIB.mib and copy it into the /usr/share/snmp/mibs/ directoryof your Zenoss server.Now we can run the original snmpwalk command again with the addition of the -m all option. This option tellsNet-SNMP tools to use all MIBs:# snmpwalk -v2c -c public -m all 127.0.1.113 systemSNMPv2-MIB::sysDescr.0 = STRING: Linux Netbotz01 2.4.26 #1 Wed Oct 31 18:09:53 CDT 2007 ppcSNMPv2-MIB::sysObjectID.0 = OID: NETBOTZV2-MIB::netBotz420ERackDISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (7275488) 20:12:34.88SNMPv2-MIB::sysContact.0 = STRING: unknownSNMPv2-MIB::sysName.0 = STRING: Netbotz01SNMPv2-MIB::sysLocation.0 = STRING: Z1 Rack02 NetBotz01Now we can see that the sysObjectID is NETBOTZV2-MIB::netBotz420ERack. This gives us a better idea of exactlywhat kind of device it is. We’ll see that as we look deeper into this device that the NETBOTZV2-MIB will prove moreuseful.Default Net-SNMP OptionsThe snmpwalk usage showed three primary command line options that we tend to use most of the time. Net-SNMPallows you to specify these in a configuration file so you don’t have to type them every time. I recommend doing this.Create /etc/snmp/snmp.conf and add the following lines:defVersion v2cdefCommunity publicmibs ALLThese lines add the following equivalent command line options respectively:• -v2c• -c public• -m allSo now we can run this command:snmpwalk 127.0.1.113And get the same results as if we ran:snmpwalk -v2c -c public -m all 127.0.1.113Trust me that this will save you time while developing this ZenPack, and others in the future.1.1. ZenPack Development Guide 11

Zenoss Labs Documentation, ReleaseDecoding and Encoding OIDsOften it can be useful to turn numeric OIDs into their human-readable equivalent, or vice-versa.snmptranslate command can be used for this. See the following examples.OID to name:The# snmptranslate .1.3.6.1.4.1.5528.100.20.10.2006NETBOTZV2-MIB::netBotz420ERackName to OID:# snmptranslate -On NETBOTZV2-MIB::netBotz420ERack.1.3.6.1.4.1.5528.100.20.10.2006Device MonitoringThis section will cover monitoring device-level metrics using SNMP. This requires no code, and you can find instructionsfor doing it in the normal Zenoss documentation. However, there are some extra considerations and stepsrequired to package your configuration in a ZenPack.Create a Device ClassTo support our new NetBotz environmental sensor device we will want to create a new device class. This will give usfull control over how these types of devices are modeled and monitored. Use the following steps to add a new deviceclass.1. Navigate to the Infrastructure view.2. Select the root of the DEVICES tree.3. Click the + button at the bottom of the list to add a new organizer.4. Set the Name to NetBotz then click SUBMIT.The new NetBotz device will now be selected. We’ll want to check on some important configuration propertiesusing the following steps.Set Device Class Properties1. Click the DETAILS button at the top of the list.2. Select Modeler Plugins.The modeler plugins are what model information about the device. We should see a list something like thefollowing. This list is being acquired from the root (/ or /Devices) device class.• zenoss.snmp.NewDeviceMap• zenoss.snmp.DeviceMap• zenoss.snmp.InterfaceMap• zenoss.snmp.RouteMapThis is a good basic list that uses standard MIB-2 support that works with most SNMP-enabled devices. However,it’s unlikely that we care about the routing table on our environmental sensors, so there’s no reason tomodel it.3. Remove zenoss.snmp.RouteMap from the list.1.1. ZenPack Development Guide 12

Zenoss Labs Documentation, Release4. Click Save.Now you can see the Path at which our modeler plugin configuration is set has changed from / to /NetBotz.This allows us to know that regardless of what the user sets their default modeler plugins to in the system thatNetBotz appliances will be collected using the set of modeler plugins we configure here.5. Select Configuration Properties from the left navigation pane.There are a lot of configuration properties. You don’t have to worry about understanding all of them. However,some will be critical to monitoring NetBotz appliances. We know that we’re going to be using SNMP so let’smake sure that it’s enabled.6. Find the zSnmpMonitorIgnore property and set its value to true.7. Now set the value for zSnmpMonitorIgnore to false.The reason for flipping the value back to it’s original value is the same as saving the list of modeler plugins.While the system default is to have SNMP monitoring enabled, a user could easily disable it globally and causeour NetBotz monitoring to stop working. By flipping the value, we’ve set it locally within our device class andwill prevent changes in the global default from affecting the operation of our ZenPack.Add Device Class to ZenPack Now that we’ve setup the NetBotz device class, it’s time to add it to our ZenPackusing the following steps. Adding a device class to your ZenPack causes all settings in that device class to be added tothe ZenPack. This includes modeler plugin configuration, configuration property values and monitoring templates.1. Make sure you’ve already created the ZenPack.2. Make sure that you have the NetBotz device class selected in the Infrastructure view.3. Choose Add to ZenPack from the gear menu in the bottom-left.4. Select your NetBotz ZenPack then click SUBMIT.Add a NetBotz Device This would be a great time to add a NetBotz device to our new /NetBotz device class. Wehaven’t done anything in the way of customer monitoring. It can often be helpful to see what Zenoss’ default settingswill return for a device before we start adding features.You can add a the device through the web interface, or on the command line using zendisc as follows:zendisc run --deviceclass=/NetBotz --device=127.0.1.113Note: I’ll often use zendisc from the command line only because the zenjobs daemon must be running to add jobsfrom the web interface. The zenjobs daemon is not required to be running when adding devices using zendisc from thecommand line because it immediately adds the device instead of scheduling a job to do it.You should now see that Zenoss was able to model some information about the device even though we haven’t addedany custom monitoring. For example, you should see the following on the device in the web interface.• Overview– Hardware Manufacturer: NetBotz– Hardware Model: .1.3.6.1.4.1.5528.100.20.10.2006– OS Manufacturer: Unknown– OS Model: Linux 2.4.26• Components - Interfaces: 2 - eth0 and loIf we were running the zenperfsnmp daemon, we’d start to see that Zenoss was monitoring the uptime and interfacemetrics after about 10 minutes.1.1. ZenPack Development Guide 13

Zenoss Labs Documentation, ReleaseConfigure Monitoring TemplatesBefore adding a monitoring template we should look to see what monitoring templates are already being used in ournew device class.Validate Existing Monitoring Templates We created the NetBotz device class directly within the root (or /) deviceclass. This means that we’ll be inheriting the system default monitoring templates and binding. Use the followingsteps to validate this.1. Select the NetBotz device class in the Infrastructure view.2. Choose Bind Templates from the gear menu in the bottom-left.You should only see Device (/Devices) in the Selected box. Depending on what other ZenPacks youhave installed in the system you may see zero or more other templates listed in the Available box.Now we investigate what this system default Device monitoring template does.3. Click CANCEL on the Bind Templates dialog.4. Click the DETAILS button at the top of the device class tree.5. Select Device (/Devices) under Monitoring Templates.You’ll see that there’s a single SNMP datasource named sysUpTime. If you expand this datasource you will seethat it contains a single datapoint which is also named sysUpTime. This single datapoint named the same asits containing datasource is always what you’ll see for SNMP datasources. The reason for having the conceptualseparation between datasources and datapoints is that other types of datasources such as COMMAND arecapable of returning multiple datapoints.You’ll note that this monitoring template has no threshold or graphs defined. This is unusual. Typically there’dbe no reason to collect data that you weren’t going to either threshold against or show in a graph. The sysUpTimedatapoint is an exception because it is shown on a device’s Overview page in the Uptime field and thereforedoesn’t need to be graphed.Let’s use snmpwalk to check if our NetBotz device supports sysUpTime. The OID listed for the sysUpTime datasourceis 1.3.6.1.2.1.1.3.0 so we run the following command:# snmpwalk 127.0.1.113 1.3.6.1.2.1.1.3.0DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (7275488) 20:12:34.88This response indicates that the NetBotz device does support the sysUpTime OID. This is a mandatory field for SNMPdevices to support so you will be able to get it in almost all cases.Add a Monitoring Template Now that we’ve validated that the existing Device monitoring template will work onour NetBotz device, we’ll add another monitoring template to collect additional information.Note: We could create a local copy of the Device monitoring template in the NetBotz device class and add newdatasources, thresholds and graphs to it. However, this prevents us from taking advantage of changes made to thesystem default Device template in the future.Follow these steps to create and bind a new template to the NetBotz device class.1. Navigate to Advanced -> Monitoring Templates.2. Click the + button in the bottom-left to add a template.1. Set the Name field to NetBotzDevice.2. Set the Template Path field to /NetBotz.1.1. ZenPack Development Guide 14

Zenoss Labs Documentation, Release3. Click SUBMIT.4. Bind this template to the NetBotz device class.1. Navigate to Infrastructure.2. Select the NetBotz device class.3. Choose Bind Templates from the gear menu in the bottom-left.4. Move NetBotzDevice from available to selected.5. Click SAVE.Build the Monitoring Template Now that we’ve created the NetBotzDevice monitoring template and bound it tothe NetBotz device class, we need to add datasources, thresholds and graphs. We don’t already know what might beinteresting to graph for each NetBotz device, so let’s go exploring with snmpwalk:# snmpwalk 127.0.1.113 .1.3SNMPv2-MIB::sysDescr.0 = STRING: Linux Netbotz01 2.4.26 #1 Wed Oct 31 18:09:53 CDT 2007 ppcSNMPv2-MIB::sysObjectID.0 = OID: NETBOTZV2-MIB::netBotz420ERack... lots of lines removed ...SNMPv2-MIB::snmpInTotalReqVars.0 = Counter32: 4406... and more removed ...There isn’t much of interest to collect at the device level. By “device-level” I mean values that only have a singleinstance for the device. Typical examples of these kinds of metrics would be memory utilization or the previoussysUpTime example. With SNMP it can be easy to find these kinds of single-instance values because their OID endsin .0 as in SNMPv2-MIB::snmpInTotalReqVars.0.Note: We’ll get into monitoring multi-instance values in the component monitoring section.Since there aren’t any extremely interesting single-instance values to collect, we’ll collect that snmpInTotalReqVarsfor illustrative purposes. We’ll need to know the numeric OID for this value. Use snmptranslate to find it:# snmptranslate -On SNMPv2-MIB::snmpInTotalReqVars.0.1.3.6.1.2.1.11.13.0Add an SNMP Datasource1. Navigate to Advanced -> Monitoring Templates.2. Expand NetBotzDevice then select /NetBotz.3. Click + on the Data Sources pane.1. Set Name to snmpInTotalReqVars2. Set Type to SNMP3. Click SUBMIT.Use the steps below to add an SNMP datasource for snmpInTotalReqVars.Note: Best practice is to name SNMP datasources according to the name of the OID that’s being polledfrom the MIB.4. Double-click to edit the snmpInTotalReqVars datasource.1. Set OID to 1.3.6.1.2.1.11.13.01.1. ZenPack Development Guide 15

Zenoss Labs Documentation, Release2. Click SAVE.Warning: A common mistake to make when setting the OID in a device-level template is to omit thetrailing .0. The reason this is common is that if you were using the MIB as reference instead of the snmpwalkabove, you’d see that the OID for SNMPv2-MIB::snmpInTotalReqVars was 1.3.6.1.2.1.11.13instead of 1.3.6.1.2.1.11.13.0. Due to this, I always recommend using snmpwalk to verify exactlywhat OID you should be polling.While Zenoss will accept the OID with the leading ., I recommend omitting it. It isn’t necessary.We now have a choice about how we want to handle the value that comes back from polling that OID. As you can seeabove in the snmpwalk output, it is a Counter32 type. This means that it starts at 0 and, in this case, increments eachtime an SNMP variable is requested. The most common way to handle counters like these is as a delta. It’s not veryinteresting to know how many variables have been requested since the device last rebooted, but it might be interestingto know how many variables are requested per second.The default type for a datapoint is GAUGE which would record the actual value you see in the snmpwalk output. Ifwe’d rather monitor the rate of requests, we’d change the datapoint type to DERIVE using the following steps.1. Double-click on the snmpInTotalReqVars.snmpInTotalReqVars datapoint.You may need to expand the snmpInTotalReqVars datasource first.1. Set RRD Type to DERIVE2. Set RRD Minimum to 03. Click SAVE.Warning: It is very important to always set the RRD Minimum to 0 for DERIVE type datapoints. If you fail to dothis, you will get large negative spikes in your data anytime the device reboots or the counter resets for any otherreason.The only time you wouldn’t set a minimum of 0 is when the value you’re monitoring can increase and decreaseand you’re interested in tracking rates of negative change as well as rates of positive change.Add a Threshold Now we can add a threshold to our monitoring template. Let’s say we want to raise a warningevent anytime the rate of SNMP variable requests exceeds 10 per second. This can be done with the following steps.1. Click + on the Thresholds pane.1. Set Name to high SNMP variable request rate2. Set Type to MinMaxThreshold3. Click ADD.2. Double-click to edit the high SNMP variable request rate threshold.1. Drag the snmpInTotalReqVars datapoint to the left box.2. Set Severity to Warning3. Set Maximum Value to 104. Set Event Class to /Perf/Snmp5. Click SAVE.Note: A MinMaxThreshold can be used to handle a variety of conditions including over a maximum value, under aminimum value, outside a defined range or within a defined range. See the regular Zenoss documentation for how touse each of these options.1.1. ZenPack Development Guide 16

Zenoss Labs Documentation, ReleaseAdd a Graph Definition Now we’ll add a graph so the user will be able to see the trend of SNMP variable requestsper second over time. This can be done with the following steps.1. Click + on the Graph Definitions pane.1. Set Name to SNMP Rates2. Click SUBMIT.2. Double-click to edit the SNMP Rates graph definition.1. Set Units to requests/sec2. Set Min Y to 03. Click SUBMIT.Note: Always set the units for your graph. Also set the minimum Y axis and maximum Y axis values ifyou know what the possible limits are for the data. This results in graphs that are easier to read.The format field should also be tweaked to best present the kind of data that is to be graphed. You can findmore information on what can be used in the format field in the RRDtool rrdgraph_graph documentationunder the PRINT section.3. Select the SNMP Rates graph definition.4. Choose Manage Graph Points from the gear menu.1. Choose Data Point from the + menu.2. Set Data Point to snmpInTotalReqVars3. Check Include Related Thresholds4. Click SUBMIT5. Double-click to edit the snmpInTotalReqVars graph point.1. Set Name to Variables2. Click SAVE.Note: The name of a graph point is what is displayed for it in the graph legend. You should alwayschoose something short that describes the data and makes sense in context of the units chosen above.You can find many more notes about how to create monitoring templates along with best practices on graph styling inthe ZenPack Standards Guide.Test Monitoring TemplateThe quick way to check if we’ve been successful in creating and binding our monitoring template is to navigate tothe NetBotz device we added to the system and verify that we see our NetBotzDevice (/NetBotz) monitoring templatelisted at the bottom of the device’s left navigation pane.Now we can test that our datasource will be collected by running the following command to do a single collection ofthe NetBotz device:zenperfsnmp run -v10 --device=Netbotz01We can look through the output to see what zenperfsnmp does. Personally I look for any lines that contain zen.RRDUtil.These lines will show the collected data being written to RRD files. If data isn’t collected, these lines won’t be present.Because of this you might run the following command instead to only see lines that contain this pattern:1.1. ZenPack Development Guide 17

Zenoss Labs Documentation, Releasezenperfsnmp run -v10 --device=Netbotz01 2>&1 | grep "zen.RRDUtil"We should see about 16 datapoints being written into RRD files. You’ll see sysUpTime, 14 interface datapoints andour custom snmpInTotalReqVars in there somewhere.Device ModelingThis section will cover creation of a custom Device subclass and modeling of device attributes.For purposes of this example, we’ll add a temp_sensor_count attribute to NetBotz devices. We’ll walk through addingthe attribute to the model, modeling it from the device, and displaying it in the overview screen for NetBotz devices.Starting in this section we’ll be working with a lot of files within the NetBotz ZenPack’s directory. To keep the pathnames short, I’ll assume the $ZP_DIR_TOP and $ZP_DIR environment variables have been set as follows.export ZP_DIR_TOP=$ZENHOME/ZenPacks/ZenPacks.training.NetBotzexport ZP_DIR=$ZP_DIR_TOP/ZenPacks/training/NetBotzCreate a Device SubclassA Device subclass should not be confused with a device class. In the previous section we created the /NetBotz deviceclass from the web interface. Creating a Device subclass means to extend the actual Python class of a Device object.You’d do this to add new attributes, methods or relationships to special device types.Use the following steps to create a NetBotzDevice class with a new attribute called temp_sensor_count.1. Create $ZP_DIR/NetBotzDevice.py with the following contents.from Products.ZenModel.Device import Deviceclass NetBotzDevice(Device):temp_sensor_count = None_properties = Device._properties + ({’id’: ’temp_sensor_count’, ’type’: ’int’},)We’re importing the base Device class then creating our own class called NetBotzDevice that extends it. We thenadd a default value of None for the new temp_sensor_count attribute. We then add to our base class’ _propertiescollection to describe the type of data we’ll be putting into our temp_sensor_count attribute. This _propertiessystem comes from Zope.2. Restart your zopectl process so the web interface can load our new module.3. Change the zPythonClass property on the /NetBotz device class toZenPacks.training.NetBotz.NetBotzDeviceThe zPythonClass property controls tells Zenoss what class of object to instantiate when a new device isadded to the device class. The default value for this property is blank. If the value is blank or invalid, Products.ZenModel.Devicewill be used.Note: Those with Python experience might notice that ZenPacks.training.NetBotz.NetBotzDevice is a module,not a class. This is true. The terminology is wrong. One can only assume that the creator of this functionalitydidn’t appreciate the difference between Python modules and classes.1.1. ZenPack Development Guide 18

Zenoss Labs Documentation, Release4. Reset the Python class of our existing device.Run zendmd and execute the following snippet.device = find("Netbotz01")print device.__class__You should see . We see this instead of the Python class we justcreated because the zPythonClass property is only used when a new device is created in a device class, or whena device is moved into a device class with a differing zPythonClass value.So we have two options for getting our NetBotz device to use the new Python class we created. We can eitherdelete the device and add it back, or move it to a different device class and back. Actually, there’s a third optionthat I use most frequently to solve this problem. I move it into the same device class using zendmd. Execute thefollowing snippet within zendmd to reset the device’s Python class.dmd.Devices.NetBotz.moveDevices(’/NetBotz’, ’Netbotz01’)commit()device = find("Netbotz01")print device.__class__Now you should see printed. This confirms that our Devicesubclass works, and that we’ve configure zPythonClass correctly for the /NetBotz device class.Find Temperature Sensor CountBefore we can write a modeler plugin to populate our new temp_sensor_count attribute, we need to figure out how toget the information. There are a few ways we could approach this. One way is to use that NETBOTZV2-MIB as areference to see if we can find anything about temperature sensors specifically.Zenoss comes with a tool called smidump that makes finding information in MIBs much easier. There are a lot ofMIB browser tools out there that make this even easier, but I primarily use a Mac and haven’t found very good optionsthere.Find temperature information in NETBOTZV2-MIB using the following command.smidump -f identifiers /usr/share/snmp/mibs/NETBOTZV2-MIB.mib | egrep -i tempYou should see the following in the output:NETBOTZV2-MIB tempSensorTable table 1.3.6.1.4.1.5528.100.4.1.1NETBOTZV2-MIB tempSensorEntry row 1.3.6.1.4.1.5528.100.4.1.1.1NETBOTZV2-MIB tempSensorId column 1.3.6.1.4.1.5528.100.4.1.1.1.1NETBOTZV2-MIB tempSensorValue column 1.3.6.1.4.1.5528.100.4.1.1.1.2NETBOTZV2-MIB tempSensorErrorStatus column 1.3.6.1.4.1.5528.100.4.1.1.1.3NETBOTZV2-MIB tempSensorLabel column 1.3.6.1.4.1.5528.100.4.1.1.1.4NETBOTZV2-MIB tempSensorEncId column 1.3.6.1.4.1.5528.100.4.1.1.1.5NETBOTZV2-MIB tempSensorPortId column 1.3.6.1.4.1.5528.100.4.1.1.1.6NETBOTZV2-MIB tempSensorValueStr column 1.3.6.1.4.1.5528.100.4.1.1.1.7NETBOTZV2-MIB tempSensorValueInt column 1.3.6.1.4.1.5528.100.4.1.1.1.8NETBOTZV2-MIB tempSensorValueIntF column 1.3.6.1.4.1.5528.100.4.1.1.1.9You’ll also see another node and a bunch of notification entries. These are related to SNMP traps, and not relevant towhat we’re interested in polling right now.What we see here is that there isn’t a single OID we can request that will tell us the number of temperature sensors.We’re going to have to do an snmpwalk of the table then count how many rows are in the response. Specifically1.1. ZenPack Development Guide 19

Zenoss Labs Documentation, Releasewe want to remember the name and OID for the row: tempSensorEntry. Due to the hierarchical nature of a MIBsrepresentation this is the most specific OID that will return the data we need.snmpwalk 127.0.1.113 1.3.6.1.4.1.5528.100.4.1.1.1You’ll see a lot of output that starts with:NETBOTZV2-MIB::tempSensorId.21604919 = STRING: nbHawkEnc_1_TEMPNETBOTZV2-MIB::tempSensorId.1095346743 = STRING: nbHawkEnc_0_TEMPNETBOTZV2-MIB::tempSensorId.1382714817 = STRING: nbHawkEnc_2_TEMP1NETBOTZV2-MIB::tempSensorId.1382714818 = STRING: nbHawkEnc_2_TEMP2NETBOTZV2-MIB::tempSensorId.1382714819 = STRING: nbHawkEnc_2_TEMP3NETBOTZV2-MIB::tempSensorId.1382714820 = STRING: nbHawkEnc_2_TEMP4NETBOTZV2-MIB::tempSensorId.1382714833 = STRING: nbHawkEnc_3_TEMP1NETBOTZV2-MIB::tempSensorId.1382714834 = STRING: nbHawkEnc_3_TEMP2NETBOTZV2-MIB::tempSensorId.1382714865 = STRING: nbHawkEnc_1_TEMP1NETBOTZV2-MIB::tempSensorId.1382714866 = STRING: nbHawkEnc_1_TEMP2NETBOTZV2-MIB::tempSensorId.1382714867 = STRING: nbHawkEnc_1_TEMP3NETBOTZV2-MIB::tempSensorId.1382714868 = STRING: nbHawkEnc_1_TEMP4NETBOTZV2-MIB::tempSensorId.2169088567 = STRING: nbHawkEnc_3_TEMPNETBOTZV2-MIB::tempSensorId.3242830391 = STRING: nbHawkEnc_2_TEMPWhat you’re seeing above is the tempSensorId column for all 14 rows in the tempSensorTable. Continuing on you willsee 14 rows for each of the other columns in the table.Create a Modeler PluginThe next step is to build a modeler plugin. A modeler plugin’s responsibility reach out into the world, gatherdata, and plug it into the attributes and relationships of our model classes. In this example, this means to makethe SNMP requests necessary to determine how many temperature sensors a NetBotz device has, and populate ourtemp_sensor_count attribute with the result.Use the following steps to create our modeler plugin.1. Make the directory that’ll contain our modeler plugin.mkdir -p $ZP_DIR/modeler/plugins/training/snmpNote that we’re using our ZenPack’s training namespace, then snmp. This is the recommended approach tomake it clear what protocol the modeler plugin will use, and to avoid our modeler plugin conflicting with onefrom someone else’s ZenPack.2. Create __init__.py or dunder-init files.touch $ZP_DIR/modeler/__init__.pytouch $ZP_DIR/modeler/plugins/__init__.pytouch $ZP_DIR/modeler/plugins/training/__init__.pytouch $ZP_DIR/modeler/plugins/training/snmp/__init__.pyThese empty __init__.py files are mandatory if we ever expect Python to import modules from these directories.3. Create $ZP_DIR/modeler/plugins/training/snmp/NetBotz.py with the following contents.from Products.DataCollector.plugins.CollectorPlugin import (SnmpPlugin, GetTableMap,)class NetBotz(SnmpPlugin):1.1. ZenPack Development Guide 20

Zenoss Labs Documentation, ReleasesnmpGetTableMaps = (GetTableMap(’tempSensorTable’, ’1.3.6.1.4.1.5528.100.4.1.1.1’, {’.1’: ’tempSensorId’,}),)def process(self, device, results, log):temp_sensors = results[1].get(’tempSensorTable’, {})return self.objectMap({’temp_sensor_count’: len(temp_sensors.keys()),})(a) Start by importing SnmpPlugin and GetTableMap from Zenoss. SnmpPlugin will handle all of the SNMPrequests for us and present the results in a format we can easily work with. GetTableMap will be used herebecause we need to request an SNMP table rather than specific OIDs.(b) Our NetBotz class extends SnmpPlugin. Note that the NetBotz class name must match the filename (modulename) of the modeler plugin.(c) By defining snmpGetTableMaps as a tuple or list on our class we can add a GetTableMap object that requeststhat 1.3.6.1.4.1.5528.100.4.1.1.1 row OID and specify that we only want to get the first (.1) columnand name it tempSensorId.(d) The process method will receive a two-element tuple containing the SNMP request results in the requestparameter. The first elememt, results[0], of this tuple would be any direct OID gets of which we didn’trequest any in this plugin. The second element, results[1] will contain a dictionary of the table results. Inthis case results[1] would look like the following.{}’tempSensorTable’: {’21604919’: ’nbHawkEnc_1_TEMP’,’1095346743’: ’nbHawkEnc_0_TEMP’,’1382714817’: ’nbHawkEnc_2_TEMP1’,’1382714818’: ’nbHawkEnc_2_TEMP2’,’1382714819’: ’nbHawkEnc_2_TEMP3’,’1382714820’: ’nbHawkEnc_2_TEMP4’,’1382714833’: ’nbHawkEnc_3_TEMP1’,’1382714834’: ’nbHawkEnc_3_TEMP2’,’1382714865’: ’nbHawkEnc_1_TEMP1’,’1382714866’: ’nbHawkEnc_1_TEMP2’,’1382714867’: ’nbHawkEnc_1_TEMP3’,’1382714868’: ’nbHawkEnc_1_TEMP4’,’2169088567’: ’nbHawkEnc_3_TEMP’,’3242830391’: ’nbHawkEnc_2_TEMP’,},(e) We then extract just the tempSensorTable results into temp_sensors to make the next return line a bit easierto understand.(f) We then return a dictionary that sets the temp_sensor_count key’s value to the number of keys intemp_sensors. Actually we return a dictionary that’s been wrapped in an ObjectMap by the modelerplugin’s objectMap utility method.The process method within all modeler plugins must return one of the following types of data.• None (makes no changes to the model)1.1. ZenPack Development Guide 21

Zenoss Labs Documentation, Release• ObjectMap (to apply directly to the device that’s being modeled)• RelationshipMap (to apply to a relationship within the device)• A list containing zero or more ObjectMap and/or RelationShipMap objects.An ObjectMap is simply a dict wrapped with some meta-data. A RelationshipMap is a list wrapped withsome meta-data and containing zero or more ObjectMap instances.4. Restart zopectl and zenhub to load the new module.5. Add our new training.snmp.NetBotz modeler plugin to the list of modeler plugins for the /NetBotz device class.Test the Modeler Plugin1. Remodel the NetBotz device.Now that we’ve created and enabled a basic modeler plugin, we should test it.You can do this from the web interface, but I usually use the command line because it can be easier to work withif further debugging is necessary.zenmodeler run --device=Netbotz012. Execute the following snippet in zendmd.device = find("Netbotz01")print device.temp_sensor_countYou should see 14 printed as the number of temperature sensors.Create the APIThe Zenoss web interface is a consumer of the Zenoss JSON API. This is now relevant to you because you have tomake sure that you extend the API to allow the web interface to know about the new class of object you’ve created.Now that you’re creating custom classes, you’ll need to instruct Zenoss how your Python objects should be translatedwhen the web interface or other API user requests information about them. This is a three part process that involvescreating an IInfo interface for your class, an Info adapter, and finally registering them for use.Create the IInfo InterfaceThis is where we define the public interface for the NetBotzDevice class we created.1. Create $ZP_DIR/interfaces.py with the following contents.from Products.Zuul.form import schemafrom Products.Zuul.interfaces.device import IDeviceInfofrom Products.Zuul.utils import ZuulMessageFactory as _tclass INetBotzDeviceInfo(IDeviceInfo):temp_sensor_count = schema.Int(title=_t(’Number of Temperature Sensors’))(a) Start by importing schema. This is how we specify the types of the attributes.(b) We then import IDeviceInfo. This is the Info interface for standard devices. By extending it, we get thestandard device API for free.(c) ZuulMessageFactory as _t allows any strings we wrap in _t() to have translations to other languagesprovided for them.(d) Next we create our Interface class, INetBotzDeviceInfo. Note that it’s our model class name, NetBotzDevice,prefixed with I and suffixed with Info. This is a best practice that can make it easier to figure outwhat’s going on for people later.1.1. ZenPack Development Guide 22

Zenoss Labs Documentation, Release(e) Finally we define our single new attribute, temp_sensor_count. It’s a number with no decimal precision sowe use Int.Create the Info Adapter Now that we’ve defined the Info interface we have to provide the implementation for it.This is what will be returned when the web interface asks for the Info for one of our NetBotzDevice objects.1. Create $ZP_DIR/info.py with the following contents.from zope.interface import implementsfrom Products.Zuul.infos import ProxyPropertyfrom Products.Zuul.infos.device import DeviceInfofrom ZenPacks.training.NetBotz.interfaces import INetBotzDeviceInfoclass NetBotzDeviceInfo(DeviceInfo):implements(INetBotzDeviceInfo)temp_sensor_count = ProxyProperty(’temp_sensor_count’)(a) Import the symbols we’ll need to implement our Info adapter.(b) Just like with the interface, we can extend DeviceInfo to get all of the standard Device functionality forfree. Note that we’re again using a best practice naming convention for our Info adapter. It should be thename of the model class suffixed with Info.(c) The implements line tells the system that this is an implementation of the interface we previously defined.(d) We then use the helpful ProxyProperty method to provide the temp_sensor_count directly from the attributeby the same name on the actual model object.Using ProxyProperty in this way is equivalent to the following.@propertydef temp_sensor_count(self):return self._adapted.temp_sensor_count@temp_sensor_count.setterdef temp_sensor_count(self, value):self._adapted.temp_sensor_count = valueAs you can see, ProxyProperty is shorter and cleaner even if you were only interested in the getter.Register the Info Adapter Now that you’ve defined your API interface and implemented the adapter to the modelclass you have to register them with the system. Otherwise they won’t be used, and the system will use the standardIDeviceInfo interface and DeviceInfo adapter because they’re the next best thing.Follow these steps to register your API interface and adapter.1. Create $ZP_DIR/configure.zcml with the following contents.1.1. ZenPack Development Guide 23

Zenoss Labs Documentation, Release(a) We open with a standard XML header and declaring that the default XML namespace (xmlns) for thedocument will be Zope’s main namespace.(b) Registering the Info adapter is the only thing we need in here at this point. We must specify the interfacethe adapter provides, the type of object that it provides the interface for, and finally the adapter factoryitself. This is boilerplate stuff that you’ll see in a lot of other areas in Zenoss and ZenPack development.Test the API We can now test the API using zendmd. Be sure to restart zendmd after making changes to interfaces.py,info.py, or configure.zcml.1. Execute the following snippet in zendmd.from Products.Zuul.interfaces import IInfodevice = find("Netbotz01")device_info = IInfo(device)print device_info.temp_sensor_count(a) Calling IInfo(device) will return the best IInfo adapter for device which is a NetBotzDevice instancein this case.You should see 14 printed if everything worked.Change the Device OverviewWe’ve come a long way, but aside from going into zendmd to test that the API works, we don’t have much to show forit. The next step will be to show the number of temperature sensors to users of the web interface. We’ll replace theMemory/Swap field in the top-left box of the device overview page with the count of temperature sensors.Follow these steps to customize the device Overview page.1. Create a directory to store our ZenPack’s JavaScript.mkdir -p $ZP_DIR/browser/resources/js2. Create __init__.py or dunder-init files.touch $ZP_DIR/browser/__init__.py3. Create $ZP_DIR/browser/resources/js/NetBotzDevice.js with the following contents.Ext.onReady(function() {var DEVICE_OVERVIEW_ID = ’deviceoverviewpanel_summary’;Ext.ComponentMgr.onAvailable(DEVICE_OVERVIEW_ID, function(){var overview = Ext.getCmp(DEVICE_OVERVIEW_ID);overview.removeField(’memory’);});});overview.addField({name: ’temp_sensor_count’,fieldLabel: _t(’# Temperature Sensors’)});(a) Wait for Ext to be ready.1.1. ZenPack Development Guide 24

Zenoss Labs Documentation, Release(b) Find the overview summary panel (top-left on Overview page)(c) Remove the memory field.(d) Add our temp_sensor_count field.Zenoss uses ExtJS as its JavaScript framework. You can find more in ExtJS’s documentation about manipulatingobjects in this way.4. Create $ZP_DIR/browser/configure.zcml with the following contents.(a) We open with a standard XML header and declaring that the default XML namespace (xmlns) for thedocument will be Zope’s browser namespace.(b) Next we register the resourceDirectory. This allows files within your ZenPack’s /browser/resources/ directoryto be served from URLs such as http://zenoss.example.com/++resource++netbotz/filename.js.(c) Next we use a viewlet register a JavaScript snippet. You can see that we’re referencing a URL within theresourceDirectory and limiting the snippet to only appear on pages where the context is a NetBotzDevice.This is the important part that keeps our customizations local to NetBotz devices.5. Edit $ZP_DIR/configure.zcml. Add the following section before the closing .This makes Zenoss load our browser/configure.zcml on startup.Test the Device Overview That’s it. We can restart zopectl and navigate to our NetBotz device’s overview page inthe web interface. You should see # Temperature Sensors label with a value of 14 at the bottom of the top-leftpanel.Component ModelingThis section will cover creation of a custom DeviceComponent subclass, creation of a relationship to our NetBotDeviceclass, and modeling of the components to fill the relationship.In the Device Modeling section we added a temp_sensor_count attribute to our NetBotz devices. This isn’t very useful.It would be more useful to monitor the temperature being reported by each of these sensors. So that’s what we’ll do.1.1. ZenPack Development Guide 25

Zenoss Labs Documentation, ReleaseModeling each sensor as a component allows Zenoss to automatically discover and monitor sensors regardless of howmany a particular device has.Find Temperature Sensor AttributesIn the Device Modeling section we used smidump to extract temperature sensor information from NETBOTZV2-MIB.This will be even more applicable as we decide what attributes and metrics are available on each sensor. Let’s usesmidump and snmpwalk for a refresher on what’s available.Find temperature information in NETBOTZV2-MIB using the following command.smidump -f identifiers /usr/share/snmp/mibs/NETBOTZV2-MIB.mib | egrep -i tempYou should see the following in the output:NETBOTZV2-MIB tempSensorTable table 1.3.6.1.4.1.5528.100.4.1.1NETBOTZV2-MIB tempSensorEntry row 1.3.6.1.4.1.5528.100.4.1.1.1NETBOTZV2-MIB tempSensorId column 1.3.6.1.4.1.5528.100.4.1.1.1.1NETBOTZV2-MIB tempSensorValue column 1.3.6.1.4.1.5528.100.4.1.1.1.2NETBOTZV2-MIB tempSensorErrorStatus column 1.3.6.1.4.1.5528.100.4.1.1.1.3NETBOTZV2-MIB tempSensorLabel column 1.3.6.1.4.1.5528.100.4.1.1.1.4NETBOTZV2-MIB tempSensorEncId column 1.3.6.1.4.1.5528.100.4.1.1.1.5NETBOTZV2-MIB tempSensorPortId column 1.3.6.1.4.1.5528.100.4.1.1.1.6NETBOTZV2-MIB tempSensorValueStr column 1.3.6.1.4.1.5528.100.4.1.1.1.7NETBOTZV2-MIB tempSensorValueInt column 1.3.6.1.4.1.5528.100.4.1.1.1.8NETBOTZV2-MIB tempSensorValueIntF column 1.3.6.1.4.1.5528.100.4.1.1.1.9Let’s now use snmpwalk to see what these values look like on our NetBotz device.snmpwalk 127.0.1.113 1.3.6.1.4.1.5528.100.4.1.1.1You should see a lot of output that begins with the following:NETBOTZV2-MIB::tempSensorId.21604919 = STRING: nbHawkEnc_1_TEMPNETBOTZV2-MIB::tempSensorId.1095346743 = STRING: nbHawkEnc_0_TEMPNETBOTZV2-MIB::tempSensorId.1382714817 = STRING: nbHawkEnc_2_TEMP1NETBOTZV2-MIB::tempSensorId.1382714818 = STRING: nbHawkEnc_2_TEMP2Note the 21604919 in the first response. This is the SNMP index of the first temperature sensor, or the first row in thetable. I like to then restrict my snmpwalk results to only show this row with a command like the following.snmpwalk 127.0.1.113 1.3.6.1.4.1.5528.100.4.1.1.1 | grep "\.21604919 ="Which will show us the value of each column for that one temperature sensor:NETBOTZV2-MIB::tempSensorId.21604919 = STRING: nbHawkEnc_1_TEMPNETBOTZV2-MIB::tempSensorValue.21604919 = INTEGER: 265NETBOTZV2-MIB::tempSensorErrorStatus.21604919 = INTEGER: normal(0)NETBOTZV2-MIB::tempSensorLabel.21604919 = STRING: TemperatureNETBOTZV2-MIB::tempSensorEncId.21604919 = STRING: nbHawkEnc_1NETBOTZV2-MIB::tempSensorPortId.21604919 = STRING:NETBOTZV2-MIB::tempSensorValueStr.21604919 = STRING: 26.500000NETBOTZV2-MIB::tempSensorValueInt.21604919 = INTEGER: 26NETBOTZV2-MIB::tempSensorValueIntF.21604919 = INTEGER: 79Now we have everything we should need to make decisions about what attributes we should model for our sensors andwhich would better be collected as datasources to have thresholds applied and plotted over time on graphs.My initial thoughts would be to model the following as attributes.1.1. ZenPack Development Guide 26

Zenoss Labs Documentation, Release• tempSensorId• tempSensorEncId (enclosure ID)• tempSensorPortIdI would then want to collect tempSensorValueStr as a datasource because it offers the best precision. Zenoss is capableof handling numeric strings so we don’t have to collect tempSensorValue and divide it by 10 like other systems might.Create a DeviceComponent SubclassUse the following steps to create a TemperatureSensor class with the attributes discovered above.1. Create $ZP_DIR/TemperatureSensor.py with the following contents.from Products.ZenModel.DeviceComponent import DeviceComponentfrom Products.ZenModel.ManagedEntity import ManagedEntityfrom Products.ZenModel.ZenossSecurity import ZEN_CHANGE_DEVICEfrom Products.ZenRelations.RelSchema import ToManyCont, ToOneclass TemperatureSensor(DeviceComponent, ManagedEntity):meta_type = portal_type = ’NetBotzTemperatureSensor’enclosure = Noneport = None_properties = ManagedEntity._properties + ({’id’: ’enclosure’, ’type’: ’string’},{’id’: ’port’, ’type’: ’string’},)_relations = ManagedEntity._relations + ((’sensor_device’, ToOne(ToManyCont,’ZenPacks.training.NetBotz.NetBotzDevice’,’temperature_sensors’,)),)factory_type_information = ({’actions’: ({’id’: ’perfConf’,’name’: ’Template’,’action’: ’objTemplates’,’permissions’: (ZEN_CHANGE_DEVICE,),},),},)def device(self):return self.sensor_device()def getRRDTemplateName(self):return ’TemperatureSensor’(a) Start by importing the symbols we’ll need.(b) Define the TemperatureSensor class. This name must match the filename.(c) Extend (inherit from) both the DeviceComponent and ManagedEntity classes.1.1. ZenPack Development Guide 27

Zenoss Labs Documentation, ReleaseDeviceComponent provides the core functionality that’s needed to associate our temperature sensor with adevice.ManagedEntity provides base functionality for objects that are to be monitored. This base class providesattributes such as snmpindex, monitor and productionState.(d) Next we set both the meta_type and portal_type of the class to NetBotzTemperatureSensor. Thisis used as the friendly name for the type of our object in various places in the web interface such as theglobal search. meta_type and portal_type should always be the same. They both exist for backwardscompatibility reasons.(e) Next we add the enclosure and port attributes in the same way as we did for our NetBotzDevice class.Note: Despite noting above that we always wanted to model the tempSensorId attribute, we aren’t addingan attribute for it here. This is because DeviceComponent already has both an id and title attribute thatwherein we can store the value of tempSensorId.(f) Next we have to setup _relations for our component type. This wasn’t mandatory for our NetBotzDeviceclass because devices automatically get a containing relationship in their device class. Component’s don’tautomatically get a containing relationship in their device because it can’t automatically be known whichone of their device’s relations they belong in, or if they’re perhaps nested under another component.The single relationship we’re adding can be expressed in English as “TemperatureSensor has a relationnamed sensor_device that refers to a single object of the ZenPacks.training.NetBotz.NetBotzDevice type.”We also describe the other side of this relationship as “NetBotzDevice has a relation named temperature_sensorsthat contains zero or more objects of the TemperatureSensor type.”(g) The factory_type_information section is boilerplate that should be used for any component types to whichyou will bind monitoring templates.(h) We must define the device method for our component. Every component must provide this method. Wecan find our device by calling the sensor_device relation.(i) Finally we override the getRRDTemplateName method. This method allows us to control the name of themonitoring template that will be automatically bound to each temperature sensor.2. Edit $ZP_DIR/NetBotzDevice.py to add the following to the bottom._relations = Device._relations + ((’temperature_sensors’, ToManyCont(ToOne,’ZenPacks.training.NetBotz.TemperatureSensor’,’sensor_device’,)),)You’ll also need to add the following imports to the top of the file.from Products.ZenRelations.RelSchema import ToManyCont, ToOneIt is mandatory that this relationship definition exist, and be an exact mirror of the definition on the other side.Note: See the Relationship Types section for more information on relationships.Test TemperatureSensor Class With our component class defined and relationships setup we can use zendmd tomake sure we didn’t make any mistakes. Execute the following snippet in zendmd.1.1. ZenPack Development Guide 28

Zenoss Labs Documentation, Releasefrom ZenPacks.training.NetBotz.TemperatureSensor import TemperatureSensorsensor = TemperatureSensor(’test_sensor_01’)device = find("Netbotz01")device.temperature_sensors._setObject(sensor.id, sensor)sensor = device.temperature_sensors._getOb(sensor.id)print sensorprint sensor.device()You’ll most likely get the following error when executing the above snippet:Traceback (most recent call last):File "", line 1, in AttributeError: temperature_sensorsThis error is indicating that we have no temperature_sensors relationship on the device object. This would seeminglymake no sense because we just added it to NetBotzDevice.py above. The key here is that existing objects like theNetbotz01 device don’t automatically get new relationships. We have to either delete the device and add it again, orexecute the following in zendmd to create the newly-defined relationship.device.buildRelations()commit()Now you can go back and run the original snippet again. You should see the name of the sensor and device objectsprinted if everything worked as planned.Update the Modeler PluginAs with the NetBotzDevice class, the next step after creating our model class is to populate it with a modeler plugin.We could create a new modeler plugin to only capture the temperature sensor components, but we’ll update the NetBotzmodeler plugin we previously created to model the sensors instead.1. Edit $ZP_DIR/modeler/plugins/training/snmp/NetBotz.py and replace its contents with thefollowing.from Products.DataCollector.plugins.CollectorPlugin import (SnmpPlugin, GetTableMap,)class NetBotz(SnmpPlugin):relname = ’temperature_sensors’modname = ’ZenPacks.training.NetBotz.TemperatureSensor’snmpGetTableMaps = (GetTableMap(’tempSensorTable’, ’1.3.6.1.4.1.5528.100.4.1.1.1’, {’.1’: ’tempSensorId’,’.5’: ’tempSensorEncId’,’.6’: ’tempSensorPortId’,}),)def process(self, device, results, log):temp_sensors = results[1].get(’tempSensorTable’, {})1.1. ZenPack Development Guide 29

Zenoss Labs Documentation, Releaserm = self.relMap()for snmpindex, row in temp_sensors.items():name = row.get(’tempSensorId’)if not name:log.warn(’Skipping temperature sensor with no name’)continuerm.append(self.objectMap({’id’: self.prepId(name),’title’: name,’snmpindex’: snmpindex.strip(’.’),’enclosure’: row.get(’tempSensorEncId’),’port’: row.get(’tempSensorPortId’),}))return rmLet’s take a closer look at how we changed the modeler plugin.(a) We added relname and modname as class attributes.These two settings control the meta-data that will automatically be set when the self.relMap andself.objectMap methods are called in the process method.Setting relname to temperature_sensors will cause the self.relMap call to create a RelationshipMapthat will be applied to the temperature_sensors relationship defined on the NetBotzDevice object.Setting modname to ZenPacks.training.NetBotz.TemperatureSensor will cause theself.objectMap calls in the process method to create ObjectMap instances that will be turned into instancesof our TemperatureSensor class.(b) We’re now requesting the tempSensorEncId and tempSensorPortId columns be returned in the SNMP tablerequest results. We’ll use these to populate their corresponding fields on the TemperatureSensor class.(c) Most of the process method has been changed.We’re now creating a RelationshipMap and appending an ObjectMap to it for each temperature sensor inthe results. We use the self.relMap and self.objectMap utility methods to make this easier.2. Restart zopectl and zenhub to load the changed module.Test the Modeler Plugin We already added the training.snmp.NetBotz modeler plugin the the /NetBotz device classin an earlier exercise. So we only need to run zenmodeler to test the temperature sensor modeling updates.1. Run zenmodeler run --device=Netbotz01We should see Changes in configuration applied near the end of zenmodeler’s output. The changes referred toshould be 14 temperature sensor objects being created and added to the device’s temperature_sensors relationship.2. Execute the following snipped in zendmd.device = find("Netbotz01")pprint(device.temperature_sensors())You should see a list of all 14 temperature sensors printed. We can test in more depth by validating that all ofour modeled attributes were set properly.for sensor in device.temperature_sensors():print "%17s: %-17s %-11s %-11s %-11s" % (1.1. ZenPack Development Guide 30

Zenoss Labs Documentation, Releasesensor.id, sensor.title, sensor.snmpindex, sensor.enclosure,sensor.port)Create the APIWe must now create and test the API for our new TemperatureSensor class much like we did for the NetBotzDeviceAPI. There’s a slight difference in the base interfaces and classes that we use because TemperatureSensor is aDeviceComponent subclass instead of a Device subclass.1. Create the IInfo interface.To create the public interface for the TemperatureSensor class we add the following class to the end of$ZP_DIR/interfaces.py.class ITemperatureSensorInfo(IComponentInfo):enclosure = schema.TextLine(title=_t(’Sensor Enclosure ID’))port = schema.TextLine(title=_t(’Sensor Port ID’))We must also add an import for our base interface, IComponentInfo.from Products.Zuul.interfaces.component import IComponentInfoNote: The attributes you add to your IComponentInfo interface control the fields that will be displayed whenthe Details for a component are viewed in the web interface.2. Create the Info adapter.To create the Info adapter for the TemperatureSensor class we add the following class to the end of$ZP_DIR/info.py.class TemperatureSensorInfo(ComponentInfo):implements(ITemperatureSensorInfo)enclosure = ProxyProperty(’enclosure’)port = ProxyProperty(’port’)We must also add an import for our base class, ComponentInfo.from Products.Zuul.infos.component import ComponentInfoAdditionally, we need to add an import of ITemperatureSensorInfo from our own interfaces module. This willrequire an update to the existing line.from ZenPacks.training.NetBotz.interfaces import (INetBotzDeviceInfo,ITemperatureSensorInfo,)3. Register the Info adapter.To register the Info adapter for TemperatureSensor we add the following before the browser include in$ZP_DIR/configure.zcml.1.1. ZenPack Development Guide 31

Zenoss Labs Documentation, ReleaseTest the API We can now test the API using zendmd. Be sure to restart zendmd after making changes to interfaces.py,info.py, or configure.zcml.1. Execute the following snippet in zendmd.from Products.Zuul.interfaces import IInfodevice = find("Netbotz01")sensor = device.temperature_sensors._getOb(’nbHawkEnc_1_TEMP1’)sensor_info = IInfo(sensor)print "id: %s, enclosure: %s, port: %s" % (sensor_info.id, sensor_info.enclosure, sensor_info.port)You should see the following line printed:id: nbHawkEnc_1_TEMP1, enclosure: nbHawkEnc_1, port: nbHawkEnc_1_DIN1Add Component Display JavaScriptTypically when a new type of component like TemperatureSensor is added, you will want to add two JavaScriptelements to provide for a more attractive display in the web interface.1. registerName to change TemperatureSensor to Temperature Sensors in the device’s left navigation pane underComponents.Add the following to $ZP_DIR/browser/resources/js/NetBotzDevice.js.(function(){var ZC = Ext.ns(’Zenoss.component’);ZC.registerName(’NetBotzTemperatureSensor’,_t(’Temperature Sensor’),_t(’Temperature Sensors’));})();(a) The JavaScript code is wrapped in an anonymous function to keep it out of the global namespace. Ideallyall ZenPack JavaScript code should be wrapped in this way.The wrapping is done using the first and last lines of the JavaScript snippet above.(b) Next we get a handle to the Zenoss.component ExtJS namespace and store it in the ZA variable.(c) Finally we actually register the name. The parameters to registerName are:i. meta_type of the class we’re registering names for.ii. Singular form of the human-friendly name of the class.iii. Plural form of the human-friendly name of the class.2. Create a ComponentGridPanel.A custom ComponentGridPanel allows us to customize the grid that will display in the top-right panel when weselect Temperature Sensors from our device’s component tree. Typically we customize what columns we wantto appear, how we want those columns to appear, and how they should be sorted.Add the following to $ZP_DIR/browser/resources/js/NetBotzDevice.js beneath theZC.registerName but before the })(); that ends the anonymous function.1.1. ZenPack Development Guide 32

Zenoss Labs Documentation, ReleaseZC.NetBotzTemperatureSensorPanel = Ext.extend(ZC.ComponentGridPanel, {constructor: function(config) {config = Ext.applyIf(config||{}, {componentType: ’NetBotzTemperatureSensor’,autoExpandColumn: ’name’,sortInfo: {field: ’name’,direction: ’ASC’},fields: [{name: ’uid’},{name: ’name’},{name: ’status’},{name: ’severity’},{name: ’usesMonitorAttribute’},{name: ’monitor’},{name: ’monitored’},{name: ’locking’},{name: ’enclosure’},{name: ’port’}],columns: [{id: ’severity’,dataIndex: ’severity’,header: _t(’Events’),renderer: Zenoss.render.severity,sortable: true,width: 50},{id: ’name’,dataIndex: ’name’,header: _t(’Name’),sortable: true},{id: ’enclosure’,dataIndex: ’enclosure’,header: _t(’Enclosure ID’),sortable: true,width: 120},{id: ’port’,dataIndex: ’port’,header: _t(’Port ID’),sortable: true,width: 120},{id: ’monitored’,dataIndex: ’monitored’,header: _t(’Monitored’),renderer: Zenoss.render.checkbox,sortable: true,width: 70},{id: ’locking’,dataIndex: ’locking’,header: _t(’Locking’),renderer: Zenoss.render.locking_icons,width: 651.1. ZenPack Development Guide 33

Zenoss Labs Documentation, Release});}]});}ZC.NetBotzTemperatureSensorPanel.superclass.constructor.call(this, config);Ext.reg(’NetBotzTemperatureSensorPanel’, ZC.NetBotzTemperatureSensorPanel);This is a length snippet of JavaScript. Let’s go through it section by section.(a) Define a new ExtJS class named NetBotzTemperatureSensorPanel.We define this class within the Zenoss.component namespace by using the ZC variable we created in theprevious step. Just like we’ve been defining our Python classes by extending existing Zenoss Pythonclasses, we do the same with JavaScript classes within the ExtJS framework. In this case we’re extendingthe ComponentGridPanel class.(b) Create the NetBotzTemperatureSensorPanel constructor.Our constructor method only does two things. First we override the config variable. Then we close bycalling our superclass’ constructor. Our superclass is the ComponentGridPanel we extended.(c) Override config variable.This is where all of the interesting stuff happens, and where we’ll be making changes. Let’s look at eachof the fields we’re changing within config.i. componentTypeMust match the meta_type on our TemperatureSensor Python class.ii. autoExpandColumnOne of the following columns can be picked to automatically expand to use the remaining space withinthe user’s web browser. Typically this is set to the name field, but it can be useful to choose a differentfield if the name is a well-known length and another field is more variable.iii. sortInfoControls which field and direction that the grid will be sorted by when it initially appears. This is anoptional field and defaults to sort by name in ascending order if not set. So in this example the sortInfofield could have been left out.iv. fieldsControls which fields will be requested from the TemperatureSensorInfo API adapter on the server todisplay a column. You can request any attribute that you made available on the Info adapter. This evenincludes fields that are not present in the IInfo interface.You must be sure to include fields needed to display all of the columns specified below. For consistencyI recommend having the following minimum set plus any that are specific to your componenttype.• uid• name• status• severity• usesMonitorAttribute1.1. ZenPack Development Guide 34

Zenoss Labs Documentation, Release• monitor• monitored• lockingv. columnsControls the visual display of columns in the grid. Each column can specify the following fields.• idA unique identifier for the field. Typically this is set to the same value as dataIndex.• dataIndexReference to one of the items from fields above.• headerText that will appear in the column’s header.• rendererJavaScript function that will be used to render the column’s data. This is an optional field and willdefault to displaying the data’s natural string representation.You can find the standard renderer choices in the following file:$ZENHOME/Products/ZenUI3/browser/resources/js/zenoss/Renderers.js• sortableWhether or not the user can choose to sort the grid by this column.• widthThe width in pixels of the column. This is an optional field, but I highly recommend setting in onall columns except for the column that’s referenced in autoExpandColumn.Test the Component DisplayWe test our component display JavaScript by looking at it in the web interface.If you’re running zopectl in the foreground, it is not necessary to restart it after making changes to existing files withinour resourceDirectory. However, you will have to force your browser to do a full refresh to make sure your browsercache isn’t interfering. This can typically be done by holding the SHIFT key while clicking the refresh button or typingthe refresh shortcut.I recommend having the browser’s JavaScript console open while testing so you don’t miss any errors.Component MonitoringThis section covers monitoring component metrics using SNMP. I assume that you’ve completed the ComponentModeling steps and now have temperature sensor components modeled for the NetBotz device. Currently there willbe no graphs for these temperature sensors.We will add collection, thresholding and graphing for the temperature monitored by each sensor.1.1. ZenPack Development Guide 35

Zenoss Labs Documentation, ReleaseFind the SNMP OIDIn the Component Modeling section we used smidump and snmpwalk to find which values would be useful to model,and which would be useful to monitor. We found tempSensorValue to be the best OID to use for monitoring a sensor’scurrent temperature.Let’s use snmpwalk again to see what tempSensorValue looks like for all of the sensors on our NetBotz device.snmpwalk 127.0.1.113 NETBOTZV2-MIB::tempSensorValueStrThis gives of the current temperature (in celsius) for each sensor:NETBOTZV2-MIB::tempSensorValueStr.21604919 = STRING: 26.500000NETBOTZV2-MIB::tempSensorValueStr.1095346743 = STRING: 27.000000NETBOTZV2-MIB::tempSensorValueStr.1382714817 = STRING: 22.100000NETBOTZV2-MIB::tempSensorValueStr.1382714818 = STRING: 21.100000NETBOTZV2-MIB::tempSensorValueStr.1382714819 = STRING: 19.600000NETBOTZV2-MIB::tempSensorValueStr.1382714820 = STRING: 19.900000NETBOTZV2-MIB::tempSensorValueStr.1382714833 = STRING: 20.500000NETBOTZV2-MIB::tempSensorValueStr.1382714834 = STRING: 20.100000NETBOTZV2-MIB::tempSensorValueStr.1382714865 = STRING: 19.700000NETBOTZV2-MIB::tempSensorValueStr.1382714866 = STRING: 20.500000NETBOTZV2-MIB::tempSensorValueStr.1382714867 = STRING: 20.100000NETBOTZV2-MIB::tempSensorValueStr.1382714868 = STRING: 20.000000NETBOTZV2-MIB::tempSensorValueStr.2169088567 = STRING: 26.600000NETBOTZV2-MIB::tempSensorValueStr.3242830391 = STRING: 27.400000As we go on to add a monitoring template below, we’ll need to know what OID to poll to collect this value. The keyto determining this for components with an snmpindex attribute like TemperatureSensor has is to find the OID for thevalues above and remove the SNMP index from the end of it. Let’s use snmptranslate to do this.snmptranslate -On NETBOTZV2-MIB::tempSensorValueStrThis results in the following output:.1.3.6.1.4.1.5528.100.4.1.1.1.7This OID (minus the leading .) is what we’ll need.Add a Monitoring TemplateIn the Component Modeling section we added a getRRDTemplateName method to our TemperatureSensor class. Wemade this method return TemperatureSensor. This means that each temperature sensor will have a template by thisname automatically bound to it.This makes life easy when adding a monitoring template to be used. All we have to do is create a monitoring templatenamed TemperatureSensor in the /NetBotz device class.1. Navigate to Advanced -> Monitoring Templates.2. Add a template.(a) Click the + in the bottom-left of the template list.(b) Set Name to TemperatureSensor(c) Set Template Path to /NetBotz(d) Click SUBMIT3. Add a data source.1.1. ZenPack Development Guide 36

Zenoss Labs Documentation, Release(a) Click the + at the top of the Data Sources panel.(b) Set Name to tempSensorValueStr(c) Set Type to SNMP(d) Click SUBMIT(e) Double-click to edit the tempSensorValueStr data source.(f) Set OID to 1.3.6.1.4.1.5528.100.4.1.1.1.7(g) Click SAVE5. Add a threshold.(a) Click the + at the top of the Thresholds panel.(b) Set Name to high temperature(c) Set Type to MinMaxThreshold(d) Click ADD(e) Double-click to edit the high temperature threshold.(f) Move the datapoint to the list on the right.(g) Set Maximum Value to 32(h) Set Event Class to /Environ(i) Click SAVE6. Add a graph.(a) Click the + at the top of the Graph Definitions panel.(b) Set Name to Temperature(c) Click SUBMIT(d) Double-click to edit the Temperature graph.(e) Set Units to degrees c.(f) Click SUBMIT7. Add a graph point.(a) Click to select the Temperature graph.(b) Choose Manage Graph Points from the gear menu.(c) Choose Data Point from the + menu.(d) Choose tempSensorValueStr then click SUBMIT(e) Double-click to edit the tempSensorValueStr graph point.(f) Set Name to Temperature(g) Set Format to %7.2lf(h) Click SAVE then SAVE again.1.1. ZenPack Development Guide 37

Zenoss Labs Documentation, ReleaseTest Monitoring Template You can now refer back to the Test Monitoring Template section of Device Monitoringfor using zenperfsnmp to test the data point collection aspect of your monitoring template.You can verify that your monitoring template is getting bound to each temperature sensor properly by navigating to oneof the temperature sensors in the web interface and choosing Templates from it’s Display drop-down box. Furthermore,you can verify that your Temperature graph is shown when choosing Graphs from the temperature sensor’s Displaydrop-down.SNMP TrapsThis section covers how to handle SNMP traps.Zenoss will accept SNMP traps from your devices as soon as you configure those devices to send traps to your Zenossserver. The zentrap daemon will listen to the standard SNMP trap port of 162/udp and create an event for every trapthat it receives.However, without you giving Zenoss more information about the contents of those traps, the events will containnumeric OIDs and be nearly impossible for a human to decipher.Importing MIBsLet’s import the NETBOTZV2-MIB that we’ve been working with through these examples.1. Copy the MIB to Zenoss’ site-specific MIB directory.cp /usr/share/snmp/mibs/NETBOTZV2-MIB.mib $ZENHOME/share/mibs/site/2. Import the MIB directory.zenmib runFrom which we should get the following output:Found 1 MIBs to import.Parsed 214 nodes and 256 notifications from NETBOTZV2-MIBLoaded MIB NETBOTZV2-MIB into the DMDLoaded 1 MIB file(s)3. Add the imported MIB to the NetBotz ZenPack.(a) Navigate to Advanced -> MIBs in the web interface.(b) Select NETBOTZV2-MIB.(c) Choose Add to ZenPack from the gear menu at the bottom of the list.(d) Choose the ZenPacks.training.NetBotz then click SUBMIT.Simulating SNMP TrapsTo more easily configure and test Zenoss’ trap handling, it’s useful to know how to simulate SNMP traps. Thealternative is breaking your real devices in various ways and hoping to be able to get the device to send all of the trapsyou need. This isn’t always possible.Let’s start by picking an SNMP trap to simulate.1. Navigate to Advanced -> MIBs in the web interface.2. Choose NETBOTZV2-MIB from the list of MIBs.1.1. ZenPack Development Guide 38

Zenoss Labs Documentation, Release3. Choose Traps from the drop-down box in the middle of the right panel.4. Choose netBotzTempTooHigh in the list of traps.We’ll now see information about this trap in the bottom-right panel. The first thing to note is the OID. This is all weneed to get started.Send a Simple Trap1. Make sure zentrap is running.Use the following steps to get your feet wet sending a basic trap.As detailed in Development Environment section, I prefer to only run the processes necessary for debugging. SoI would run zentrap in the foreground in a new terminal session with the following command.zentrap run -v10 --cycle2. Run the following snmptrap command.snmptrap localhost 0 NETBOTZV2-MIB::netBotzTempTooHighIf you were running zentrap in the foreground you should have seen a log message similar to the following aftersending the trap. This is Zenoss using the MIB to turn the trap into an event:Queued event (total of 1) {’sysUpTime.0’: 0L,’firstTime’: 1343072200.36731,’severity’: 3,’eventClassKey’: ’netBotzTempTooHigh’,’oid’: ’1.3.6.1.4.1.5528.100.10.2.1.0.2’,’component’: ’’,’community’: ’public’,’summary’: ’snmp trap netBotzTempTooHigh’,’eventGroup’: ’trap’,’sysUpTime’: 0L,’device’: ’127.0.0.1’,’lastTime’: 1343072200.36731,’monitor’: ’localhost’}You can see how Zenoss has maintained the numeric OID in the event’s oid field. It has also decoded it tonetBotzTempTooHigh using the MIB we imported and used that value in the eventClassKey and summary fields.3. Find this netBotzTempTooHigh event in web interface’s event console.Send a Full Trap Now that we’ve proved out a simple trap, we should add variable bindings or varbinds to the trap.If you look at the netBotzTempTooHigh trap in the Zenoss web interface’s MIB explorer again, you’ll see that there’san extensive list of Objects associated with the trap definition. These are variable bindings.A variable binding allows the device sending the SNMP trap to attach additional information to the trap. In thisexample, one of the variable bindings for the netBotzTempTooHigh trap is netBotzV2TrapSensorID. This will give usa way to know which one of the sensors has exceeded it’s high temperature threshold.1. Run the following snmptrap command.snmptrap localhost 0 NETBOTZV2-MIB::netBotzTempTooHigh \NETBOTZV2-MIB::netBotzV2TrapSensorID s ’nbHawkEnc_1_TEMP1’As you can see, this zentrap command starts exactly the same as in the example. We then add the followingthree fields.(a) NETBOTZV2-MIB::netBotzV2TrapSensorID (OID)1.1. ZenPack Development Guide 39

Zenoss Labs Documentation, Release(b) s (type)(c) ’nbHawkEnc_1_TEMP1’ (value)We can continue to add sets of these three parameters to add as many other variable bindings to the trap as wewant.Assuming you were running zentrap in the foreground you should see a log that looks like the following:Queued event (total of 1) {’sysUpTime.0’: 0L,’firstTime’: 1343073249.083523,’severity’: 3,’netBotzV2TrapSensorID’: ’nbHawkEnc_1_TEMP1’,’eventClassKey’: ’netBotzTempTooHigh’,’oid’: ’1.3.6.1.4.1.5528.100.10.2.1.0.2’,’component’: ’’,’community’: ’public’,’summary’: ’snmp trap netBotzTempTooHigh’,’eventGroup’: ’trap’,’sysUpTime’: 0L,’device’: ’127.0.0.1’,’lastTime’: 1343073249.083523,’monitor’: ’localhost’}Note that the only difference between this event and the simple event is the addition of the net-BotzV2TrapSensorID field. So now you see how Zenoss take the name/value pairs that are the SNMP trap’svariable bindings and turn them into name/value pairs within the resulting event.Mapping SNMP Trap EventsNow that we’re able to create SNMP traps anytime we want, it’s time to use Zenoss’ event mapping system to makethem more useful. The most important field on an incoming event when it comes to mapping is the eventClassKeyfield. Fortunately for us, SNMP traps get that great eventClassKey set that gives us a big head start.1. Map the event.(a) Navigate to Events in the web interface.(b) Select the netBotzTempTooHigh event you just created.(c) Click the toolbar button that looks like a hierarchy. If you hover over it, the tooltip will say Reclassify anevent.(d) Choose the /Environ event class then click SUBMITNow the next time a netBotzTempTooHigh trap is received it will be put into the /Environ event class insteadof /Unknown.2. Enrich the event.(a) Click the Go to new mapping link to navigate to the new mapping.(b) Click Edit in the left navigation pane.(c) Set Transform to the following:evt.component = getattr(evt, ’netBotzV2TrapSensorID’, ’’)This will use the name of the sensor as described by the netBotzV2TrapSensorID variable binding as theevent’s component field.1.1. ZenPack Development Guide 40

Zenoss Labs Documentation, ReleaseThere are endless possibilities of what you could do within the transform for this event and others. This is just onepractical example.1.1.6 Monitoring an HTTP APIThe following sections will describe an efficient approach to monitoring data via a HTTP API. We’ll start by usingzenpacklib to more easily extend the Zenoss object model. Then we’ll use a Python modeler plugin to fill out theobject model. Then we’ll use PythonCollector to monitor for events, datapoints and even to update the model.For purposes of this guide we’ll be building a ZenPack that monitors the weather using The Weather Channel’s WeatherUnderground API.Note: It is recommended that you have already finished the Monitoring an SNMP-Enabled Device section as itprovides much more detail and troubleshooting advice. This exercise builds on that experience.Exercises:Wunderground APIThe Weather Underground provides an API that can be used to get all sorts of data related to the weather. Before youcan use most endpoints on the API you must first create an account. Fortunately you can get a Developer account withall of the bells and whistles for free by signing up at http://www.wunderground.com/weather/api. So go sign up andget your API key. You’ll need it for the rest of this exercise.We’ll be using the following APIs for this exercise.1. AutoComplete2. Alerts3. ConditionsAutoComplete APIBoth the Alerts and Conditions APIs require that you query for a specific location. It can be hard to know what thename or code for a location is without doing some manual research. That’s where the AutoComplete API comes in.You can provide a reasonable name for a location and it will return a list of possible matches along with a unique linkfor that location.We’ll use the AutoComplete API during modeling so that the Zenoss user can enter nearly any city or county namethen let Zenoss do the work of converting that into the link that we’ll subsequently use to query for weather alerts andconditions.Here’s an example query for Austin, TX:http://autocomplete.wunderground.com/aq?query=Austin%2C%20TXNote: “Austin%2C%20TX” is the URL encoded version of “Austin, TX”. We won’t have to worry about that whenwe work with it because our HTTP library automatically encodes URLs.Here’s the response to that example query for Austin, TX:{"RESULTS": [{"c": "US",1.1. ZenPack Development Guide 41

Zenoss Labs Documentation, Release}]}"l": "/q/zmw:78701.1.99999","lat": "30.271158","ll": "30.271158 -97.741699","lon": "-97.741699","name": "Austin, Texas","type": "city","tz": "America/Chicago","tzs": "CDT","zmw": "78701.1.99999"There are a few things to note about this request and response. The first is that we didn’t need to use our API key.This is because the AutoComplete API doesn’t require an API key. The second is that there’s only a single result forAustin, TX. The third is the l value which is the unique link to Austin, TX that we can use when accessing the otherAPI endpoints such as Alerts and Conditions.Alerts APIThe Alerts API provides information about severe weather alerts such as tornado warnings, flood warnings and otherspecial weather statements. We’ll be collecting these alerts to create corresponding Zenoss events. This way operatorscan know when severe weather may be impacting areas of concern.Here’s an example query for alerts in Austin, TX:http://api.wunderground.com/api//alerts/q/zmw:78701.1.99999.jsonNote: Note how the URL ends with /alerts/.json using the l link value from the AutoComplete query for Austin,TX above.Here’s the relevant portion of the response to an alerts query. Of course Austin doesn’t have severe weather so we’llbe looking at Des Moines alerts instead:{}"alerts": [{"date": "1:07 PM CDT on June 16, 2014","date_epoch": "1402942020","description": "Severe Thunderstorm Warning","expires": "2:15 PM CDT on June 16, 2014","expires_epoch": "1402946100","message": "\nThe National Weather Service in Des Moines has issued a\n\n* Severe Thunder"phenomena": "SV","significance": "W","type": "WRN","tz_long": "America/Chicago","tz_short": "CDT"}]It’s easy to imagine turning this alert into a Zenoss event. We’ll see how to do this a bit later. The Alerts APIdocumentation has a link to a document that describes what the phenomena, significance, and type values represent.1.1. ZenPack Development Guide 42

Zenoss Labs Documentation, ReleaseConditions APIThe Conditions API provides information about current weather conditions for a given location. The Conditions APIis used in exactly the same way as the Alerts API, and accepts the same link to specify the location. There’s a lot ofnumeric data that would be useful to graph and threshold as Zenoss datapoints.Here’s an example query for conditions in Austin, TX:http://api.wunderground.com/api//conditions/q/zmw:78701.1.99999.jsonHere’s the relevant portion of the response to a conditions query:{"current_observation": {"UV": "1","dewpoint_c": 11,"dewpoint_f": 51,"dewpoint_string": "51 F (11 C)","display_location": {"city": "San Francisco","country": "US","country_iso3166": "US","elevation": "47.00000000","full": "San Francisco, CA","latitude": "37.77500916","longitude": "-122.41825867","magic": "1","state": "CA","state_name": "California","wmo": "99999","zip": "94101"},"estimated": {},"feelslike_c": "13.9","feelslike_f": "57.0","feelslike_string": "57.0 F (13.9 C)","forecast_url": "http://www.wunderground.com/US/CA/San_Francisco.html","heat_index_c": "NA","heat_index_f": "NA","heat_index_string": "NA","history_url": "http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KCASANFR58","icon": "partlycloudy","icon_url": "http://icons.wxug.com/i/c/k/partlycloudy.gif","image": {"link": "http://www.wunderground.com","title": "Weather Underground","url": "http://icons.wxug.com/graphics/wu2/logo_130x80.png"},"local_epoch": "1402931138","local_time_rfc822": "Mon, 16 Jun 2014 08:05:38 -0700","local_tz_long": "America/Los_Angeles","local_tz_offset": "-0700","local_tz_short": "PDT","nowcast": "","ob_url": "http://www.wunderground.com/cgi-bin/findweather/getForecast?query=37.773285,-122.4"observation_epoch": "1402931132","observation_location": {"city": "SOMA - Near Van Ness, San Francisco","country": "US",1.1. ZenPack Development Guide 43

Zenoss Labs Documentation, Release}}"country_iso3166": "US","elevation": "49 ft","full": "SOMA - Near Van Ness, San Francisco, California","latitude": "37.773285","longitude": "-122.417725","state": "California"},"observation_time": "Last Updated on June 16, 8:05 AM PDT","observation_time_rfc822": "Mon, 16 Jun 2014 08:05:32 -0700","precip_1hr_in": "0.00","precip_1hr_metric": " 0","precip_1hr_string": "0.00 in ( 0 mm)","precip_today_in": "0.00","precip_today_metric": "0","precip_today_string": "0.00 in (0 mm)","pressure_in": "29.89","pressure_mb": "1012","pressure_trend": "+","relative_humidity": "81%","solarradiation": "--","station_id": "KCASANFR58","temp_c": 13.9,"temp_f": 57.0,"temperature_string": "57.0 F (13.9 C)","visibility_km": "16.1","visibility_mi": "10.0","weather": "Scattered Clouds","wind_degrees": 238,"wind_dir": "WSW","wind_gust_kph": 0,"wind_gust_mph": 0,"wind_kph": 4.8,"wind_mph": 3.0,"wind_string": "From the WSW at 3.0 MPH","windchill_c": "NA","windchill_f": "NA","windchill_string": "NA"Using ZenPackLib (zenpacklib)In the Device Modeling and Component Modeling sections of the Monitoring an SNMP-Enabled Device documentationwe covered what could be called the “long way” to extend the Zenoss object model to add the new NetBotzDevicedevice type, it’s TemperatureSensor component type and the relationship between them. A new library called zenpacklibis now available that we use to make this process easier.In this case we want to add a new WundergroundDevice device type, a WundergroundLocation component type anda relationship between them so we can add a Weather Underground device such as wunderground.com to Zenoss andadd one or more locations where we want to monitor weather conditions. This section will describe how we do thatusing zenpacklib.Create the Weather Underground ZenPackSince we don’t yet have a Weather Underground ZenPack to work with, let’s create that first.1.1. ZenPack Development Guide 44

Zenoss Labs Documentation, Release1. Go to the Advanced -> Settings -> ZenPacks in the Zenoss web interface.2. Choose Create a ZenPack... from the gear menu above the list of ZenPacks.3. Name the ZenPack ZenPacks.training.WeatherUnderground then click OK.Download zenpacklibCreating the ZenPack will create the ZenPack’s source directory under $ZEN-HOME/ZenPacks/ZenPacks.training.WeatherUnderground. Run the following commands as the zenoss user tomake this directory, and it’s important subdirectory more easily accessible:export ZP_TOP_DIR=$ZENHOME/ZenPacks/ZenPacks.training.WeatherUndergroundexport ZP_DIR=$ZP_TOP_DIR/ZenPacks/training/WeatherUndergroundThen download the latest zenpacklib from GitHub into the ZenPack’s main code directory with the following commands.cd $ZP_DIRwget https://raw.githubusercontent.com/zenoss/zenpacklib/master/zenpacklib.pyCreate the ZenPackSpecOnce zenpacklib.py is in $ZP_DIR you can create what is called a ZenPackSpec in the ZenPack’s __init__.py. ThisZenPackSpec will define the specification for the ZenPack. Specifically its name, zProperties, classes and class relationships.Replace the contents of $ZP_DIR/__init__.py with the following:# Import zenpacklib from the current directory (zenpacklib.py).from . import zenpacklib# Create a ZenPackSpec and name it CFG.CFG = zenpacklib.ZenPackSpec(name=__name__,zProperties={’DEFAULTS’: {’category’: ’Weather Underground’},’zWundergroundAPIKey’: {},},’zWundergroundLocations’: {’type’: ’lines’,’default’: [’Austin, TX’, ’San Jose, CA’, ’Annapolis, MD’],},classes={’WundergroundDevice’: {’base’: zenpacklib.Device,’label’: ’Weather Underground API’,},’WundergroundLocation’: {’base’: zenpacklib.Component,’label’: ’Location’,1.1. ZenPack Development Guide 45

Zenoss Labs Documentation, Release’properties’: {’country_code’: {’label’: ’Country Code’,’order’: 4.0,},’timezone’: {’label’: ’Time Zone’,’order’: 4.1,},},},}’api_link’: {’label’: ’API Link’,’order’: 4.9,’grid_display’: False,},)class_relationships=zenpacklib.relationships_from_yuml("""[WundergroundDevice]++-[WundergroundLocation]""")# Create the specification.CFG.create()You can see this ZenPackSpec defines the following important aspects of our ZenPack.1. The name is set to __name__ which will evaluate to ZenPacks.training.WeatherUnderground. This shouldalways be set in this way as it will helpfully figure out the name for you.2. The zProperties contains configuration properties we want the ZenPack to add to the Zenoss system when it isinstalled.Note that DEFAULTS is not added as configuration property. It is a special value that will cause it’s properties tobe added as the default for all of the other listed zProperties. Specifically in this case it will cause the categoryof zWundergroundAPIKey and zWundergroundLocations to be set to Weather Underground. This is aconvenience to avoid having to repeatedly type the category for each added property.The zWundergroundAPIKey zProperty has an empty dictionary ({}). This is because we want it to be a stringtype with an empty default value. These happen to be the defaults so they don’t need to be specified.The zWundergroundLocations property uses the lines type which allows the user to specify multiple lines of text.Each line will be turned into an element in a list which you can see is also how the default value is specified. Theidea here is that unless the user configures otherwise, we will default to monitoring weather alerts and conditionsfor Austin, TX, San Jose, CA, and Annapolis, MD.3. The classes contains each of the object classes we want the ZenPack to add.In this case we’re adding WundergroundDevice which because base is set to zenpacklib.Device will be a subclassor specialization of the standard Zenoss device type. We’re also adding WundergroundLocation which becausebase is set to zenpacklib.Component will be a subclass of the standard component type.The label for each is simply the human-friendly name that will be used to refer to the resulting objects whenthey’re seen in the Zenoss web interface.The properties for WundergroundLocation are extra bits of data we want to model from the API and show to theuser in the web interface. order will be used to show the properties in the defined order, and setting grid_display1.1. ZenPack Development Guide 46

Zenoss Labs Documentation, Releaseto false for api_link will allow it be shown in the details panel of the component, but not in the component grid.4. class_relationships uses the relationships_from_yuml helper to succinctly define a relationship stating WundergroundDevicecan contain many WundergroundLocation objects.5. Finally CFG.create() create what has been defined in the ZenPackSpec. Without this call, nothing wouldhappen.Reinstall the ZenPackBecause you added new zProperties you must now reinstall the ZenPack. This is required because new zPropertiesonly get installed into Zenoss when the ZenPack is installed. Everything else in the ZenPackSpec is created each timeZenoss starts.Run the following command to reinstall the ZenPack and keep it in development mode.zenpack --link --install $ZP_TOP_DIRPython Modeler PluginNow that we’ve created a WundergroundLocation component type, we need to create a modeler plugin to createlocations in the database. We’re dealing with a custom HTTP API, so we’ll want to base our modeler plugin on thePythonPlugin class. This gives us full control of both the collection and processing of the modeling data.The modeler plugin will pass each location the user has specified in the zWundergroundLocations property to WeatherUnderground’s AutoComplete API to retrieve some basic information about the location, and very importantly the l(link) that uniquely identifies the location. The link will later be used to monitor the alerts and conditions for thelocation.Create WeatherUnderground.Locations Modeler PluginUse the following steps to create our modeler plugin.1. Make the directory that will contain our modeler plugin.mkdir -p $ZP_DIR/modeler/plugins/WeatherUnderground2. Create __init__.py or dunder-init files.touch $ZP_DIR/modeler/__init__.pytouch $ZP_DIR/modeler/plugins/__init__.pytouch $ZP_DIR/modeler/plugins/WeatherUnderground/__init__.pyThese empty __init__.py files are mandatory if we ever expect Python to import modules from these directories.3. Create $ZP_DIR/modeler/plugins/WeatherUnderground/Locations.py with the followingcontents."""Models locations using the Weather Underground API."""# stdlib Importsimport json# Twisted Importsfrom twisted.internet.defer import inlineCallbacks, returnValuefrom twisted.web.client import getPage1.1. ZenPack Development Guide 47

Zenoss Labs Documentation, Release# Zenoss Importsfrom Products.DataCollector.plugins.CollectorPlugin import PythonPluginclass Locations(PythonPlugin):"""Weather Underground locations modeler plugin."""relname = ’wundergroundLocations’modname = ’ZenPacks.training.WeatherUnderground.WundergroundLocation’requiredProperties = (’zWundergroundAPIKey’,’zWundergroundLocations’,)deviceProperties = PythonPlugin.deviceProperties + requiredProperties@inlineCallbacksdef collect(self, device, log):"""Asynchronously collect data from device. Return a deferred."""log.info("%s: collecting data", device.id)apikey = getattr(device, ’zWundergroundAPIKey’, None)if not apikey:log.error("%s: %s not set. Get one from http://www.wunderground.com/weather/api",device.id,’zWundergroundAPIKey’)returnValue(None)locations = getattr(device, ’zWundergroundLocations’, None)if not locations:log.error("%s: %s not set.",device.id,’zWundergroundLocations’)returnValue(None)rm = self.relMap()for location in locations:try:response = yield getPage(’http://autocomplete.wunderground.com/aq?query={query}’.format(query=location))response = json.loads(response)except Exception, e:log.error("%s: %s", device.id, e)returnValue(None)for result in response[’RESULTS’]:rm.append(self.objectMap({1.1. ZenPack Development Guide 48

Zenoss Labs Documentation, ReleasereturnValue(rm)’id’: self.prepId(result[’zmw’]),’title’: result[’name’],’api_link’: result[’l’],’country_code’: result[’c’],’timezone’: result[’tzs’],}))def process(self, device, results, log):"""Process results. Return iterable of datamaps or None."""return resultsWhile it looks like there’s quite a bit of code in this modeler plugin, a lot of that is the kind of error handlingyou’d want to do in a real modeler plugin. Let’s walk through some of the highlights.(a) ImportsWe import the standard json module because the Weather Underground API returns json-encoded responses.We import inlineCallBacks and returnValue because the PythonPlugin.collect method should return a Deferredso that it can be executed asynchronously by zenmodeler. You don’t need to use inlineCallbacks, butI find it to be a nice way to make Twisted’s asynchronous callback-based code look more procedural andbe easier to understand. I recommend Dave Peticolas’ excellent Twisted Introduction for learning moreabout Twisted. inlineCallback is covered in part 17.We also import Twisted’s getPage function. This is an extremely easy to use function for asynchronouslyfetching a URL.We import PythonPlugin because it will be the base class for our modeler plugin class. It’s the best choicefor modeling data from HTTP APIs.(b) Locations ClassRemember that your modeler plugin’s class name must match the filename or Zenoss won’t be able to loadit. So because we named the file Locations.py we must name the class Locations.(c) relname and modname PropertiesThese should be defined in this way for modeler plugins that fill a single relationship like we’re doing inthis case. It states that this modeler plugin creates objects in the device’s wundergroundLocations relationship,and that it creates objects of the ZenPacks.training.WeatherUnderground.WundergroundLocationtype within this relationship.Where does relname come from? It comes from the [WundergroundDevice]++-[WundergroundLocation]relationship we defined in __init__.py. Because it’s a to-many relationship to the WundergroundLocationtype, zenpacklib will name the relationship by lowercasing the first letter and adding an “s” to theend to make it plural.Where does modname come from? It will be ..So because we defined the WundergroundLocation class in __init__.py, and theZenPack’s name is ZenPacks.training.WeatherUnderground, the modname will be Zen-Packs.training.WeatherUnderground.WundergroundLocation.(d) deviceProperties PropertiesThe class’ deviceProperties property provides a way to get additional device properties available to yourmodeler plugin’s collect and process methods. The default properties that will be available for a Python-Plugin are: id, manageIp, _snmpLastCollection, _snmpStatus, and zCollectorClientTimeout. Our modeler1.1. ZenPack Development Guide 49

Zenoss Labs Documentation, Releaseplugin will also need to know what values the user has set for zWundergroundAPIKey and zWunderground-Locations. So we add those to the defaults.(e) collect MethodThe collect method is something PythonPlugin has, but other base modeler plugin types like SnmpPlugindon’t. This is because you must write the code to collect the data to be processed, and that’s exactly whatyou should do in the collect method.While the collect method can return either normal results or a Deferred, it is highly recommend to returna Deferred to keep zenmodeler from blocking while your collect method executes. In this examplewe’ve decorated the method with @inlineCallbacks and have returned out data at the endwith returnValue(rm). This causes it to return a Deferred. By decorating the method with@inlineCallbacks we’re able to make an asynchronous request to the Weather Underground APIwith response = yield getPage(...).The first thing we do in the collect method is log an informational message to let the user know whatwe’re doing. This log will appear in zenmodeler.log, or on the console if we run zenmodeler in theforeground, or in the web interface when the user manually remodels the device.Next we make sure that the user has configured a value for zWundergroundAPIKey. This isn’t strictlynecessary here because the modeler plugin only uses Weather Underground’s AutoComplete API whichdoesn’t require an API key. I put this check here because I didn’t want to get into a situation where thelocations modeled successfully, but then failed to collect because an API key wasn’t set.Next we make suer that the user as configured at least one location in zWundergroundLocations. This ismandatory because this controls what locations will be modeled.Next we create rm which is a common convention we use in modeler plugins and stands for RelationshipMap.Because we set the relname and modname class properties this will create a RelationshipMapwith it’s relname and modname set to the same.Now we iterate through each location making a call to the AutoComplete API for each. For each matchinglocation in the response we will append an ObjectMap to rm with some key properties set.• id is mandatory and should be set to a value unique to all components on the device. If you look backthe example AutoComplete response you’ll see that the zmw property is useful for this purpose. Notethat prepId should always be used for id. It will make any string safe to use as a Zenoss id.• title will default to the value of id if it isn’t set. It’s usually a good idea to explicitly set it as we’redoing here. It should be a human-friendly label for the component. The location’s name is a goodcandidate for this. It will look something like “Austin, Texas”.• api_link is a property we defined for the WundergroundLocation class in __init__.py. This iswhere we’ll store the returned link or l property. This will be important for monitoring the alerts andconditions of the location later on.• country_code is another property we defined. It’s purely informational and will simply be shown tothe user when they’re viewing the location in the web interface.• timezeone is another property we defined just for informational purposes.(f) process MethodThe process method is usually where you take the data in the results argument and process it into DataMapsto return. However, in the case of PythonPlugin modeler plugins, the data returned from the collect methodwill be passed into process as the results argument. In this case that is already complete processed data.So we just return it.4. Restart Zenoss.1.1. ZenPack Development Guide 50

Zenoss Labs Documentation, ReleaseAfter adding a new modeler plugin you must restart Zenoss. If you’re following the Running a Minimal Zenossinstructions you really only need to restart zopectl and zenhub.That’s it. The modeler plugin has been created. Now we just need to do some Zenoss configuration to allow us to useit.Add WeatherUnderground Device ClassTo support adding our special WundergroundDevice devices that we defined in __init__.py to Zenoss we mustcreate a new device class. This will give us control of the zPythonClass configuration property that defines what typeof devices will be created. It will also allow us to control what modeler plugins and monitoring templates will be used.Use the following steps to add the device class.1. Navigate to the Infrastructure view.2. Select the root of the DEVICES tree.3. Click the + button at the bottom of the list to add a new organizer.4. Set the Name to WeatherUnderground then click SUBMIT.The new WeatherUnderground device will now be selected. We’ll want to check on some important configurationproperties using the following steps.Set Device Class Properties1. Click the DETAILS button at the top of the list.2. Select Configuration Properties. Set the following properties.• zPythonClass: ZenPacks.training.WeatherUnderground.WundergroundDevice• zPingMonitorIgnore: true• zSnmpMonitorIgnore: true3. Select Modeler Plugins from the left navigation pane.You’ll likely find a list of selected modeler plugins that looks something like the following.• zenoss.snmp.NewDeviceMap• zenoss.snmp.DeviceMap• zenoss.snmp.InterfaceMap• zenoss.snmp.RouteMap4. Remove all of the modeler plugins from the Selected list.5. Move WeatherUnderground.Locations from the Available to the Selected list.6. Click Save.Add the WeatherUnderground Device Class to the ZenPack Now that we’ve setup the WeatherUnderground deviceclass, it’s time to add it to our ZenPack using the following steps. Adding a device class to your ZenPack causesall settings in that device class to be added to the ZenPack. This includes modeler plugin configuration, configurationproperty values and monitoring templates.1. Make sure that you have the WeatherUnderground device class selected in the Infrastructure view.2. Choose Add to ZenPack from the gear menu in the bottom-left.1.1. ZenPack Development Guide 51

Zenoss Labs Documentation, Release3. Select ZenPacks.training.WeatherUnderground then click SUBMIT.4. Export the ZenPack. (Exporting a ZenPack)Add the wunderground.com Device This would be a good time to add a device to the new device class. There aremany ways to add devices to Zenoss, but if you’re Running a Minimal Zenoss you may not be running zenjobs andsome of them won’t work. During ZenPack development it’s often easiest to use zendisc to add devices.Run the following command to add a wunderground.com device.zendisc run --deviceclass=/WeatherUnderground --device=wunderground.comYou should see output similar to the following:INFO zen.ZenModeler: Collecting for device wunderground.comINFO zen.ZenModeler: No WMI plugins found for wunderground.comINFO zen.ZenModeler: Python collection device wunderground.comINFO zen.ZenModeler: plugins: WeatherUnderground.LocationsINFO zen.PythonClient: wunderground.com: collecting dataERROR zen.PythonClient: wunderground.com: zWundergroundAPIKey not set. Get one from http://www.wunderINFO zen.PythonClient: Python client finished collection for wunderground.comWARNING zen.ZenModeler: The plugin WeatherUnderground.Locations returned no results.INFO zen.ZenModeler: No change in configuration detectedINFO zen.ZenModeler: No command plugins found for wunderground.comINFO zen.ZenModeler: SNMP monitoring off for wunderground.comINFO zen.ZenModeler: No portscan plugins found for wunderground.comINFO zen.ZenModeler: Scan time: 0.02 secondsINFO zen.ZenModeler: Daemon ZenModeler shutting downNote: The error about zWundergroundAPIKey not being set is expected because we haven’t set it. The solutionis to go to the wunderground.com device in the web interface and add your API key to the zWundergroundAPIKeyconfiguration property. After adding the API key you should remodel the device.Another good way to add device to Zenoss is with zenbatchload. Using zenbatchload also allows us to set configurationproperties such as zWundergroundAPIKey as the device is added.Create a wunderground.zenbatchload file with the following contents:/Devices/WeatherUndergroundwunderground.com zWundergroundAPIKey=’’, zWundergroundLocations=[’Austin, TX’, ’Des MoiNow run the following command to load from that file:zenbatchload wunderground.zenbatchloadYou should now be able to see a list of locations on the wunderground.com device!Using PythonCollectorThe PythonCollector ZenPack adds the capability to write high performance datasources in Python. They will becollected by the zenpython daemon that comes with the PythonCollector ZenPack.I’d recommend reading the PythonCollector Documentation for more information.1.1. ZenPack Development Guide 52

Zenoss Labs Documentation, ReleaseInstalling PythonCollectorThe first thing we’ll need to do is to make sure the PythonCollector ZenPack is installed on our system. If it isn’t,follow these instructions to install it.1. Download the latest release from the PythonCollector page.2. Run the following command to install the ZenPack:zenpack --install ZenPacks.zenoss.PythonCollector-.egg3. Restart Zenoss.Add PythonCollector Dependency Since we’re going to be using PythonCollector capabilities in our ZenPack wemust now update our ZenPack to define the dependency.Follow these instructions to define the dependency.1. Navigate to Advanced -> Settings -> ZenPacks.2. Click into the ZenPacks.training.WeatherUnderground ZenPack.3. Check ZenPacks.zenoss.PythonCollector in the list of dependencies.4. Click Save.5. Export the ZenPack. (Exporting a ZenPack)Python Collector Plugin (Events)Now that we have one or more locations modeled on our wunderground.com device, we’ll want to start monitoringeach location. Using PythonCollector we have the ability to create events, record datapoints and even update themodel. We’ll start with an example that creates events from weather alert data.The idea will be that we’ll create events for locations that have outstanding weather alerts such as tornado warnings.We’ll try to capture severity data so tornado warnings are higher severity events than something like a frost advisory.Create Alerts Data Source PluginTo make Zenoss able to run the data source plugin we’re going to create, we must first make sure the PythonCollectorZenPack is installed, and that our ZenPack depends on it. See the Using PythonCollector section if you haven’t alreadydone this.Follow these steps to create the Alerts data source plugin:1. Create $ZP_DIR/dsplugins.py with the following contents."""Monitors current conditions using the Weather Underground API."""# Loggingimport loggingLOG = logging.getLogger(’zen.WeatherUnderground’)# stdlib Importsimport jsonimport time# Twisted Importsfrom twisted.internet.defer import inlineCallbacks, returnValue1.1. ZenPack Development Guide 53

Zenoss Labs Documentation, Releasefrom twisted.web.client import getPage# PythonCollector Importsfrom ZenPacks.zenoss.PythonCollector.datasources.PythonDataSource import (PythonDataSourcePlugin,)class Alerts(PythonDataSourcePlugin):"""Weather Underground alerts data source plugin."""@classmethoddef config_key(cls, datasource, context):return (context.device().id,datasource.getCycleTime(context),context.id,’wunderground-alerts’,)@classmethoddef params(cls, datasource, context):return {’api_key’: context.zWundergroundAPIKey,’api_link’: context.api_link,’location_name’: context.title,}@inlineCallbacksdef collect(self, config):data = self.new_data()for datasource in config.datasources:try:response = yield getPage(’http://api.wunderground.com/api/{api_key}/alerts{api_link}.json’.format(api_key=datasource.params[’api_key’],api_link=datasource.params[’api_link’]))response = json.loads(response)except Exception:LOG.exception("%s: failed to get alerts data for %s",config.id,datasource.location_name)continuefor alert in response[’alerts’]:severity = Noneif int(alert[’expires_epoch’])

Zenoss Labs Documentation, Releaseseverity = 2data[’events’].append({’device’: config.id,’component’: datasource.component,’severity’: severity,’eventKey’: ’wu-alert-{}’.format(alert[’type’]),’eventClassKey’: ’wu-alert’,returnValue(data)’summary’: alert[’description’],’message’: alert[’message’],’wu-description’: alert[’description’],’wu-date’: alert[’date’],’wu-expires’: alert[’expires’],’wu-phenomena’: alert[’phenomena’],’wu-significance’: alert[’significance’],’wu-type’: alert[’type’],})Let’s walk through this code to explain what is being done.(a) LoggingThe first thing we do is import logging and create LOG as our logger. It’s important that the name of thelogger in the logging.getLogger() begins with zen.. You will not see your logs otherwise.The stdlib and Twisted imports are almost identical to what we used in the modeler plugin, and they’reused for the same purposes.Finally we import PythonDataSourcePlugin from the PythonCollector ZenPack. This is the class our datasource plugin will extend, and basically allows us to write code that will be executed by the zenpythoncollector daemon.(b) Alerts ClassUnlike our modeler plugin, there’s no need to make the plugin class’ name the same as the filename. Aswe’ll see later when we’re setting up the monitoring template that will use this plugin, there’s no specificname for the file or the class required because we configure where to find the plugin in the datasourceconfiguration within the monitoring template.(c) config_key Class MethodThe config_key method must have the @classmethod decorator. It is passed datasource, and context.The datasource argument will be the actual datasource that the user configures in the monitoring templatessection of the web interface. It has properties such as eventClass, severity, and as you can see a getCycleTime()method that returns the interval at which it should be polled. The context argument will be theobject to which the monitoring template and datasource is bound. In our case this will be a location objectsuch as Austin, TX.The purpose of the config_key method is to split monitoring configuration into tasks that will be executedby the zenpython daemon. The zenpython daemon will create one task for each unique value returned fromconfig_key. It should be used to optimize the way data is collected. In some cases it is possible to make asingle query to an API to get back data for many components. In these cases it would be wise to removecontext.id from the config_key so we get one task for all components.In our case, the Weather Underground API must be queried once per location so it makes more sense toput context.id in the config_key so we get one task per location.1.1. ZenPack Development Guide 55

Zenoss Labs Documentation, ReleaseThe value returned by config_key will be used when zenpython logs. So adding something likewunderground-alerts to the end makes it easy to see logs related to collecting alerts in the log file.The config_key method will only be executed by zenhub. So you must restart zenhub if you make changesto the config_key method. This also means that if there’s an exception in the config_key method it willappear in the zenhub log, not zenpython.(d) params Class MethodThe params method must have the @classmethod decorator. It is passed the same datasource andcontext arguments as config_key.The purpose of the params method is to copy information from the Zenoss database into the config.datasources[*]that will be passed as an argument to the collect method. Since the collect methodis run by zenpython it won’t have direct access to the database, so it relies on the params method toprovide it with any information it will need to collect.In our case you can see that we’re copying the context’s zWundergroundAPIKey, api_link and title properties.All of these will be used in the collect method.Just like the config_key method, params will only be executed by zenhub. So be sure to restart zenhub ifyou make changes, and look in the zenhub log for errors.(e) collect MethodThe collect method does all of the real work. It will be called once per cycletime. It gets passed a configargument which for the most part has two useful properties: config.id and config.datasources. config.idwill be the device’s id, and config.datasources is a list of the datasources that need to be collected.You’ll see in the collect method that each datasource in config.datasources has some useful properties.datasource.component will be the id of the component against which the datasource is run, or blank inthe case of a device-level monitoring template. datasource.params contains whatever the params methodreturned.Within the body of the collect method we see that we create a new data variable using data =self.new_data(). data is a place where we stick all of the collected events, values and maps. datalooks like the following:data = {’events’: [],’values’: defaultdict(, {}),’maps’: [],}Next we iterate over every configured datasource. For each one we make a call to Weather Underground’sAlerts API, then iterate over each alert in the response creating an event for each.The following standard fields are being set for every event. You should read Zenoss’ event managementdocumentation if the purpose of any of these fields is not clear. I highly recommend setting all of thesefields to an appropriate value for any event you send into Zenoss to improve the ability of Zenoss andZenoss’ operators to manage the events.• device: Mandatory. The device id related to the event.• component: Optional. The component id related to the event.• severity: Mandatory. The severity for the event.• eventKey: Optional. A further uniqueness key for the event. Used for de-duplication and clearing.• eventClassKey: Optional. An identifier for the type of event. Used during event class mapping.• summary: Mandatory: A (hopefully) short summary of the event. Truncated to 128 characters.1.1. ZenPack Development Guide 56

Zenoss Labs Documentation, Release• message: Optional: A longer text description of the event. Not truncated.You will also see many wu-* fields being added to the event. Zenoss allows arbitrary fields on events soit can be a good practice to add any further information you get about the event in this way. It can makeunderstanding and troubleshooting the resulting event easier.Finally we return data with all of events we appended to it. zenpython will take care of getting the eventssent from this point.2. Restart Zenoss.After adding a new datasource plugin you must restart Zenoss. If you’re following the Running a MinimalZenoss instructions you really only need to restart zenhub.That’s it. The datasource plugin has been created. Now we just need to do some Zenoss configuration to allow us touse it.Configure Monitoring TemplatesRather than use the web interface to manually Add a Monitoring Template, we’ll jump to the next section on UsingYAML Templates to show how we can describe a monitoring template using YAML.Using YAML TemplatesOrdinarily monitoring templates are built within the Zenoss web interface. If you want one of these templates includedwith a ZenPack, you simply use the web interface to add the template to any of the ZenPacks you have in development.Sometimes you know that you have a lot of monitoring templates to create, or the monitoring templates have a lot ofdatasource, datapoints, thresholds and graphs. It can be tedious and error-prone to add all of these through the webinterface. In these cases it may be preferable to load monitoring templates from files.There’s a script available called load-templates that makes it possible to load monitoring templates from YAML sourcefiles.Download load-templatesRun the following commands to download the load-templates script into your ZenPack:cd $ZP_TOP_DIRwget https://raw.githubusercontent.com/zenoss/zenpacklib/master/load-templateschmod 755 load-templatesNote: Note that we’re putting this script into $ZP_TOP_DIR, not $ZP_DIR. This isn’t required, but it’s a goodpractice. Files located in $ZP_TOP_DIR won’t by default be included in your ZenPack once it’s built. Since loadtemplatesis only used by the ZenPack developer, it doesn’t need to be included in the built ZenPack.This script requires that the PyYAML library be installed on your system. You can run the following command as thezenoss user to install it:easy_install PyYAML1.1. ZenPack Development Guide 57

Zenoss Labs Documentation, ReleaseCreate templates.yamlWe now want to create the YAML file that can be used to create a monitoring template that will be bound to ourlocations and execute our Alerts data source plugin.Follow these steps to create this monitoring template:1. Create $ZP_TOP_DIR/templates.yaml with the following contents./WeatherUnderground/Location:description: Weather Underground location monitoring.targetPythonClass: ZenPacks.training.WeatherUnderground.WundergroundLocationdatasources:alerts:type: Pythonplugin_classname: ZenPacks.training.WeatherUnderground.dsplugins.Alertscycletime: "600"At least some of this should be self-explanatory. The YAML vocabulary has been designed to be as intuitive andconcise as possible. Let’s walk through it.(a) The highest-level element (based on indentation) is /WeatherUnderground/Location. This means to createa Location monitoring template in the /WeatherUnderground device class.Note: Because we’re using zenpacklib the monitoring template must be called Location because the isthe label for the WundergroundLocation class to which we want the template bound.(b) The description is for documentation purposes and should describe the purpose of the monitoring template.(c) The targetPythonClass is a hint to what type of object the template is meant to be bound to. Currently thisis only used to determine if users should be allowed to manually bind the template to device classes ordevices. Providing a valid component type like we’ve done prevents users from making this mistake.(d) Next we have datasources with a single alerts datasource defined.The alerts datasource only has three properties:• type: This is what makes zenpython collect the data.• plugin_classname: This is the fully-qualified class name for the PythonDataSource plugin we createdthat will be responsible for collecting the datasource.• cycletime: The interval in seconds at which this datasource should be collected.2. Run the following commands to create the monitoring template defined in templates.yaml.cd $ZP_TOP_DIR./load-templates templates.yaml3. Navigate to Advanced -> Monitoring Templates in the web interface to verify that the Location monitoringtemplate has been created as defined.4. Export the ZenPack. (Exporting a ZenPack)The /WeatherUnderground device class is already part of our ZenPack, and we put the Location monitoring templateinto that device class. So exporting causes the template to be dumped into the ZenPack’s objects.xmlfile.1.1. ZenPack Development Guide 58

Zenoss Labs Documentation, ReleaseTest Monitoring Weather AlertsTesting this is a bit tricky since we’ll have to be monitoring a location that currently has an active weather alert.Fortunately there’s an easy way to find one of these locations.Follow these steps to test weather alert monitoring:1. Go to the following URL for the current severe weather map of the United States.http://www.wunderground.com/severe.asp2. Click on one of the colored areas. Orange and red are more exciting. This will take you to the text of thewarning. It should reference city or county names.3. Update zWundergroundLocations on the wunderground.com device to add one of the cities or counties that hasan active weather alert. For example, “Buffalo, South Dakota”.4. Remodel the wunderground.com device then verify that the new location is modeled.5. Run the following command to collect from wunderground.com.zenpython run -v10 --device=wunderground.comThere will be a lot of output from this command, but we’re mainly looking for an event to be sent for the weatheralert. It will look similar to the following output:DEBUG zen.zenpython: Queued event (total of 1) {’rcvtime’: 1403112635.631883, ’wu-type’: u’FIR’,Python Collector Plugin (Data Points)We’ve already created a data source plugin that creates Zenoss events for weather alerts. Now we want to use theWeather Underground Conditions API to monitor current weather conditions for each location. The purpose of this isto illustrate that these Python data source plugins can also be used to collect datapoints.Create Conditions Data Source PluginFollow these steps to create the Conditions data source plugin:1. Add the following contents to the end of $ZP_DIR/dsplugins.py.Most of the Conditions plugin is almost identical to the Alerts plugin so I won’t repeat whatcan be read back in that section. The main difference starts at the current_observation =response[’current_observation’] line of the collect method.It grabs the current_observation data from the response then iterates over every datapoint configured on thedatasource. This is a nice approach because it allows for some user-flexibility in what datapoints are capturedfrom the Conditions API. If the API made temp_c and temp_f available, we could choose to collect temp_c justby adding a datapoint by that name.The following line is the most important in terms of explaining how to have your plugin return datapoint values.Basically we just stick (value, ’N’) into the component’s datapoint dictionary. The ’N’ is the timestampat which the value occurred. If you know the time it should be specified as the integer UNIX timestamp. Use’N’ if you don’t know. This will use the current time.2. Restart Zenoss.After adding a new datasource plugin you must restart Zenoss. If you’re following the Running a MinimalZenoss instructions you really only need to restart zenhub.1.1. ZenPack Development Guide 59

Zenoss Labs Documentation, ReleaseThat’s it. The datasource plugin has been created. Now we just need to do some Zenoss configuration to allow us touse it.Add Conditions to Monitoring TemplateTo use this new plugin we’ll add a new datasource and corresponding graphs to the existing Location monitoringtemplate defined in templates.yaml.Follow these steps to update the monitoring template:1. Update $ZP_TOP_DIR/templates.yaml with the following content. This includes what should alreadybe in the file./WeatherUnderground/Location:description: Weather Underground location monitoring.targetPythonClass: ZenPacks.training.WeatherUnderground.WundergroundLocationdatasources:alerts:type: Pythonplugin_classname: ZenPacks.training.WeatherUnderground.dsplugins.Alertscycletime: "600"conditions:type: Pythonplugin_classname: ZenPacks.training.WeatherUnderground.dsplugins.Conditionscycletime: "600"datapoints:temp_c: GAUGEfeelslike_c: GAUGEheat_index_c: GAUGEwindchill_c: GAUGEdewpoint_c: GAUGErelative_humidity: GAUGEpressure_mb: GAUGEprecip_1hr_metric: GAUGEUV: GAUGEwind_kph: GAUGEwind_gust_kph: GAUGEvisibility_km: GAUGEgraphs:Temperatures:units: degrees C.graphpoints:Temperature:dpName: conditions_temp_cformat: "%7.2lf"Feels Like:dpName: conditions_feelslike_cformat: "%7.2lf"Heat Index:dpName: conditions_heat_index_cformat: "%7.2lf"1.1. ZenPack Development Guide 60

Zenoss Labs Documentation, ReleaseWind Chill:dpName: conditions_windchilltemp_cformat: "%7.2lf"Dewpoint:dpName: conditions_dewpoint_cformat: "%7.2lf"Relative Humidity:units: percentminy: 0maxy: 100graphpoints:Relative Humidity:dpName: conditions_relative_humidityformat: "%7.2lf%%"Pressure:units: millibarsminy: 0graphpoints:Pressure:dpName: conditions_pressure_mbformat: "%7.0lf"Precipitation:units: centimetersminy: 0graphpoints:1 Hour:dpName: conditions_precip_1hr_metricformat: "%7.2lf"UV Index:units: UV indexminy: 0maxy: 12graphpoints:UV Index:dpName: conditions_UVformat: "%7.0lf"Wind Speed:units: kphminy: 0graphpoints:Sustained:dpName: conditions_wind_kphformat: "%7.2lf"Gust:dpName: conditions_wind_gust_kphformat: "%7.2lf"1.1. ZenPack Development Guide 61

Zenoss Labs Documentation, ReleaseVisibility:units: kilometersminy: 0graphpoints:Visibility:dpName: conditions_visibility_kmformat: "%7.2lf"Only the first 9 lines previously existed for the alerts support. Adding the conditions datasource with its 12datapoints and corresponding 9 graphs accounts for the remaining 104 lines.2. Run the following commands to create the monitoring template defined in templates.yaml.cd $ZP_TOP_DIR./load-templates templates.yaml3. Navigate to Advanced -> Monitoring Templates in the web interface to verify that the Location monitoringtemplate has been updated with the conditions datasource and corresponding graphs.4. Export the ZenPack. (Exporting a ZenPack)The /WeatherUnderground device class is already part of our ZenPack, and we put the Location monitoring templateinto that device class. So exporting causes the template to be dumped into the ZenPack’s objects.xmlfile.Test Monitoring Weather ConditionsFollow these steps to test weather condition monitoring:1. Run the following command to collect from wunderground.com.zenpython run -v10 --device=wunderground.comThere will be a lot of output from this command, but we’re mainly looking for at least one datapoint beingwritten. If one works, it’s likely that they all work. Look for a line similar to the following:DEBUG zen.RRDUtil: /opt/zenoss/perf/Devices/wunderground.com/80901.1.99999/conditions_temp_c.rrdPython Collector Plugin (Modeling)The final capability of Python data source plugins is to make changes to the Zenoss model. This allows a data sourceto make changes to the model in the same way that zenmodeler does. Having this capability in a data source allowsmodeling more frequently than the normal 12 hour zenmodeler interval.To demonstrate this through an exercise, we’ll extend the existing Conditions plugin to capture the what the ConditionsAPI calls weather which is some text that looks like “Scattered Clouds” or “Sunny”. We’ll then show this value foreach location in the web interface.AddAdd Modeling to Conditions Data Source PluginFollow these steps to add modeling to the Conditions data source plugin:1.1. ZenPack Development Guide 62

Zenoss Labs Documentation, Release1. Edit $ZP_DIR/__init__.py.Add the following weather property to the WundergroundLocation class between the existing timezone andapi_link properties.’weather’: {’label’: ’Weather’,’order’: 4.2,},2. Edit $ZP_DIR/dsplugins.py.Add the following needed import to the top of dsplugins.py.from Products.DataCollector.plugins.DataMaps import ObjectMapAdd the following code to the Conditions class’ collect method right above the returnValue(data) lineindented one level further. The returnValue(data) line is included in the following update to show wherethe new code should be placed.data[’maps’].append(ObjectMap({’relname’: ’wundergroundLocations’,’modname’: ’ZenPacks.training.WeatherUnderground.WundergroundLocation’,’id’: datasource.component,’weather’: current_observation[’weather’],}))returnValue(data)# existing lineThe maps concept here is exactly the same as it is in modeler plugins. data[’maps’] can contain anythingthat a modeler plugin’s process method can return.2. Don’t update the Location monitoring template.We’re adding capability to a datasource that’s already configured. No updates are required to the monitoringtemplate.3. Restart Zenoss.If we had only updated the collect method of the Conditions plugin we would only need to restart zenpython.However, because we added the new weather property to the WundergroundLocation class, we must restartnearly everything, so it’s simpler to restart everything.Test Modeling Current WeatherFollow these steps to test weather condition monitoring:1. Run the following command to collect from wunderground.com.zenpython run -v10 --device=wunderground.comThere will be a lot of output from this command, but we’re looking for the following line which indicates thatour maps were applied:DEBUG zen.python: Applying 1 datamaps to wunderground.com2. Navigate to the Locations on the wunderground.com device and verify that each location shows something in itsWeather column.1.1. ZenPack Development Guide 63

Zenoss Labs Documentation, Release1.1.7 Exporting a ZenPackNow that we’ve created a ZenPack and added some configuration to it, we need to export it. Exporting a ZenPack takesall of the object’s you’ve added to your ZenPack through the web interface and compiles them into an objects.xmlfile that gets saved into your ZenPack’s source directory in the file system.Follow these steps to export the NetBotz ZenPack.1. Navigate to Advanced -> ZenPacks -> NetBotz ZenPack in the web interface.2. Scroll to the bottom of the page to see what objects the ZenPack provides.All objects listed in the ZenPack Provides section and objects contained within them will be exported.3. Choose Export ZenPack from the gear menu in the bottom-left of the screen.4. Choose to only export and not download then click OK.You could also choose to download the ZenPack through your web browser. However, the downloaded file willbe the built egg distribution format of the ZenPack. This means that it can be installed into other Zenoss systems,but is not suitable for further development.This will export everything under ZenPack Provides to a file within your ZenPack’s source called objects.xml. Noother files in your ZenPack’s source directory are created or modified. You can find this file in the following path:$ZENHOME/ZenPacks/ZenPacks.training.NetBotz/ZenPacks/yourname/NetBotz/objects/objects.xmlEach time you add a new object to you ZenPack within the web interface, or modify an object that’s already containedwithin your ZenPack, you should export the ZenPack again to update objects.xml. If you’re using version control onyour ZenPack’s source directory this would be a good time to commit the resulting change to objects.xml.Warning: Exporting a ZenPack completely overwrites the objects.xml file that previously existed within theZenPack’s source directory. For this reason it is recommended that the objects.xml file never be modified by hand.1.1.8 TroubleshootingUsing the Python DebuggerOne of the most powerful tools when debugging the Python portions of a ZenPack is the Python debugger (pdb). Withpdb you can set breakpoints in your code. When the breakpoints are hit, you get a (pdb) prompt that has full access toexamine the stack and any local or global variables.To set a breakpoint in your code you add the following line:import pdb; pdb.set_trace()As with any code change, you must restart the Zenoss process that executes the code in question.1.2 ZenPack DocumentationZenPacks must be documented using reStructuredText. The minimum documentation requirement is that each Zen-Pack have a README.rst located in its top-level directory.An optional top-level docs/ directory containing at least one file named index.rst can also be created to suplementthe README.rst. This would be the recommended approach if a ZenPack’s documentation requires theadditional complexity of additional structure, files, or Sphinx extensions.1.2. ZenPack Documentation 64

Zenoss Labs Documentation, ReleaseContents:1.2.1 ZenPack Standards GuideThis document describes all requirements and recommendations for ZenPack development. The intended audience isall Zenoss, Inc. employees who create or modify ZenPacks including engineering, services and support. The intendedaudience also includes any third-parties that create or modify ZenPacks that are delivered to Zenoss customers byZenoss, Inc.AssumptionsSome assumptions are made in this document regarding access to test facilities.Product InclusionZenPacks must always be developed with the understanding that the potential exists for them to become part of theproduct. Even in cases were ownership of the ZenPack does not rest with Zenoss this must be observed becauseownership can change in the future. For this reason, special attention must be paid to document the inclusion of anysensitive company data in ZenPacks.Access to Test EnvironmentIn cases where the ZenPack developer(s) do not have access to endpoints necessary for integration and testing worksome standard operating procedures must be suspended.File LocationsThe location of specific files within a ZenPack’s directory structure is technically mandated in some circumstances,and open to the developer’s desires in others. To make it easier for other developers to more easily get up to speedwith the ZenPack in the future, the following recommendations for file locations should be used.• ZenPacks./• ZenPacks./ZenPacks/namespace/PackName/– browser/* configure.zcml All ZCML definitions related to defining browser views or wiring browser-onlyconcerns. This configure.zcml should be included by the default configure.zcml in itsparent directory.* resources/· css/ - All stylesheets loaded by users’ browsers.· img/ - All images loaded by users’ browsers.· js/ - All javascript loaded by users’ browser.– lib/ Any third-party modules included with the ZenPack should be located in this directory. In the caseof pure-Python modules they can be located directly here. In the case of binary modules the build processmust install them here. See the section of License Compliance below for more information on how toproperly handle third-party content.– libexec/ Any scripts intended to be run by the zencommand daemon must be located in this directory.1.2. ZenPack Documentation 65

Zenoss Labs Documentation, Release– objects/ There should only ever be a single file called objects.xml in this directory. While the ZenPackinstallation code will load objects from any file in this directory with a .xml extension, the ZenPack exportcode will dump all objects back to objects.xml so creating other files only creates future confusionbetween installation and export.– facades.py All facades (classes implementing Products.Zuul.interfaces.IFacade) shouldbe defined in this file. In ZenPacks where this single file becomes hard to maintain, a facades/ directoryshould be created containing individual files named for the group of facades they contain.– info.py All info adapters (classes implementing Products.Zuul.interfaces.IInfo) shouldbe defined in this file. In ZenPacks where this single file becomes hard to maintain, an info/ directoryshould be created containing individual files named for the group of info adapters they contain.– interfaces.py All interfaces (classes extending zope.interface.Interface) should be definedin this file. In ZenPacks where this single file becomes hard to maintain, an interfaces/ directoryshould be created containing individual files named for the group of interfaces they contain.– routers.py All routers (classes extending Products.ZenUtils.Ext.DirectRouter) shouldbe defined in this file. In ZenPacks where this single file becomes hard to maintain, a routers/ directoryshould be created containing individual files named for the group of routers they contain.License ComplianceAll ZenPack content must be compliant with the license of the ZenPack being developed. If you intend to include athird-party module with a GPL license, the ZenPack must also carry a GPL license and not include any other codethat would violate the GPL license. Always run third-party module inclusion through legal to make sure there is noconflict.Coding StandardsAll code and configuration in ZenPacks should be developed according to the following public style guides.• Python– PEP 8 – Style Guide or Python Code– PEP 257 – Docstring Conventions• ZCML– Zope’s ZCML Style GuideMonitoring Template StandardsPerformance templates are one of the easiest places to make a real user experience difference when new features areadded to Zenoss. Spending a very small amount of time to get the templates right goes a long way towards improvingthe overall user experience. For this reason, the following checklist should be used to determine if your monitoringtemplate is acceptable.Templates1. Is the template worthwhile? Should it be removed?2. Is the template at the correct point in the model?3. Does the template have a description? Is the description a good one?1.2. ZenPack Documentation 66

Zenoss Labs Documentation, ReleaseData Sources1. Can your datasource be named better?1. Is it a common metric that is being collected from other devices in another way? If so, name yours the same.This makes global reporting much easier.2. camelCaseNames seem to be the standard. Use them.2. Never use absolute paths for COMMAND datasource command templates. This will end up causing problemson one of the three platforms we deal with. Link your plugin into zenPath(‘libexec’) instead.Data Points1. Using a COUNTER? You might want to think otherwise.1. Unnoticed counter rollovers can result in extremely skewed data.2. Using a DERIVE with a minimum of 0 will record unknown instead of wrong data.2. Enter the minimum and/or maximum possible values for the data point if you know them.1. This again will allow unknown to be recorded instead of bad data.Data Point Aliases1. Include the unit in the alias name if it is in any way not obvious. For example, use cpu_percent instead ofcpu_usage.2. Use an RPN to calculate the base unit if the data point isn’t already collected that way. For example, use1024,* to convert a data point collected in KBytes to bytes.Thresholds1. Don’t include a number in your threshold’s name.1. This makes people have to recreate the threshold if they want to change it.Graph Definitions1. Have you entered the units? Do it!1. This will become the y-axis label and should be all lowercase.2. Always use the base units. Never kbps or MBs. bps or bytes are better.2. Do you know the minimum/maximum allowable values? Enter them!1. Common scenarios include percentage graphing with minimum 0 and maximum 100.3. Think about the order of your graph points. Does it make sense?4. Are there other templates that show similar data to yours?1. If so, you should try hard to mimic their appearance to create a consistent experience.1.2. ZenPack Documentation 67

Zenoss Labs Documentation, ReleaseGraph Points1. Have you changed the legend? Do it!2. Adjust the format so that it makes sense.1. %5.2lf%s is good for values you want RRDTool to auto-scale.2. %6.2lf%% is good for percentages.3. %4.0lf is good for four digit numbers with no decimal precision or scaling.3. Should you be using areas or lines?1. Lines are good for most values.2. Areas are good for things that can be thought of as a volume or quantity.4. Does stacking the values to present a visual aggregate makes sense?ETL StandardsETL is an acronym for Extract, Transform, Load. When writing ETL adapters you’re defining how Zenoss modeldata is extracted and transformed into the Zenoss Analytics schema. The following guidelines should be used to keepreporting consistent.1. The reportProperties implementation in IReportable adapters must include the units in the name ifnot immediately obvious. For example, use cpu_used_percent instead of cpu_used.DocumentationZenPacks must be documented according to the ZenPack.example.Name template. The Zen-Packs.zenoss.SolarisMonitor documentation can be used as an example of a ZenPack that has been documented usingthis template.Code DocumentationPython code must be documented in docstrings in the locations specified in PEP-8 and according to the style of PEP-257. Links to these standards can be found in the Coding Standards section. Inline code comments should also beused when the code isn’t obvious.TestingThe following types of testing must be performed. All test results should be recorded in the ZenPack’s test resultmatrix. The matrix will have the ZenPack version on one axis and the Zenoss version on the other axis. At theintersection will be the result of unit testing, internal integration testing and live integration testing.Unit TestsUnit tests must be written for all public interfaces of ZenPack-specific code. Unit tests will be the only mechanism forautomated regression testing in some cases, and the primary source in all others.1.2. ZenPack Documentation 68

Zenoss Labs Documentation, ReleaseInternal Integration TestingZenPacks must be tested internally using the packaged .egg that is will be delivered to the customer. The test servermust be the exact same version of Zenoss being used by the customer. The test environment must match the customer’senvironment as closely as possible. The only exception to internal integration testing is cases where it is not possibleto replicate the test environment internally.Live Integration TestingZenPacks must be tested in their live deployment environment. A development or staging instance of Zenoss thatmatches the production environment as closely as possible should be used.VersioningThe first feature-complete ZenPack delivered to a customer should be version 1.0.0. Subsequent versions must incrementthe micro version if they contain only bugfixes or tweaks (i.e. 1.0.1.) Subsequent versions must increment theminor version if the contain new features (i.e. 1.1.0.)A ZenPack’s version must be incremented each time it is delivered to a customer if there has been any change to itwhatsoever.ReviewsPeer review is a strong mechanism to catch potential issues before integration testing is performed. To that end thefollowing reviews must be performed.Design ReviewThe initial design of a ZenPack must be peer reviewed before coding begins.Code ReviewAll code, including updates, must be peer reviewed before being committed to the mainline development branch orany stable release branch.Packaging & DeliveryAll ZenPacks must be delivered in their packaged .egg format. If arrangements have been made for the customer toalso get the source for the ZenPack it should be provided in addition to the packaged egg as a tarball of the developmentdirectory.ZenPacks must be built using the same environment that the customer will be installing them into. If the customer isinstalling into multiple environments a separate egg should be built and delivered for each environment. In this contextthe same environment is defined as the following.• Exact same version of Zenoss• Same major version of operating system• Same architecture (i.e. i386 or x86_64)All files including documentation must be delivered to customers in a Parature ticket.1.2. ZenPack Documentation 69

Zenoss Labs Documentation, Release1.2.2 ZenPack.example.NameAboutIntroduction to the ZenPack and any potentially foreign concepts such as the target that is being monitored or integratedwith.FeaturesHigh-level overview of the ZenPack’s features and functionality. Focus on the value proposition. Target audiencewould be people trying to understand Zenoss’ capabilities, not Zenoss administrators.PrerequisitesRequirements and dependencies that must be satisfied. This section should cover the following at a minimum andwhere applicable.• Minimum required Zenoss version• Maximum supported Zenoss version if known• ZenPack dependencies (with specific versions if known)• Other installation dependencies if known (e.g. operating system packages)• Supported versions of the monitoring or integration targetLimitationsNote any shortcomings that the user might otherwise be left curious about. This section is optional.UsageThe target audience for the entire Usage section is a Zenoss administrator.InstallingStandard installation steps plus any other installation steps or notes specific to this ZenPack.UsingOne or more ad-hoc usage related sections. This (or these) sections will likely contain the bulk of the ZenPack’scustom documentation. The section(s) will not necessarily be called Using.RemovingStandard ZenPack removal steps plus any removal steps or notes specific to this ZenPack. Be especially careful tocover anything that will result in data loss such as removal of device classes and their contained devices.1.2. ZenPack Documentation 70

Zenoss Labs Documentation, ReleaseTroubleshootingDocument common problems users of the ZenPack may run into such as what happens in the result of authenticationfailures or other configuration mistakes.AppendixThe two examples appendixes below will very commonly be used. Additional reference material can be made availablein additional appendixes. The Appendixes section can only be omitted if the ZenPack installs no items as describedbelow, and requires no non-platform daemons as described below.Appendex A: Installed ItemsDetail the items installed by the ZenPack. Items include the following.• Device Classes• Configuration Properties• Modeler Plugins• Command Parsers• Monitoring Templates• Process Classes• IP Service Classes• Windows Service Classes• Event Classes• Event Mappings• MIBs• ReportsAppendex B: Related DaemonsDetail the daemons outside of the core platform required to take advantage of all of the ZenPack’s functionality. Thecore platform daemons listed below should not be explicitly listed.• zeoctl• zeneventserver• zeneventd• zenhub• zenjobs• zendisc• zenmodeler• zenimpactserver• zenimpactgraph1.2. ZenPack Documentation 71

Zenoss Labs Documentation, Release• zenimpactstate• zenjserverDaemons such as the following, plus any daemons delivered with the ZenPack should be listed.• zencommand• zenperfsnmp• zenprocess1.2.3 ZenPacks.zenoss.SolarisMonitorAboutThe SolarisMonitor ZenPack enables Resource Manager to use Secure Shell (SSH) to monitor Solaris hosts. ResourceManager models and monitors devices placed in the /Server/SSH/Solaris device class by running commands andparsing the output. Parsing of command output is performed on the Resource Manager server (if using a local collector)or on a distributed collector. The account used to monitor the device does not require root access or special privileges.In addition to the previously described modeling and monitoring features this ZenPack also enables Resource Managerto model and monitor Sun Solaris LDOM servers. Resource Manager will model devices utilizing the Simple NetworkManagement Protocol (SNMP) to collect LDOM information when a divice resides in either the /Server/Solaris or/Server/SSH/Solaris device classes. The discovered LDOM information will be displayed as components of the LDOMhost server.FeaturesThe SolarisMonitor ZenPack provides:• File system and process monitoring• Network interfaces and route modeling• CPU utilization information• Hardware information (memory, number of CPUs, and model numbers)• OS information (OS-level, command-style information)• Pkginfo information (such as installed software)• LDOM monitoringPrerequisitesPrerequisite RestrictionZenoss Platform 3.1 or greaterInstalled ZenPacks ZenPacks.zenoss.SolarisMonitorFirewall Acccess Collector server to 22/tcp and 161/udp of Solaris serverSolaris Releases OpenSolaris 5.11, Solaris 9 and 10LimitationsThe SolarisMonitor ZenPack does not support monitoring in Solaris Zones or systems containing Solaris Zones. (Implementedwith Solaris 10, Solaris Zones act as isolated virtual servers within a single operating system instance.)1.2. ZenPack Documentation 72

Zenoss Labs Documentation, ReleaseUsageInstallationThis ZenPack has no special installation considerations. Depending on the version of Zenoss you’re installing theZenPack into, you will need to verify that you have the correct package (.egg) to install.• Zenoss 4.1 and later: The ZenPack file must end with -py2.7.egg.• Zenoss 3.0 - 4.0: The ZenPack file must end with -py2.6.egg.To install the ZenPack you must copy the .egg file to your Zenoss master server and run the following command asthe zenoss user:zenpack --install After installing you must restart Zenoss by running the following command as the zenoss user on your masterZenoss server:zenoss restartIf you have distributed collectors you must also update them after installing the ZenPack.ConfiguringDepending on the version of Solaris you may be able to monitor the server using either SSH or SNMP. For OpenSolarisand Solaris 10, you can choose to use either SSH or SNMP monitoring. For Solaris 9, only SSH monitoring issupported.Configuring SSH Monitoring Use the following steps to configure Zenoss to monitor your Solaris server(s) usingSSH.1. Navigate to the /Server/SSH/Solaris device class’ configuration properties.2. Verify that the zCommandUsername and zCommandPassword are set to valid login credentials.3. Add your Solaris server(s) to the /Server/SSH/Solaris device class.Configuring SNMP Monitoring Use the following steps to configure Zenoss to monitor your Solaris server(s) usingSNMP.1. Verify that the snmpd process is running on your Solaris server(s).2. Navigate to the /Server/Solaris device class’ configuration properties.3. Verify that your Solaris server(s) SNMP community strings are listed in the zSnmpCommunities property.4. Add your Solaris server(s) to the /Server/Solaris device class.Configuring LDOM Monitoring For OpenSolaris and Solaris 10 servers you will also get support for monitoringLDOMs if they’re used on the server. However, this monitoring is always performed using SNMP. If you’re alreadymonitoring your Solaris server using SNMP there is no additional configuration required to monitor its LDOMs. If youconfigured Zenoss to monitor your Solaris server using SSH you should take the following steps to monitor LDOMs.1. Verify that the snmpd process is running on your Solaris server(s).2. Navigate to the /Server/SSH/Solaris device class’ configuration properties.3. Verify that your Solaris server(s) SNMP community strings are listed in the zSnmpCommunities property.1.2. ZenPack Documentation 73

Zenoss Labs Documentation, Release4. Remodel your Solaris server(s) if they’re already in the system. Otherwise add them to the/Server/SSH/Solaris device class.RemovalUse caution when removing this ZenPack• Will permanently remove devices located in /Server/SSH/Solaris device class.• Will permanently remove LDOM modeled components for devices located in /Server/Solaris.• Will permanently remove associated monitored data for LDOM components.• Will permanently remove the /Server/SSH/Solaris device class.To remove this ZenPack you must run the following command as the zenoss user on your master Zenoss server:zenpack --remove ZenPacks.zenoss.SolarisMonitorYou must then restart the master Zenoss server by running the following command as the zenoss user:zenoss restartTroubleshootingResolving CHANNEL_OPEN_FAILURE Issues The zencommand daemon’s log file($ZENHOME/collector/zencommand.log) may show messages stating:ERROR zen.SshClient CHANNEL_OPEN_FAILURE: Authentication failure WARNING:zen.SshClient:Open of commanIf the sshd daemon’s log file on the remote device is examined, it may report that the MAX_SESSIONS numberof connections has been exceeded and that it is denying the connection request. In the OpenSSH daemons, thisMAX_SESSIONS number is a compile-time option and cannot be reset in a configuration file.To work around this sshd daemon limitation, use the configuration property zSshConcurrentSessions to control thenumber of connections created by zencommand to the remote device:1. Navigate to the device or device class in the Resource Manager interface.• If applying changes to a device class:(a) Select the class in the devices hierarchy.(b) Click Details.(c) Select Configuration Properties.• If applying changes to a device:(a) Click the device in the device list.(b) Select Configuration Properties.2. Set the zSshConcurrentSessions property. Try 10 first, and 2 if that doesn’t resolve the problem.Resolving Command Timeout Issues The zencommand daemon’s log file ($ZEN-HOME/collector/zencommand.log) may show messages stating:WARNING:zen.zencommand:Command timed out on device device_name: command1.2. ZenPack Documentation 74

Zenoss Labs Documentation, ReleaseIf this occurs, it usually indicates that the remote device has taken too long to return results from the commands.To increase the amount of time to allow devices to return results, change the configuration propertyzCommandCommandTimeout to a larger value.1. Navigate to the device or device class in the Resource Manager interface.• If applying changes to a device class:(a) Select the class in the devices hierarchy.(b) Click Details.(c) Select Configuration Properties.• If applying changes to a device:(a) Click the device in the device list.(b) Select Configuration Properties.2. Increase the zCommandCommandTimeout property incrementally to a maximum of 240 until the timeout isresolved.AppendixesAppendix A: Installed ItemsType Name LocationDevice Class /SSH/Solaris /Devices/ServerModeler Plugin df_ag zenoss.cmd.solarisModeler Plugin kstat zenoss.cmd.solarisModeler Plugin memory zenoss.cmd.solarisModeler Plugin netstat_an zenoss.cmd.solarisModeler Plugin netstat_r_vn zenoss.cmd.solarisModeler Plugin pkginfo zenoss.cmd.solarisModeler Plugin process zenoss.cmd.solarisModeler Plugin uname_a zenoss.cmd.solarisModeler Plugin hostid zenoss.snmp.solarisModeler Plugin ldommap zenoss.snmp.solarisMonitoring Template Device /Server/SSH/SolarisMonitoring Template FileSystem /Server/SSH/SolarisMonitoring Template OSProcess /Server/SSH/SolarisMonitoring Template ethernetCsmacd /Server/SSH/SolarisMonitoring Template LDOM /ServerMonitoring Template LDOMVcpu /ServerMonitoring Template LDOMVds /ServerEvent Class /Status/LDOM /Event Class /Status/LDOM/vCPU /Event Mapping ldomStateChange /ChangeEvent Mapping ldomVCpuChange /ChangeEvent Mapping ldomVccChange /ChangeEvent Mapping ldomVconsChange /ChangeEvent Mapping ldomVdiskChange /ChangeEvent Mapping ldomVdsChange /ChangeEvent Mapping ldomVmemChange /ChangeEvent Mapping ldomVnetChange /ChangeContinued on next page1.2. ZenPack Documentation 75

Zenoss Labs Documentation, ReleaseTable 1.1 – continued from previous pageType Name LocationEvent Mapping ldomVswChange /ChangeEvent Mapping ldomCreate /Change/AddEvent Mapping ldomDestroy /Change/RemoveMIB SUN-LDOM-MIB /Monitoring Templates Device (/Server/SSH/Solaris)• Data Points– cpu_ssCpuIdle– cpu_ssCpuInterrupt– cpu_ssCpuSystem– cpu_ssCpuUser– io_read– io_written– percent_memory_percentMemUsed– percent_swap_percentSwapUsed– uptime_laLoadInt1– uptime_laLoadInt5– uptime_laLoadInt15– uptime_sysUpTime• Thresholds– CPU Utilization– high load• Graphs– Load Average– CPU Utilization– Memory Utilization– IOFileSystem (/Server/SSH/Solaris)• Data Points– disk_availBlocks– disk_availNodes– disk_percentInodesUsed– disk_totalBlocks– disk_totalInodes– disk_usedBlocks– disk_usedInodes1.2. ZenPack Documentation 76

Zenoss Labs Documentation, Release• Thresholds– high_disk_usage• Graphs– Utilization– Inode UtilizationOSProcess (/Server/SSH/Solaris) - Data Points• ps_count• ps_cpu• ps_mem• Graphs– CPU Utilization– Memory– Process CountethernetCsmacd (/Server/SSH/Solaris) - Data Points• intf_ifInErrors• intf_ifInPackets• intf_ifOutErrors• intf_ifOutPackets• intf_octets_ifInOctets• intf_octets_ifOutOctets• Thresholds– Utilization 75 perc• Graphs– Throughput– PacketsLDOM (/Server)• Data Sources– ldomOperState• Thresholds– operational stateLDOMVcpu (/Server)• Data Sources– ldomVcpuOperationalStatus– ldomVcpuUtilPercent• Threshold– operational status1.2. ZenPack Documentation 77

Zenoss Labs Documentation, Release• Graph Definition– CPU UtilizationLDOMVds (/Server)• Data Source– ldomVdsNumofAvailVolume– ldomVdsNumofUsedVolume• Graph Definition– VolumesAppendex B: Required DaemonsIn addition to the core platform daemons the following optional daemons are required for this ZenPack to fully function.• zenperfsnmp• zencommand1.3 ZenPack TaxonomyZenPacks can be used to override almost any functionality of the Zenoss platform, or to add any new features. Thiswide-open extensibility is much like Firefox’s add-on system and has the natural result of people building somesurprising things. For this reason it can be difficult to answer the question, “What is a ZenPack?”This document will outline the various ways ZenPacks can be classified to make it easier to find the ZenPack you need,describe a ZenPack that already exists, and to serve as an example for what is possible.1.3.1 ZenPack ClassificationsEvery ZenPack will be classified using one or more of the elements listed under each classifier property. Some classifierproperties have mutually exclusive elements, and some do not. This distinction will be made in the definition of eachclassifier property. For examples of how some existing representative ZenPacks are classified see Example ZenPackClassifications.FunctionalityFunctionality classifies the high-level type of feature(s) provided. Specifically its type of interaction with the environmentoutside of the Zenoss platform. It is highly recommended that these categories be treated as mutually exclusive.While it is possible for a ZenPack to build a ZenPack that delivers functionality in more than one of the followingcategories, this can lead to less modularity and confusion about what a ZenPack does.1.3. ZenPack Taxonomy 78

Zenoss Labs Documentation, ReleaseMonitoringMonitoring is one or more of status, event and performance collection. The collection can be through active polling,passive receiving or both. A monitoring ZenPack provides functionality to perform this collection for a specific targettechnology.Example ZenPacks.zenoss.ApacheMonitorIntegrationIntegration is defined as being any interaction with systems outside of Zenoss not deemed to be a Monitoring interaction.Examples include pushing or pulling non-monitoring data to or from an external system, or causing action in aremote system or allowing a remote system to cause action within Zenoss.Example ZenPacks.zenoss.RANCIDIntegratorPlatform ExtensionA Zenoss platform extension is defined as any functionality that doesn’t interact with outside systems. The providedfunctionality is instead used directly by users or by other parts of the Zenoss platform, or by other ZenPacks.Example ZenPacks.zenoss.DistributedCollectorMaintainerThe maintainer of a ZenPack is the organization or individual that controls the code repository for a ZenPack and is thegate for all changes including defect and enhancement resolution. The following categories are mutually exclusive.Zenoss EngineeringMaintained by the product engineering organization at Zenoss, Inc. Ongoing support is provided by Zenoss, Inc. atthe same level as the Zenoss platform software for customers with an active subscription.Example ZenPacks.zenoss.ImpactZenoss ServicesMaintained by the services organization at Zenoss, Inc. Ongoing support is provided under a statement of work.Example ZenPacks.zenoss.ServiceNowIntegratorZenoss CommunityMaintained by a member of the Zenoss community. Ongoing support is subject to the individual maintainers’ control,and is typically provided in the community web forums, mailing lists and IRC channel.Example ZenPacks.community.ZenODBC1.3. ZenPack Taxonomy 79

Zenoss Labs Documentation, ReleaseAvailabilityWho has access, license and permission to use the ZenPack. The following elements are mutually exclusive.Open SourceZenPack source and packages are available as free open source. Designed to function properly on a Zenoss systemwith or without commercial-only ZenPacks installed.Example ZenPacks.zenoss.ApacheMonitorAvailable with Zenoss SubscriptionZenPack packages are available at no extra cost to anyone with a Zenoss subscription, but are not installed by default.May have dependencies on Open Source ZenPacks or other ZenPacks that are Available with Zenoss Subscription.Example ZenPacks.zenoss.DatabaseMonitorAdditional Cost with Zenoss SubscriptionZenPack packages are available at an additional cost on top of an existing Zenoss subscription. May have dependencieson Open Source ZenPacks, ZenPacks that are Available with Zenoss Subscription, or other ZenPacks that areAdditional Cost with Zenoss Subscription.Example ZenPacks.zenoss.ImpactQA LevelThe level of automated, manual and field testing A ZenPack has. The elements are mutually exclusive.UntestedInsufficient automated testing to qualify as Automatically Tested, and insufficient manual testing to qualify as Q.A.Tested.Example ZenPacks.zenoss.OpenStackAutomatically TestedStandard automated testing passes plus a minimum of 90% unit test code coverage with all tests passing.Q.A. TestedTested, and passed, by the quality assurance group of Zenoss, Inc.1.3. ZenPack Taxonomy 80

Zenoss Labs Documentation, ReleaseComplexityDefined by the technical difficulty of implementing specific types of functionality within the ZenPack. The elementsare not mutually exclusive, and most ZenPacks will implement multiple types of functionality as defined below. Arough total complexity score could be created for each ZenPack by summing the complexity score of all implementedelements.ConfigurationBuilt entirely in the web interface. No programming knowledge required.Complexity 1Skills ZenossExample ZenPacks.zenoss.IISMonitorScriptsScripts can be written in any language and do anything. Since all Zenoss customizations should be packaged asZenPacks, they’re only included in ZenPacks as a packaging mechanism. They might not have any direct interactionwith the Zenoss platform.Complexity 2Skills Scripting (Any Language)Example ZenPacks.zenoss.RANCIDIntegratorCommand DataSource PluginsCommand datasource plugins can be written in any language and executed either on the Zenoss server, or remotelyusing SSH. Without writing a custom parser (see next item) they must write to STDOUT using either the Nagios orCacti output formats and exit using the appropriate Nagios or cacti exit code.Complexity 2Skills Scripting (Any Language)Example ZenPacks.zenoss.ApacheMonitorEvent Class Transforms and MappingsBuilt in the web interface. Basic Python knowledge required.Complexity 2Skills Zenoss, Basic PythonExample ZenPacks.zenoss.OpenStack1.3. ZenPack Taxonomy 81

Zenoss Labs Documentation, ReleaseCommand DataSource ParsersCommand datasource parsers must be written in Python and conform to the Zenoss CommandParser API. Theseparsers must be written to extract extended data from the output of command datasource plugins (see previous item),or to handle output that doesn’t conform to the Nagios or Cacti output formats.Complexity 3Skills Zenoss, PythonExample ZenPacks.zenoss.SolarisMonitorDataSource TypesWhen a new datasource is added in the web interface you must choose the type. Creating a DataSource type in aZenPack is a way to add new types to this list. The ApacheMonitor ZenPack listed as the example below adds theability to collect performance metrics from an Apache httpd server using mod_status.New DataSource types are written in Python and must subclass RRDDataSource or one of its existing subclasses.Additionally an API adapter must also be written in Python to define the user interface to the datasource properties.Complexity 4Skills Zenoss, ZCML, PythonExample ZenPacks.zenoss.ApacheMonitorImpact AdaptersThere are three types of impact adapters. All are written in Python and added to the system configuration throughZCML directives.The first is a state provider. These implement the IStateProvider interface and allow manipulation of how agiven node type’s state within the impact graph is calculated.The second is a relations provider. These implement the IRelationshipDataProvider interface and allowmanipulation of what other nodes a given node type impacts, and what other nodes impact it.The third is a triggers provider. These implement the INodeTriggers interface and allow manipulation of thedefault impact policies set on a given type of node.Complexity 5Skills Zenoss, ZCML, PythonExample ZenPacks.zenoss.ZenVMwareETL AdaptersETL is used to export model, performance and event data from a Zenoss instance to a Zenoss Analytics instance.However, ETL adapters only need to be written to manipulate the model data that is exported. There are two types ofETL adapters. They’re both written in Python and added to the system configuration through ZCML directives.The first type is a reportable. These implement the IReportable interface and allow precise control over whichproperties of an object type are exported, and how they’re named and manipulated for export.The second type is a reportable factory. These implement the IReportableFactory interface and all manipulationof which objects are considered for export. By default all devices and components are considered for extraction so1.3. ZenPack Taxonomy 82

Zenoss Labs Documentation, Releasea reportable factory is usually only used when fine-grained control over the relationships between these objects isneeded.Complexity 4Skills Zenoss, ZCML, PythonExample ZenPacks.zenoss.ZenVMwareUser InterfaceModifications to the existing user interface, or entirely new sections of user interface. The difficulty of these changesvaries considerably. See the Skills field below for the range of skills that could be required to make these kinds ofchanges.The ServiceNowIntegrator example given below adds a new button to the event console that pops up a new dialog boxwith some custom options available. Only ZCML and JavaScript were required for this type of change.TAL is usually only required when editing or creating old-style pages that aren’t entirely built using ExtJS.Complexity 5Skills Zenoss, ZCML, TAL, JavaScript, ExtJSExample ZenPacks.zenoss.ServiceNowIntegratorModeler Plugins - SNMP, COMMAND, WMIModeler plugins provide the mapping between data collected from the environment and the Zenoss model. In thecase where the data can be collected using SNMP, COMMAND (run a command remotely via SSH) or WMI, there isexisting infrastructure to make these tasks easier. However, the modeler plugins are still written in Python.If collecting using SNMP the SnmpPlugin class can be extended to do the hard parts of SNMP gets or walks for you.If collecting by running a command on a remote system via SSH, the CommandPlugin class can be extended to dothe hard parts of SSH and output parsing for you. If collecting from a Windows system using WMI, the WmiPluginclass can be extended to do the hard parts of WQL querying for you.The only significant logic that must be implemented in these cases is turning the returned data structures intoObjectMap and RelationshipMap objects to apply to the Zenoss model.Complexity 6Skills Zenoss, Python, (SNMP, Scripting or WMI)Example ZenPacks.zenoss.SolarisMonitorModeler Plugins - PythonSee Model Extensions above for what modeler plugins are. Python modeler plugins only differ in that you extend thePythonPlugin class, and must implement the collection logic in addition to the processing logic.The collect method implementation may return data normally, or it may return a Twisted deferred to takeadvantage of the asynchronous modeling engine. It is recommended to use the deferred approach whenever possibleto avoid blocking the zenmodeler daemon while the collect method executes.Complexity 7Skills Zenoss, Python, TwistedExample ZenPacks.zenoss.OpenStack1.3. ZenPack Taxonomy 83

Zenoss Labs Documentation, ReleaseModel ExtensionsWhen the standard model of the Zenoss platform doesn’t cover an object or property you need in your ZenPack, themodel can be extended. Existing model classes such as Device, FileSystem or IpInterface can be extended, and entirelynew types of components can be created.The typical requirements for extended the model include at least the following steps.1. Create a Python class2. Create an API interface and adapter3. Wire up the API with ZCML4. Write JavaScript to tailor the display of your component5. Write a modeler pluginComplexity 8Skills Zenoss, ZCML, Python, JavaScriptExample ZenPacks.zenoss.OpenStackDaemonsA new daemon must be written only if none of the existing daemons can perform the task required by your ZenPack.The zencommand daemon is the usual last resort for custom collection requirements if none of the more specializeddaemons will work. See Command DataSource Plugins and Command DataSource Parsers for what can be done byzencommand.There is a common collector framework that should be used to perform much of the typical daemon functionality suchas configuration and scheduling in a consistent way. To use this you should create a CollectorDaemon object,configure it with a class that implements the ICollectorPreferences interface and create a task class thatimplements the IScheduledTask interface.In almost all cases you will also need to create a ZenHub service to build the configuration for your new daemon. Thisservice should subclass HubService or one of its existing more specialized subclasses.Complexity 9Skills Zenoss, Python, TwistedExample ZenPacks.zenoss.ZenVMwarePlatform ExtensionPlatform extensions are any implementations added to a ZenPack that doesn’t fall into any of the previously-definedcomplexity elements. Due to the flexibility of ZenPacks, these could be almost anything.The DistributedCollector example given below falls into this category because it extends the simple flat collectorstructure in the core Zenoss platform to be a tiered hub and collector structure. It also adds extensive hub and collectormanagement capabilities.Complexity 10Skills Zenoss, ZCML, Python, JavaScript, etc.Example ZenPacks.zenoss.DistributedCollector1.3. ZenPack Taxonomy 84

Zenoss Labs Documentation, Release1.3.2 Example ZenPack ClassificationsZenPacks.zenoss.ApacheMonitorClassificationFunctionalityMaintainerAvailabilityQA LevelComplexityValueMonitoringZenoss EngineeringOpen SourceQ.A. TestedConfigurationCommand DataSource PluginsDataSource TypesZenPacks.zenoss.IISMonitorClassificationFunctionalityMaintainerAvailabilityQA LevelComplexityValueMonitoringZenoss EngineeringAvailable with Zenoss SubscriptionQ.A. TestedConfigurationZenPacks.zenoss.DistributedCollectorClassificationFunctionalityMaintainerAvailabilityQA LevelComplexityValuePlatform ExtensionZenoss EngineeringAvailable with Zenoss SubscriptionQ.A. TestedConfigurationUser InterfacePlatform Extension1.3. ZenPack Taxonomy 85

Zenoss Labs Documentation, ReleaseZenPacks.zenoss.RANCIDIntegratorClassificationFunctionalityMaintainerAvailabilityQA LevelComplexityValueIntegrationZenoss EngineeringAvailable with Zenoss SubscriptionQ.A. TestedConfigurationEvent Class Transforms and MappingsScriptsZenPacks.zenoss.DatabaseMonitorClassificationFunctionalityMaintainerAvailabilityQA LevelComplexityValueMonitoringZenoss EngineeringAvailable with Zenoss SubscriptionQ.A. TestedConfigurationCommand DataSource PluginsDataSource TypesZenPacks.zenoss.ZenVMwareClassificationFunctionalityMaintainerAvailabilityQA LevelComplexityValueMonitoringZenoss EngineeringAvailable with Zenoss SubscriptionQ.A. TestedConfigurationEvent Class Transforms and MappingsDataSource TypesUser InterfaceImpact AdaptersETL AdaptersModel ExtensionsDaemons1.3. ZenPack Taxonomy 86

Zenoss Labs Documentation, ReleaseZenPacks.zenoss.SolarisMonitorClassificationFunctionalityMaintainerAvailabilityQA LevelComplexityValueMonitoringZenoss EngineeringAvailable with Zenoss SubscriptionQ.A. TestedConfigurationCommand DataSource PluginsCommand DataSource ParsersModeler Plugins - SNMP, COMMAND, WMIZenPacks.zenoss.ImpactClassificationFunctionalityMaintainerAvailabilityQA LevelComplexityValuePlatform ExtensionZenoss EngineeringAdditional Cost with Zenoss SubscriptionQ.A. TestedConfigurationUser InterfaceImpact AdaptersDaemonsPlatform ExtensionZenPacks.zenoss.OpenStackClassificationFunctionalityMaintainerAvailabilityQA LevelComplexityValueMonitoringZenoss EngineeringOpen SourceUntestedConfigurationEvent Class Transforms and MappingsCommand DataSource PluginsCommand DataSource ParsersUser InterfaceImpact AdaptersModeler Plugins - PythonModel Extensions1.3. ZenPack Taxonomy 87

Zenoss Labs Documentation, ReleaseZenPacks.zenoss.ServiceNowIntegratorClassificationFunctionalityMaintainerAvailabilityQA LevelComplexityValueIntegrationZenoss ServicesAvailable with Zenoss SubscriptionQ.A. TestedConfigurationUser InterfaceModel ExtensionsDaemonsZenPacks.community.ZenODBCClassificationFunctionalityMaintainerAvailabilityQA LevelComplexityValuePlatform ExtensionZenoss CommunityOpen SourceQ.A. TestedDataSource TypesModeler Plugins - Python1.3. ZenPack Taxonomy 88

CHAPTER 2Documentation FormatsThis documentation is available in the following formats.• HTML• PDF• Epub• Manpage89

CHAPTER 3ContributionThis documentation is generated automatically every time a change is made to the zenosslabs repository. The mostdirect route for contribution would be to fork that repository, make the desired change to the documentation and senda pull request.See GitHub’s Fork a Repo documentation if you’re unfamiliar with this process. Otherwise, email the address belowto have someone incorporate your changes for you.90

CHAPTER 4ContactQuestions and comments related to this documentation should sent to labs@zenoss.com.91