Setting up Splunk for VMware Monitoring - VMware Communities

2009Setting up Splunk forVMware MonitoringProven Practice GuideA how-to guide for monitoring VMware ESX, ESXi and vCenter Servers andWindows and Linux Virtual Machines using Splunk Server.David Converywww.DailyHypervisor.com6/30/2009

Setting up Splunk forVMware MonitoringTable of ContentsSummary ....................................................................................................................................................... 3Requirements ................................................................................................................................................ 3Preparing the Splunk Server ......................................................................................................................... 3Installing Splunk Server ................................................................................................................................. 4Setting up ESX Servers and Linux VMs for Monitoring ................................................................................. 7Setting up ESXi Servers for Monitoring ......................................................................................................... 9Setting up vCenter Servers and Windows VMs for Monitoring .................................................................. 10Setting up SNARE .................................................................................................................................... 10Setting up Epilog ..................................................................................................................................... 13Confirming a System is Monitored ......................................................................................................... 17Adding Miscellaneous Log Files to Splunk for Troubleshooting ................................................................. 18David Convery Pg 2 of 18 www.dailyhypervisor.com

Setting up Splunk forVMware MonitoringSummaryGathering and maintaining log files is an important part of a server administrator’s duties. Using acentralized logging server, such as a sylog server offers several benefits. The log files become useful fortroubleshooting purposes, if needed. Also, keeping an unaltered set of logs in a different location can aidin forensic activities after an attack.This document explains how to set up Splunk for monitoring a VMware Environment. This includesmonitoring the ESX/ESXi Server logs, the vCenter Server Logs and some of the add-on services tovCenter. It also includes generic event logging for Windows and Linux guest operating systems.RequirementsThe Splunk Server can be downloaded from the web site - http://www.splunk.com/ . For this document,we will be using the Linux version on an Ubuntu server VM. Refer to the documentation for systemrequirements, but current recommendation call for a dual processor system with 4GB RAM. There areversions of Splunk available for several operating systems, including Windows. I chose Ubuntu for thisdocument because it is free and easy to install.Windows Event Logs will be monitored using SNARE. Windows text based log files will be monitoredusing Epilog. Both of these agents can be downloaded from http://sourceforge.net/projects/snare/ .Preparing the Splunk ServerCreate a VM with 2 vCPUs and 4GB vRAM. Install Ubuntu server, choose the OpenSSH package duringthe installation if you wish to perform remote administration. No other packages are required. Once theinstallation is complete, update your system by running sudo apt-get update and then sudo apt-getupgrade. For more information, refer to the official server guide at:https://help.ubuntu.com/9.04/serverguide/C/index.html orhttps://help.ubuntu.com/9.04/serverguide/C/serverguide.pdfAfter the system is updated, install VMware Tools following this guide:https://help.ubuntu.com/community/VMware/ToolsDownload the latest copy of Splunk Server from the Splunk web site and copy it to the /tmp directory onthe Ubuntu server using SCP.David Convery Pg 3 of 18 www.dailyhypervisor.com

Installing Splunk ServerSetting up Splunk forVMware MonitoringCopy the latest installation package tothe /tmp directory using SCP.Use the dpkg utility to install thepackage:sudo dpkg –I splunk-3.x.x-xxxxx-linux-2.6-amd64.debEnter your password.David Convery Pg 4 of 18 www.dailyhypervisor.com

Setting up Splunk forVMware MonitoringNote the installation path and the url foryour new Splunk Server:$SPLUNK_HOME = /opt/splunkhttp://SplunkServerFQDN:8000Start the Splunk Server and accept thelicense:sudo /opt/splunk/bin/splunk start --accept-licenseConfirm the Splunk Server has started.David Convery Pg 5 of 18 www.dailyhypervisor.com

Setting up Splunk forVMware MonitoringEnable the Splunk Server to start on boot:sudo /opt/splunk/bin/splunk enable bootstartConfirm that auto start is enabled.CONGRATULATIONS! You have installed your Splunk Server. Easy huh?David Convery Pg 6 of 18 www.dailyhypervisor.com

Setting up Splunk forVMware MonitoringSetting up ESX Servers and Linux VMs for MonitoringSetting up an ESX or a Linux Server for monitoring requires you to edit the /etc/syslog.conf configurationfile. In this example, we will use vi, but nano can be used as well.Login to the console as root (or use sudo)and begin to edit the /etc/syslog.conf file:vi /etc/syslog.conf-ORsudovi /etc/syslog.confGo to the end of the file by entering:GAPPEND the line by entering:aAdd a line to point to the Splunk Syslogserver:# send all to syslog server*.* @192.168.23.140(Substitute the IP address for your SplunkServer)David Convery Pg 7 of 18 www.dailyhypervisor.com

Setting up Splunk forVMware MonitoringWRITE and QUIT by entering::wqRestart the syslog service, enter:/etc/init.d/syslog restartThe syslog service will restartDavid Convery Pg 8 of 18 www.dailyhypervisor.com

Setting up Splunk forVMware Monitoring(ESX Only) Configure and restart thefirewall:esxcfg-firewall –o 514,udp,out,syslogesxcfg-firewall –lSetting up ESXi Servers for MonitoringSetting up ESXi servers requires connection using the VI Client, either directly to the ESXi Server orthrough the vCenter Server. As an alternative, you can use the vCLI or PowerCLI for this task.Open the VI Client and select the ESXiserver. Click on the Configuration taband then on the Advanced link. Browseto Syslog > Remote and enter the IPaddress and port for the Splunk Server.David Convery Pg 9 of 18 www.dailyhypervisor.com

Setting up Splunk forVMware MonitoringSetting up vCenter Servers and Windows VMs for MonitoringMonitoring a vCenter Server, a VMware Update Manager Server and any Windows based physical orvirtual machine will require agents to parse the logs and send them to the syslog server. We will useSNARE and Epilog for this purpose. These agents can be downloaded fromhttp://sourceforge.net/projects/snare/ .Setting up SNARESNARE will be used to monitor the Windows Event Logs. It will convert them and send to the syslogserver.Download the SNARE installer package for yourversion of Windows. Use the Vista package forWindows 2008.For remote control, select:Yes – with password, local access onlyDavid Convery Pg 10 of 18 www.dailyhypervisor.com

Setting up Splunk forVMware MonitoringOpen a browser and go to:http://localhost:6161Login credentials are snare / snareClick on the Network Configuration link in theleft pane.Enter the IP address and port of the SplunkServer.Enable SYSLOG HeaderSet SYSLOG Facility to SyslogSet SYSLOG Priority to DYNAMICDavid Convery Pg 11 of 18 www.dailyhypervisor.com

Setting up Splunk forVMware MonitoringClick on the Remote Control Configuration linkin the left pane.Enter a secure passwordEdit the Filtering Objectives Configuration asnecessary by clicking Modify.Edit as needed.David Convery Pg 12 of 18 www.dailyhypervisor.com

Setting up Splunk forVMware MonitoringAfter a few minutes, click on the Latest Eventslink in the left pane and verify that the EventLog is being monitored.Setting up EpilogEpilog will be used to monitor all of the text based log files associated with the vSpere services and anyother services that you wish to monitor.Check out http://viops.vmware.com/home/community/management/logging for some good logginginformation. Check http://kb.vmware.com/kb/1010956 for the VMO logs.Download the Epilog installer package for yourversion of Windows. Use the Vista package forWindows 2008.David Convery Pg 13 of 18 www.dailyhypervisor.com

Setting up Splunk forVMware MonitoringOpen a browser and go to:http://localhost:6162Login with snare / snareClick on the Log Configuration link in the leftpane.Click on the Add button.Enter the path and log name format for the logfile you wish to monitor and click ChangeConfiguration.David Convery Pg 14 of 18 www.dailyhypervisor.com

Setting up Splunk forVMware MonitoringRepeat for each additional log file.Click on the Network Configuration link in theleft pane, edit the settings and click ChangeConfiguration.Click on the Remote Control Configuration linkin the left pane, reset the snare password andclick Change Configuration.David Convery Pg 15 of 18 www.dailyhypervisor.com

Setting up Splunk forVMware MonitoringClick on the Objectives Configuration link in theleft pane. Click on the Add button.Enter * and click Change Configuration.When finished, click on the Apply the LatestAudit Configuration link in the left pane.Click on the Latest Events link in the left paneto monitor the logs.David Convery Pg 16 of 18 www.dailyhypervisor.com

Confirming a System is MonitoredSetting up Splunk forVMware MonitoringAt the Splunk console, type the followingto check that a server is being monitored:/opt/splunk/bin/splunk list udpThis will return a list of UDP ports beingmonitored and the systems beingmonitored.David Convery Pg 17 of 18 www.dailyhypervisor.com

Setting up Splunk forVMware MonitoringAdding Miscellaneous Log Files to Splunk for TroubleshootingSometimes, it may be necessary to add log files from an unmonitored or isolated source to aid introubleshooting. The steps listed below demonstrate how to do this.Add a directory for your log files:sudo mkdir /var/SplunkMonitorSet the proper permissions:sudo chmod 755 /var/SplunkMonitorIn this case, I am copying a VizionCore vConverterlog bundle to the directory.Unzip the log bundle if required.Add the directory within Splunk:/opt/splunk/bin/splunk add monitor /var/SplunkMonitorSplunk will return that the directory is beingmonitored. Any text based files placed in thisdirectory will be automatically scanned.For a good article about how to collect diagnostic data from all of the VMware products, check out thisarticle -> http://kb.vmware.com/kb/1008524David Convery Pg 18 of 18 www.dailyhypervisor.com