Caché Monitoring Guide - InterSystems Documentation

intersystems.com

Caché Monitoring Guide - InterSystems Documentation

Caché Monitoring GuideVersion 5.115 June 2006InterSystems Corporation 1 Memorial Drive Cambridge MA 02142 www.intersystems.com


Caché Monitoring GuideCaché Version 5.1 15 June 2006Copyright © 2006 InterSystems Corporation.All rights reserved.This book was assembled and formatted in Adobe Page Description Format (PDF) using tools and information fromthe following sources: Sun Microsystems, RenderX, Inc., Adobe Systems, and the World Wide Web Consortium atwww.w3c.org. The primary document development tools were special-purpose XML-processing applications builtby InterSystems using Caché and Java.The Caché product and its logos are trademarks of InterSystems Corporation.The Ensemble product and its logos are trademarks of InterSystems Corporation.The InterSystems name and logo are trademarks of InterSystems Corporation.This document contains trade secret and confidential information which is the property of InterSystems Corporation,One Memorial Drive, Cambridge, MA 02142, or its affiliates, and is furnished for the sole purpose of the operationand maintenance of the products of InterSystems Corporation. No part of this publication is to be used for any otherpurpose, and this publication is not to be reproduced, copied, disclosed, transmitted, stored in a retrieval system ortranslated into any human or computer language, in any form, by any means, in whole or in part, without the expressprior written consent of InterSystems Corporation.The copying, use and disposition of this document and the software programs described herein is prohibited exceptto the limited extent set forth in the standard software license agreement(s) of InterSystems Corporation coveringsuch programs and related documentation. InterSystems Corporation makes no representations and warrantiesconcerning such software programs other than those set forth in such standard software license agreement(s). Inaddition, the liability of InterSystems Corporation for any losses or damages relating to or arising out of the use ofsuch software programs is limited in the manner set forth in such standard software license agreement(s).THE FOREGOING IS A GENERAL SUMMARY OF THE RESTRICTIONS AND LIMITATIONS IMPOSED BYINTERSYSTEMS CORPORATION ON THE USE OF, AND LIABILITY ARISING FROM, ITS COMPUTERSOFTWARE. FOR COMPLETE INFORMATION REFERENCE SHOULD BE MADE TO THE STANDARD SOFTWARELICENSE AGREEMENT(S) OF INTERSYSTEMS CORPORATION, COPIES OF WHICH WILL BE MADE AVAILABLEUPON REQUEST.InterSystems Corporation disclaims responsibility for errors which may appear in this document, and it reserves theright, in its sole discretion and without notice, to make substitutions and modifications in the products and practicesdescribed in this document.Caché, InterSystems Caché, Caché SQL, Caché ObjectScript, Caché Object, Ensemble, InterSystems Ensemble,Ensemble Object, and Ensemble Production are trademarks of InterSystems Corporation. All other brand or productnames used herein are trademarks or registered trademarks of their respective companies or organizations.For Support questions about any InterSystems products, contact:InterSystems Worldwide Customer SupportTel: +1 617 621-0700Fax: +1 617 374-9391Email: support@InterSystems.com


Table of ContentsIntroduction ........................................................................................................................ 11 Monitoring Caché Using the System Management Portal ........................................... 31.1 Monitoring System Dashboard Indicators ............................................................... 31.2 Monitoring System Performance ............................................................................. 71.3 Monitoring Locks ..................................................................................................... 81.4 Monitoring Log Files ............................................................................................. 101.5 Viewing CSP Sessions ........................................................................................... 111.6 Viewing Background Tasks .................................................................................... 112 Using the Caché Monitor .............................................................................................. 132.1 Monitor Architecture .............................................................................................. 132.2 System Monitor ...................................................................................................... 142.2.1 System Monitor Manager ............................................................................. 142.3 Application Monitor ............................................................................................... 162.4 Caché Metrics ........................................................................................................ 162.5 Alerts ...................................................................................................................... 172.6 Managing the Application Monitor ........................................................................ 172.6.1 Manage Application Monitor ....................................................................... 182.6.2 Manage Monitor Classes .............................................................................. 182.6.3 Manage Email Options ................................................................................. 192.6.4 Manage Alerts ............................................................................................... 192.7 Writing User-defined Monitor Classes .................................................................. 202.7.1 Example of a User-defined Monitor Class ................................................... 212.8 Generating Metrics ................................................................................................. 242.9 Viewing Metrics Data ............................................................................................. 252.10 Caché Monitor Errors .......................................................................................... 253 Gathering Global Activity Statistics Using ^GLOSTAT ............................................ 293.1 Running ^GLOSTAT .............................................................................................. 293.2 Overview of ^GLOSTAT Statistics ........................................................................ 313.3 Examples of ^GLOSTAT Output ........................................................................... 323.3.1 Example A .................................................................................................... 333.3.2 Example B .................................................................................................... 333.3.3 Example C .................................................................................................... 34Caché Monitoring Guideiii


3.3.4 Example D .................................................................................................... 343.3.5 Example E .................................................................................................... 373.3.6 Example F ..................................................................................................... 373.3.7 Example G .................................................................................................... 384 Monitoring System Performance Using ^PERFMON ............................................... 414.1 Running PERFMON .............................................................................................. 434.1.1 PERFMON Report ....................................................................................... 455 Examining Routine Performance Using ^%SYS.MONLBL ..................................... 515.1 Start Monitoring ..................................................................................................... 515.2 Monitoring Options ................................................................................................ 545.3 Sample Output ....................................................................................................... 555.4 Programming Interface .......................................................................................... 56Appendix A: Monitoring Caché Using BMC PATROL ................................................ 57A.1 Running PATROL with Caché .............................................................................. 57A.1.1 Configure PATROL Settings ....................................................................... 58A.1.2 Caché PATROL Routine .............................................................................. 59A.2 Caché PATROL Knowledge Modules ................................................................... 59A.2.1 Adding Caché Modules to PATROL ............................................................ 60A.3 Caché Metrics Used with BMC PATROL ............................................................. 61Appendix B: Monitoring Caché Using SNMP ............................................................... 67B.1 Running SNMP with Caché .................................................................................. 67B.2 Managing SNMP in Caché .................................................................................... 68B.3 Caché as a Subagent .............................................................................................. 69B.4 Caché MIB Structure ............................................................................................. 70B.4.1 Extending the Caché MIB ........................................................................... 71ivCaché Monitoring Guide


List of TablesSystem Performance Indicators ............................................................................................ 4ECP and Shadowing Indicators ............................................................................................ 5System Time Indicators ........................................................................................................ 5System Usage Indicators ...................................................................................................... 6Errors and Alerts Indicators .................................................................................................. 6Licensing Indicators ............................................................................................................. 7Task Manager Upcoming Tasks ............................................................................................ 7System Usage Statistics ........................................................................................................ 8Lock Modes ........................................................................................................................ 10Statistics Produced by ^GLOSTAT .................................................................................... 31Fields for Custom Report ................................................................................................... 47Caché PATROL Metrics ..................................................................................................... 62Caché Monitoring Guidev


IntroductionThis book describes several tools available for monitoring Caché. The following chaptersdescribe how to monitor Caché with tools included with Caché:• Monitoring Caché Using the System Management Portal — The System Dashboard ofthe System Management Portal contains many metrics that show you the state of yourCaché instance at a glance. You can also click a details link to see more details of thesemetrics. Many of these metrics are also accessible in other locations in the Operationslist of the System Management Portal.• Using the Caché Monitor — The Caché Monitor is a system monitor facility that providesservices for gathering and monitoring performance and system data, generating alerts,handling notifications, and generating and maintaining a persistent repository of suchdata.The following chapters describe how to monitor Caché with system routines:• Gathering Global Activity Statistics Using ^GLOSTAT• Monitoring System Performance Using ^PERFMON• Examining Routine Performance Using ^%SYS.MONLBLThe following appendixes describe how to monitor Caché with various third-party tools:• Monitoring Caché Using BMC PATROL• Monitoring Caché Using SNMPCaché Monitoring Guide 1


1Monitoring Caché Using theSystem Management PortalYou can monitor many aspects of your Caché instance starting at the System Dashboard ofthe System Management Portal. From this page, you can view performance indicators andthen, for selected indicators, navigate to more detailed information. This chapter describesthe following monitoring tasks:• Monitoring System Dashboard Indicators• Monitoring Locks• Monitoring Log Files• Monitoring System Performance• Viewing CSP Sessions• Viewing Background Tasks1.1 Monitoring System Dashboard IndicatorsThe [Home] > [System Dashboard] page of the System Management Portal displays real-timestatus of key system performance indicators in the following categories. Each category isdescribed in one of the tables that follow.• System Performance IndicatorsCaché Monitoring Guide 3


Monitoring Caché Using the System Management Portal• ECP and Shadowing Indicators• System Time Indicators• System Usage Indicators• Errors and Alerts Indicators• Licensing Indicators• Task Manager IndicatorsIn most cases, you can click any of these indicators to display a description of the metric inthe bottom detail box at the lower left corner of the page.System Performance IndicatorsIndicatorGlobals/SecondGlobal RefsGlobal SetsRoutine RefsLogical RequestsDisk ReadsDisk WritesCache EfficiencyDefinitionMost recently measured number of global references per second.Number of global references since system startup.Number of global Set and Kill operations since system startup.Number of routine loads and saves since system startup.Number of logical block requests since system startup.Number of physical block read operations since system startup.Number of physical block write operations since system startup.Most recently measured cache efficiency (Global references /(physical reads + writes)).In the description box, click the details link to display the [Home] > [System Usage] page inthe bottom detail box. See the Monitoring System Performance section for details.4 Caché Monitoring Guide


ECP and Shadowing IndicatorsMonitoring System Dashboard IndicatorsIndicatorApplication ServersApplication ServerTrafficData ServersData Server TrafficShadow ClientsShadow ServersDefinitionSummary status of ECP (Enterprise Caché Protocol) applicationservers connected to this system.Most recently measured ECP application server traffic inbytes/second.Summary status of ECP data servers to which this system isconnected.Most recently measured ECP data server traffic in bytes persecond.Summary status of shadow clients connected to this system.Summary status of shadow servers connected to this systemas a data source.For more information on the first four indicators, the ECP indicators, see the “ConfiguringDistributed Systems” chapter of the Caché Distributed Data Management Guide.In the description box for the last two indicators, the shadow indicators, click the details linkto display the [Home] > [Shadow Servers] page. For more information on shadowing, see the“Shadow Journaling” chapter of the Caché High Availability Guide.System Time IndicatorsIndicatorSystem Up TimeLast BackupDefinitionElapsed time since this system was started.Date and time of last system backup.You can run backups or view the backup history from the [Home] > [Backup] page. For moreinformation on developing a backup plan, see the “Backup and Restore” chapter of theCaché High Availability Guide.Caché Monitoring Guide 5


Monitoring Caché Using the System Management PortalSystem Usage IndicatorsIndicatorDatabase SpaceJournal SpaceJournal EntriesLock TableWrite DaemonProcessesCSP SessionsMost Active ProcessesDefinitionIndicates whether there is a reasonable amount of disk spaceavailable for database files. Clicking details displays the [Home]> [Databases] > [Freespace] page.Indicates whether there is a reasonable amount of disk spaceavailable for journal files. Clicking details displays the [Home] >[Journals] page.Number of entries written to the system journal. Clicking detailsdisplays the [Home] > [Journals] page.Indicates the current status of the system Lock Table. Clickingdetails displays the [Home] > [Locks] page.Indicates the current status of the system Write daemon.Most recent number of running processes. Clicking detailsdisplays the [Home] > [Processes] page.Most recent number of CSP sessions. Clicking details displaysthe [Home] > [CSP Sessions] page.Running processes with highest amount of activity (lines of codeexecuted). Clicking details displays the [Home] > [Processes] page.For more information on each of these topics, click Help under the InterSystems logo on theportal page displayed when you click for more details.Errors and Alerts IndicatorsIndicatorSerious AlertsApplication ErrorsDefinitionNumber of serious alerts that have been raised. Clicking detailsdisplays the [Home] > [System Logs] > [View Console Log] page.Number of application errors that have been logged. Clickingdetails displays the [Home] > [System Logs] > [View Application ErrorLog] page.See the Monitoring Log Files section in this chapter for more details.6 Caché Monitoring Guide


Monitoring System PerformanceLicensing IndicatorsIndicatorLicense LimitCurrent License UseHighest License UseDefinitionMaximum allowed license units for this system.License usage as a percentage of available license units.Highest license usage as a percentage of available license units.In the description box, click the details link to display the [Home] > [License Usage] > [LicenseActivity Summary] page. For more information on licensing, see the “Managing CachéLicenses” chapter of the Caché System Administration GuideTask Manager Upcoming TasksIndicatorUpcoming TasksTaskTimeStatusDefinitionLists a maximum of five tasks scheduled to run within the nexttwenty-four hours.Name of the upcoming task.Time the task is scheduled to run.Task status, one of: scheduled, completed, running.In the description box, click the details link to display the [Home] > [Task Manager] > [ViewUpcoming Tasks] page. For details on the Task Manager, see the Using the Task Managersection of the “Managing Caché” chapter of the Caché System Administration Guide.1.2 Monitoring System PerformanceThis section describes the output of the System Usage page of the System Management Portal.Navigate to the [Home] > [System Usage] page to view several metrics that help you monitorsystem performance. See the “Gathering Global Activity Statistics with ^GLOSTAT”chapter for an alternative method of retrieving these statistics.Caché Monitoring Guide 7


Monitoring Caché Using the System Management PortalSystem Usage StatisticsStatisticGlobal references (all)Global updatereferencesRoutine callsRoutine buffer loadsand savesLogical block requestsBlock readsBlock writesJournal entriesJournal block writesDefinitionLogical count of accesses to globals, including Sets, Kills,$Data, $Order, $Increment, $Query, and global references inexpressions.Logical count of global references that are Set, Kill, or$Increment operations.Number of calls to a routine.Total number of routine loads and saves as a result of ZLoad,ZSave, and running routines. (In a well-tuned environment, thisnumber increases slowly, since most routine loads are satisfiedby the routine cache memory without accessing the disk. Eachroutine load or save transfers up to 32 KB of data (64 KB forUnicode).)Number of database blocks read by the global database code.(In a well-tuned environment, many of these reads are satisfiedwithout disk access.)Number of physical database blocks (2-KB or 8-KB) read fromdisk for both global and routine references.Number of physical database blocks (2-KB or 8-KB) written todisk for both global and routine references.Number of journal records created—one for each databasemodification (Set , Kill, etc.) or transaction event (TStart,TCommit) or other event that is saved to the journal.Number of 64-KB journal blocks written to the journal file.Routine linesLast UpdateDate and timestamp of the displayed statistics.1.3 Monitoring LocksCaché locks are created when a Caché process issues a Lock command on a Caché entity,such as a local or global variable, as long as the entity is not already locked by another process.Entities need not exist in the database to lock them.8 Caché Monitoring Guide


To display or delete locks, navigate to the [Home] > [Locks] page from the Operations sectionof the System Management Portal home page. The table lists one row for each lock ownerand lock waiter. If one lock is owned by more than one process, each owner has its own row.The Lock Table has the following column entries.Monitoring LocksColumn HeadingOwnerMode/CountReferenceDirectorySystemRemoteRemoveDefinitionThe process ID of the process holding or waiting for the lock.Contains the client system name if it is a remote lock.Lock mode and count of the lock item. If the lock count is 1 thecount is not displayed. For details, see the Lock Modes table.Lock reference string of the lock item (does not include thedatabase name).The database location of the lock item.The system name of the lock located, if it is the local system thecolumn is blank.Indicates whether the lock is located on a remote system. a zero(0) indicates the lock is local.If this lock is removable, this option along with the Remove alllocks for process option appears in the row. Click the appropriatebutton to remove the lock or all locks for the process.In most cases, the only time you need to remove locks is as a result of an application problem.For a more in-depth description of the Lock command and its features, see the Lock entryof the Caché ObjectScript Reference guide.You may need to enlarge the size of the lock table if your system uses a large number oflocks. You can do this using the System Management Portal:1. Navigate to the [Home] > [Configuration] > [Advanced Settings] page and enter Lock inthe Filter box to shorten the list.2. In the LockTableSize row, click Edit to display the Configuration Setting page.3. In the Value box, update the amount of memory allocated on your system for locks (inbytes), and click OK.The minimum is 65536; the maximum value depends on the value of GenericHeapSize.Increase the heap size if you need more room for the lock table. Caché rounds up thevalue to the next multiple of 64 KB. The default range is from 65536–1769472.Caché Monitoring Guide 9


Monitoring Caché Using the System Management Portal4. Click Apply Changes to activate the new value when Caché restarts.The following table lists the possible values of the mode portion of the Mode/Count column.Lock ModesModeExclusiveSharedLockZAWaitLockWaitShareWaitLockZALockPendingSharePendingDelockPendingLostDescriptionExclusive lock mode.Share lock mode.ZALLOCATE lock mode.Waiting for exclusive lock mode.Waiting for share lock mode.Waiting for ZALLOCATE lock mode.Exclusive lock pending, waiting for server to grant the exclusivelock.Share lock pending, waiting for server to grant the share lock.Delock pending, waiting for server to release the lock.Lock lost due to network reset.For more detailed information, see the SYS.Lock class documentation in the Caché ClassReference.1.4 Monitoring Log FilesCaché reports general messages, system errors, certain operating system errors, and networkerrors through an operator console facility. For Caché Windows-based systems, there is nooperator console as such and all console messages are sent to a log file, cconsole.log, in thesystem manager directory, typically CacheSys\Mgr. For Caché systems on UNIX-based orOpenVMS platforms, you can send operator console messages to the console log file or theconsole terminal.The console log file is a plain text file and may be viewed with any editor or text viewer andby using the System Management Portal:10 Caché Monitoring Guide


Viewing CSP Sessions1. Navigate to the [Home] > [System Logs] page by clicking System Logs in the Operationscolumn of the home page.2. Click View Console Log under the System Logs column to display the text output of theconsole log file.If you have any trouble starting Caché, it is recommended that you view the console log file.The cconsole.log file grows until it reaches a maximum size, when it is renamed cconsole.old.The value is configurable (default is 5 MB) from the [Home] > [Configuration] > [AdvancedSettings] page of the System Management Portal. Update the number of MBs for the ConsoleLogSizesetting.You may also view log information about application errors and ODBC errors from the [Home]> [System Logs] > [View Application Error Log] and the [Home] > [System Logs] > [View ODBCError Log] pages of the System Management Portal.1.5 Viewing CSP SessionsNavigate to the [Home] > [CSP Sessions] page to view all currently active CSP sessions.1.6 Viewing Background TasksNavigate to the [Background Tasks] page to view the logs of any completed or runningbackground tasks.Caché Monitoring Guide 11


2Using the Caché MonitorThe Caché System Monitor is an extendable system monitor facility. It provides services forgathering and monitoring performance and system data, generating alerts, handling notifications,and generating and maintaining a persistent repository of such data.2.1 Monitor ArchitectureThe Caché Monitor is a modular system that can run multiple instances of the Monitor, inmultiple namespaces.Each Monitor instance acts the same in each namespace in which it is activated:• At startup time, the Monitor examines a list of which Monitor Metrics are active. Foreach active Metric, the Monitor reads metadata that tells the Monitor which Metricsclasses to invoke, and the alerts that have been defined for each such class.• The Monitor then invokes standard methods in each of the active Metrics classes. TheMetrics data returned from each Metrics class is examined to determine the following:- If the data is marked as persistent, save the persistent sample class.- If an alert is defined on that class, pass the sample data to the alert handler. The alerthandler determines if a threshold has been reached and, if so, execute a notificationper the user instructions.Caché Monitoring Guide 13


Using the Caché Monitor2.2 System MonitorThe System Monitor is a variant of the standard Caché Monitor, adapted to special systemusage.At Caché startup, a System Monitor is started in the %SYS namespace. This Monitor unconditionallymonitors the internal Caché system traps which have been designed to work withthe System Monitor. No persistent data is kept.Alerts are handled in the same way as for the standard Monitor. An alert is generated for anysystem trap that occurs, and notification provided per the user-defined instructions for emailnotification. See below for instructions on setting email options.See the Monitor Errors section on the system traps that are handled by the System Monitor.2.2.1 System Monitor ManagerThe System Monitor Manager (^MONMGR) is a character-based application that allowsyou to perform System Monitor management functions. It must be executed in the %SYSnamespace, and sets parameters for System Monitor only.The options provided in ^MONMGR are:%SYS>Do ^MONMGR1) Manage MONITOR2) Set Email options3) List Email options4) Test Email options5) Set Sample Interval6) ExitOption?Start, Stop, Refresh the System Monitor. Refreshing the Monitor causes it to pick up newemail options and sample interval, without having to restart the System Monitor.Option? 11) Refresh MONITOR2) Halt MONITOR3) Start MONITOR4) ExitOption?Email options provide for sending email on notification. The user specifies the following:14 Caché Monitoring Guide


System Monitor1. Mail server — The user must specify the name of the email server that handles email intheir environment. The user’s IT staff should be able to provide this information.2. Recipients — A list of recipients of the email. The list must be a list of valid emailaddresses.3. Sender — This is text the defines the sender of the email. It need not be a replyable emailaccount.Option? 2Mail server? doc.serverRecipients? userdoc@company.comSender? userdoc@company.com1) Manage MONITOR2) Set Email options3) List Email options4) Test Email options5) Set Sample Interval6) ExitOption? 3Mail server: doc.serverRecipients: userdoc@company.comSender: userdoc@company.com1) Manage MONITOR2) Set Email options3) List Email options4) Test Email options5) Set Sample Interval6) ExitOption? 4Sending email on Mail Server doc.serverFrom: userdoc@company.comTo: userdoc@company.com1) Manage MONITOR2) Set Email options3) List Email options4) Test Email options5) Set Sample Interval6) ExitOption?The Monitor runs repeatedly once started. The default time between sample gathering andalerts processing is ten seconds. This time may be changed to any interval by setting thisoption (5).Caché Monitoring Guide 15


Using the Caché Monitor2.3 Application MonitorThe Application Monitor provides for monitoring Caché system-defined and user-definedmetrics. You first registers system classes in the local namespace, and create your own userdefinedmetric classes. Next, you activate the classes you want to actively monitor. You thenstart an instance of the Application Monitor (%MONAPP) in the local namespace.When an Application Monitor is started in the local namespace, it searches for all classes thathave been activated. For each such class, it creates sample data in local namespace globals.Monitor metadata is kept in globals ^ISCMonitor(). Sample data is kept as persistent data forthe sample class.You may view this data by invoking class methods, or via a browser. All sample classes areautomatically CSP-enabled.See the sections on Managing the Application Monitor, and writing your own User-DefinedMonitor Classes, below.^ISCMonitor("Monitor","AppInterval")=300^ISCMonitor("Monitor","Interval")=10^ISCMonitor("Monitor","Metric","%Monitor.System.Dashboard","Active")=1^ISCMonitor("Monitor","Recipients")=""^ISCMonitor("Monitor","SmtpServer")=""2.4 Caché MetricsThere are the following types of metrics with corresponding predefined Caché sample systemmetric classes. View the Caché Class Reference entry for a list of properties correspondingto the sample metrics in each category:• System Activity Counters — %Monitor.System.Sample.SystemMetrics• Global Metrics — %Monitor.System.Sample.Globals• Server Metrics — %Monitor.System.Sample.Servers• Client Metrics — %Monitor.System.Sample.Clients• Free Space Metrics — %Monitor.System.Sample.Freespace• Audit Metrics — %Monitor.System.Sample.AuditCount16 Caché Monitoring Guide


• Console Metrics — %Monitor.System.Sample.Dashboard Monitors the cconsole.log file.Alerts may be defined on severity level. The System Monitor always monitors thecconsole.log file, and provides alerts for severity level of Severe or higher.These classes monitor counters of system activity. You can obtain global metrics for all theglobals in the current namespace (%Monitor.System.Globals) and system metrics for totalsystem activity (%Monitor.System.SystemMetrics). You can also obtain global and systemmetrics relative to:• Routines — %Monitor.System.Sample.Routines) – for each routine currently executing• Processes (%Monitor.System.Sample.Processes)– for each Caché processAlerts2.5 AlertsAn alert specifies a list of properties in a class to monitor. An expression is specified, forexample, “%1 < 100 && %1 > 20” . During Monitor sampling, the values of the propertiesin the class are evaluated. If the expression evaluates to true, an alert is triggered. A notificationaction is taken whenever an alert is triggered.The expression provides for parameter substitution. %1 indicates the first property in the listof monitored properties, %2 the second property, etc.An Alert is defined by using the Monitor Manager Alert management functions.2.6 Managing the Application MonitorThe Monitor Manager (^%MONAPPMGR) is a character-based utility that allows the userto do Management functions. It may be executed in a user’s namespace, and sets parametersfor that namespace only. In this way, multiple Managers, each with their own activated classes,and notification options, may all run at the same time.Caché Monitoring Guide 17


Using the Caché Monitor%SYS>Do ^%MONAPPMGR1) Manage Application Monitor2) Manage Monitor Classes3) Manage Email options4) Manage Alerts5) ExitOption?2.6.1 Manage Application MonitorOption? 11) Start Application Monitor2) Halt Application Monitor3) Refresh Application Monitor4) Activate/Deactivate a Monitor Class5) Set Sample Interval6) ExitOption?Start, Stop, Refresh the Monitor — Refreshing the Monitor causes it to pick up new lists ofactivated classes, and notification options, without having to restart the Monitor.Activate/Deactivate a Monitor Class — Of all the classes defined in the namespaces asMonitor-enabled, only those that are marked as Active will actually be sampled by theApplication MonitorInterval — The Monitor runs repeatedly once started. The default time between samplegathering and alerts processing is 20 seconds. This time may be changed to any interval bysetting this option.2.6.2 Manage Monitor ClassesOption? 21) Activate/Deactivate a Monitor Class2) List Monitor Classes3) Register Monitor System Classes4) ExitOption?1. Activate/Deactivate a Monitor Class Of all the classes defined in the namespaces asMonitor-enabled, only those that are marked as Active will actually be sampled by theApplication Monitor2. List Monitor Classes — Lists all the Monitor classes defined in the namespace, andwhether or not they are activated.18 Caché Monitoring Guide


Managing the Application Monitor3. Register Monitor System Classes — System classes must be registered in the localnamespace in which they will be monitored. Registration sets Monitor globals in the localnamespace that tells the Monitor of the activation status of defined classes.Option? 2ClassActive----- ------%Monitor.System.AuditCountN%Monitor.System.ClientsN%Monitor.System.DashboardY%Monitor.System.DatabaseN%Monitor.System.FreespaceN%Monitor.System.GlobalsN%Monitor.System.ProcessesN%Monitor.System.RoutinesN%Monitor.System.ServersN%Monitor.System.SystemMetricsN1) Activate/Deactivate a Monitor Class2) List Monitor Classes3) Register Monitor System Classes4) ExitOption?2.6.3 Manage Email OptionsOption? 31) Set Email options2) List Email options3) Test Email options4) ExitOption?Email notification may be configured in the same way as for the System Monitor. See thesection on the System Monitor for email definitions.When alerts are defined, they may be configured to use email as a means of notification. SeeAlerts below.2.6.4 Manage AlertsOption? 41) Create Alert2) Delete Alert3) List Alerts4) ExitOption?Caché Monitoring Guide 19


Using the Caché Monitor1) Create Alert — To create an alert, specify the following:Input FieldAlert NameApplication NameActionNotify MethodClassPropertiesExpressionPurposeName of the alertQualifier to group alerts. Give a set of alerts a group name to gatherthose alerts into a user-defined “Application.”One of: None, Notify via email, or Notify via user-class method.(Full class name and method, for example, A.B.MyNotify) If anaction is selected as Notify via class method, then this Notify Methodis called if the alert is triggered and the Notify Method displays alist of its values.Name of the Metric class to monitor for alerts.Property of the Monitor Class to be evaluated.Expression to use to evaluate the Property. For example, “%1 =“User” && %2 < 100”. %1 refers to the first property in the list ofproperties, %2 the second property, etc.2.7 Writing User-defined Monitor ClassesThe Cache Monitor provides predefined system classes for monitoring system metrics. Usersmay also write their own Monitor Classes to monitor user application data and counters.A Monitor Class is any class that inherits from the abstract Monitor class, %Monitor.Abstract.The %Monitor.System classes are examples of Monitor Classes. Users may also write theirown Monitor Classes by following these steps:1. Write a class that inherits from %Monitor.Adaptor. The inheritance provides persistence,parameters, properties, code generation, and a projection that generates the monitormetadata from the user’s class definition. See the %Monitor.Adaptor class documentationin the Caché Class Reference for full details on this class, and the code that the user mustwrite.2. Compiles your class. Compiling classes that inherit from %Monitor.Adaptor generatesnew classes in a subpackage of the users class called Sample. So, for instance, if a useris compiling A.B.MyMetric, a new class is generated in A.B.Sample.MyMetric. The userneed do nothing with this class.20 Caché Monitoring Guide


All Sample classes are automatically CSP-enabled, so that sample data for the users metricmay be viewed by pointing to A.B.Sample.MyMetric.cls.The Monitor automatically invokes this class and generates data and alerts if the class hasbeen Activated. See Monitor Manager for class Activation.2.7.1 Example of a User-defined Monitor ClassOur example gets the free space for each dataset in a Caché instance. At each sampling, wewould want n instances of sample data objects, each instance corresponding to a dataset. Inour example, each instance has only a single property, the free space available in that datasetwhen the sample was collected.1. Create a class that inherits from %Monitor.Adaptor:Writing User-defined Monitor ClassesClass MyMetric.Freespace Extends %Monitor.Adaptor [ ProcedureBlock ]{}2. Add properties that you want to be part of the sample data. They must be of %Monitortypes:Class MyMetric.Freespace Extends %Monitor.Adaptor [ ProcedureBlock ]{/// Name of datasetProperty DBName As %Monitor.String;/// Current amount of FreespaceProperty FreeSpace As %Monitor.String;}3. Add an index parameter that tells which fields form a unique key among the instances ofthe samples:Class MyMetric.Freespace Extends %Monitor.Adaptor [ ProcedureBlock ]{Parameter INDEX = "DBName";}4. Add control properties as needed:Class MyMetric.Freespace Extends %Monitor.Adaptor [ ProcedureBlock ]{/// Result SetProperty Rspec As %Library.ResultSet;}5. Override a method named Initialize(). Initialize is called at the start of each metricsgathering run.Caché Monitoring Guide 21


Using the Caché MonitorClass MyMetric.Freespace Extends %Monitor.Adaptor [ ProcedureBlock ]{/// Initialize the list of datasets and freespace.Method Initialize() As %Status{s ..Rspec = ##class(%Library.ResultSet).%New("SYS.Database:FreeSpace")}}d ..Rspec.Execute("*",0)Quit $$$OK6. Override a method named GetSample(). GetSample() is called repeatedly until a statusof 0 is returned. The user writes code to populate the metrics data for each sample instance.Class MyMetric.Freespace Extends %Monitor.Adaptor [ ProcedureBlock ]{/// Get dataset metric sample./// A return code of $$$OK indicates there is a new sample instance./// A return code of 0 indicates there is no sample instance.Method GetSample() As %Status{// Get freespace data}Set stat = ..Rspec.Next(.sc)// Quit if we have done all the datasetsIf 'stat Q 0// populate this instanceSet ..DBName = ..Rspec.Get("Directory")Set ..FreeSpace = ..Rspec.Get("Available")// quit with return value indicating the sample data is readyQ $$$OK7. Compile the class. The class is shown below:22 Caché Monitoring Guide


Class MyMetric.Freespace Extends %Monitor.Adaptor [ ProcedureBlock ]{Parameter INDEX = "DBName";/// Name of datasetProperty DBName As %Monitor.String;/// Current amount of FreespaceProperty FreeSpace As %Monitor.String;/// Result SetProperty Rspec As %Library.ResultSet;/// Initialize routine metrics.Method Initialize() As %Status{s ..Rspec = ##class(%Library.ResultSet).%New("SYS.Database:FreeSpace")}d ..Rspec.Execute("*",0)Quit $$$OK/// Get routine metric sample./// A return code of $$$OK indicates there is a new sample instance./// Any other return code indicates there is no sample instance.{// Get freespace data}Set stat = ..Rspec.Next(.sc)// Quit if we have done all the datasetsIf 'stat Q $$$Error($$$GeneralError,”End of Sample”)// populate this instanceSet ..DBName = ..Rspec.Get("Directory")Set ..FreeSpace = ..Rspec.Get("Available")// quit with return value indicating the sample data is readyQ $$$OK8. Additionally, you may override the Startup() and Shutdown() methods. These methodsare called once when sampling begins, so the user may open channels or perform otherone-time-only initialization:Class MyMetric.Freespace Extends %Monitor.Adaptor [ ProcedureBlock ]{/// Open a tcp/ip device to send warningsMethod Startup() As %Status{Set ..io="|TCP|2"Set host="127.0.0.1"Open ..io:(host:^serverport:"M"):200}Method Shutdown() As %Status{Close ..io}Writing User-defined Monitor Classes9. Compiling the class has created a new class, Freespace, in the package MyMetric.Sample:Caché Monitoring Guide 23


Using the Caché Monitor/// Persistent sample class for MyMetric.FreespaceClass MyMetric.Sample.Freespace Extends %Monitor.Sample [ClassType=persistent]{Parameter INDEX = "DBName";Property Application As %String [ InitialExpression = "MyMetric" ];/// Name of datasetProperty DBName As %Monitor.String(CAPTION = "");/// Current amount of FreespaceProperty FreeSpace As %Monitor.String(CAPTION = "");Property GroupName As %String [InitialExpression = "Freespace"];Property MetricsClass As %String [InitialExpression = " MyMetric.Freespace "];}Note:You should not modify this class. You may, however, inherit from this class to writecustom queries against your sample data.2.8 Generating MetricsThe %Monitor.SampleAgent class is the class that does the actual sampling. It is this class thatinvokes the Initialize() and GetSample() methods of the metrics classes.The %Monitor.SampleAgent.%New(n) constructor takes one argument, the name of themetrics class it is to run. It creates an instance of that class, and invokes the Startup() methodon that class. Then, each time the %Monitor.SampleAgent.Collect() method is invoked,the Sample Agent invokes the class Initialize() method for the class, then repeatedly invokesthe GetSample() method for the class. On each call to GetSample(), the SampleAgent createsa sample class for the metrics class. The pseudo-code for these operations follows:Set sampler = ##class(%Monitor.SampleAgent).%New(“MyMetrics.Freespace”)/* at this point, the sampler has created an instance of MyMetrics.Freespace,and invoked its Startup method */for I=1:1:10 d sampler.Collect() h 10/* at each iteration, sampler calls MyMetrics.Freespace.Initialize(), then loopson GetSample(). Whenever GetSample() returns $$$OK, sampler creates a newMyMetrics.Sample.Freespace instance, with the sample data. When GetSample()returns an error value, no sample is created, and sampler.Collect() returns. */24 Caché Monitoring Guide


Viewing Metrics Data2.9 Viewing Metrics DataThe simplest way to view metrics is using a Web browser; all metrics classes are CSP-enabled.This means that every sample is available for viewing through a browser: In our example,the CSP url is:http://localhost:8972/csp/user/MyMetrics.Sample.Freespace.clsThe output looks like:Monitor - Freespace c:\cache51\Name of dataset: c:\cache51\Current amount of Freespace: 8.2MBMonitor - Freespace c:\cache51\mgr\Name of dataset: c:\cache51\mgr\Current amount of Freespace: 6.4MBThe CSP code is generated automatically when the sample class is generated.Alternatively, you use the Display() method of the %Monitor.View class:%SYS>s mclass="Monitor.Test.Freespace"%SYS>s col=##class(%Monitor.SampleAgent).%New(mclass)%SYS>w col.Collect()1%SYS>w ##class(%Monitor.View).Display(mclass)Monitor - Freespace c:\cache51\Name of dataset: c:\cache51\Current amount of Freespace: 8.2MBMonitor - Freespace c:\cache51\mgr\Name of dataset:c:\cache51\mgr\Current amount of Freespace: 6.4MB2.10 Caché Monitor ErrorsThe following system errors are notified by the System Monitor unconditionally:1. Process halt due to segment violation (access violation).2. in database %3. AUDIT: ERROR: FAILED to change audit database to '%.. Still auditing to '%.4. AUDIT: ERROR: FAILED to set audit database to '%.5. Sync failed during expansion of sfn #, new map not addedCaché Monitoring Guide 25


Using the Caché Monitor6. Sync failed during expansion of sfn #, not all blocks added7. WD failed to allocate wdqlist...freezing system8. WD: CP has exited - freezing system9. Write Daemon encountered serious error - System Frozen10. Insufficient global buffers - WD in panic mode11. WD Panic: SFN x Block y written directly to database12. Unexpected Write Error: dkvolblk returned %d for block #%d in %13. Unexpected Write Error: dkswrite returned %d for block #%d in %14. Unexpected Write Error: %d for block #%d in %.15. Cluster crash - All Cache systems are suspended16. System is shutting down poorly, because there are open transactions, or ECP failed topreserve its state17. SERIOUS JOURNALING ERROR: JRNSTOP cannot open %.* Stopping journaling ascleanly as possible, but you should assume that some journaling data has been lost.18. Unable to allocate memory for journal translation table19. Journal file has reached its maximum size of %u bytes and automatic rollover has failed20. Write to journal file has failed21. Failed to open the latest journal file22. Sync of journal file failed23. Journaling will be disabled in %d seconds OR when journal buffers are completely filled,whichever comes first. To avoid potential loss of journal data, resolve the cause of theerror (see ^SYSLOG) or switch journaling to a new device.24. Error logging in journal25. Journaling Error x reading attributes after expansion26. ECP client daemon/connection is hung27. Cluster Failsoft failed, couldn't determine locksysid for failed system - all cluster systemsare suspended28. enqpijstop failed, declaring a cluster crash29. enqpijchange failed, declaring a cluster crash26 Caché Monitoring Guide


Caché Monitor Errors30. Failure during WIJ processing - Declaring a crash31. Failure during PIJ processing - Declaring a crash32. Error reading block – recovery read error33. Error writing block – recovery write error34. WIJ expansion failure: System Frozen -The system has been frozen because WIJ expansionhas failed for too long. If space is created for the WIJ, the system will resume otherwiseyou need to shut it down with cforce35. CP: Failed to create monitor for daemon termination36. CP: WD has been on pass %d for %d seconds - freezing system. System will resume ifWD completes a pass37. WD: CP has died before we opened its handle - Freezing system38. WD: Error code %d getting handle for CP monitor - CP not being monitored39. WD: Control Process died with exit code %d - Freezing system40. CP: Daemon died with exit code %d - Freezing system41. Performing emergency Cache shutdown due to Operating System shutdown42. CP: All processes have died - freezing system43. cforce failed to terminate all processes44. Failed to start mlock process (OpenVMS)45. Failed to start slave write daemon46. ENQDMN exiting due to reason #47. Daemon %c found alive lock already owned by another process (OpenVMS)48. Daemon %c failed to acquire alive lock. (OpenVMS)Caché Monitoring Guide 27


3Gathering Global Activity StatisticsUsing ^GLOSTATCaché provides the ^GLOSTAT utility, which gathers global activity statistics and displaysa variety of information about disk I/O operations. This chapter describes how to use theroutine; it covers the following topics:• Running ^GLOSTAT• Overview of ^GLOSTAT Statistics• Examples of ^GLOSTAT OutputYou can also use the InterSystems System Management Portal to view the statistics reportedby ^GLOSTAT. Logon to the portal application for the system you are monitoring andnavigate to the [Home] > [System Usage] page.3.1 Running ^GLOSTATTo run the ^GLOSTAT routine you must be in the %SYS namespace. The name of the routineis case-sensitive.1. Enter the following command:Do ^GLOSTAT2. The following prompt appears:Caché Monitoring Guide 29


Gathering Global Activity Statistics Using ^GLOSTATShould detailed statistics be displayed for each block type? No =>As shown, the default is No. Press Enter to show summary block totals (Example A), ortype Yes and press Enter to display detailed statistics by block type (Example B).3. The ^GLOSTAT routine displays statistics according to your request. Each time Cachéstarts, it initializes the ^GLOSTAT statistical counters; therefore, the output of the firstrun reflects operations since Caché has started.After the first and subsequent displays of the ^GLOSTAT report, the following promptappears:Continue (c), zero statistics (z), timed stats (# sec > 0), quit?You may enter one of the following:Responseczq# (a positive integerindicating number ofseconds)ActionDisplays the report again with updated cumulative statisticssince the last initialization.Initializes the ^GLOSTAT statistical counters to zero.Quits the ^GLOSTAT routine.Initializes statistics, counts statistics for the indicated numberof seconds, and reports statistics as an average per second(Example C and Example G).4. To report statistics for a determinate interval (Example D):a. Enter z to initialize the statistics to zeroes.b. Either enter q to exit the utility, or remain at the prompt without entering anything.c. Wait for desired period of time.d. If you quit ^GLOSTAT, run it again. Otherwise, enter c to continue.e. The ^GLOSTAT report displays statistics that reflect operations since you initializedthe counters.Note:The ^GLOSTAT utility uses 32-bit counters for most statistics; extremely largetime intervals may cause some counters to overflow.30 Caché Monitoring Guide


3.2 Overview of ^GLOSTAT StatisticsEach ^GLOSTAT statistic represents the number of times a type of event has occurred sinceCaché has started, since the counters have been initialized, or per second during a definedinterval. You may run ^GLOSTAT at any time from the system manager's namespace. Inmost cases, it is significant to run the utility on an active system, not an idle one.If the Caché instance is a stand-alone configuration or an ECP data server, then the reportdisplays only the “Total” column. If it is an ECP application server (that is, it connects to aremote database) then three columns are shown: “Local,” “Remote,” and “Total”(Example E).The following table defines the ^GLOSTAT statistics.Statistics Produced by ^GLOSTATOverview of ^GLOSTAT StatisticsStatisticGlobal references (all)Global updatereferencesRoutine callsRoutine buffer loadsand savesCache EfficiencyJournal EntriesJournal Block WritesLogical BlockRequestsDefinitionLogical count of accesses to globals, including Sets, Kills,$Data, $Order, $Increment, $Query, and global references inexpressions.Logical count of global references that are Sets, Kills, or$Increments.Number of calls to a routine.Total number of routine loads and saves as a result of ZLoad,ZSave, and running routines. (In a well-tuned environment, thisnumber increases slowly, since most routine loads are satisfiedby the routine cache memory without accessing the disk. Eachroutine load or save transfers up to 32 KB of data (64 KB forUnicode).)Number of all global references divided by the number ofphysical block reads and writes. Not a percentage.Number of journal records created—one for each databasemodification (Set , Kill, etc.) or transaction event (TStart,TCommit) or other event that is saved to the journal.Number of 64-KB journal blocks written to the journal file.Number of database blocks read by the global database code.(In a well-tuned environment, many of these reads are satisfiedwithout disk access.)Caché Monitoring Guide 31


Gathering Global Activity Statistics Using ^GLOSTATStatisticPhysical Block ReadsPhysical Block WritesBlocks Queued to beWrittenDefinitionNumber of physical database blocks (2-KB or 8-KB) read fromdisk for both global and routine references.Number of physical database blocks (2-KB or 8-KB) written todisk for both global and routine references.Number of 2-KB or 8-KB database blocks that have been queuedto be written to disk.If the option is selected, ^GLOSTAT also reports the following detailed block statistics forthe “Logical Block Requests,” “Physical Block Reads,” and “Blocks Queued to be Written”statistics. A total for all block types within each category appears in the right-most column(Example F).Block types include:LabelDataDirBdataMapUpper ptrBottom ptrOtherDescriptionData blocks.Directory blocks.Big data blocks—blocks that contain big strings.Map blocks.Upper pointer blocks.Bottom pointer blocks.Other blocks that are not any of the above. (For example, incrementalbackup or storage allocation information.)3.3 Examples of ^GLOSTAT OutputThe following output samples show the various options when running the ^GLOSTATutility routine:• Example A — Initial running with no detailed statistics on a stand-alone or server configuration.• Example B — Initial running with detailed block statistics on a stand-alone or serverconfiguration.32 Caché Monitoring Guide


Examples of ^GLOSTAT Output• Example C — Subsequent running with no detailed statistics at a timed interval.• Example D — Subsequent runnings with no detailed statistics showing the use of the c,z, and q responses.• Example E — Initial running with no detailed statistics on a client configuration.• Example F — Initial running with detailed block statistics on a client configuration.• Example G — Initial and subsequent runnings with detailed block statistics at a timedinterval.3.3.1 Example AThe following is sample output of the initial running of the ^GLOSTAT routine. No detailedstatistics are chosen and the Caché instance is either a stand-alone configuration or a server:%SYS>Do ^GLOSTATShould detailed statistics be displayed for each block type? No =>nStatisticsTotal------------------------------- -------------Global references (all): 6,445,330Global update references: 1,322,207Routine calls: 896,625Routine buffer loads and saves: 9,008Logical block requests: 2,997,805Block reads: 34,674Block writes: 1,918Cache Efficiency: 176Journal Entries: 211,164Journal Block Writes: 363Continue (c), zero statistics (z), timed stats (# sec > 0), quit?3.3.2 Example BThe following is sample output of the initial running of the ^GLOSTAT routine. Detailedblock statistics have been chosen and the Caché instance is either a stand-alone configurationor a server:Caché Monitoring Guide 33


Gathering Global Activity Statistics Using ^GLOSTAT%SYS>do ^GLOSTATShould detailed statistics be displayed for each block type? No =>yStatisticsTotal------------------------------- -------------Global references (all): 6,444,341Global update references: 1,322,017Routine calls: 896,130Routine buffer loads and saves: 9,008Cache Efficiency: 176Journal Entries: 211,164Journal Block Writes: 363Logical Block Requests Data: 1,643,739Dir: 13,147 Upper ptr: 138,837BData: 164,102 Bottom ptr: 1,034,472Map: 3,043 Other: 1 2,997,341Physical Block Reads Data: 33,846Dir: 42 Upper ptr: 31BData: 452 Bottom ptr: 290Map: 12 Other: 1 34,674Physical Block Writes 1,918Blocks Queued to be Written Data: 1,952Dir: 11 Upper ptr: 0BData: 317 Bottom ptr: 65Map: 22 Other: 6 2,373Continue (c), zero statistics (z), timed stats (# sec > 0), quit?3.3.3 Example CThe following example shows ^GLOSTAT statistics per second for a 30–second timedinterval. The user has not requested detailed statistics for each block type. The Caché instanceis either a stand-alone configuration or a server:Continue (c), zero statistics (z), timed stats (# sec > 0), quit? 30Counts per Second for 30 Seconds...Statistics (per second)Total------------------------------- -------------Global references (all): 1,369.8Global update references: 299.6Routine calls: 1,057.6Routine buffer loads and saves: 12.0Logical block requests: 696.9Block reads: 0.7Block writes: 3.2Cache Efficiency: 354.3Journal Entries: 277.5Journal Block Writes: 0.5Continue (c), zero statistics (z), timed stats (# sec > 0), quit?3.3.4 Example DThe following example shows ^GLOSTAT results from running the routine multiple times.34 Caché Monitoring Guide


Examples of ^GLOSTAT OutputAfter the initial output, entering z initializes the counters and then q quits the routine. Afterwaiting the desired amount of time, the routine is run again and displays statistics reflectingoperations since z was entered.Entering z again initializes the statistics a second time, and entering c to continue immediatelydisplays the initialized statistics. Entering c once more shows the statistics once again accumulating.Caché Monitoring Guide 35


Gathering Global Activity Statistics Using ^GLOSTAT%SYS>Do ^GLOSTATShould detailed statistics be displayed for each block type? No =>StatisticsTotal------------------------------- -------------Global references (all): 85,414Global update references: 23,073Routine calls: 67,092Routine buffer loads and saves: 477Logical block requests: 64,587Block reads: 140Block writes: 49Cache Efficiency: 452Journal Entries: 25,341Journal Block Writes: 44Continue (c), zero statistics (z), timed stats (# sec > 0), quit? zContinue (c), zero statistics (z), timed stats (# sec > 0), quit? q%SYS>Do ^GLOSTATShould detailed statistics be displayed for each block type? No =>StatisticsTotal------------------------------- -------------Global references (all): 4,332Global update references: 568Routine calls: 3,487Routine buffer loads and saves: 95Logical block requests: 1,423Block reads: 0Block writes: 130Cache Efficiency: 33Journal Entries: 218Journal Block Writes: 1Continue (c), zero statistics (z), timed stats (# sec > 0), quit? zContinue (c), zero statistics (z), timed stats (# sec > 0), quit? cStatisticsTotal------------------------------- -------------Global references (all): 0Global update references: 0Routine calls: 0Routine buffer loads and saves: 0Logical block requests: 0Block reads: 0Block writes: 0Cache Efficiency:no i/oJournal Entries: 0Journal Block Writes: 0Continue (c), zero statistics (z), timed stats (# sec > 0), quit? cStatisticsTotal------------------------------- -------------Global references (all): 9,362Global update references: 1,742Routine calls: 7,382Routine buffer loads and saves: 96Logical block requests: 4,404Block reads: 4Block writes: 0Cache Efficiency: 2,341Journal Entries: 1,54836 Caché Monitoring Guide


Examples of ^GLOSTAT OutputJournal Block Writes: 3Continue (c), zero statistics (z), timed stats (# sec > 0), quit?3.3.5 Example EThe following is sample output of the initial running of the ^GLOSTAT routine. No detailedstatistics are chosen and the Caché instance is a client:%SYS>DO ^GLOSTATShould detailed statistics be displayed for each block type? No =>nStatistics Local Remote Total------------------------------- ------------- ------------- -------------Global references (all): 1,558,696 0 1,558,696Global update references: 531,676 0 531,676Routine calls: 95,987 0 95,987Routine buffer loads and saves: 747 0 747Logical block requests: 1,121,187 n/a 1,121,187Block reads: 3,155 0 3,155Block writes: 1,450 n/a 1,450Cache Efficiency: 338 no getsJournal Entries: 525,177 n/a 525,177Journal Block Writes: 12,546 n/a 12,546Continue (c), zero statistics (z), timed stats (# sec > 0), quit?3.3.6 Example FThe following is sample output of the initial running of the ^GLOSTAT routine. Detailedblock statistics are chosen and the Caché instance is a client:Caché Monitoring Guide 37


Gathering Global Activity Statistics Using ^GLOSTAT%SYS>DO ^GLOSTATShould detailed statistics be displayed for each block type? No =>yStatistics Local Remote Total------------------------------- ------------- ------------- -------------Global references (all): 1,558,611 0 1,558,611Global update references: 531,676 0 531,676Routine calls: 95,979 0 95,979Routine buffer loads and saves: 747 0 747Cache Efficiency: 338 no getsJournal Entries: 525,177 n/a 525,177Journal Block Writes: 12,546 n/a 12,546Logical Block Requests Data: 608,909Dir: 792 Upper ptr: 2,475BData: 81,750 Bottom ptr: 426,330Map: 879 Other: 1 1,121,136Physical Block Reads Data: 2,823Dir: 11 Upper ptr: 9BData: 240 Bottom ptr: 66Map: 5 Other: 1 3,155Physical Block Writes 1,450Blocks Queued to be Written Data: 959Dir: 10 Upper ptr: 0BData: 430 Bottom ptr: 38Map: 17 Other: 7 1,461Continue (c), zero statistics (z), timed stats (# sec > 0), quit?3.3.7 Example GThe following example shows ^GLOSTAT statistics per second for a 30-second timedinterval after the initial running when the user requests detailed statistics for each block type.The Caché instance is either a stand-alone configuration or a server:38 Caché Monitoring Guide


Examples of ^GLOSTAT Output%SYS>Do ^GLOSTATShould detailed statistics be displayed for each block type? No =>yStatisticsTotal------------------------------- -------------Global references (all): 50,818Global update references: 8,920Routine calls: 25,279Routine buffer loads and saves: 1,027Cache Efficiency: 100Journal Entries: 236Journal Block Writes: 4Logical Block Requests Data: 16,375Dir: 1,760 Upper ptr: 263BData: 165 Bottom ptr: 2,538Map: 83 Other: 1 21,185Physical Block Reads Data: 153Dir: 6 Upper ptr: 5BData: 165 Bottom ptr: 38Map: 4 Other: 1 372Physical Block Writes 134Blocks Queued to be Written Data: 104Dir: 6 Upper ptr: 0BData: 0 Bottom ptr: 29Map: 9 Other: 5 153Continue (c), zero statistics (z), timed stats (# sec > 0), quit? 30Counts per Second for 30 Seconds...Statistics (per second)Total------------------------------- -------------Global references (all): 4,232.7Global update references: 1,453.4Routine calls: 446.8Routine buffer loads and saves: 14.1Cache Efficiency: 507.9Journal Entries: 33.3Journal Block Writes: 0.1Logical Block Requests Data: 909.8Dir: 15.6 Upper ptr: 21.3BData: 0.2 Bottom ptr: 589.3Map: 1.3 Other: 0 1,537.5Physical Block Reads Data: 6.8Dir: 0 Upper ptr: 0BData: 0.2 Bottom ptr: 0.1Map: 0 Other: 0 7.1Physical Block Writes 1.2Blocks Queued to be Written Data: 2.6Dir: 0.0 Upper ptr: 0BData: 0 Bottom ptr: 0.2Map: 0.1 Other: 0 2.9Continue (c), zero statistics (z), timed stats (# sec > 0), quit?Caché Monitoring Guide 39


4Monitoring System PerformanceUsing ^PERFMONCaché provides a utility called ^PERFMON, which provides extrinsic functions and a rudimentarymenu to call functions used to control the system MONITOR facilityNote:The MONITOR facility is an internal system feature that was accessible throughthe ^MONITOR routine in previous versions of Caché. The ^PERFMON functionswere available in previous versions in the ^VPMON routine.The MONITOR facility provides performance data for the Caché system by collecting countsof events at the system level and also sorted by process, global, routine, and network nodes.Since there is some overhead involved in collecting this data, the collection of counters mustbe specifically enabled and data is collected for a specific number of processes, globals,routines, and network nodes. Memory is allocated at MONITOR startup to create slots forthe number of processes, routines, globals, and nodes specified. The first process to triggeran event counter allocates the first slot and continues to add to that set of counters. Once allthe available slots have been allocated to processes, any subsequent process counts are includedin an "Other" slot. The same procedure is followed for globals, routines, and nodes.Reports of the data may be viewed while collection is in progress. Once collection stops,memory is de-allocated and the counter slots are gone. So, any retention of the numbers needsto be handled by writing the reports to a file (or a global). Data is given as rates per secondby default, although there is also an option for gathering the raw totals. There are also functionswhich allow you to pause/resume the collection, and zero the counters.Caché Monitoring Guide 41


Monitoring System Performance Using ^PERFMONThe menu items available by running ^PERFMON correspond directly to functions availablein the ^PERFMON routine, and the input collected is used to directly supply the parametersof these functions.Note that similar functions which control the same MONITOR facility are available throughthe classes in the %Monitor.System package, which also allow you to save the data as a namedcollection in a persistent object format. See the package%SYSTEM.Monitor.Collection,%Monitor.System.Sample, and %SYSTEM.Monitor.Counters class documentation in the CachéClass Reference for more details.The following functions are available:Startturns on collection of the statisticsstatus =$$Start^PERFMON(process, global, routine, network)Where:• process = the number of process slots to reserve• global = the number of global slots to reserve• routine = the number of routine slots to reserve• network = the number of network node slots to reservestatus = $$Start^PERFMON(process, global, routine, network) .process = the number of process slots to reserveglobal = the number of global slots to reserveroutine = the number of routine slots to reservenetwork = the number of network node slots to reservestatus = 1 if successful or,“-1,Somebody else is using Monitor."“-2,Monitor is already running."“-3,Memory allocation failed"“-4,Couldn't enable stats collection"status = $$Stop^PERFMON() stops collectionstatus = 1 if successful or,“-1,Somebody else is using Monitor."“-2,Monitor is not running."status = $$Pause^PERFMON() momentarily pauses collection(allows a consistent state for viewing data)status = 1 if successful or,“-1,Somebody else is using Monitor."“-2,Monitor is not running."42 Caché Monitoring Guide


Running PERFMONstatus = $$Resume^PERFMON() resumes collectionstatus = 1 if successful or,“-1,Somebody else is using Monitor."“-2,Monitor is not running."status = $$Clear^PERFMON() clear all countersstatus = 1 if successful or,“-1,Somebody else is using Monitor."“-2,Monitor is not running."4.1 Running PERFMONTake the following steps to run PERFMON:1. Enter the following command:DO ^PERFMON2. The following menu appears. Enter the number of your choice. Press Enter to exit theroutine.1. Start Monitor2. Stop Monitor3. Pause Monitor4. Resume Monitor5. Clear Counters6. Report StatisticsEnter the number of your choice:The first four options start, stop, pause, and resume the monitor respectively.status = $$Start^PERFMON(process, global, routine, network)turns on collection of the statistics.process = the number of process slots to reserveglobal = the number of global slots to reserveroutine = the number of routine slots to reservenetwork = the number of network node slots to reserveCaché Monitoring Guide 43


Monitoring System Performance Using ^PERFMONStatus code1-1-2-3-4DescriptionSuccessfulSomebody else is using MonitorMonitor is already runningMemory allocation failedCould not enable statistics collection44 Caché Monitoring Guide


Running PERFMON4.1.1 PERFMON ReportCaché Monitoring Guide 45


Monitoring System Performance Using ^PERFMON;;The menu items available from ^PERFMON correspond directly to callable;;extrinsic functions in ^PERFMON, and the input collected is used to;;directly supply the parameters of these functions. Each function returns;;a success/failure status, where success is "1" and failure is indicated by;;a string with a negative number followed by a comma and a brief message.;;*PAGE*;;The following functions are available:;;;;$$Start^PERFMON(process, global, routine, network) turns on collection;; of the statistics.;;;; - process = the number of process slots to reserve;; - global = the number of global slots to reserve;; - routine = the number of routine slots to reserve;; - network = the number of network node slots to reserve;;;;$$Stop^PERFMON() stops collection;;;;$$Pause^PERFMON() pauses collection;; (allows a consistent state for viewing data);;;;$$Resume^PERFMON() resumes collection (after viewing data);;;;$$Clear^PERFMON() clear all counters;;*PAGE*.;;;;$$Report^PERFMON(report, sort, format, output, [list], [data]);; gathers and outputs a report of the counters;;;; - report = type of report to output.;; - "G" = Global Activity;; - "R" = Routine Activity;; - "N" = Network Activity;; - "C" = Custom report (user selects columns);;;; - sort = sort order;; - "P" = organize report by Process;; - "R" = organize report by Routine;; - "G" = organize report by Global;; - "I" = organize report by Incoming Node;; - "O" = organize report by Outgoing Node;;;; - format = output format;; - "P" = printable/viewable report;; - "D" = comma delimited dataformat = output format“P” = printable/viewable report (no pagination)“D” = comma delimited data (can be read into spreadsheet)“A” = return data in local array;;;; - output = file name, or 0 for output to screen;; - list = comma separated list of numbers to specify cols in Custom report;;*END*data = 1 for standard rates/second, or 2 for raw totalsstatus = 1 if successful or,“-1, Monitor is not running"“-2, Missing input parameter."“-3, Invalid report category."“-4, Invalid report organization."“-5, Invalid report format.”“-6, Invalid list for Custom report.”“-7, Invalid data format (1=rates or 2=raw only).”46 Caché Monitoring Guide


The following table lists the metrics available for a custom report (type “C” ). The global (“G” ), routine ( “R” ), and network ( “N” ) activity reports each display a predefined subsetof this list. The list parameter in the Report function must be a comma-separated list of thenumbers indicated in the left column of this table.Fields for Custom ReportRunning PERFMONNumber12345678910111213141516171819202122Column TitleGloRefGloSetGloKillTotBlkRdDirBlkRdUpntBlkRdBpntBlkRdDataBlkRdRouBlkRdMapBlkRdOthBlkRdDirBlkWtUpntBlkWtBpntBlkWtDataBlkWtRouBlkWtMapBlkWtOthBlkWtDirBlkBufUpntBlkBufBpntBlkBufDataBlkBufDescriptionGlobal referencesGlobal setsGlobal killsTotal physical block reads (sum of next seven counters)Directory block readsUpper pointer block readsBottom pointer block readsData block readsRoutine block readsMap block readsOther block readsDirectory block writesUpper pointer block writesbottom pointer block writedata block writesroutine block writesmap block writesother block writesdirectory block requests satisfied from a globalupper pointer block requests satisfied from a globalbufferbottom pointer block requests satisfied from a globalbufferdata block requests satisfied from a global bufferCaché Monitoring Guide 47


Monitoring System Performance Using ^PERFMONNumber232425262728293031323334353637383940414243444546474849Column TitleRouBlkBufMapBlkBufOthBlkBufJrnEntryBlkAllocNetGloRefNetGloSetNetGloKillNetReqSentNCacheHitNCacheMissNetLockRtnLineRtnLoadRtnFetchLockComLockSuccLockFailTermReadTermWriteTermChRdTermChWrtSeqReadSeqWrtIJCMsgRdIJCMsgWtIJCNetMsgDescriptionroutine block requests satisfied from a global buffermap block requests satisfied from a global bufferother block requests satisfied from a global bufferjournal entriesblocks allocatednetwork global refsnetwork setsnetwork killsnetwork requests sentnetwork cache hitsnetwork cache missesnetwork locksM commandsroutine loadsroutine fetcheslock commandssuccessful lock commandsfailed lock commandsterminal readsterminal writesterminal read charsterminal write charssequential readssequential writeslocal IJC messages readlocal IJC messages writtennetwork IJC messages written48 Caché Monitoring Guide


Running PERFMONNumber5051Column TitleRetransmitBuffSentDescriptionnetwork retransmitsnetwork buffers sentCaché Monitoring Guide 49


5Examining Routine PerformanceUsing ^%SYS.MONLBLThe routine ^%SYS.MONLBL provides a user interface to the Caché MONITOR facility.This utility provides a way to diagnose where time is spent executing selected code in CachéObjectScript and Caché Basic routines, helping to identify lines of code that are particularlyresource intensive. It is an extension of the existing MONITOR utility accessed through^PERFMON and the %Monitor.System package classes. Because these utilities share thesame memory allocation, you can only run one of them at a time on a Caché instance.This document contains the following sections:• Start MonitoringMonitoring Options• Sample Output• Programming Interface5.1 Start MonitoringIf the monitor is not running when you call ^%SYS.MONLBL, the routine displays awarning message and gives you the one option to start the monitor. For example:Caché Monitoring Guide 51


Examining Routine Performance Using ^%SYS.MONLBL%SYS>Do ^%SYS.MONLBL1.) Start MonitorWARNING ! Starting the line-by-line monitor will enable thecollection of statistics for *every* line of code executedby the selected routines and processes. This can have a majorimpact on the performance of a system, and it is recommendedthat you do this on a 'test' system.The line-by-line monitor also allocates shared memory to trackthese statistics for each line of each routine selected. This isNOT taken from the memory already allocated by Cache, but a new,separate section of shared memory is created. This should be consideredif you are using '*' wildcards and trying to analyze a large numberof routines.Enter '1' to select routines and enable the monitor:Enter 1 to begin the dialog to provide the appropriate information to start monitoring.You can select which routines and processes to monitor and which metrics to collect. Youprovide monitoring information to the routine in the following order:1. Routine Names – Enter a list of routine names to monitor. You can only select routinesaccessible from your current namespace. You may use asterisk (*) wild cards to selectmultiple routines. Press Enter twice after entering the last routine name to end the list.Enter routine names to monitor on a line by line basis.Patterns using '*' are allowed.Enter '?L' to see a list of routines already selected.Press 'Enter' to terminate input.Routine Name:2. Select Metrics to monitor – Enter the number of your choice: 1 (the default) fordefault metrics, 2 for all available metrics, and 3 to choose a custom list of metrics.Select Metrics to monitor1.) Monitor Default Metrics2.) Monitor All Metrics3.) Customize Monitor MetricsEnter the number of your choice: The default metrics collected are:• RtnLine — the number of times a line is executed• Time — the clock time spent in executing that line52 Caché Monitoring Guide


Start Monitoring• TotalTime — the total clock time for that line including time spent in subroutinescalled by that lineIf you choose to customize the list, you can select any of the standard performance metricssupported by the %Monitor.System class. Enter a question mark (?) when prompted forthe metric item number to see a list of available metrics. For example:Enter the number of your choice: 3Enter metrics item number (blank to terminate, ? for list)Metric#: ?1.) GloRef: global refs2.) GloSet: global sets...34.) RtnLine: lines of Cache Object Script...51.) Time: elapsed time on wall clock52.) TotalTime: total time used (including sub-routines)Metric#:This example does not show the full list; it is best for you to retrieve the most up to datelist when you run the routine. See Programming Interface for a method for retrieving thelist.3. Select Processes to monitor – Enter the number of your choice: 1 (the default) forall processes, 2 for the current process, and 3 to choose a list of processes by process IDnumber (PID).Select Processes to monitor1.) Monitor All Processes2.) Monitor Current Process Only3.) Enter list of PIDsEnter the number of your choice: ^%SYS.MONLBL does not currently provide a list or a way to select PIDs; however,you can use the ^%SS utility or the [Home] > [Processes] page of the System ManagementPortal to find specific process ID numbers.Enter the number of your choice: 3Enter PID (blank to terminate)PID: 1640PID: 2452PID:Caché Monitoring Guide 53


Examining Routine Performance Using ^%SYS.MONLBLPress Enter twice after entering the last routine name to end the list.Once you provide the necessary information, ^%SYS.MONLBL allocates a special sectionof shared memory for counters for each line per routine, and notifies the selected processesthat monitoring is activated.Monitor started.Press RETURN to continue ...5.2 Monitoring OptionsYou now have additional menu options:• 2.) Stop Monitor — Stops all ^%SYS.MONLBL monitoring. Deallocates the countermemory and deletes collected data.• 3.) Pause Monitor — Pauses the collection and maintains any collected data. Thismay be useful when viewing collected data to ensure that counts are not changing as thereport is displayed. This option only appears if ^%SYS.MONLBL is running.• 4.) Resume Monitor — Resumes collection after a pause. This option only appears if^%SYS.MONLBL has been paused.• 5.) Clear Counters — Clears any collected data, but continues monitoring and collectingnew data.• 6.) Report Statistics — Displays a report of collected data. Presents a list of allmonitored routines, and asks you to select one or more routines. Press Enter to displayall monitored routines. After entering the last routine, press Enter again to end the list.You can enter a file name for the output, or enter nothing and press Enter to display thereport on your terminal. For each line of the selected routine(s), the report displays a linenumber, the counts for each metric, and the text of that line of code (if source code isavailable).54 Caché Monitoring Guide


Sample Output5.3 Sample OutputThe following is a partial sample of output from monitoring the ^%SYS.MONLBL routineitself:Routine ^%SYS.MONLBL ...Line RtnLine Time TotalTime..79 0 0 0 read !,"FileName: ", filename s:...80 0 0 0 open filename:"NW":1 if '$t set ...81 0 0 0 use filename82 1 0.000010 0.000010 ; Write Rtn Data83 1 0.000005 0.000005 if (rtnnum > 0) {84 0 0 0 do writertndata(rtnnum)85 0 0 0 write !!86 0 0 0 }87 0 0 0 else {88 1 0.000011 0.000011 for rtnnum=1:1:($zu(84,16)) {89 163 0.001803 0.852597 do writertndata(rtnnum)90 162 0.004931 0.004931 write !!91 162 0.000428 0.000428 }92 0 0 0 }93 0 0 0 goto writertndataend94 163 0.000337 0.000337 writertndata(rtnnum)95 163 0.000333 0.000333 ; Write Data For a Single Rtn96 163 0.000512 0.000512 set rtnname = $zu(84,16,2,rtnnum)97 163 0.000334 0.000334 s sp=1198 163 0.030024 0.030024 w !!,"Routine ^",rtnname," ..."99 163 0.000601 0.000601 set lns = $zu(84, 16, 1, rtnnum)100 163 0.003862 0.003862 if lns=0 w " no data yet." quit101 7 0.000015 0.000015 ; Write Column Headers102 7 0.000309 0.000309 write !!,"Line " s col=5103 7 0.000039 0.000039 for metric=0:1:($zu(84,13)-1) {104 21 0.000059 0.000059 s column = $zu(84,13,11,metric) + 1105 21 0.000146 0.000146 s out=$piece($text(@("Flist+"_column)...106 21 0.000571 0.000571 w ?col,$j(out,sp-1)107 21 0.000060 0.000060 s col=col+sp108 21 0.000049 0.000049 }109 7 0.000019 0.000019 for line=0:1:(lns-1) {110 2958 0.103596 0.103596 write !,(line+1)111 2959 0.007729 0.007729 s col=5112 2960 0.009972 0.009972 for metric=0:1:($zu(84,13)-1) {113 8880 0.029456 0.029460 s out=$zu(84,16,3,line,metric)114 8883 0.019621 0.019623 ; Convert clock/CPU time to seconds115 8886 0.021176 0.021178 s n=$zu(84,13,11,metric)116 8889 0.027933 0.027938 i (n=50)!(n=51) s out=$select(out=0...117 8892 0.306696 0.306726 write ?col,$j(out,sp-1)118 8895 0.024750 0.024753 s col=col+sp119 8898 0.020944 0.020947 }120 2967 0.169251 0.169251 write ?col,$TEXT(@("+" _ (line+1) _ ...121 2968 0.015208 0.015208 }122 6 0.000034 0.000034 quitTotal Time for Recursive CodeWhen a routine contains recursive code, the TotalTime counter for the line which calls backinto the same subroutine may seem too large in relation to the total time of the entire routine.Caché Monitoring Guide 55


Examining Routine Performance Using ^%SYS.MONLBLThis effect is caused by the accumulation on one line, of TotalTime for multiple iterationsof the same code. That is, the TotalTime includes the total amount of time spent on all niterations of the subroutine, plus the total time spent on n-1 iterations, plus the total time forn-2, etc. While this is not technically wrong, it may be confusing. Future enhancements tothe routine may display a more useful number, the TotalTime for the outer most loop of therecursion.5.4 Programming InterfaceProgrammers can also interface with the Caché MONITOR facility through the%Monitor.System.LineByLine class. Methods are provided for each menu option in^%SYS.MONLBL. For example, start monitoring by calling:Set status=##class(%Monitor.System.LineByLine).Start(Routine,Metric,Process)You can select which routines and processes to monitor. You may also select any of the otherstandard performance metrics supported by the %Monitor.System classes. Use theMonitor.System.LineByLine.GetMetrics() method to retrieve a list of metric names:Set metrics=##class(%Monitor.System.LineByLine).GetMetrics(3)Selecting 3 as the parameter prints a list of all available metrics with a short description foreach to the current device.Stop monitoring by calling:Do ##class(%Monitor.System.LineByLine).Stop()You can retrieve the collected counts using the %Monitor.System.LineByLine:Resultquery, where the counters for each line are returned in $LIST format.See the Caché online Class Reference for %Monitor.System.LineByLine for more details.56 Caché Monitoring Guide


AMonitoring Caché Using BMCPATROLThis appendix describes the interface between Caché and BMC PATROL.BMC PATROL is a tool for monitoring and managing various software systems. Cachésupplies PATROL extensions so that you can monitor and collect information about Caché.This interface allows users to monitor metrics of one or multiple Caché systems from thePATROL Console. The interface requires that the PATROL daemon is running on the Cachésystem to collect and output the metric values and that the Caché knowledge module files(*.km) are loaded into the PATROL Console to read and display these values.This appendix provides information on the following topics:• Running PATROL with CachéCaché PATROL Knowledge Modules• Caché Metrics Used with BMC PATROLA.1 Running PATROL with CachéRun the ^PATROL Caché ObjectScript routine on each Caché installation that you want tomonitor using the start and stop entry points, or by setting it to automatically run at systemstartup.Caché Monitoring Guide 57


Monitoring Caché Using BMC PATROLThe routine starts a background process that outputs metrics to a file, patrol.dat, located inthe Caché manager’s directory, c:\CacheSys\Mgr by default. The file is rewritten for eachcollection period, so the file size is static. The file also includes an identifying header and atime stamp so that the PATROL Console can determine that it is active and up-to-date.There are two ways to run and configure PATROL in Caché:• Configure PATROL Settings• Caché PATROL RoutineA.1.1 Configure PATROL SettingsAutomatic PATROL StartupYou can set an option to automatically start the PATROL daemon at Caché startup using theSystem Management Portal:1. Navigate to the [Home] > [Configuration] > [Advanced Settings] page and click Monitoringin the Category box to shorten the list.2. In the PatrolStart row, click Edit to display the Configuration Setting page.3. In the Value list box, click true to start Patrol at Caché startup.PATROL ArgumentsFrom the same [Home] > [Configuration] > [Advanced Settings] page of the System ManagementPortal, you can also set the following PATROL settings:• PatrolTopProcesses —Number of processes displayed in the Process Status window onthe PATROL console. This window shows the top processes as sorted by global or routineactivity. The default number of processes is 20. A value of 0 tells the PATROL utility tostop calculating the top processes, potentially saving significant work on systems withmany processes. The valid range is 1–10000 processes.• PatrolDisplayMode — Controls how the monitoring data is displayed in the PATROLconsole. The default option is Total. Options are as follows:- Total — Displays the total counts since the collection was started.- Delta — Displays the count for the last collection period.- Rate — Displays a calculated count per second.58 Caché Monitoring Guide


Caché PATROL Knowledge Modules• PatrolCollectionInterval — Number of seconds between each time Caché collects dataand makes it available to PATROL. The default is 30 seconds; the valid range is 1–900seconds.A.1.2 Caché PATROL RoutineCaché provides entry points to the ^PATROL routine to start and stop PATROL.To start PATROL:Do start^PATROL(display,process,timer)The arguments are described in the following table:ArgumentdisplayprocesstimerDescriptionDisplay mode. The literals total, delta, or rate toindicate the type of numbers to output.How many processes for which to pass %SS statistics.Collection period in seconds.Defaulttotal2030For example:Do start^PATROL("total",20,30)Sets the PATROL console to display the total statistic counts, since the collection started, forthe top 20 processes; Caché sends the information every 30 seconds. The collection periodargument is passed to the PATROL console so that the collection and display update aresynchronized.To stop PATROL:Do stop^PATROLA.2 Caché PATROL Knowledge ModulesThe architecture of PATROL is based on the concept of knowledge modules. A knowledgemodule contains a set of commands, parameters to monitor, and actions used by PATROL.The Caché plug-in for PATROL consists of several knowledge modules, that you load intothe PATROL Console.Caché Monitoring Guide 59


Monitoring Caché Using BMC PATROL• ISC_CACHE.km• ISC_CACHE_CONFIG.km• ISC_CACHE_DISK.km• ISC_CACHE_GLOBAL.km• ISC_CACHE_NETWORK.km• ISC_CACHE_OTHER.km• ISC_CACHE_OVERVIEW.km• ISC_CACHE_ROUTINE.kmOnce these KMs are loaded, the Console automatically attempts to discover Caché installationson all connected systems. The discovery process either searches the Registry on NT or parsesthe output from the ccontrol list command on UNIX and OpenVMS. For each Caché installationit finds it checks to see if the patrol.dat file exists in the Caché manager's directory andif the time stamp within that file is current. Caché installations which are currently reportingCaché metrics for PATROL appear in the PATROL Console.A.2.1 Adding Caché Modules to PATROLTo add a Caché knowledge module into the Console and activate it:1. From the PATROL Console File menu, click Load KM.2. Select all the *.km files, located in the Caché Patrol directory, c:\CacheSys\Patrol, bydefault.3. The ISC_CACHE module should appear in the Desktop tab of the Console in a few seconds.4. Right-click ISC_CACHE and choose Add Configuration from the KM Commands menu.5. In the Add Configuration dialog box, enter a configuration name and the Caché directory,C:/CacheSys, as the install directory.6. You may need to wait for, at most, 30 seconds (PATROL default sync period), beforePATROL recognizes Caché statistics.For more information you can consult the BMC Web site.If any Caché installations are discovered on a system, then the main entry for Caché (theCaché class) appears under that system entry. Each Caché instance (each Caché configuration60 Caché Monitoring Guide


installed on that system) appears under the Caché class. Under each Caché instance are thegeneral metric categories of Overview, Global, Routines, Disk Activity, Network, and Other.For example:- PATROLMainMap- TEST1- ISC_CACHE- ISC_Config_CACHE+ ISC_DiskActivity+ ISC_Global+ ISC_Network+ ISC_Other+ ISC_Overview+ ISC_RoutineCaché Metrics Used with BMC PATROLThe categories expand to show all the individual metrics. The metrics under Overview aregauges showing current levels. The others are graphs showing values over time.Right clicking the Caché configuration allows the user to select Caché-specific commands,to either Remove Configuration or show a Process Status window.Manually adding a configuration should not normally be necessary, since all Caché installationsshould be automatically discovered, but might be useful if there is a question or a problemwith a specific installation.Error messages from the Caché KMs may be output to the System Output window. Checkthese messages if you have questions about Caché installations that are not automaticallydiscovered.A.3 Caché Metrics Used with BMC PATROLThe list of metrics for Caché:Caché Monitoring Guide 61


Monitoring Caché Using BMC PATROLCaché PATROL MetricsCategoryOverviewMetricGlobal Refs (gauge)Global Sets, Reads, Kills (graph)Net Global Refs (gauge)Net Global Sets, Reads, Kills (graph)Routine Lines (gauge)Routine Loads (gauge)Locks (gauge)Process Count (graph)Cache Efficiency (graph) (= 100*(LogicalReads/(LogicalReads+ Physical Reads)) )Licenses Used (gauge)62 Caché Monitoring Guide


Caché Metrics Used with BMC PATROLCategoryGlobalMetricGlobal RefsGlobal SetsGlobal KillsGlobal ReadsBlocks AllocatedLocksSuccessful LocksFailed LocksJob InGlobalWD QueSizeGlobal AvailBufsQue GaccessQue GaccUpdQue GBFAnyQue GBFSpecJournal EntriesJrn FileSizeJrn EndOffsetTot Global BufsGThrottle CurGThrottle MaxGThrottle CntRoutineRoutine LinesRoutine LoadsRoutine FetchesCaché Monitoring Guide 63


Monitoring Caché Using BMC PATROLCategoryDisk ActivityMetricPhysical Directory ReadsPhysical U-Ptr ReadsPhysical B-Ptr ReadsPhysical Data ReadsPhysical Routine ReadsPhysical Map ReadsPhysical Other ReadsPhysical Directory WritesPhysical U-Ptr WritesPhysical B-Ptr WritesPhysical Data WritesPhysical Routine WritesPhysical Map WritesPhysical Other WritesLogical Directory ReadsLogical U-Ptr ReadsLogical B-Ptr ReadsLogical Data ReadsLogical Routine ReadsLogical Map ReadsLogical Other Reads64 Caché Monitoring Guide


Caché Metrics Used with BMC PATROLCategoryNetworkMetricNet Global RefsNet Global SetsNet Global KillsNet Global ReadsNet Requests SentNet Cache HitsNet Cache MissesNet LocksNet RetransmitsNet BufferNet GblJobsOtherTerminal ReadsTerminal WritesTerminal Read CharTerminal Write CharSequential ReadSequential WriteCaché Monitoring Guide 65


BMonitoring Caché Using SNMPThis document describes the interface between Caché and SNMP (Simple Network ManagementProtocol). SNMP is a communication protocol that originated and has gained widespreadacceptance as a method of managing TCP/IP networks, including individual network devices,and computer devices in general. Its popularity has expanded its use as the underlying structureand protocol for many enterprise management tools. This is its main importance to Caché: astandard way to provide management and monitoring information to a wide variety of managementtools.SNMP is both a standard message format and a standard set of definitions for managed objects.It also provides a standard structure for adding custom-managed objects, a feature that Cachéuses to define its management information for use by other applications.The interface description includes the following topics:• Running SNMP with Caché• Managing SNMP in CachéCaché as a Subagent• Caché MIB StructureB.1 Running SNMP with CachéSNMP defines a client/server relationship where the client (a network management application)connects to a server program (called the SNMP agent) which executes on a remote networkCaché Monitoring Guide 67


Monitoring Caché Using SNMPdevice or a computer system. The client requests and receives information from that agent.There are four basic types of SNMP messages:• GET – fetch the data for a specific managed object• GETNEXT – get data for the next managed object in a hierarchical tree, allowing systemmanagers to walk through all the data for a device• SET – set the value for a specific managed object• TRAP – an asynchronous alert sent by the managed device or systemThe SNMP MIB (Management Information Base) contains definitions of the managed objects.Each device publishes a file, also referred to as its MIB, which defines which portion of thestandard MIB it supports, along with any custom definitions of managed objects. For Caché,this is the ISC-CACHE.mib file, located in the SNMP subdirectory of the Caché install directory.B.2 Managing SNMP in CachéSince SNMP is a standard protocol, the management of the Caché subagent is minimal. Themost important task is to verify that the SNMP master agent on the system is compatible withthe AgentX protocol, and is active and listening for connections on the standard AgentX TCPport 705. On Windows systems, Caché automatically installs a DLL to connect with thestandard Windows SNMP service. Verify that the Windows SNMP service is installed andstarted either automatically or manually.Next, enable the Caché monitoring service using the following steps:1. Navigate to the [Home] > [Security Management] > [Services] page of the System ManagementPortal.2. Click the %System_Monitor service.3. Select the Service enabled check box and click Save.4. You return to the list of services page and see that the %System_Monitor service isenabled.Finally, configure the Caché SNMP subagent to start automatically at Caché startup usingthe following steps:1. Navigate to the [Home] > [Configuration] > [Advanced Settings] page of the SystemManagement Portal.68 Caché Monitoring Guide


Caché as a Subagent2. Click Monitoring in the Category box to shorten the list.3. In the SNMPStart setting row click Edit to display the Configuration Setting page.4. In the Value list box, click true to start the SNMP subagent at Caché startup.5. Click OK and Apply Changes. When you edit this setting, the Caché part of the SNMPinterface immediately stops and starts.You can also start and stop the Caché SNMP subagent manually or programmatically usingthe ^SNMP routine:Do start^SNMP(,)Do stop^SNMPw $$start^SNMP(port,timeout)"w !," port = TCP port for connection (default is 705)"w !," timeout = TCP port read timeout (default is "_20_" seconds)"w !!," w $$stop^SNMP()"Caché logs any problems in establishing a connection or answering requests in the SNMP.LOGfile in the Caché Manager’s directory.B.3 Caché as a SubagentThe SNMP client connects to the SNMP agent which is listening on a well-known address,UDP port 161. Since the client expects to connect on this particular port, there can only beone SNMP agent on a computer system. To allow access to multiple applications on thesystem, developers can implement master agents, which may be extended or connected tomultiple subagents. InterSystems has implemented the Caché SNMP interface as a subagent,designed to communicate through an SNMP master agent.Most operating systems that Caché supports provide an SNMP master agent which is extensiblein some way to support multiple subagents. Many of these agents, however, implementtheir extensibility in a proprietary and incompatible manner. Caché implements its subagentusing the Agent Extensibility (AgentX) protocol, an IETF-proposed standard as described inRFC 2741.Some of the standard SNMP master agents support AgentX (most notably OpenVMS andTru64 UNIX). If the SNMP master agent supplied by an operating system is not AgentXcompatible,it can be replaced by the public domain NET-SNMP agent. The exception is theWindows standard agent which does not support AgentX and for which the NET-SNMPversion may not be adequate. For this we supply a Windows extension agent DLL, iscsnmp.dll,Caché Monitoring Guide 69


Monitoring Caché Using SNMPwhich handles the connection between the standard Windows SNMP service extension APIand the Caché AgentX server.B.4 Caché MIB StructureAll of the managed object data available through the Caché SNMP interface is defined in theCaché MIB file, ISC-CACHE.mib which can be found in the SNMP subdirectory under theCaché manager’s directory. Typically an SNMP management application must load the MIBfile for the managed application to understand and appropriately display the information.This procedure varies between applications; consult the documentation for your managementapplication for the appropriate way to load the Caché MIB.The specific data defined in the Caché MIB is documented in the file itself, and is not repeatedhere. But it may be valuable to understand the overall structure of the Caché MIB tree, especiallyas it relates to multiple instances on the same system.SNMP defines a specific, hierarchical, tree structure for all managed objects called theStructure of Management Information (SMI) which is detailed in RFC 1155. Each managedobject is named by a unique object identifier (OID), which is written as a sequence of integersseparated by periods, for example: 1.3.6.1.2.1.1.1. The MIB translates this dotted integeridentifier into a text name.The standard SNMP MIB defines many standard managed objects. To define applicationspecificextensions to the standard MIB, as Caché does, an application uses the “enterprise”branch which is defined as:iso.org.dod.internet.private.enterprises (1.3.6.1.4.1)The IANA (Internet Assigned Numbers Authority) assigns each organization a privateenterprise number as the next level in the hierarchy. For Caché this is 16563 which representsintersystems.Below this, Caché implements its enterprise private sub-tree as follows:• The level below intersystems is the product ID. For Caché this is .1 (iscCache). Thisserves as the MIB module identity and is also known as the application ID.• The next level separates data objects from notifications. These are: .1 (cacheObjects)and .2 (cacheTraps). By convention, the intersystems tree uses a brief lowercase prefixadded to all data objects and notification names. For Caché this is cache.70 Caché Monitoring Guide


Caché MIB Structure• The next level is the “table” or group level. All data objects are organized into tables,even if there is only one instance or “row” to the table. This serves to organize themanagement data objects into groups. This is also necessary to support multiple Cachéinstances on one machine. All tables use the Caché instance name as the first index ofthe table. The tables may also have one or more additional indexes.• The next level is the conceptual row for the table (as required by the SNMP SMI). Thisis always .1.• Finally, the individual data objects contained in that table, including any that are designatedas indexes.• The notifications (traps) are defined as individual entries at the same hierarchical levelas the table above.For example, the size of a database would be encoded as1.3.6.1.4.1.16563.1.1.3.1.6.4.84.69.83.84.1; this translates to:iso.org.dod.internet.private.enterprises.intersystems.iscCache.cacheObjects.cacheDBTab.cacheDBRow.cacheDBSize.TEST(instname).1(DBindex)B.4.1 Extending the Caché MIBApplication programmers can add managed object definitions and extend the MIB for whichthe Caché subagent provides data:1. Create Caché object definitions in classes that inherit from the %Monitor.Adaptor class.See the %Monitor documentation in the Caché Class Reference for details about addingmanaged objects to the %Monitor package.2. Execute an SNMP class method to enable these managed objects in SNMP and createan MIB definition file for management applications to use. The method to accomplishthis is:$SYSTEM.MonitorTools.SNMP.CreateMIB()See the MonitorTools.SNMP class documentation in the Caché Class Reference for details ofthe CreateMIB() method parameters.The method creates a branch of the private enterprise MIB tree for a specific applicationdefined in the %Monitor database. In addition to creating the actual MIB file for the application,the method also creates an internal outline of the MIB tree. The Caché subagent usesthis to register the MIB sub-tree, walk the tree for GETNEXT requests, and reference specificobjects methods for gathering the instance data in GET requests.Caché Monitoring Guide 71


Monitoring Caché Using SNMPAll the managed object definitions use the same general organization as the Caché enterpriseMIB tree, that is: application.objects.table.row.item.indexes. The first index forall tables is the Caché application ID. All applications must register with the IANA to obtaintheir own private enterprise number, which is one of the parameters in the CreateMIB()method.To disable the application in SNMP, use the MonitorTools.SNMP.DeleteMIB() method.This deletes the internal outline of the application MIB, so the Caché subagent no longerregisters or answers requests for that private enterprise MIB sub-tree.72 Caché Monitoring Guide

More magazines by this user
Similar magazines