Skip to content
CA Application Performance Management - 10.7
Documentation powered by DocOps

Cluster Supportability Metrics

Last update January 31, 2019

A cluster integrates the resources of two or more computing devices (that could otherwise function separately) together for a common purpose. Clustering allows an Enterprise Manager to manage other Enterprise Managers, or serve as a Manager of Managers (MOM). APM measures application supportability by measuring the performance of the various application components and provides the following metrics:

The following data is reported for the computer hosting the Enterprise Manager and for each computer with instrumented methods.

  • EM Host--Enterprise Manager server host name
  • EM Port--Enterprise Manager port to which the agent is connected
  • Java Version
  • Virtual machine
  • Launch time
  • Process ID
  • Host IP address
  • Host operating system
  • Host wall clock time

Supportability metrics display information about the Enterprise Manager rather than the application it is monitoring. These metrics appear in the Investigator tree, under:

Custom Metric Host (Virtual)

Custom Metric Process (Virtual)

Custom Metric Agent (Virtual)(SuperDomain)

Beneath this level, supportability metrics are arranged in the following hierarchy. Definitions for some of these metrics follow the list.

See also: Memory-related metrics for GC Heap and GC Monitor.

Agent Node

Agents have the following hierarchy:

<Host_Name>
   <Process_Name>
    <Agent_Name>

<Process_Name> and <Agent_Name> are configurable in IntroscopeAgent.profile.

For each <Agent_Name>, the following metrics are available:

  • ConnectionStatus, one of:
    • 3 = disconnected
    • 2 = connected, slowly or no data
    • 1 = connected
    • 0 = unmounted
  • Historical Metric Count
  • Metric Count
  • Raw Metric Count

Agent Metrics

<Agent_Name> | Transaction Tracing Events Per Interval

The total number of Transaction Trace events per agent per interval.

<Agent_Name> | Transaction Tracing Events Limit Exceeded

The number of times the clamp limit defined in the introscope.enterprisemanager.agent.trace.limit property was exceeded for a particular interval. This metric is displayed for each agent.

Note: You can define the clamp limit for introscope.enterprisemanager.agent.trace.limit property in the apm-events-thresholds-config.xml file. The apm-events-thresholds-config.xml file is located in the < EM_Home>\config directory.

<Agent_Name> | Error Snapshot Events Per Interval

The total number of error snapshot events per agent per interval.

<Agent_Name> | Error Snapshot Events Limit Exceeded

The number of times the clamp limit defined in the introscope.enterprisemanager.agent.error.limit property was exceeded for a particular interval. This metric is displayed for each agent.

Note: You can define the clamp limit for introscope.enterprisemanager.agent.error.limit property in the apm-events-thresholds-config.xml file. The apm-events-thresholds-config.xml file is located in the < EM_Home>\config directory.

Enterprise Manager Node

Under the Enterprise Manager node, the following supportability metrics are available:

  • Host
  • Name
  • Overall Capacity (%)
  • Port
  • CPU
    • EM CPU Used (%)
  • Configuration
    • Agent Clusters Metric Load
    • Number of Agent Clusters
    • Number of Metric Groupings
  • Connections
    • Cross-Cluster Data Viewer Clamped metric -- Indicates whether the maximum number of CDVs connected to the Collector or Standalone Enterprise Manager has been exceeded. If the value is 0, the clamp is not in effect. If the value is 1, then the clamp is in effect.
    • Disallowed Agents Clamped metric -- Indicates whether the maximum number of disallowed agents; connected to the given MOM, Collector, or Standalone Enterprise Manager has been exceeded. If the value is 0, the clamp is not in effect. If the value is 1, then the clamp is in effect.
    • EM Historical Metric Clamped
    • EM Live Metric Clamped
    • Max Number of Agent Connection Limit Exceeded Per Interval metric -- Indicates whether the maximum number of agents; connected to the MOM, Collector, or Standalone Enterprise Manager has been exceeded. If the value is 0, the clamp is not in effect. If the value is 1, then the clamp is in effect.
    • Metrics From External Agents -- Shows the EM metric count that EM virtual agents generate.
    • Metrics Queued (%)
    • Number of Agents
    • Number of Applications -- The number of agent applications currently reporting data.
    • Number of Cross-Cluster Data Viewers
    • Number of Disallowed Agents
      • Displays the number of disallowed agents that are connected to the given MOM, Collector, or Standalone Enterprise Manager. Passively connected agents do not send metric data.
    • Number of Events Processed
    • Number of Events Processed Limit Exceeded
    • Number of Historical Metrics
    • Number of Metrics Handled -- Number of incoming Metrics processed per 15-second interval. This number varies, but could be around the same value as Number of Metrics. A value less than Number of Metrics indicates the Enterprise Manager might be overloaded and is not processing incoming data fast enough.
    • Number of Metrics -- Displays the total metric load on the Enterprise Manager.
    • Number of Unique Applications
    • Number of Workstations

Data Store Node

Under the Data Store node, the following metrics are available:

  • SmartStor
    • Metrics Appended To Query Per Interval
    • Metrics Converted From Spool to Query Per Interval
    • SmartStor Disk Usage (mb)
    • MetaData
      • Agent Metric Cache Size
        The number of agent metrics in cache.
      • Cached Metric Count
        The number of metrics in cache.

      • Items Iterated Per Interval
        The number of items iterated from DB per interval.

      • Items Matched Per Interval
        The number of items received from queries per interval. 

      • Metrics Created Per Interval
        The number of created metrics in DB per interval.

      • Metrics Received from DB Per Interval
        The number of received metrics from DB per interval.
      • Sort Duration (ms)
        The time the Enterprise Manager took to sort metric data.
    • Tasks
      • Converting Spool To Data
      • Data Append
      • Reperiodizing
  • Transactions
    • Number of Dropped Per Interval
    • Number of Inserts Per Interval
    • Number of Queries Per Interval
    • Number of Traces in Database
    • Number of Traces in Insert Queue
    • TT Database Disk usage (mb)
    • Total Data Insertion Duration Per Interval (ms)
    • Total Index Insertion Duration Per Interval (ms)
    • Total Query Duration Per Interval (ms)
  • Volume Space Free
    • Baseline Volume Free (mb)
    • Log Volume Free (mb)
    • SmartStor Archive Volume Free (mb)
    • Traces Volume Free (mb)

Database Sub-node

Metric Data Points Sent per Interval

Queued Metric Data Points

Health Sub-node

CPU Capacity (%)

GC Capacity (%)

Harvest Capacity (%)

Heap Capacity (%)

Incoming Data Capacity (%)

SmartStor Capacity (%)

Internal Sub-node

The following metrics appear under the Internal sub-node:

  • Number of Connection Tickets
  • Number of Dependent Calculator Input Metrics
    Total number of metrics that are inputs to dependent calculators. Dependent calculators use for input the metric values that other calculators produce. This count refers to all the metrics given to the dependent calculators, not only the metrics produced by other calculators.
  • Number of Non Dependent Calculator Input Metrics
    Total number of metrics that are inputs to non-dependent calculators. Non-dependent calculators do not use metric values that other calculators produce. For example, metrics coming from agents.
  • Number of metric Data Queries per Interval
  • Number of Queued Async Data Queries
  • Number of Registered Async Data Queries
  • Number of Registered Async MG Queries
  • Number of Registered Async Path Queries
  • Number of Transaction Trace Action Sessions
  • Number of Transaction Trace Session Clients
  • Number of Virtual Metrics
  • AlertID
  • Query memory in transit (bytes)
  • Alerts
    • <Management_Module_Name>
      • Agent Connection Status
        - Number of Evaluated Metrics
      • Backend Heuristics
      • Browser Heuristics
      • CPU Heuristic
      • Console Summary Alert
      • Frontend Errors Heuristic
      • Frontend Heuristics
      • Frontend Response Time Heuristic
      • JDBC Heuristic
      • JMS Heuristics
      • JVM Heuristics
      • Thread Pool Heuristic
    • Total Number of Evaluated Metrics
      The total number of metrics that are evaluated for all alerts.
  • Calculators
    • Total Number of Evaluated Metrics
      The total number of metrics that are evaluated for all calculators. This metric is the Sum of Number of Dependent Calculator Input Metrics and Number of Non Dependent Calculator Input Metrics. When this count spikes, the Enterprise Manager is performing many real-time calculations, which can overload the CPU resources.
    • <calculator name> | Total Number of Evaluated Metrics
      The total number of metrics that are evaluated for an individual calculator.

      Note: This metric appears in the Investigator only when the calculator is defined.
  • GC Heap
    • Collectors
      • <Collector_Name>
        - Collection Count Per Interval
        - GC Duration (ms)
    • Pools
  • Harvest
    • Alert Action Processing Time (ms)
      Elapsed time the Enterprise Manager takes to process all alert actions.
    • Calculator Queries Wait Time (ms)
      Elapsed time for the calculator queries thread to complete its current work including waiting for the non-calculator query loop to finish. New calculator query processing starts after all the previous time slice non-calculator deliveries to clients complete.
    • Non Calculator Queries Delivery Time (ms)
      Time the Enterprise Manager took to run and deliver non-calculator queries to all requesting clients in a time slice.
      After all the calculator queries are run, the Enterprise Manager runs non-calculator queries and sends the results to all the clients that requested them.
    • Non Calculator Queries Excess Time (ms)
      Excess waiting time for non-calculator queries to complete beyond a time slice.
      Clients send non-calculator query requests to the Enterprise Manager, which sends results back. If this process does not finish within a time slice, it is carried over until completed. This metric shows how long beyond a time slice the non-calculator queries extended.
    • Metrics From All Agents
      Total number of unique metrics that are generated by all connected agents that have sent data in the last time slice. This count does not include historical metrics. Clamp settings do not affect this count.
    • Spooling Data File Write Time (ms)
      Time the Enterprise Manager took to write the harvested data to the spooling (.spool) file in a time slice.
    • Spooling Preparation Time (ms)
      Time the Enterprise Manager took to prepare the harvested data to write to the spooling (.spool) file in a time slice.
  • Management Module Calculators
    • Total Number of Evaluated Metrics
      Number of metrics that are input to the Management Module calculators.
  • Messaging
    • Active Incoming Threads
    • Active Outgoing Threads
    • Corrupted Messages Per Interval
    • Post Offices
      • <Post_Office_Name>
        - Number of Mailboxes
        - Queued Messages
  • Metric Group
    • Metric Matches Per Interval
      Total number of metrics that have been evaluated in all queries in the last time slice.
    • Queued Queries Per Interval
      Number of queries currently waiting for processing in the harvest cycle interval. The value is generally zero after startup.
  • Query
    • Cache Queries Duration (ms)
    • Cache Queries Per Interval
    • SmartStor Queries Duration (ms)
    • SmartStor Queries Per Interval
  • Threads
    • <Thread_name>
      • Blocked Count
      • Blocked Time (ms)
      • CPU Time (ms)
      • User Time (ms)
      • Wait Count
      • Wait Time (ms)

Problems Sub-node

Management Modules

  • Warning Count

Tasks Sub-node

Harvest Duration (ms)

SmartStor Duration (ms)

Harvest Metrics

Harvest Capacity

The Harvest Capacity metric displays the percent of time needed for the data harvest in a 15-second time slice. For example, if the data harvest takes 15 seconds, the metric value would be 100. The Investigator displays this metric at the location.

Custom Metric Host (Virtual)| Custom Metric Process (Virtual)| Custom Metric Agent (Virtual)(*SuperDomain*)| Enterprise Manager | Health | Harvest Capacity (ms)

Harvest Duration

The Harvest Duration metric shows the time in milliseconds (during a 15-second time slice) spent harvesting data. The metric is generally a good indicator in determining whether the Enterprise Manager is keeping up with the current workload. You can find this metric at the following location in the Investigator tree.

Custom Metric Host (Virtual)| Custom Metric Process (Virtual)| Custom Metric Agent (Virtual)(*SuperDomain*)| Enterprise Manager | Tasks | Harvest Duration (ms)

For more information about this metric, see CA APM Performance Monitoring Using Supportability Metrics.

Incoming Data Capacity (%)

The capacity of the Enterprise Manager to handle incoming data. The metric is calculated by multiplying the total metric capacity by 2. For example, if 150,000 metrics are in queue waiting to be processed, and the Enterprise Manager has a capacity to handle 300,000 metrics, Incoming Data Capacity is 25 percent.

You can find this metric at the following location in the Metric Browser tree:

Custom Metric Host (Virtual)| Custom Metric Process (Virtual)| Custom Metric Agent (Virtual)(*SuperDomain*)| Enterprise Manager | Health | Incoming Data Capcity (%)

For more information about this metric, see CA APM Performance Monitoring Using Supportability Metrics.

Collector Metrics

The following metrics are Collector metrics.

  • Collector Metrics Received Per Interval
    The Collector Metrics Received Per Interval metric is a simple way of gauging how much load metric data queries are placing on the cluster. This metric is the total sum of Collector metric data points that the MOM has received each 15-second time period, including data queries. You can find the Collector Metrics Received Per Interval metric here in the Investigator tree:

    Custom Metric Host (Virtual)| Custom Metric Process (Virtual)| Custom Metric Agent (Virtual)(*SuperDomain*)| Enterprise Manager | MOM | Collector Metrics Received Per Interval
    

    A large Collector Metrics Received Per Interval metric value, that is coupled with degradation of the cluster, indicates that the MOM has been asked to read too much metric data from the Collectors.
    For more information about this metric, see CA APM Performance Monitoring Using Supportability Metrics.

  • Collector | <Collector name> | Skew Time (ms)
    Indicates the clock skew for a specific Collector.
  • Collector Number of Async Queries per Interval
    The total number of asynchronous queries for all Collectors.
  • Collector Async Queries Duration (ms)
    The total time taken for all asynchronous queries for all Collectors.
  • Collector Number of Sync Queries per Interval
    The total number of synchronous queries for all Collectors.
  • Collector Sync Queries Duration (ms)
    The total time taken for all synchronous queries for all Collectors.
  • Collector Number of Sync Queries by CLW per Interval
    The total number of synchronous queries for all Collectors requested from all Command Line Workstations.
  • EM Live Metric Clamped
    Indicates if the number of live metrics that are handled by Enterprise Manager is less than or greater than the maximum limit specified in the introscope.enterprisemanager.metrics.live.limit property for Enterprise Manager clamps. The metric value is 0 if the number of live metrics for the Enterprise Manager is less than the specified limit. The metric value is 1 if the number of live metrics for the Enterprise Manager is greater than the specified limit.

    Note: You can define the clamp limit for introscope.enterprisemanager.metrics.live.limit property in the apm-events-thresholds-config.xml file. The apm-events-thresholds-config.xml file is located in the < EM_Home>\config directory.
  • EM Historical Metric Clamped
    Indicates if the number of live metrics that are handled by Enterprise Manager is less than or greater than the maximum limit specified in the introscope.enterprisemanager.metrics.historical.limit property for Enterprise Manager clamps. The metric value is 0 if the number of live metrics for the Enterprise Manager is less than the specified limit. The metric value is 1 if the number of live metrics for the Enterprise Manager is greater than the specified limit.

    Note: You can define the clamp limit for introscope.enterprisemanager.metrics.historical.limit property in the apm-events-thresholds-config.xml file. The apm-events-thresholds-config.xml file is located in the < EM_Home>\config directory.
  • Max Number of Agent Connection Limit Exceeded Per Interval
    The number of times the clamp limit defined in the introscope.enterprisemanager.agent.connection.limit property was exceeded for a particular interval.

    Note: You can define the clamp limit for introscope.enterprisemanager.agent.connection.limit property in the apm-events-thresholds-config.xml file. The apm-events-thresholds-config.xml file is located in the < EM_Home>\config directory.
  • Number of Events Processed
    Indicates the total number of all events such as Transaction Traces and errors that the Enterprise Manager processes in each interval.
  • Number of Events Processed Limit Exceeded
    The number of times the clamp limit defined in the introscope.enterprisemanager.events.limit property was exceeded for a particular interval.

    Note: You can define the clamp limit for introscope.enterprisemanager.events.limit property in the apm-events-thresholds-config.xml file. The apm-events-thresholds-config.xml file is located in the < EM_Home>\config directory.
  • Number of Collector Metrics
    The Number of Collector Metrics metric shows the total number of metrics currently being tracked in the cluster. You can find the Number of Collector Metrics metric here in the Investigator tree:

    Custom Metric Host (Virtual)| Custom Metric Process (Virtual)| Custom Metric Agent (Virtual)(*SuperDomain*)| Enterprise Manager | MOM | Number of Collector Metrics.
    

    Note: For more information about this metric, see CA APM Performance Monitoring Using Supportability Metrics.

Query Metrics

Data Points Retrieved From Disk Per Interval

The number of data points that are retrieved from SmartStor per interval.

Data Points Returned Per Interval

The number of data points that the Enterprise Manager returned to clients per interval.

Metrics Read From Disk Per Interval

Number of metrics that are read from SmartStor per interval.

Metrics Returned Per Interval

The number of unique metrics that the Enterprise Manager returned to clients.

Queries Exceeding Max Data Points Read From Disk Limit Per Interval

Indicates whether the maximum number of metric data points specified in the introscope.enterprisemanager.query.datapointlimit property that an Enterprise Manager returns for a particular batch query is exceeded. If the number of metric data points that Enterprise Manger returns is less than the specified limit, the metric value is 0. If the number of metric data points that Enterprise Manager returns exceeds the specified limit, the metric value is 1.

Note: Define the clamp limit for the introscope.enterprisemanager.query.datapointlimit property in the IntroscopeEnterpriseManager.properties file. Locate the IntroscopeEnterpriseManager.properties file in the <EM_Home>\config directory. To set the clamps, see Configure the System.

Queries Exceeding Max Data Points Returned Limit Per Interval

Indicates if the maximum number of metric data points that are specified in the queryintroscope.enterprisemanager.query.returneddatapointlimit property that an Enterprise Manager can return for a particular batch query is exceeded. If the number of metric data points that Enterprise Manager returns is less than the specified limit, the metric value is 0. If the number of metric data points that Enterprise Manager returns exceeds the specified limit, the metric value is 1.

Note: Define the clamp limit for the introscope.enterprisemanager.query.returneddatapointlimit property in the IntroscopeEnterpriseManager.properties file. The IntroscopeEnterpriseManager.properties file is located in the <EM_Home>\config directory. To set the clamps, see Configure the System.

Converting Spool to Data Metric

The Converting Spool to Data metric tracks whether the spool to data conversion task is running. You can find this metric at the following location in the Investigator tree:

Custom Metric Host (Virtual)| Custom Metric Process (Virtual)| Custom Metric Agent (Virtual)(*SuperDomain*)| Enterprise Manager | Data Store | SmartStor | Tasks | Converting Spool to Data

If this metric remains at 1 for more than 10 minutes per hour, this metric indicates that reorganizing the SmartStor spool file is taking too long.

Overall Capacity (%) Metric

The Enterprise Manager Overall Capacity (%) metric estimates the percentage of the Enterprise Manager capacity that is consumed. You can find it at this location in the Investigator tree:

Custom Metric Host (Virtual)| Custom Metric Process (Virtual)| Custom Metric Agent (Virtual)(*SuperDomain*)| Enterprise Manager: Overall Capacity (%)

Note: For more information about this metric, see CA APM Performance Monitoring Using Supportability Metrics.

SmartStor Capacity (%) Metric

The SmartStor Capacity (%) metric displays the percent of time that is needed for the SmartStor write process in a 15-second time slice, where 15 seconds equals 100 percent.

You can find it at this location in the Investigator tree:

Custom Metric Host (Virtual)| Custom Metric Process (Virtual)| Custom Metric Agent (Virtual)(*SuperDomain*)| Enterprise Manager | Health | SmartStor Capacity (%)

Note: For more information about this metric and about SmartStor, see CA APM Performance Monitoring Using Supportability Metrics.

Heap Capacity (%) Metric

The Heap Capacity (%) metric is determined by what percentage of heap the JVM is currently using (based on the GC Heap: In Use Post GC (mb) metric).

Note: For more information about this metric, see CA APM Performance Monitoring Using Supportability Metrics.

Write Duration (ms) Metric

The Write Duration (ms) metric displays the duration, in milliseconds, of the SmartStor write process. This is the integer version of the SmartStor Capacity metric. You can find it in this location in the Investigator tree:

Custom Metric Host (Virtual)| Custom Metric Process (Virtual)| Custom Metric Agent (Virtual)(*SuperDomain*)| Data Store | SmartStor | MetaData | Write Duration (ms)

Number of Agents Metric

This metric displays the number of currently connected agents. The metric location is:

Custom Metric Host (Virtual)| Custom Metric Process (Virtual)| Custom Metric Agent (Virtual)(*SuperDomain*)| Enterprise Manager | Connections

Number of Metrics

This metric displays the total metric load on the Enterprise Manager. The metric location is:

Custom Metric Host (Virtual)| Custom Metric Process (Virtual)| Custom Metric Agent (Virtual)(*SuperDomain*)| Enterprise Manager | Connections

Historical Metric Count Metric

The Historical Metric Count metric shows the total number of metrics from an agent that are live or recently active. The metric location is:

Custom Metric Host (Virtual)| Custom Metric Process (Virtual)| Custom Metric Agent (Virtual)| Agent | Historical Metric Count

Number of Historical Metrics

The Number of Historical Metrics metric displays the total number of metrics an Enterprise Manager is tracking across all agents. The metric location is:

Custom Metric Host (Virtual)| Custom Metric Process (Virtual)| Custom Metric Agent (Virtual)(*SuperDomain*)| Enterprise Manager | Connections | Number of Historical Metrics.

Note: For more information about this metric, see CA APM Performance Monitoring Using Supportability Metrics.

Was this helpful?

Please log in to post comments.