Skip to content
CA Application Performance Management - 10.7
Documentation powered by DocOps

Metric Requirements and Recommendations

Last update October 8, 2018

Most agents report metrics to the Enterprise Manager at 7.5-second intervals. The Enterprise Manager processes metric data at 15-second intervals called time slices. The processing of time slices is named the harvest cycle. During the harvest cycle, the Enterprise Manager aggregates the two sets of metrics data that the agents reported. This aggregated data is processed to perform calculations, check alerts, update heuristics, and update Workstation views. Raw (agent-based) and processed metrics are persisted to the SmartStor database.

Introscope metric load bases resource consumption on the following workload factors:

  • Total number of metrics the Enterprise Manager receives per interval.
  • Total number of metrics that calculators process.
    Metric groupings is the term for metrics that calculators process.
  • Number of metrics persisted per interval.

Understanding metric groupings and metric matching can be helpful in avoiding performance problems.

Metric Groupings and Metric Matching

Every metric in Introscope has a string identifier that includes its host, process, agent, and resource name such as this string identifier:

*SuperDomain* | Custom Metric Host (Virtual)| Custom Metric Process (Virtual)| 
Custom Metric Agent (Virtual)(*SuperDomain*) |<Host_Name>|<Process_Name>|
<Agent_Name>|JSP|_Shopping_Cart_JSP:Average Response Time (ms)

The structure of your Investigator tree reflects the metric identifier. A metric identifier is also referred to as the metric path.

Introscope business logic is built around the concept of metric groupings. Introscope users can define metric groupings using the Management Module Editor in the Workstation.

In the Management Module Editor metric groupings are specified using regular expressions that match metric identifiers. Introscope also creates internal metric groupings to support CA APM functionality. Every frontend is an implicit metric grouping.

A metric received from an agent can match no metric groupings, one grouping, or many groupings. Metric groupings imply aggregation of metrics. Calculators perform operations on metric groupings.

Note: For more information about defining metric groupings, see Configure Metric Groupings in the Workstation.

Monitor Generated Metrics and Calculators

Calculators subscribe to metrics coming from agents or other calculators, perform calculations on those metrics, and then output new metrics. Calculator-generated metrics are metrics that calculators output. These metrics are stored in SmartStor and returned in response to queries, as are agent-based metrics. Calculator-generated metrics can result in a larger than expected metric load on your Enterprise Manager or MOM, and can in extreme circumstances contribute to performance problems. You view calculator-related metrics as well as agent-based metrics in the metric browser tree.

Calculator execution can be a large part of each harvest cycle. If the average harvest duration exceeds 3500 milliseconds, an Enterprise Manager or MOM is in danger of overload. You can see the Harvest Duration metric here:

*SuperDomain*|Custom Metric Host (Virtual)|Custom Metric Process (Virtual)|
Custom Metric Agent (Virtual)(*SuperDomain*)|
Enterprise Manager|Tasks:Harvest Duration (ms)

When the average harvest duration exceeds 3500 milliseconds, one or more of these causes are the most likely:

  • Metric overload
  • Calculator overload
  • Resource contention with other processes.

Metric Overload

Persisting metric data to SmartStor is also a critical activity during each harvest cycle. The SmartStor duration metric tracks how long it takes to write data to SmartStor during every 15-second metric harvest cycle. If an Enterprise Manager is experiencing metric overload, you also see high SmartStor durations consistently exceeding 3500 ms. You can see the SmartStor Duration metric here:

*SuperDomain*|Custom Metric Host (Virtual)|Custom Metric Process (Virtual)|Custom Metric Agent (Virtual)(*SuperDomain*)|Enterprise Manager|Tasks:Smartstor Duration (ms)

Calculator Overload

You can use these metrics under the Enterprise Manager | Internal node to monitor the calculator load:

  • Calculator Harvest Time to see the time spent executing calculators in each harvest cycle. See the Calculator Harvest Time metric here:

    *SuperDomain*|Custom Metric Host (Virtual)|Custom Metric Process (Virtual)|Custom Metric Agent (Virtual)(*SuperDomain*)|Enterprise Manager|Internal:Calculator Harvest Time (ms)
    
  • Number of Calculator-generated Metrics to see the number of metrics output by calculators. See the Number of Calculator-generated Metrics metric here:

    *SuperDomain*|Custom Metric Host (Virtual)|Custom Metric Process (Virtual)|Custom Metric Agent (Virtual)(*SuperDomain*)|Enterprise Manager|Internal:Number of Calculator-generated Metrics
    
  • Number of Calculators to see the number of calculators that execute each harvest cycle. See the Number of Calculators metric here:

    *SuperDomain*|Custom Metric Host (Virtual)|Custom Metric Process (Virtual)|Custom Metric Agent (Virtual)(*SuperDomain*)| Enterprise Manager| Internal:Number of Calculators
    
  • Number of Evaluated Metrics to see the number of metrics each alert is evaluating. Alerts, being calculators, can subscribe to a variable number of metrics from one to as many there are in the system. This metric allows you to identify the individual system impact of each alert. You can see the Number of Evaluated Metrics metric here:

    *SuperDomain*|Custom Metric Host (Virtual)|Custom Metric Process (Virtual)|Custom Metric Agent (Virtual)(*SuperDomain*)|Enterprise Manager|Internal|Alerts|...:Number of Evaluated Metrics
    

    Note: The path to individual alert metrics varies with the alert definition and the Management Module in which the alert is defined.

You can use this metric to monitor cluster metrics load:

  • Collector Metrics Received per Interval to see the total number of metrics a MOM receives from the Collectors each 15-second harvest interval. Non-Collector metrics, such as those that internal virtual agents (metric groupings) and calculators generate, are not counted. You can see the Collector Metrics Received per Interval metric here:

    *SuperDomain*|Custom Metric Host (Virtual)|Custom Metric Process (Virtual)|Custom Metric Agent (Virtual)(*SuperDomain*)|Enterprise Manager|MOM:Collector Metrics Received per Interval
    

Performance Problems and Poorly-Defined Metric Groupings

Metric groupings define the set of metrics that the Enterprise Manager must query on behalf of calculators during each harvest cycle. In a cluster, metric groupings define the metric subscription load on the MOM. For best performance and efficient use of computing resources, define metric groupings as narrowly as possible. Include only the metrics that are required to provide the information you need.

Do not create metric groupings or regular expressions that use only the wildcard or asterisk (.*) symbol and no other specifiers. Running the search term (.*) creates a metric grouping that matches all metrics in the system. As the monitoring load grows, a metric grouping that matches all metrics in the system can quickly overload harvest cycle processing.

You can use the SmartStor tools test_regex utility with the “-metrics” option to see how many metrics are matched by the regular expressions you use in your metric groupings. You can find the SmartStor tools utilities in the <EM_Home>/tools directory.

Note: For more information about the test_regex utility, see Configure and Manage SmartStor Data.
Was this helpful?

Please log in to post comments.