Skip to content
CA Application Performance Management - 10.0
Documentation powered by DocOps

Configure and Monitor Differential Analysis

Last update December 12, 2017

Differential Analysis lets you monitor changes in the performance and the stability of your applications.

Differential Analysis

Differential Analysis automatically identifies important changes in the performance of your applications. Legacy baselining predicts what is normal. Differential Analysis tracks the stability of your applications. It looks for uncontrolled variance in the Average Response Time of your frontend and business transaction metrics. The results are not very different from how a seismometer displays earthquakes. Uncontrolled variance appears as dark areas in an otherwise stable white strip, letting you pick out periods of instability across minutes, hours, or days -- even for hundreds of applications.

The core of Differential Analysis was invented and put into practical use by Walter Shewhart to address the problem of quality control on buried telephone lines. Shewhart invented statistical control charts and the Western Electric Rules. Western Electric Rules are decision rules that detect out-of-control variance on control charts.

Note: By default, Differential Analysis works on Average Response Time for application frontends and business transaction metrics. While you can configure other metrics, good results are not guaranteed. For example, you can configure backend Average Response Time for Differential Analysis. Feel free to experiment and get back to us with your results. Good results are not likely when measuring Responses Per Interval, GC Heap, or similar counter type metrics.

Differential Analysis Map

The Differential Analysis map is a visual exploration of the stability and responsiveness of many applications: dozens, hundreds, or even thousands of them. Each strip in the map corresponds to a single metric, for example, the Average Response Time for a specific frontend application. In a quiet, stable period the strip is a light shade. When instability occurs, the shade of the strip darkens progressively according to the severity of the instability. Thus, a single strip lets you see the stability of a single application or business transaction over time. The map sorts the most unstable strips to the top and makes it easy to scan through dozens of applications for unstable response times.

Variance intensity determines the color of the strip at a point in time. It is represented as follows:

  • Gray -- total inactivity, the application received no transactions over the interval measured.

    Note: When data suddenly becomes unavailable, the shade of the last cell with data is retained until the number of empty cells is equal to the Window Length value. This value (20 by default) is configured in the differential control.
  • White -- good stability
  • Light -- moderate instability
  • Dark -- severe instability, a problem that requires immediate attention may exist.

Differential Analysis also reports peak instability over an interval of lower instability (a lighter shade). In this case, the strip shows the shade of the overall stability as usual. When the maximum variance intensity is greater than the average variance intensity for a period, a vertical line appears in the cell. The line is the shade of the maximum state. This depiction helps to ensure that you do not miss an important period of instability while looking at a longer time range. For example, during a long interval, any instability at shorter intervals can be averaged out. The map cell is striped with a vertical line indicating the peak variance, so that an important period of instability is not missed.

  • Map Special Cases

    Be aware of the following special cases:

    • During the learning period of baseline calculation, the baseline engine does not report variance intensity. This behavior reduces false positives. The map may either not show a strip or show a yellow learning pattern to indicate that the differential control is not ready to report its results.
    • If a metric is idle for an interval, the number of transactions that are monitored is zero -- the map displays a gray, hashed pattern. This pattern indicates the absence of activity during that interval.
    • Complex queries can return partial results. If a query is too complex, Differential Analysis applies an intelligent clamp. A truncated set of results appears with a red message at the top of the map. This message indicates that:
      – The map does not display the full result set.
      – Traversing further down the Investigator tree limits the number of matching metrics and increases the chances of retrieving a full result set.

      APM includes a data point limit clamp that limits all query results beyond a specified number:

      introscope.enterprisemanager.query.datapointlimit

      Clamps the maximum number of data points that the Enterprise Manager retrieves from the SmartStor disk during a batch of metric queries.

      introscope.enterprisemanager.query.returneddatapointlimit

      Clamps the number of data points that are returned from a batch of metric queries.

    Important! If you use the data point limit clamps, you receive incomplete results in the Differential Analysis maps when the data point limit clamps are reached. Use the Differential Analysis intelligent clamping, and turn on data point limit clamps only when necessary.

How to Configure Differential Analysis

Differential Analysis is active by default. However, as an administrator you configure the default settings, or create and configure Differential Analysis elements. As an application triager, you monitor performance using the information in the Differential Analysis map or chart.

Perform the following steps:

  1. Verify the Differential Analysis prerequisites.
  2. Configure a metric grouping.
  3. Create a differential control.
  4. Configure a differential control.
  5. Configure a simple alert from a management module.
  6. (Optional) Tune the configuration:
  7. Monitor performance using Differential Analysis.

Verify the Differential Analysis Prerequisites

Before you implement Differential Analysis, verify that your CA APM implementation includes:

  • An Enterprise Manager
  • Agents configured for application frontends, business transactions, or both.
  • WebView
  • For the CA CEM installation, you have installed a Transaction Impact Monitor (TIM).

    Note: You can have one or more TIMs in a CA CEM environment.

Configure a Metric Grouping

You use the Perl regular expression standard for configuring metric groupings. A regular expression is a text string that describes a search pattern. Wildcards let you condense the collection of objects to a short expression.

Follow these steps:

  1. In WebView, click MANAGEMENT.
  2. Navigate the Management Modules tree to the Metric Groupings node which includes the metric grouping that you want to configure, for example: Frontend URLs and BTs Average Response Times.
  3. Click the metric grouping.
    The Preview pane appears.
  4. Select one of the following options. The following fields require explanation:
    • Use Management Module Agent Expressions
      Defines the use of agent expressions that are defined at the management module level, for example: (.*)\|(.*)\|(.*)
    • Use Metric Grouping Agent Expressions
      Defines the use of agent expressions at the metric grouping level. Each metric grouping can have its own set of agent expressions.
  5. Enter a regular expression:
    • Metric Grouping Agent Expressions
      Specifies a regular expression in Perl 5 that filters input to the metric by specifying the data up to and including the agent name.
      This expression determines to which agents the grouped data is limited. For example, the expression (.*)\|WPS2-0[1,2] looks for any agents in all domains that are named WPS2-01 or WPS2-02. The agent expression is important when multiple JVMs report to the same Enterprise Manager, but serve different purposes such as different websites in the same corporation. In this case, you use a regular expression to limit which agents your metric grouping filters.

      Note: These fields are disabled when you select the Use Management Module Agent Expressions option.
    • Metric Expressions
      Specifies a regular expression in Perl 5 that specifies the resource (the chain of folders leading to the metric) and the metric. For example:
      Frontends\|Apps\|[^|]+\|URLs\|[^|]+:Average Response Time\(ms\) looks for all URLs for frontends and only for the Response Time (ms) metric.
      Business Service\|[^|]+\|Business Transactions\|.*:Average Response Time\(ms\) looks for CEM business transaction metrics coming from the TIM and only for the Response Time (ms) metric.
  6. Click Add.
    An additional field opens so you can enter a regular expression for Metric Grouping Agent Expressions and Metric Expressions.
  7. Click Remove.
    If more than one set of expressions are available, you can remove them. One expression must exist in the metric grouping definition. In this case, Remove is disabled.
  8. Click Apply.
    The metric grouping is configured.

Create a Differential Control

Differential controls are elements that detect instability in matching metrics. Caution thresholds and danger thresholds control variance intensity, which is a measure of the stability for the metric. This behavior makes differential controls different from traditional alerts that combine a state of green, yellow, and red and imply that problems exist. Differential controls let you determine how much instability must be registered to enter caution and danger variance intensity zones. The advanced options provide even more control over Differential Analysis variance intensities. In general, Differential Analysis users can apply low thresholds and view the full feed of stability changes, or use high thresholds and limit the output to a trickle of only the most extreme changes (which are most likely to indicate a problem).

We recommend that you start with the default thresholds and change the default alert threshold settings to indicate the intensities at which you would like a notification.

The Differential Analysis map only appears for a metric if a differential control exists for the metric. You create a differential control element in a management module matching your metric. Management modules can exist for each domain or in all domains and they contain elements that organize metric data for presentation in WebView. CA APM includes a preconfigured differential control for all frontends and business transactions.

Follow these steps:

  1. In WebView, click MANAGEMENT.
  2. Select Elements, New Differential Control.
    The New Differential Control dialog appears.
  3. Complete the following options and click OK. The following fields require explanation:
    • Name
      Specifies a name for the element.

      Note: To identify the source of an element, use a descriptive name.
    • Force Uniqueness
      Appends a number to the name when the same name exists within the management module.
    • Management Module
      Specifies a management module for containing the element.

    The element appears under the differential control node in the management module that you specified. The editor appears in the Preview pane.

    Important! In the Workstation, no differential control elements appear. In WebView, you can create, read, update, and delete the differential control elements.

Configure a Differential Control

A differential control management module lets you use a common configuration for Differential Analysis, such as actions for caution and danger states.

Note: The Management Modules include a preconfigured differential control for all frontends and business transactions. You can edit this module that is under the Differential Controls folder and named Differential Analysis Control.

Follow these steps:

  1. In WebView, click MANAGEMENT.
  2. In the tree, navigate to Management Modules, <Management Module>, Default Differential Controls.
  3. Select the metric group that you want to configure for a differential control, for example: Frontend URLs and BTs Average Response Time.
  4. Adjust the thresholds by using the Danger and Caution sliders. The top (very high value) of the one slider automatically adjusts the bottom (very low value) of the other slider. Configure more caution and less danger, or more danger and less caution: Caution very high equals Danger very low. The generated thresholds from the slider values auto-adjust to the window length.
    The following graphic shows a quick reference for Differential Analysis.
  5. (Optional) Click Advanced and complete the options:
    • Rule Application
      Specifies a rule for triggering spikes:
      1 Any single data point falls outside the 3σ limit from the centerline.
      2 Two out of three consecutive points fall beyond the 2σ limit.
      3 Four out of five consecutive points fall beyond the 1σ limit.
      4 Ten out of ten consecutive points increasing or decreasing.
    • Window Length
      Specifies the number of previous 15-second cells that influence the weight of a window. The total weight of the window determines caution or danger regions. Each cell in the window contains the largest weight of all rules that are broken in that 15-second period, minus any decay. The oldest cell is aged out and dropped as new cells are added.
      Default: 20
    • Decay
      Specifies the rate at which the breached rules decay with their cell over time. As new cells are added, decay reduces the effect of the previously added weights in the window. The newest one is always full weight. Based on the decay value, only parts of old weights are considered for final calculation. So each cell in the window has a reduction percentage following the decay level. Based on that reduction percentage value, the weights are reduced on each new instance addition to the window.

      100 represents the fastest decay—rule breaches drop in value quickly as they age. From newest to oldest, the cells have progressive reductions from 0 to 100 decay in their weights. For example, oldest cells have complete reduction in their weight. This setting reduces the likelihood of caution and danger notification, especially when a period of instability passes quickly.

      0 represents the highest sensitivity (no decay)—breaches maintain their full value over the entire window: oldest has the same value as it did when it was newest. Using this setting, a short period of instability can keep the spike in a caution or danger state until it has aged out completely from the window.

      The default value is 20. Recent rule breaches are worth moderately more than older rule breaches.

  6. Click Apply.
    The differential control for a baseline is configured.

Configure a Simple Alert From a Management Module

You can configure a default simple alert for differential analysis and define the conditions under which the alert is triggered.

Follow these steps:

  1. In WebView, click MANAGEMENT.
  2. Navigate to Management Modules, Sample, Alerts.

  3. Click Default Differential Analysis Alert.

    The Preview pane shows the default values for the alert.

  4. (Optional) Change the following options:
    • Resolution
      Specifies the time period resolution in hours, minutes, or seconds.
      An alert uses input data from a selected metric grouping. For the time resolution, the alert gathers information and summarizes a value for that time period. The resulting value depends on the type of data in the metric. For example, if the metric is a rate, the summarized value is the average rate during that time period. Or if the metric is a counter, it produces the most recent value of the counter.

      Note: Time resolution values must be in 15-second increments.
    • Combination
      Specifies whether an alert is triggered when a metric exceeds a threshold (any) or all metrics exceed a threshold (all).

      Note: The Combination field is ignored when the Notify by individual metric check box is selected.
    • Comparison Operator
      Specifies the condition in which the alert is triggered. The comparison operator relates to the danger and caution threshold values.
    • Notify by Individual Metric
      Triggers individual metric alerts. You can use the individual metric alert notification and resolution alert together.
    • Trigger Alert Notification
      Triggers an alert notification when a problem or resolution state is detected. Select one of the following options:
      • Each Period While Problem Exists -- This option produces a problem message every period that the simple alert is in caution or danger.
      • When Severity Increases -- This option produces a problem message on any period when the state of the alert escalates.
      • Whenever Severity Changes (resolution alert) -- This option produces problem or resolution messages on any state transition.
      • Report Only Final State Whenever Severity Changes (resolution alert) -- This option produces a problem or resolution message only for the final state of an alert transition.
  5. Set the threshold options for Danger and Caution alerts.
    • Threshold
      Specifies a value that triggers a danger alert or caution alert.

      Note: For Differential Analysis, the underlying metric is variance intensity, which has a range from 10 to 40. 10 - 20 is stable range. 20 - 30 is moderate instability. Over 30 is a period of severe instability. A good default can be 25 for Caution and 35 for Danger.
    These options set the ratio of excessive periods that must be met for CA Introscope to trigger the alert. For example, enter 8 and 10. The danger alert triggers only when the metric exceeds the danger threshold in eight of the ten observed periods.
    • Periods Over Threshold
      Specifies the maximum number of periods that the threshold can exceed before an alert is triggered.
    • Observed Periods
      Specifies the total number of periods being monitored for each alert iteration.
  6. Add actions to danger or caution conditions. Under either Danger actions or Caution actions in the settings pane, click Add.
    The Choose Action dialog appears and lists available actions.
  7. Select one or more actions and click Choose.

    Note: To select multiple actions, press the Ctrl key while selecting actions.

    The actions appear in the Actions list.

  8. In the list, select the alert that you want to test and click Test.
    A message appears for all the actions in the list.

    Note: Test messages appear only when the actions are active.
  9. Click Close on the message dialog.
    The actions are set for the alert.
  10. Enter the action delay time in hours, minutes.
  11. Click Apply.
    The simple alert configuration is set.

Reduce Notifications for a Frontend

You can reduce or stop the action notifications for a frontend by:

  • Excluding the specific frontend applications from Differential Analysis (which stops all notifications).
  • Excluding the specific frontend applications and then creating a separate differential control element specifically for that frontend (which allows fine-tuning of notifications).
Note: A simple way to reduce the number of notifications is to add actions to the danger list only.

Follow these steps:

  1. In WebView, click MANAGEMENT.
  2. In the tree, navigate to Management Modules, <Management Module>, Differential Control.
  3. Note the name of the metric group that you want to edit for a differential control, for example: Frontend Average Response Time.
  4. Navigate the Management Modules tree to the Metric Groupings node you noted.
  5. Click the metric grouping.
    The Preview pane appears.
  6. Select the Use Metric Grouping Agent Expressions option.
  7. Replace the values in the following fields:
    1. Use your existing agent expression.
    2. Enter a pattern in Metric Expressions.
      For example, this pattern excludes a frontend that is named Foo:

      Frontends\|Apps\|(?!Foo)\w+:Average Response Time \(ms\)

      To exclude more frontends, use a pipe character (|) to separate each frontend:

      Frontends\|Apps\|(?!Baz|Foo|Bar)\w+:Average Response Time \(ms\)

      If you have a frontend with a long name or special characters, use a unique, short prefix for the name instead of the entire name. If the prefix matches more than one name, all matching frontends are excluded.

  8. Click Apply.
    The metric grouping is configured to exclude the specified frontend applications. The frontends are not included in Differential Analysis and the map.

If you still want notifications but you want different differential control settings (for example, greater caution or danger thresholds), follow these steps:

  1. Exclude the frontend from the default metric grouping by performing the previous steps.
  2. Create a metric grouping that matches your frontend only.
  3. Create a differential control element that uses your new metric grouping, and then adjust the configuration as appropriate.

Increase the Metric Count Limit

Each collector has a default limit to the number of collected metrics. Differential controls have a default maximum of 20,000 metrics. If the limit is reached, no new metrics appear in the Differential Analysis map. You can increase the metric count limit for a differential control so that the metrics appear.

Follow these steps:

  1. In WebView, click INVESTIGATOR, Metric Browser.
  2. Navigate the tree to Variance.
  3. Select each of the management modules that have differential control elements and select the Metric Count tab for each one.
  4. Note the number that appears in the Metric Count column.
    The sum of these counts is the number of metrics that you have configured for analysis. If that number equals 20,000 metrics or greater, continue to the next step.
  5. Go to <EM_Home>/config and open the IntroscopeEnterpriseManager.properties file in a text editor.
  6. Locate the following section:

    ###############################################################################

    # Metric Baselining Settings

    # Configures values for the metric baseline engine.

  7. Increase the value in the following property, for example:

    introscope.enterprisemanager.baseline.max.metrics.baselined=21000

    Note: If a collector is at capacity and you lower this value, the number of metrics do not decrease immediately. The metrics decrease over time as metrics age out.
  8. Save and close the file.

Turn Seasonality On or Off

Differential Analysis uses previous data in 30-minute periods, 48 periods per day, and 336 (48 times 7 days) periods per week for seasonal prediction. For the first two weeks, the prediction is not seasonal. The third week starts seasonal prediction using the previous two weeks, and any invariant data appears as a variant in the map. You can turn seasonality on or off. Seasonality is on by default.

Follow these steps:

  1. Go to <EM_Home>/config and open the IntroscopeEnterpriseManager.properties file in a text editor.
  2. Change the introscope.enterprisemanager.baseline.core.seasonality.enabled property value to true or false.
  3. Save and close the file.

Monitor Performance Using Differential Analysis

Real-time performance data displays in the Differential Analysis map. This map shows a baseline summary of the actual values, the prediction, and the standard deviation. You can also investigate performance for individual components.

Note: A standard deviation is the statistical difference between each data point and the predicted value.

Follow these steps:

  1. In APM Team Center, click WebView.
    APM WebView appears.
  2. Click INVESTIGATOR.
    The Metric Browser tree shows a hierarchical view of your system.
  3. In the tree, select the agent for which you want performance information, for example:

    SuperDomain | Host | Process |Agent | Frontends | Apps | Adaptor

    Note: The Variance node and subnodes do not show Differential Analysis.

  4. Click the Differential Analysis tab.
    The map shows a graphical representation of performance data and shows the last 8 minutes of data. The top 100 problematic metrics appear in decreasing instability. The data refreshes when you query, change the time period, or select a different node.
  5. (Optional) Adjust the time range using the Change Start Time arrow controls.
    The latest data appears.
  6. Click a strip of interest in the map.
    The Differential Analysis chart appears. This chart helps you understand the metric stability over the timeline that the strip represents. The chart shows the status of the monitored component, so you can quickly detect normal and abnormal performance:
    • A line represents an actual metric value.
    • The shaded regions correspond to standard deviation bands 1, 2, and 3. The darker the band, the higher the deviation from the predicted value. Any metric in the white is better than predicted. Any metric that appears above the bottom white area has exceeded the predicted value. For example, if the metric exceeds the top band, it is exceeding 3x the standard deviation.
  7. Mouseover the line.
    Tooltips display the metric values.
Was this helpful?

Please log in to post comments.

  1. Luiz Freire Emmerich
    2015-07-29 09:29

    When talking about the map, besides the described (gray, white, light, dark) I see some other variations like yellow, blue (I know that the light/dark refers to blue but the color was not described at the list), diagonal striped blue, etc. What do those variations mean?

    yellow: differential analysis calibration?

    diagonal striped blue: differential analysis idle?

    Would be great to have a screenshot (images) pointing to the differences in color.

     

    Thanks in advance

    1. Anonymous
      2015-07-30 07:04

      Hi Luiz,

      Please refer to this information in the documentation:

      Map Special Cases

      Be aware of the following special cases:

      • During the learning period of baseline calculation, the baseline engine does not report variance intensity. This behavior reduces false positives. The map may either not show a strip or show a yellow learning pattern to indicate that the differential control is not ready to report its results.
      • If a metric is idle for an interval, the number of transactions that are monitored is zero -- the map displays a gray, hashed pattern. This pattern indicates the absence of activity during that interval.

      Thanks for your feedback,

      Carey 

       

      1. Luiz Freire Emmerich
        2015-07-31 08:38

        Cool, missed that...

        Those bullet points really caught my attention when checking for that information and I quit there... maybe adding another bullet at "Differential Analysis Map" with something like:

        • Special cases: check next item or
        • Yellow and diagonal striped blue: ...

        I believe Nikunj had an image for that as well, that would also be useful to enhance the documentation.

        (smile)

        Thank you for your time!

  2. Nikunj Mahesh Jain
    2015-07-29 08:55

    Hello Team,

    To Luiz's question and to enhance the detailed explanation of Differential analysis Maps. Please review the attachment and update it on the page. This will allow clarify basic doubts about the information DA Map cells exhibit in different scenarios.

    Unfortunately, I am unable to attach the description chart here. So I will email the admin to allow me the permissions required to add those.

    Thanks.

    1. Carey Feldstein
      2015-07-30 07:29

      Hi Nikunj,

      Please see my comment to Luiz.

      Thanks,

      Carey

  3. Janelle Shaw
    2015-08-05 03:34

    I agree Nik's diagram would be a nice visual to this description.

    Also, I would consider reordering the steps as one would create a Metric Grouping to organize the metrics for which we want to apply a DA Control and THEN create the DA Control, where you can select the (newly created) Metric Grouping from the drop-down list.  Or perhaps just have the Create Metric Grouping step referenced/link in the Create DA Control step. Otherwise, someone might create a control, then realize they want a new metric grouping, so they move to create it, and then have to go back to DA control to set it to the new grouping. Nuance but you see what I'm saying.

    Also, in the Configure Differential Control section - you may want a gentle second reminder that MOST users do NOT need to adjust the thresholds initially.

    Last comment - I love the seismometer analogy. Great way to describe why DA is different than just detecting any baseline deviation without context of severity and duration.

     

    1. Carey Feldstein
      2015-08-06 09:59

      Hi Janelle,

      Thank you for your feedback. I will work with you and my team to make the improvements.

      Carey

  4. Robert Gauthier
    2015-09-30 07:57

    Would you add more information on how base lining works.  It is not clearly described in the documentation, unless I missed it.

    thank you,

     

    Rob

    1. Carey Feldstein
      2016-07-08 10:27

      Hi Robert,

      I checked with the engineering team and they need to know more details about what you want to know. Please send us your questions.

      Thank you, Carey