Skip to content
CA Application Performance Management - 10.3
Documentation powered by DocOps

Configure and Manage SmartStor Data

Last update November 15, 2018

SmartStor stores metric and transaction data. The CA Introscope SmartStor database records all application performance data (CA Introscope metrics) always. This database information lets users analyze historical data, identify root causes of application downtime, or perform capacity analysis without the need for an external database.

SmartStor is enabled by default during CA Introscope installation. SmartStor data is set to age out over time, so the data store will not get excessively large. Multiple data files can grow in number as more data is generated.

Note: For questions and answers about SmartStor, see TEC1651685.

SmartStor Data Storage

SmartStor records all application performance data (Introscope metrics) while it is in production, without the need for an external database. SmartStor lets you analyze historical data, identify the root causes of application downtime, or perform a capacity analysis. SmartStor is enabled with default settings when installing Introscope:

  • Once an hour, on the hour, SmartStor converts .spool files to .data files to optimize query processing.
  • Once a day, at midnight, SmartStor reformats historical metric data based on the Enterprise Manager tier settings and deletes aged out metrics and metadata.

SmartStor uses a storage hierarchy that assumes that recent data is accessed the most and has the highest resolution. The following data resolutions are based on the default property settings:

  • Store 1 week of data in 15-second chunks (uncompressed)
  • Store 30 days of data in minute (60 seconds) chunks (compressed)
  • Store 1 year of data in 15 minute (900 seconds) chunks (compressed)

SmartStor Data Files

The data files for SmartStor are:

  • The metrics.metadata file -- Tracks information about the various metrics that the agents have reported. This file tracks the names of the metrics.
  • Files with a .spool extension -- Holds the 15-second metric values sent to SmartStor since the last spool-to-query conversion on the Enterprise Manager. The .spool file never holds more than one hour worth of data. An example name is 1251226800000.spool.
  • Files with a .data extension -- Holds metric values in the read-only query optimized format in SmartStor. These files hold the data point values for 15 seconds, 1 minute, and 1 hour. An example name is 1251183600000.data.

By default the Enterprise Manager stores all SmartStor data in the <EM_Home>/data directory.

Warning! Stop the Enterprise Manager before you back up or you work with the SmartStor data files. Copying or editing Enterprise Manager data files while the Enterprise Manager is running results in data loss.

When an Enterprise Manager is shut down, you can do all of the following tasks to back up and work with Enterprise Manager data:

  • Save data to backup media.
  • Edit SmartStor data using SmartStor tools.
  • Compress data.
  • Restore data.
  • Copy data for offline viewing by another Enterprise Manager.

Set Up a Dedicated Disk for SmartStor Data

Configure SmartStor to write to a dedicated disk. A dedicated disk avoids severely affecting Enterprise Manager performance and responsiveness. Disk contention with other Enterprise Manager activities or other processes running on the host can cause poor performance.

Follow these steps:

  1. Go to the <EM_Home>/config directory.
  2. Open the IntroscopeEnterpriseManager.properties file in a text editor.
  3. Set the following property to true:

    introscope.enterprisemanager.smartstor.dedicatedcontroller

  4. Save and close the file.
  5. Restart the Enterprise Manager.

Automatic Metadata Deletion

During the SmartStor reperiodization, Enterprise Managers automatically delete metadata (metric and agent descriptions) for which there are no associated metric data points. This process frees disk and memory space for reuse during SmartStor tiering.

Enterprise Managers do not delete historical agents that are manually mounted in a Workstation, including agents without metric data in SmartStor.

Configure SmartStor Tiers

The tiers in SmartStor define how long to store metric data. You can have a maximum of three tiers, each with the same or increasing frequency periods. The frequency represents the metric reporting granularity.

Each tier consists of a property pair:

  • Storage frequency (in seconds)
    • The frequency must be a multiple of 15 seconds.
    • The frequency must not be greater than 1800 seconds (30 minutes).
    • The frequency of each tier must be an even multiple of the previous tier frequency. This multiple can be one; each tier can have the same frequency.
  • Data aging period (in days). The age must not be zero.

Tier 1 is the minimum period. You configure Tier 2 based on the configuration of the frequency of Tier 1. For example, Tier 2 is as a summary of Tier 1, and Tier 3 is a summary of Tier 2. The following table shows how the tiers, frequency, and age relate:

Tier Frequency in Seconds Age in Days
1 15 7
2 15 14
3 30 10
  1. Nightly -- SmartStor collects metric data in an archive file, for example, a file named A1. A1 is stored in Tier 1.
  2. Day 8 -- A1 is promoted to Tier 2.
  3. Day 21 -- A1 is promoted to Tier 3. The frequency of data is converted to 30 seconds.
  4. Day 31 -- A-1 is removed from SmartStor.

Follow these steps:

  1. Go to the <EM_Home>/config directory.
  2. Open the IntroscopeEnterpriseManager.properties file in a text editor.
  3. Enter values for the following properties:

    introscope.enterprisemanager.smartstor.tier1.frequency

    introscope.enterprisemanager.smartstor.tier1.age

    introscope.enterprisemanager.smartstor.tier2.frequency

    introscope.enterprisemanager.smartstor.tier2.age

    introscope.enterprisemanager.smartstor.tier3.frequency

    introscope.enterprisemanager.smartstor.tier3.age

  4. Save and close the file.
  5. Restart the Enterprise Manager.

    Note: If you later change your data tiers and restart the Enterprise Manager, SmartStor attempts to adjust data to your new configuration. SmartStor can lose up to one day of data.

Create a Query-only Duplicate Cluster

You can copy SmartStor data files to duplicate cluster machines to serve as a query-only source historical (non-live) data.

Note: We recommend creating duplicate clusters only in production environments where the service demand considerably exceeds the capacity of the hardware on which the cluster is running.

Follow these steps:

  1. Create an exact duplicate of your production or test cluster. Use one computer for the MOM and one for each collector in your cluster.
  2. Shut down your production cluster.
  3. Copy the SmartStor, baselines, and Transaction Events data files from:
    • The production or test MOM to the duplicate MOM.
    • From each of the production or test collectors to the duplicate collectors.
  4. Start up both the production cluster and the query-only duplicate cluster.
  5. Run any desired queries on the query-only duplicate cluster.
  6. Repeat necessary steps once a day to keep current the data on the query-only duplicate cluster.

Use the SmartStor Command-Line Tools

The SmartStor tools let you perform different maintenance and administration tasks on the SmartStor database. These command-line tools that let you:

  • Clean up from metric leaks and explosions

    Note: For information about metric leaks and explosions, see CA APM Sizing and Performance.
  • Split a single Enterprise Manager into multiple Enterprise Managers
  • Merge multiple Enterprise Managers into a single Enterprise Manager

Follow these steps:

  1. Shut down the source Enterprise Manager.

    Important! CA Technologies recommends shutting down the source and destination Enterprise Managers when running SmartStor tools. Corruption can occur if the Enterprise Manager is writing data to SmartStor when SmartStor tools are also changing SmartStor data.
  2. Go to the <EM_Home>/tools/ directory.
  3. Open a command prompt and use one of the following start scripts:
    • Windows -- SmartStorTools.bat
    • UNIX -- SmartStorTools.sh
  4. Use the following syntax:

    <SmartStorTools start script> <command> <arguments>

    SmartStorTools uses the following arguments:

    • -agents -- A regular expression for agents
    • -dest -- Destination SmartStor directory
    • -help -- A help message
    • -metrics -- A regular expression for metrics
    • -silent -- Silent mode
    • -src -- Source SmartStor directory

    The special characters must be escaped using a backslash in the names of agents and metrics. The special characters are:

    ([{\^-$|]})?*+

    For example, specify the metric name SuperDomain|machineName|WebLogic|WebLogic as SuperDomain\|machineName\|WebLogic\|WebLogic.

    Warning! Use the remove and prune tools with caution because these tools delete data.

SmartStor Tool Commands

SmartStor provides the following tool commands:

  • merge
  • remove_metrics
  • remove_agents
  • keep_agents
  • prune
  • list-agents
  • test_regex
Warning! Use the prune tool with caution. Since this tool deletes data, take care that you do not accidentally delete important data.

merge

Merges a specified SmartStor source directory into a different destination directory.

Arguments:

  • <destination SmartStor directory>
  • -help Prints a help message
  • <source SmartStor directory>

Examples:

Windows:

SmartStorTools.bat merge <source directory> <destination directory>

For example:

SmartStorTools.bat merge <EM_Home>\data  C:\data

When the command executes successfully, all the SmartStor data present in the source directory is merged with the data in the destination directory.

UNIX:

SmartStorTools.sh merge <source directory> <destination directory>

remove_metrics

Removes metrics matching the given regular expression. Use this tool to clean up metrics leaks and explosions.

Warning! Use the remove_metrics tool with caution. Since this tool deletes data, take care that you do not accidentally delete important data.

Arguments:

  • -dest -- Destination SmartStor directory.
  • -help -- Prints a help message.
  • -metrics -- Regular expression for metrics to remove.
  • -src -- Source SmartStor directory.

Examples:

Windows

SmartStorTools.bat remove_metrics  -dest <destination directory> -metrics <metrics to be removed> -src <source directory>

For example:

<EM_Home>\tools\SmartStorTools.bat remove_metrics  -dest C:\SWDump\destination -metrics ".*Socket.*" -src <EM_Home>\data

When the command executes successfully, all the metrics except the socket metrics are present in the destination directory and the source directory has all the metrics intact.

UNIX

SmartStorTools.sh remove_metrics  -dest < destination directory> -metrics <metrics to be removed> -src <source directory>

remove_agents

Removes agents matching the given regular expression.

Warning! Use the remove_agent tool with caution. Since this tool deletes data, take care that you do not accidentally delete important data.

Arguments:

  • -agents -- Regular expression for agents to remove.
  • -dest -- Destination SmartStor directory.
  • -help -- Prints a help message.
  • -src -- Source SmartStor directory.

Examples:

Windows

SmartStorTools.bat remove_agents  -agents <Agent name>  -dest <destination directory>  -src <source directory>

For example:

<EM_Home>\tools>SmartStorTools.bat remove_agents  -agents  ".*WebSphere.*" -dest C:\SWDump\destination -src <EM_Home>\data

When the command executes successfully, all the metrics related to WebSphere agents are removed from the destination directory and all the metrics in the source directory remain intact.

UNIX

SmartStorTools.sh remove_agents  -agents  ".*WebSphere.*" -dest <destination directory> -src <source directory>

keep_agents

Keeps agents matching the given regular expression

Arguments:

  • -agents -- Regular expression for agents to keep
  • -dest -- Destination SmartStor directory
  • -help -- Prints a help message
  • -src -- Source SmartStor directory

Examples:

Windows

SmartStorTools.bat keep_agents  -agents  <agent(s) name>   -dest <destination directory>  -src <source directory>

For example:

<EM_Home>\tools>SmartStorTools.bat keep_agents -agents ".*Websphere.*" -dest C:\SWDump\destination\keepagents -src data

When the command executes successfully, all the metrics related to Websphere agents are present in the destination directory and the contents of source directory are intact.

UNIX

SmartStorTools.sh keep_agents -agents <Agent(s) name> -dest <destination directory> -src <source directory>

prune

Finds agents that have no historical data in SmartStor, and removes all associated metric metadata. Use this tool to clean up metrics leaks and explosions.

Warning! Use the prune tool with caution to avoid losing important data. The tool removes the metric definition and you lose access to historical data.

Arguments:

  • -dest -- directory containing pruned metadata file along with unchanged *.data files of src.
  • -help -- prints a help message
  • -silent -- silent mode
  • -src -- source SmartStor directory

Examples:

Windows

SmartStorTools.bat Prune   -dest <destination directory>  -src <source directory>

For example:

<EM_Home>\tools>SmartStorTools.bat Prune -dest C:\SWDump\destination\backup -src data

When the command executes successfully, metadata files are backed up to the dest directory with the name metrics.metadata.

UNIX

SmartStorTools.sh prune -dest <destination directory> -src <source directory>

test_regex

Prints to the Windows or UNIX console the metrics or agents matching the given regular expression.

Arguments:

  • -help -- Prints a help message
  • -src -- Source SmartStor directory
  • -agents -- Regular expression for agents to match
  • -metrics -- Regular expression for metrics to match

Examples:

Windows

SmartStorTools.bat test_regex -agents <agent(s) name>  -src <sourcedirectory>

For example:

<EM_Home>\tools>SmartStorTools.bat test_regex -agents ".*Websphere.*" -src <EM_Home>\data

When the command executes successfully, all the metrics for WebSphere agent are printed.

UNIX

SmartStorTools.sh test_regex -agents <agent(s) name> -src <source directory>

list_agents

Lists the agents matching a given regular expression.

The SmartStor tool list_agents tools list the agents matching the given regular expression.

Arguments:

  • -agents -- Regular expression for agents to match
  • -src -- Source SmartStor directory
  • -help -- Prints a help message

Examples:

Windows

SmartStorTools.bat list_agents  -agents  <agent(s) name>  -src <source directory>

For example:

<EM_Home>\tools>SmartStorTools.bat list_agents -agents ".*Websphere.*" -src <EM_Home>\data

When the command executes successfully, all the agent names matching the regular expression (in this case ".*WebSphere.*") are listed with their last reported date.

UNIX

SmartStorTools.sh list_agents -agents <Agents name> -src <source directory>

Redirect a Pruned Metric List to a Text File

You can redirect the Windows or UNIX pruned metrics list to a text file.

Follow these steps:

  1. Shut down the Enterprise Manager.
  2. From a command prompt, run the prune command and include the >> argument, for example:

    <EM_Home>\tools>SmartStorTools.bat prune -src <path to data directory> -dest D:\backup >>D:\console.txt

  3. Rename the metrics.metadata file in the <EM_Home>/data directory, for example: noprunemetrics.metadata.
  4. Run the test_regex command on the <EM_Home>/data directory.
    The Windows or UNIX console displays the command output showing all the metrics that are based on the regular expression you included in the command. Notice that the pruned metrics are no longer present.
    For example, if you were pruning many socket metrics, you run this test_regex command:
    On Windows:

    <EM_Home>\tools>SmartStorTools.bat test_regex -metrics ".*Socket.*" -src <path to data directory>

    On UNIX:

    <EM_Home>/tools>SmartStorTools.sh test_regex -metrics ".*Socket.*" -src <path to data directory>

  5. Restart the Enterprise Manager.

Trim SmartStor Data and Metadata

To trim excess SmartStor data and metadata, use the SmartStor tools.

Follow these steps:

  1. Shut down the Enterprise Manager.
  2. From a command prompt, run the test_regex command to print the current metric list.
    For example, if Introscope is installed in the <EM_Home> directory and SmartStor is configured to save data in the <EM_Home>\data directory, run this command:

    SmartStorTools.sh test_regex -metrics <agents regexp> -src ../data

    Example:

    SmartStorTools.sh test_regex -metrics ".*" -src ../data

    A list of fully qualified metric names appears. The total line count indicates the total number of metrics. However, the line count is not as accurate as the metric count you get from the supportability metrics in the Investigator.

  3. Compare line counts against specific regular expressions matching an agent or metric name. Determine what metrics occupy most of the space.
  4. From the list, pick metrics from specific agents to remove.
  5. Run the SmartStor tools remove_metrics command for the excess metrics.
    For example, if you have high SQL metric counts, use the following command:

    <EM_Home>\tools>SmartStorTools.bat remove_metrics  -dest c:\sql_metrics_removed -metrics ".*SQL.*" -src <EM_Home>\data

    When the command executes successfully, all the metrics except the SQL metrics are present in the destination directory and the source directory has all the metrics intact.

    Note: This command needs approximately as much target space to run as the space occupied by the original data (specified by the -src option). In the example, the new data without the SQL metrics (specified by the -dest option) is in the c:\sql_metrics_removed directory.

    This command is a long-running operation and can run for a few hours.

  6. Repeat the remove_metrics command on a different set of excess metrics.
    Sockets are another common source of leaked metrics. Use the following command to remove socket metrics:

    <EM_Home>\tools>SmartStorTools.bat remove_metrics  -dest c:\sql_and_sockets_metrics_removed -metrics".*Sockets.*" -src c:\sql_metrics_removed

    This command uses the data that is derived from the first command that removed SQL metrics as the source data.. The data that has both SQL and socket metrics removed is in the -dest option location: c:\sql_and_sockets__metrics_removed. This command needs almost as much extra space to run as the original data.

  7. Move any backup metadata files from the <EM_Home>\data directory to a backup directory.
    The Enterprise Manager automatically uses the largest metadata file, which is the file with the largest number of defined metrics. Reducing metrics can result in backup metadata files being larger than the current data file.

Check the Metrics Data Value

Check the metrics data value for the cleaned metrics data and metadata after you trim excess SmartStor data and metadata.

Follow these steps:

  1. Verify that the Enterprise Manager is shut down.
  2. Open the apm-events-thresholds-config.xml file in the <EM_Home>\config directory. Note the current property setting, and then set the property:

    introscope.enterprisemanager.metrics.historical.limit=500000

  3. Save and close the file.
  4. Open the IntroscopeEnterpriseManager.properties file in the <EM_Home>\config directory. Note the current setting, and then set the property:

    introscope.enterprisemanager.smartstor.directory=<destination for the metrics removal>

    For example, the example destination is c:\sql_and_sockets_metrics_removed. This property setting is:

    introscope.enterprisemanager.smartstor.directory=c:\sql_and_sockets_metrics_removed

  5. Save and close the file.
  6. Restart the Enterprise Manager.
  7. Check the value of the following metric:

    Enterprise Manager | Data Store | SmartStor | Metadata | Metrics with Data

    The value should be far below the original value. If the value is not lower, repeat the trimming steps. When the historical metric count is below 300 K, the Enterprise Manager runs more efficiently.

  8. (Optional) Reset the changed property values back to their original settings.

    Note: Shut down the Enterprise Manager before resetting the values.

Example Use the test_regex Command to Find a SmartStor Data Problem

The text_regex command produces a readable version of metrics.metadata. In this example, you use the text_regex command to discover that JMX metrics data have exploded.

Follow these steps:

  1. Run the test_regex command, for example:

    <EM_Home>\tools>SmartStorTools.bat test_regex -metrics -src <EM_Home>\data

    The result is a list of the metadata metrics.

  2. Review the metadata metrics list and determine the probable source of the metric leak or explosion.
    For example, many metrics metadata entries can exist for JMX metrics.
  3. (Optional) Run a script to sort and group the metrics. For example, to verify that JMX metrics are the source of the metric explosion, run this script:

    grep '|'.* | awk -F'|' '{print $1}' | sort | uniq -c

    On UNIX, use the full test_regex command with a sort script:

    SmartStorTools.sh test_regex -metrics <agents regexp> -src ../data | grep '|'.* | awk-F'|' '{print $1}' | sort | uniq  -c

    Example:

    SmartStorTools.sh test_regex -metrics ".*" -src ../data | grep '|'.* | awk -F'|' '{print $1}' | sort | uniq -c

  4. Review the sort results. For example, a list of metric counts and types appears:

    1054327 JMX

    1564 JSP

    1051 OlamWebApp

    889 EJB

    569 CEF

    496 Struts

    173 Edocs

    167 Servlets

    36 CPU

    29 Sockets

    In this case, SmartStor is storing over a million metrics and most are JMX metrics. This result indicates that a JMX metric explosion occurred. You can clean up the metrics using the SmartStor tools.

    Note: For more information about metric explosions, see Java Agent or Microsoft .NET Agent.

Upgrade 8.x Data to 10.x

You can use a SmartStor Tools command to upgrade SmartStor data from 8.x agents to 10.x. The tool updates the 8.x frontend and backend metric IDs with 10.x frontend and backend metric IDs. Doing this upgrade prevents duplicate metrics from showing up in the application triage map display.

You use the upgrade command after upgrading the Enterprise Manager and the agent to 10.x.

Follow these steps:

  1. Shut down the Enterprise Manager.
  2. From a command prompt, browse to the <EM_Home>/tools directory.
  3. Back up the SmartStor data directory.
  4. Run the upgrade command. Use the following syntax and arguments:

    upgrade <sourceDir> <destDir>

    • sourceDir -- path to the existing directory containing SmartStor 8.x metrics
    • destDir -- path to a new directory

      Note: The utility creates the directory. Do not create the directory in advance.

    Example:

    C:\Program Files\CA APM\Introscope<new_version(10.x)>\tools>SmartStorTools.bat upgrade C:\Program Files\CA APM\Introscope<new_version(10.x)>\data C:\Program Files\CA APM\Introscope<new_version(10.x)>\upgradeddata

  5. Open <EM_Home>/config/IntroscopeEnterpriseManager.properties in a text editor and locate the SmartStor Settings section.
  6. Set the value of introscope.enterprisemanager.smartstor.directory to the destination directory.

    Note: The default value is data. This value refers to <EM_Home>/data/.
  7. Restart the Enterprise Manager.
Note: To migrate SmartStor data from 9.x to 10.x, no additional upgrade steps are necessary.
Was this helpful?

Please log in to post comments.

  1. Sergio Morales Correa
    2016-10-25 05:39

    Hello, The selectivemetricsloadestimator.xlsx, smartstorsizing.xls APMDiskSpaceCalculator.xls have always been available from the below location: https://support.ca.com/irj/portal/anonymous/phpsupcontent?contentID=98d908b5-686f-450b-8c26-80f223940f98&productID=5974

    It seems after the recent changes in CSO the above link is no longer available.
    Below some of the affected pages:

    https://docops.ca.com/ca-apm/10-3/en/administrating/configure-enterprise-manager/configure-and-manage-smartstor-data https://docops.ca.com/ca-apm/10-3/en/installing/apm-sizing-and-performance-guide/monitored-application-requirements#MonitoredApplicationRequirements-EstimateApplicationsMetricsRequirements https://docops.ca.com/ca-apm/10-3/en/installing/apm-sizing-and-performance-guide/implement-enterprise-manager-and-cluster-sizing https://docops.ca.com/ca-apm/10-3/en/installing/apm-sizing-and-performance-guide/monitored-application-requirements

    Could you correct the issue in all the APM books please? Thanks, Sergio

  2. Sergio Morales Correa
    2016-10-25 09:20

    Hello Replace: https://support.ca.com/irj/portal/anonymous/phpsupcontent?contentID=98d908b5-686f-450b-8c26-80f223940f98&productID=5974 With new: http://www.ca.com/us/services-support/ca-support/ca-support-online/product-content/recommended-reading/product-related-technical-information/ca-application-performance-management-sizing-tools.html This change needs to be applied in all APM books Thanks, Sergio

    1. Ruth Wunderle
      2018-11-08 05:18

      Hello. It seems the link for the Smartstor calculator is again invalid (link "https://support.ca.com/us/product-content/recommended-reading/product-related-technical-information/ca-application-performance-management-sizing-tools.html"). In APM 10.7 docu it points to https://sizer.ca.com/APMSizingTool/index.html which works.

      1. Maria Wasilewska
        2018-11-15 05:05

        Thanks Ruth Wunderle,

        These changes have already been made in the source space and are awaiting the next publishing to go live.

  3. Wren Withers
    2018-11-09 11:31

    Thanks for the link URLs Sergio and Ruth. We will update the outdated links. TA1188686

    Wren