Skip to content
CA Application Performance Management - 10.0
Documentation powered by DocOps

CA APM Data Storage Requirements

Last update July 23, 2018

Each Enterprise Manager Requires SmartStor on a Dedicated Disk or I/O Subsystem

The following requirement is critical for optimal CA APM capacity.

Point the SmartStor location to a dedicated disk or disk-array. Do not share the SmartStor disk with any of these other data stores:

  • Transaction Events database (traces.db)
  • Differential Analysis database (variance.db)

You can choose the SmartStor location at install time or by using the introscope.enterprisemanager.smartstor.directory property in the IntroscopeEnterpriseManager.properties file.

In addition, set the introscope.enterprisemanager.smartstor.dedicatedcontroller property in the IntroscopeEnterpriseManager.properties file to true.

Enterprise Manager Internal Database Settings and Capacity

Background information and specifics can help you understand sizing and performance-related recommendations, settings, and limits for your SmartStor, Transaction Events, and variance databases. You can use the Enterprise Manager-related settings and capacity limits to set up, maintain, and configure your CA APM environment.

Disk Space Requirements for Enterprise Manager Internal Databases

To answer the question "How much disk space do I need for all my Introscope databases?", calculate your disk space needs for the three databases in which Introscope stores data: SmartStor, traces.db, and variance.db.

  • SmartStor is used to store metric data coming from agents.

  • traces.db contains all Transaction Traces and events data, such as error snapshots.
    This database spans multiple files. One file is created per day and this data is kept for the number of days that are specified in the IntroscopeEnterpriseManager.properties file. In this example setting, the daily file is stored for 14 days.

    introscope.enterprisemanager.transactionevents.storage.max.data.age=14
    
  • variance.db stores Differential Analysis data for configured metrics in a single file.

The traces.db and variance.db databases collect and maintain data at different rates. To determine the database disk space needs for your Enterprise Manager, perform disk space calculations for traces.db and variance.db separately. Then sum the two calculations.

traces.db Disk Space Calculation Example

Estimating the disk space needs for your Introscope traces.db file starts by answering these questions:

  • "How many events do I want to keep?"
  • "How many days do I want to keep these events?".

After you have answered these questions, determining the needed disk space involves the mathematical calculations that are shown in this example.

By substituting the Total days to store data and Events/day values with the values you have determined for your system, you can estimate your Enterprise Manager traces.db disk space requirements.

Agent average bytes/event = 4096

Total days to store data = 36.

Events/day = 1000 events/minute x 60 min/hr x 24 hrs/day = 1,440,000

Note: This number is the maximum load number for events/day based on 1,000 events/minute, which is the maximum recommended load for a Windows machine. The apm-events-thresholds-config.xml file contains the introscope.enterprisemanager.events.limit clamp, which limits the number of events an Enterprise Manager can process to 1,250 (by default) per 15-second interval.

Bytes required/day = 4096 (bytes/event) x 1,440,000 events/day = 5,898,240,000

GB required/day = 5,898,240,000 bytes required/day/(1024 x 1024 x 1024) =5.49 GB

Total disk space required = 36 (total days to store data) x 5.49 GB required/day = 198 GB.

variance.db Disk Space Calculation Example

The variance.db file should rarely exceed 400 MB. The equation for estimating the variance.db disk space requirement uses the following inputs:

  • Number of metrics being baselined because of the enabled configuration (Differential Controls)

Here are the equation variables:

M = number of unique metrics being baselined, live and historical combined

D = required disk space (bytes)

You can estimate the variance.db disk space requirement using this equation:

D = 20000 * M

Here is an example:

Metrics = 5 K unique over time (both live and historical combined)

The variance.db file size = 20000 *  5000 = 100,000,000 bytes = 95 MB.

You can see the current number of baselined metrics in each collector from the following supportability metric custom metric agent.

Enterprise |Baseline Engine: Total Metrics Baseline (count)

Important! The variance database is not pruned regularly. A configuration with differential controls matching many metrics (more than the limit configured in collectors, which is 20 K by default), can result in the following behavior. At runtime (live), differential analysis works, and you are not able to monitor more than 20 K at any point. But because the matched metrics are higher, the active 20 K may change over time resulting in more entries in variance.db. If you see this behavior, revisit the configuration and reduce the values to have realistic matched metrics.

Note: These numbers are only examples; they are not provided as recommendations for any or all Introscope environments.

Set the SmartStor Dedicated Controller Property

A dedicated controller property tells the Collector that there is a dedicated SmartStor disk. You can see the property in the IntroscopeEnterpriseManager.properties file as shown:

introscope.enterprisemanager.smartstor.dedicatedcontroller=true

Taking both of the following actions allows the Enterprise Manager to make full use of concurrency optimizations that are unsafe when the SmartStor disk is shared:

  • Providing a separate disk for each SmartStor.
  • Setting the dedicated controller property to true.

The dedicated controller property is set to false by default. You must provide a dedicated disk I/O path for SmartStor to set this property to true; it cannot be set to true when there is only a single disk for each Collector.

When the dedicated controller property is set to false, the metric capacity can decrease up to 50 percent.

Note: For instructions about how to set the SmartStor dedicated controller property to true, see Implement Enterprise Manager and Cluster Sizing.

All these restrictions apply to all the varied storage choices available (local disks, external storage solutions such as SAN). The SmartStor requirement for a separate disk or controller does not mean that a separate host adapter is required. For example, a separate fiber channel adapter or SCSI adapter.

I mportant! Multiple partitions on the same drive share a controller, which is not an appropriate environment for a SmartStor instance.

Plan for SmartStor Storage Using SAN

The most important consideration for SmartStor disk storage is that the I/O path is free of contention, and that performance is fast and consistent. In a SAN storage environment, the optimal configuration maps each SmartStor to a unique logical unit number (LUN) that represents a dedicated physical disk. With this configuration, it is safe to set introscope.enterprisemanager.smartstor.dedicatedcontroller=true.

If you have configured two or more LUNs to represent partitions or subsets of the same physical disk, this configuration does not meet the requirements for a SmartStor dedicated disk.

SmartStor on Network File Systems

Do not use NFS for SmartStor storage or replace SAN with NFS for SmartStor storage. NFS is optimized for file-sharing, not throughput. NFS is unacceptable for SmartStor because NFS automatically defers writes to make most efficient use of the I/O capacity. This deferred action causes major synchronization problems.

Plan for SmartStor Storage Using SAS Controllers

One SAS controller can be used for the Enterprise Managers that store both SmartStor, and the traces.db and variance.db data. Having a dedicated disk for SmartStor is important; in this case meaning that SmartStor has its own dedicated SAS port.

SmartStor I/O Disk Usage

CA APM writes metrics to the SmartStor database every 15 seconds. In most monitoring environments, sequential data write operations are the dominant SmartStor activity. Write operation efficiency is a major determinant of Enterprise Manager metric capacity. Use this table to estimate your SmartStor disk I/O requirements that are based on your anticipated or actual metrics load.

Metric workload Average disk writes/second Peak disk writes/ second
100,000 50 100
200,000 100 187
300,000 165 210
500,000 250 350
700,000 400 560

SmartStor and Flat File Archiving

The flat file archiving is an alternate format that can be used for metric data storage instead of SmartStor. Unlike SmartStor, the flat file format writes the data in readable ascii format, which is considerably more expensive than the SmartStor format.

CA Technologies has the following recommendations about SmartStor and flat file archiving:

  • Avoid flat file archive compression, unless minimal disk space is a dominant consideration.
    The Enterprise Manager requires noticeable CPU resources when the flat file archiving compression feature is enabled. The high CPU usage can adversely affect Enterprise Manager performance when the compression feature periodically runs. If you must use flat file archiving, archive the smallest possible number of metrics.
  • Do not use flat file archiving in production.
    Readable metric values are most useful in a QA debug environment.
  • Do not locate SmartStor on the same disk as a flat file archive.
    Locate SmartStor on its own dedicated disk and set the SmartStor dedicated controller property.

APM Database Storage Requirements and Recommendations

Background information and specifics can help you understand the sizing and performance-related recommendations, settings, and limits for your APM database.

APM Database Disk Requirements

For best performance, follow standard database server disk configuration strategies. Use separate disks for data, indexes, and logs.

APM Database Server Location

If you are running Introscope-only, you must install the APM database, which is required for each standalone Enterprise Manager. In an environment with multiple standalone Enterprise Managers, each must have its own APM database. For Introscope-only the APM database activity and storage needs are minor because the APM database primarily stores application triage map data. 

If you are running CA CEM, the APM database activity and storage needs are much more significant. In this case, CA Technologies recommends that you run the APM database on a separate server from any Enterprise Managers.

Tip: If you expect an Enterprise Manager to run close to its capacity, avoid resource contention by locating the APM database on a separate server.

Multiple APM Databases on One Computer

In some environments, it can be advantageous to optimize hardware utilization by installing multiple APM database instances in the same database server. You can use the Enterprise Manager installer to create multiple schemas (for example, apmdb1, apmdb2, apmdb3) that share PostgreSQL port 5432. Be sure that each schema or database instance has a unique name.

If you run multiple instances of the APM database on one computer, the memory, number of CPU cores, and disk space requirements for each database are not reduced. Provide resources adequate for the sum of the requirements for each database instance. For example, use separate disks for transaction logs and data for each database instance.

If you deploy full CA APM with TIM monitoring, CA Technologies does not recommend multiple APM database instances in the same database server. However, if your deployment requires a shared APM database make sure that the resource allocation follows the guidelines just stated.

APM Database Connection Pool Settings

For broadest platform compatibility, the APM database is configured by default to use a maximum of 125 database connections.

The connection pool settings for Introscope and CA CEM are configured in separate files, both of which are located in the <EM_Home>/config directory. You configure the following files:

  • APMEnterpriseManager.properties file, which defines the APM database connection pool settings for the application triage map.
    The default values are set to be sufficient in all cases.
  • tess-default.properties file, which defines the APM database connection pool settings for individual Enterprise Manager services. Values for the c3p0 properties in this file are used to initialize connection pools during Enterprise Manager start up and if Enterprise Manager services are reassigned from one Enterprise Manager to another.
Important! If you must configure any of the default TESS properties, create your own version of the tess-default.properties file; name it tess-customer.properties. The Enterprise Manager knows to load files with this name. The tess-default.properties file is overwritten during future upgrades. The tess-customer.properties file is not overwritten during future upgrades. For these reasons, instructions in this guide refer to the tess-customer.properties file.

The Enterprise Manager Services provide supportability metrics describing allocation and usage of the connection pools. These APM database connection pool supportability metrics can be used to verify correct configuration of the connection pools. The supportability metrics can also help detect certain capacity issues.

Standalone Enterprise Manager APM Database Connection Pool Settings

The default Enterprise Manager services connection pool settings that are installed with each Enterprise Manager have a minimum connection pool size of 20 connections and a maximum of 44 connections. These defaults are the connection pool settings to run all the Enterprise Manager services within a single Enterprise Manager.

For a standalone Enterprise Manager handling an agent metrics and applications load only (no data from TIMs), the Enterprise Manager services and associated APM database connections, are not used.

Collector or MOM APM Database Connection Pool Settings

The role a Collector or MOM plays in the cluster, with respect to the placement of Enterprise Manager services, determines the number of database connections that a Collector or MOM requires. By default, all Enterprise Manager services are assigned to the first Collector that connects to the APM database. After all the Collectors and the MOM in a cluster are installed and the MOM is running, the CA CEM administrator can reassign Enterprise Manager services.

Since a Collector role is not known at install time, all Collectors are configured, by default, with database connection pool settings sufficient to support a CEM console and a standalone Enterprise Manager.

In general, the out-of-the box default property values, which are specified in the <EM_Home>/config/tess-default.properties file, are sufficient for most deployments. If necessary to conform to your site-specific needs or policies, you can configure the values using the tess-customer.properties file.

After the initial connection pool configuration, CA APM automatically readjusts each Enterprise Manager APM database connection pool settings if an Enterprise Manager service is moved from one Collector to another in a cluster.

APM Database Sizing for an Introscope-only Workload

When your deployment handles only Introscope agent data (no CA CEM workload), APM disk space requirements are modest.

CA Technologies recommends the following PostgreSQL APM database sizing when your deployment is running Introscope only:

  • At least 1 GB disk space
  • 200 MB memory

APM Database Sizing for CA CEM Workloads

You can determine the APM database disk space requirements for handling data that TIMs report based on these factors:

  • Defect rate
  • Data included in defects
  • Defect data retention settings
  • Amount of user and user group transaction traffic
  • Statistics retention settings

The number of defects that are defined and the frequency of defined thresholds being crossed determines the defect rate. Defects are defined based on the business requirements at your organization. Defects are stored in the APM database. A reported defect containing no response body information requires approximately 650 bytes of disk space.

You can see the defect rate by viewing the Defects Processor:Processed metric here in the metric browser tree:

Note: In this path, machine and port refer to the Collector hosting the TIM Collection service.
*SuperDomain*|Custom Metric Host (Virtual)|Custom Metric Process (Virtual)|
Custom Metric Agent (Virtual)(machine@port)|Enterprise Manager|CEM|Processors|
Defects Processor:Processed

Defects can be configured to include response body information up to 10,000 bytes in size. Response body information is named defect metadata and is also stored in the APM database.

Note: For information about configuring defects to include response body information, see the Configure Enterprise Manager.

PostgreSQL compresses large string data objects such as defect metadata; it is difficult to predict the resulting compression ratio for general usage. However, assume that each defect with metadata requires about 4 KB of APM database disk space.

To get information about the number of defects that have been processed containing metadata, query the ts_defect_meta_values_<date> APM database table in the APM database.

The defect retention settings specify the length of time defect data is retained in the APM database. The period is configurable from 1 to 30 days. The business requirements of your organization typically determine these settings, but it is also important that you consider the impact on database disk space requirements.

Note: For information about defect data retention settings, see Define the CA CEM Domain.

Statistics information is aggregated at various granularities (interval, daily, weekly, monthly). Therefore, APM database disk space requirements do not directly correlate with the number of statistics records processed. In general, each statistics record requires about 3 K of APM database disk space. Statistics records are processed once per hour. To obtain the statistics numbers that are processed run a historical query against the Stats Processor:Processed metric, which you can see here in the metric browser tree:

*SuperDomain*|Custom Metric Host (Virtual)|Custom Metric Process (Virtual)|
Custom Metric Agent (Virtual)(machine@port)|Enterprise Manager|CEM|Processors|
Stats Processor:Processed

As with defect data, consider the statistics retention periods your organization has determined when calculating APM database disk space requirements.

APM Database Sizing for a Small CA CEM Workload

When your deployment is running less than half of the CA CEM-only reference workload, CA Technologies recommends the following PostgreSQL APM database sizing:

  • 50 GB disk space
  • 1 GB memory
  • 2 CPU cores
  • Separate disk for the APM database log

APM Database Sizing for a Medium to Large CA CEM Workload

When your deployment is running a workload similar to the CA CEM-only reference workload, CA Technologies recommends the following PostgreSQL APM database sizing:

  • 8 CPU cores
  • 32 GB RAM
  • Separate disks for the APM database logs and CA APM data
  • Hard disk primary partition: 100 GB
  • Hard disk CA APM data partition: 300 GB
  • Log data partition: 100 GB
    Provide a data disk that is a RAID array or SAN capable of 500+ disk writes/second at peak usage.

Determine APM Database Disk Space Requirements

The APM database stores two kinds of data:

  • Introscope business service and business transaction data, which is used in the Investigator application triage map.
  • CA CEM data about configuration, recording, logins, defects, and statistics.

These factors affect the volume of data in the APM database:

  • The statistics and defect retention periods.
  • The number of defects that are defined and their thresholds.
  • The volume of monitored traffic
  • Transaction definitions
  • User and user group definitions

CA CEM defects are generated when configured thresholds are exceeded. Web based application traffic typically follows a pattern of heavy usage periods that are separated by relatively light usage periods. Thresholds are likely to be exceeded more often during heavy usage periods. Periods of relatively high defect traffic are named defect storms. The defect rate between defect storms is named the steady-state defect rate.

The APM disk space calculator uses CA CEM transaction definition and expected traffic information to estimate the disk space required.

Follow these steps:

  1. Input the values for the CA CEM and traffic factors that affect the size and quantity of defect and statistics data.
    • Input information describing the measured or expected defects rate.
    • Input statistics data using one of these methods:
      • Measured or expected Statistics Received value.
      • Configured values for Users/User Groups, Business Transactions, and Business Services.
        The APM database disk-space projection that is based on this method provides the worst case scenario for the statistics gathered using the specified configuration.
      If you input data using both methods, the calculator uses the Statistics Received value for the projection.
    The resulting projected PostgreSQL and Oracle disk space appears in gigabytes.
  2. Estimate the calculator input data for your deployment by using the guidance in the following table.
CA CEM or traffic input factor How to estimate calculator input data
Defects Data Inputs  
Steady-state(# per minute) Analyze the CEM.Processors.Defects Processor.Processed supportability metric over time to calculate the average steady state defect arrival rate. Find the metric in these locations:
The metric browser tree
<EM_Home>/logs/tessperflog.txt file on the Enterprise Manager running the TIM Collection service
Storm(# per minute) Analyze the CEM.Processors.Defects Processor.Processed supportability metric over time to calculate the average peak (storm) defect arrival rate. Find the metric in these locations:
The metric browser tree
<EM_Home>/logs/tessperflog.txt file on the Enterprise Manager running the TIM Collection service
Storm Duration(in minutes) Analyze the CEM.Processors.Defects Processor.Processed supportability metric over time to calculate the average duration of defect storms. Find the metric in these locations:
The metric browser tree
<EM_Home>/logs/tessperflog.txt file on the Enterprise Manager running the TIM Collection service
Time between Storms(in minutes) Analyze the CEM.Processors.Defects Processor.Processed supportability metric over time to calculate the average time period between defect storms. Find the metric in these locations:
The metric browser tree
<EM_Home>/logs/tessperflog.txt file on the Enterprise Manager running the TIM Collection service
Include Metadata? Choose Yes or No from the picklist based on whether most of the defects are configured to include metadata such as response body information.
Note: This factor greatly impacts APM database capacity.
Retention Period(days) In the CEM console, select Setup > Incident Settings > Delete Defects after.
The valid values are 1 through 30 days.
Note: If the retention period value is 0, the disk space projection is 0 GB. The APM database requires disk space only when you retain data.
Statistics Received(per hour) Analyze the CEM|Processors|.Stats Processor.Processed supportability metric over time to calculate the average statistics arrival rate. Find the metric in these locations:
The metric browser tree
<EM_Home>/logs/tessperflog.txt file on the Enterprise Manager running the TIM Collection service
# of Users/User Groups In the CEM console, select Administration > User Groups.
Input the number of users if you use Enterprise Mode.
Input the number of user groups if you use eCommerce mode.
# of Business Transactions In the CEM console, go to the Business Transactions page at this URL, where <MOM_IP> is the IP address of the MOM computer:
http://<MOM_IP>:8081/wily/cem/tess/app/admin/tranSetDefSearch.html?pPropertyName=businessValue&pFocusId=restoreFocus%28%27search%27%29
Count the number of business transactions.
The number of business transactions defined should be greater than zero and reflect the monitoring configuration you set up.
# of Business Services In the CEM console, select Administration > Business Services.
Hourly Stats Retention In the CEM console, select Setup > Domain > Data Retention Settings.
Daily Stats Retention In the CEM console, select Setup > Domain > Data Retention Settings.
Weekly Stats Retention In the CEM console, select Setup > Domain > Data Retention Settings.
Was this helpful?

Please log in to post comments.

  1. Guenter Grossberger
    2015-09-03 02:19

    baselines.db is variance.db in APM 10

  2. Billy Cole
    2016-03-04 08:57

    Within APM 10 with Team Center and also pre-10 with Triage Map, the application map or vertices are stored within the APM DB. This adds a very noticeable increase in Postgres memory needs. The estimates on the amount of RAM the postgres instance requires needs to be adjusted.

    Also might want to add that if not using CEM, the contents of the APM DB are rebuilt over time and if upgrading without CEM, you do not need to import your previous exported data.

    1. Lucie Stehnova
      2016-06-29 08:00

      Dear Billy, Apologies for not getting back earlier. Thank you very much for your comment, we will address your suggestions. Lucy