Disk Usage and Drain of Events Health Monitor Alerts

The Disk Usage health module compares disk usage on a managed device’s hard drive and malware storage pack to the limits configured for the module and alerts when usage exceeds the percentages configured for the module. This module also alerts when the system excessively deletes files in monitored disk usage categories, or when disk usage excluding those categories reaches excessive levels, based on module thresholds.

This topic describes the symptoms and troubleshooting guidelines for the Drain of Unprocessed Events health alerts generated by the Disk Usage health module.

The disk manager process manages the disk usage of a device. Each type of file monitored by the disk manager is assigned with a silo. Based on the amount of disk space available on the system the disk manager computes a High Water Mark (HWM) and a Low Water Mark (LWM) for each silo.

To display detailed disk usage information for each part of the system, including silos, LWMs, and HWMs, use the show disk-manager command.

Health Alert Format

When the Health Monitor process on the management center runs (once every 5 minutes or when a manual run is triggered), the Disk Usage module looks into the diskmanager.log file and, if the correct conditions are met, the health alert is triggered.

Additional symptoms besides the alerts can include:

• Slowness on the management center user interface

• Loss of events

Common Troubleshoot Scenarios

A Drain of unprocessed events of <SILO NAME> health alert is caused by a bottleneck in the event processing path.

There are three potential bottlenecks with respect to these Disk Usage alerts:

• Excessive logging ― The EventHandler process on threat defense is oversubscribed (it reads slower than what Snort writes).

• Sftunnel bottleneck ― The Eventing interface is unstable or oversubscribed.

• SFDataCorrelator bottleneck ― The data transmission channel between the management center and the managed device is oversubscribed.

Excessive Logging

One of the most common causes for the health alerts of this type is excessive input. The difference between the Low Water Mark (LWM) and High Water Mark (HWM) gathered from the show disk-manager command shows how much space there is available to take on that silo to go from LWM (freshly drained) to the HWM value. If there are drain of unprocessed events, review the logging configuration.

• Check for double logging ― Double logging scenarios can be identified if you look at the correlator perfstats on the management center:

  admin@FMC:~$ sudo perfstats -Cq < /var/sf/rna/correlator-stats/now

• Check logging settings for the ACP ― Review the logging settings of the Access Control Policy (ACP). If the logging setting includes both "Beginning" and "End" of connection, modify the setting to log only the end to reduce the number of events.

Communications Bottleneck ― Sftunnel

Sftunnel is responsible for encrypted communications between the management center and the managed device. Events are sent over the tunnel to the management center. Connectivity issues and/or instability in the communication channel (sftunnel) between the managed device and the management center can be due to:

• Sftunnel is down or is unstable (flaps).

  Ensure that the management center and the managed device have reachability between their management interfaces on TCP port 8305.

  The sftunnel process should be stable and should not restart unexpectedly. Verify this by checking the /var/log/message file and search for messages that contain the sftunneld string.

• Sftunnel is oversubscribed.

  Review trend data from the Heath Monitor and look for signs of oversubscription of the management center's management interface, which can be a spike in management traffic or a constant oversubscription.

  Use as a secondary management interface for eventing. To use this interface, you must configure its IP address and other parameters at the threat defense CLI using the configure network management-interface command.

Communications Bottleneck ― SFDataCorrelator

The SFDataCorrelator manages data transmission between the management center and the managed device; on the management center, it analyzes binary files created by the system to generate events, connection data, and network maps. The first step is to review the diskmanager.log file for important information to be gathered, such as:

• The frequency of the drain.

• The number of files with Unprocessed Events drained.

• The occurrence of the drain with Unprocessed Events.

Each time the disk manager process runs it generates an entry for each of the different silos on its own log file, which is located under [/ngfw]/var/log/diskmanager.log. Information gathered from the diskmanager.log (in CSV format) can be used to help narrow the search for a cause.

Additional troubleshooting steps:

• The command stats_unified.pl can help you to determine if the managed device does have some data which must be sent to management center. This condition can happen when the managed device and the management center experience a connectivity issue. The managed device stores the log data on to a hard drive.

  admin@FMC:~$ sudo stats_unified.pl

• The manage_proc.pl command can reconfigure the correlator on the management center side.

  root@FMC:~# manage_procs.pl

Before You Contact Cisco Technical Assistance Center (TAC)

It is highly recommended to collect these items before you contact Cisco TAC:

• Screenshots of the health alert seen.

• Troubleshoot file generated from the management center.

• Troubleshoot file generated from the affected managed device.

  Date and Time when the problem was first seen.

• Information about any recent changes done to the policies (if applicable).

  The output of the stats_unified.pl command as described in the Communications Bottleneck ― SFDataCorrelator.