Sysdig Monitor Documentation

Alerts

Alerts form the framework for Sysdig's monitoring capabilities, notifying users when an event/issue occurs that requires their attention. The Alerts module displays a complete list of all existing alerts, as well as providing create/edit functionality for users to create their own alerts, or modify existing ones, to tailor the monitoring facility to the infrastructure's particular needs.

Note

Host comparison alerts are also configurable for one-to-many comparisons between an individual host and its associated group.

Setting up alerts involves two basic steps:

  1. Prerequisite: Configure the notification channels you want to use for alert notification.

  2. Configure alerts by defining an alert type, name, parameters, notification channel to be used, etc.

When an alert notification is sent (for example, when it is triggered, marked as resolved in the web UI, or when it ceases triggering) it contains the following information:

  • The name of the alert.

  • The type of notification: active, resolved, or OK.

  • The value of the segmentation. For example, when segmenting on host.hostName the relevant hostName will be provided.

Note

Sysdig will at times deprecate metrics that are no longer utilized. Alerts that use these metrics will not be modified or disabled, but will no longer be updated. For more information, refer to the Heuristic and Deprecated Metrics documentation.

Contents

Alert Types

There are currently five types of alerts available in Sysdig Monitor:

Alert Type

Description

Downtime

Monitor any type of entity (e.g. host, container, process, service, etc), and alert when the entity goes down.

245628934.png

Metric

Monitor time-series metrics, and alert if they violate user-defined thresholds.

245596187.png

Event

Monitor occurrences of specific events, and alert if the total number of occurrences violates a threshold. Useful for alerting on container, orchestration, and service events like restarts and deployments.

245858316.png

Anomaly Detection

Monitor hosts based on their historical behaviors, and alert when they deviate.

245661739.png

Group Outlier

Monitor a group of hosts and be notified when one acts differently from the rest.

245923870.png

Configure Alerts

Open the Alert Wizard

There are multiple ways to access the Alert wizard:

  • From the Explore table, select the Alert (bell) icon beside an entity:

    245661796.png

  • From the Explore table, click the More Options (three dots) icon for the table, and select Create a New Alert:

    245891118.png

  • From the Explore table or the Dashboards module, click the More Options (three dots) icon for a panel, and select Create Alert:

    245596253.png

  • From the Alerts module, click the Add Alert button:

    245628995.png

  • From the Alerts module, click on an existing alert, either by clicking directly on the alert, or by selecting the check box beside the alert, and clicking the Edit button:

    245661832.png
Configure an Alert

The Alert wizard provides users with the ability to create a new alert, or edit an existing alert.

Configure the notification channels you want to use before you begin, so the channels are available for assignment during alert configuration.

Note

Sysdig Monitor also allows for alerts to be created via the Sysdig API. For more information, refer to the API Alerts documentation.

To configure an alert:

  1. Select the desired Alert Type.

    Note

    When editing an existing alert, the wizard hides the ability to change the alert type

  2. Define the alert name.

  3. Optional: Add a description for the alert.

  4. Define the alert, by configuring what the alert should look for, the scope of the alert, and the boundaries for triggering the alert.

  5. Configure the notification channels the alert should use if triggered, whether a notification should be sent every 30 minutes if the issue remains unresolved, and the format of the message.

  6. Optional: Configure a Sysdig capture.

    Note

    Sysdig capture files are not available for event alerts.

  7. Click the Create or Save button to save the alert.

Manage Alerts

Alerts can be managed individually, or as a group, by using the checkboxes on the left hand side of the module interface, and the customization bar at the bottom of the interface. The columns of the table can also be configured, to provide users with the necessary data for their use cases.

Enable/Disable Alerts

Alerts can be enabled or disabled using the customization bar:

  1. From the Alerts module, check the boxes beside the relevant alert/s.

  2. Click the Enable button or the Disable button as necessary.

    Note

    The Enable /Disable buttons are only visible if a relevant alert is selected. For example, in the first image below, only the Disable button is visible, as the alert selected is currently enabled:

    244645964.png

    In the image below, both buttons are visible, as an enabled alert and a disabled alert are visible:

    244580395.png
Configure the Alerts Table Columns

To configure the visible columns:

  1. From the Alerts module, click the Table Columns Configuration (three dots) icon.

    242712581.png

  2. Check the boxes beside each desired column.

  3. Click the Apply button to save the changes, the Restore button to return the table to the original configuration, or the Cancel button to revert to the previous configuration.

Search for an Alert

The Alerts table can be searched using partial or full strings. For example, the search below displays only events that contain kubernetes:

242712671.png
Edit an Existing Alert

To edit an existing alert:

  1. Click the checkbox beside the alert:

    243007548.png

  2. Click the Edit button on the customization bar:

    242843743.png

  3. Edit the alert, and click the Save button to confirm the changes.

Copy an Alert

Alerts can be copied within the current team to allow for similar alerts to be created quickly, or copied to a different team to share alerts.

Copy an Alert to the same team

To copy an alert within the current team:

  1. Click the checkbox beside the alert to be copied.

  2. Click the Copy button on the customization bar:

    242745471.png

  3. Check that the Current Team option is selected.

  4. Rename the alert, and click the Copy and Open button to save the changes.

Copy an Alert to a Different Team

To copy an alert within the current team:

  1. Click the checkbox beside the alert to be copied.

  2. Click the Copy button on the customization bar:

    242745471.png

  3. Select the Other Team(s) option.

  4. Open the Select Team drop-down menu, and click the checkbox beside the team/s that the alert should be copied to:

    243204135.png

  5. Rename the alert, and click the Send Copy button to save the changes.

Export Alert JSON

A JSON file can be exported to a local machine, containing JSON snippets for each selected alert:

  1. Click the checkbox/es beside the relevant alert/s to be exported.

  2. Click the Export JSON button on the customization bar:

    243367967.png
Delete Alerts

To delete one or more alerts:

  1. Click the checkbox/es beside the relevant alert/s to be deleted.

  2. Click the Delete button on the customization bar:

    243138680.png

  3. Click the Yes, Delete Alert/s button to confirm the changes.

Advanced Alert Thresholds

Advanced alerts allow users to define alert thresholds as custom boolean expressions that can involve multiple conditions. These advanced alerts require specific syntax, as described in the examples below.

Format and Operations

The correct format is:

condition1 AND condition2
condition1 OR condition2
NOT condition1

The order of operations can also be altered via parenthesis:

NOT (condition1 AND (condition2 OR condition3))

Conditions take the following form:

groupAggregation(timeAggregation(metric.name)) operator value

Each condition has five parts:

  • The metric name

    Warning

    Exact metric names must be used. To avoid typos, click the HELP link to access drop-down list of available metrics. Selecting a metric from the list will automatically add the name to the threshold expression being edited.

  • Group aggregation (optional)

    Note

    If no group aggregation type is selected, the appropriate default for the metric will be applied (either sum or average).

    Note

    Group aggregation functions must be applied outside of time aggregation functions.

  • Time aggregation

  • Operator

  • Value

The table below displays supported time aggregation functions, group aggregation functions, and relational operators:

Time Aggregation Function

Group Aggregation Function

Relational Operator

timeAvg()

avg()

=

min()

min()

<

max()

max()

>

sum()

sum()

<=

>=

!=

Several example advanced alerts are shown below:

timeAvg(cpu.used.percent) > 50 AND timeAvg(memory.used.percent) > 75
timeAvg(cpu.used.percent) > 50 OR timeAvg(memory.used.percent) > 75
timeAvg(container.count) != 10  
min(min(cpu.used.percent)) <= 30 OR max(max(cpu.used.percent)) >= 60
sum(file.bytes.total) > 0 OR sum(net.bytes.total) > 0
timeAvg(cpu.used.percent) > 50 AND (timeAvg(mysql.net.connections) > 20 OR timeAvg(memory.used.percent) > 75)
In this section: