Sysdig Monitor Documentation

Sysdig is able to monitor node.js applications by linking a library to the node.js code, which then creates a server in the code to export the StatsD metrics.

The example below shows a node.js application that exports metrics using the Prometheus protocol:

{
          "name": "node-example",
          "version": "1.0.0",
          "description": "Node example exporting metrics via Prometheus",
          "main": "index.js",
          "scripts": {
            "test": "echo \"Error: no test specified\" && exit 1"
          },
          "license": "BSD-2-Clause",
          "dependencies": {
            "express": "^4.14.0",
            "gc-stats": "^1.0.0",
            "prom-client": "^6.3.0",
            "prometheus-gc-stats": "^0.3.1"
          }
}

The index.js library function is shown below:

        // Use express as HTTP middleware
        // Feel free to use your own
        var express = require('express')
                var app = express()

        // Initialize Prometheus exporter
                const prom = require('prom-client')
                const prom_gc = require('prometheus-gc-stats')
                prom_gc()

        // Sample HTTP route
                app.get('/', function (req, res) {
                res.send('Hello World!')
                })

        // Export Prometheus metrics from /metrics endpoint
                app.get('/metrics', function(req, res) {
                res.end(prom.register.metrics());
                });

                app.listen(3000, function () {
                console.log('Example app listening on port 3000!')
                })

To integrate an application:

Once the Sysdig agent is deployed, node.js metrics will be automatically retrieved. The image below shows an example of key node.js metrics visible on the Sysdig Monitor UI:

Apache web server is an open-source, web server creation, deployment, and management software. If Apache is installed on your environment, the Sysdig agent will connect using the mod_status module on Apache. You may need to edit the default entries in the agent configuration file to connect. See Default Configuration.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Install mod_status on your Apache servers and enable ExtendedStatus.

The following configuration is required. If it is already present, then un-comment the lines, otherwise add the configuration.

LoadModule status_module modules/mod_status.so
...

<Location /server-status>
    SetHandler server-status
    Order Deny,Allow
    Deny from all
    Allow from localhost
</Location>
...

ExtendedStatus On

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

Apache has a common default for exposing metrics. The process command name can be either apache2 or httpd. By default, the Sysdig agent will look for the process apache2. If named differently in your environment (e.g. httpd), edit the configuration file to match the process name as shown in Example 1.

By default, Sysdig's dragent.default.yaml uses the following code to connect with Apache and collect all metrics.

app_checks:
  - name: apache
    check_module: apache
    pattern: 
      comm: apache2
    conf:
      apache_status_url: "http://localhost:{port}/server-status?auto"
    log_errors: false 

If it is necessary to edit dragent.yaml to change the process name, use the following example and update the comm with the value httpd.

app_checks:
  - name: apache
    check_module: apache
    pattern:
      comm: httpd
    conf:
      apache_status_url: "http://localhost/server-status?auto"
    log_errors: false

The Apache metrics are listed in the metrics dictionary here: Apache Metrics.

Apache Kafka is a distributed streaming platform. Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies. If Kafka is installed on your environment, the Sysdig agent will automatically connect. See Default Configuration.

The Sysdig agent automatically collects metrics from Kafka via JMX polling. You need to provide consumer names and topics in the agent config file to collect consumer-based Kafka metrics.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Kafka will automatically expose all metrics. You do not need to add anything on the Kafka instance.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

Metrics from Kafka via JMX polling are already configured in the agent's default-settings configuration file. Metrics for consumers, however, need to use app-checks to poll the Kafka and Zookeeper API. You need to provide consumer names and topics in dragent.yaml file.

Since consumer names and topics are environment-specific, a default configuration is not present in dragent.default.yaml.

Refer to the following examples for adding Kafka checks to dragent.yaml.

A basic example with sample consumer and topic names:

app_checks:
  - name: kafka
    check_module: kafka_consumer
    pattern:
      comm: java
      arg: kafka.Kafka
    conf:
      kafka_connect_str: "127.0.0.1:9092" # kafka address, usually localhost as we run the check on the same instance
      zk_connect_str: "localhost:2181" # zookeeper address, may be different than localhost
      zk_prefix: /
    consumer_groups:
      sample-consumer-1: # sample consumer name
        sample-topic-1: [0, ] # sample topic name and partitions
      sample-consumer-2: # sample consumer name
        sample-topic-2: [0, 1, 2, 3] # sample topic name and partitions

From Kafka 9 onwards, you can store consumer group config info inside Kafka itself for better performance.

app_checks:
  - name: kafka
    check_module: kafka_consumer
    pattern: 
      comm: java
      arg: kafka.Kafka
    conf: 
      kafka_connect_str: "localhost:9092"
      zk_connect_str: "localhost:2181"
      zk_prefix: /
      kafka_consumer_offsets: true
      consumer_groups: 
        sample-consumer-1: # sample consumer name
          sample-topic-1: [0, ] # sample topic name and partitions

To enable aggregation of partitions at the topic level, use kafka_consumer_topics with aggregate_partitions = true.

In this case the app check will aggregate the lag & offset values at the partition level, reducing the number of metrics collected.

Set aggregate_partitions = false to disable metrics aggregation at the partition level. In this case, the appcheck will show lag and offset values for each partition.

app_checks:
  - name: kafka
    check_module: kafka_consumer
    pattern: 
      comm: java
      arg: kafka.Kafka
    conf: 
      kafka_connect_str: "localhost:9092"
      zk_connect_str: "localhost:2181"
      zk_prefix: /
      kafka_consumer_offsets: true
      kafka_consumer_topics:
        aggregate_partitions: true
      consumer_groups:
        sample-consumer-1: # sample consumer name
          sample-topic-1: [0, ] # sample topic name and partitions
        sample-consumer-2: # sample consumer name
          sample-topic-2: [0, 1, 2, 3] # sample topic name and partitions

Optional tags can be applied to every emitted metric, service check, and/or event.

app_checks:
  - name: kafka
    check_module: kafka_consumer
    pattern: 
      comm: java
      arg: kafka.Kafka
    conf: 
      kafka_connect_str: "localhost:9092"
      zk_connect_str: "localhost:2181"
      zk_prefix: /
      consumer_groups:
        sample-consumer-1: # sample consumer name
          sample-topic-1: [0, ] # sample topic name and partitions
    tags:  ["key_first_tag:value_1", "key_second_tag:value_2", "key_third_tag:value_3"]

See Apache Kafka Consumer Metrics.

See Apache Kafka JMX Metrics.

Consul is a distributed service mesh to connect, secure, and configure services across any runtime platform and public or private cloud. If Consul is installed on your environment, the Sysdig agent will automatically connect and collect basic metrics. If the Consul Access Control List (ACL) is configured, you may need to edit the default entries to connect. Also, additional latency metrics can be collected by modifying default entries. See the Sysdig Agent Configuration section.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Consul is ready to expose metrics without any special configuration.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's dragent.default.yaml uses the following code to connect with Consul and collect basic metrics.

app_checks:
  - name: consul
    pattern:
      comm: consul
    conf:
      url: "http://localhost:8500"
      catalog_checks: yes

With the dragent.default.yaml file, the below set of metrics are available in the Sysdig Monitor UI:

Additional metrics and event can be collected by adding configuration in dragent.yaml file. The ACL token must be provided if enabled. See the following examples.

self_leader_check An enabled node will watch for itself to become the leader and will emit an event when that happens. It can be enabled on all nodes.

app_checks:
  - name: consul
    pattern:
      comm: consul
    conf:
      url: "http://localhost:8500"
      catalog_checks: yes
      self_leader_check: yes
    logs_enabled: true

If the network_latency_checks flag is enabled, then the Consul network coordinates will be retrieved and the latency calculated for each node and between data centers.

app_checks:
  - name: consul
    pattern:
      comm: consul
    conf:
      url: "http://localhost:8500"
      catalog_checks: yes
      network_latency_checks: yes
    logs_enabled: true

With the above changes, you can see the following additional metrics:

When the ACL System is enabled in Consul, the ACL Agent Token must be added in dragent.yaml in order to collect metrics.

Follow Consul's official documentation to Configure ACL, Bootstrap ACL and Create Agent Token.

app_checks:
  - name: consul
    pattern:
      comm: consul
    conf:
      url: "http://localhost:8500"
      acl_token: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" #Add agent token
      catalog_checks: yes
      logs_enabled: true

In addition to the metrics from the Sysdig app-check, there are many other metrics that Consul can send using StatsD. Those metrics will be automatically collected by the Sysdig agent's StatsD integration if Consul is configured to send them.

Add statsd_address under telemetry to the Consul config file. The default config file location is /consul/config/local.json

{
...
  "telemetry": {
     "statsd_address": "127.0.0.1:8125"
  }
...
}

See Telemetry Metrics for more details.

See Consul Metrics.

Couchbase Server is a distributed, open-source, NoSQL database engine. The core architecture is designed to simplify building modern applications with a flexible data model and simpler high availability, high scalability, high performance, and advanced security. If Couchbase is installed on your environment, the Sysdig agent will automatically connect. If authentication is configured, you may need to edit the default entries to connect. See Default Configuration.

The Sysdig agent automatically collects all bucket and node metrics. You can also edit the configuration to collect query metrics.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Couchbase will automatically expose all metrics. You do not need to configure anything on the Couchbase instance.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's dragent.default.yaml uses the following code to connect with Couchbase and collect all bucket and node metrics.

app_checks:
  - name: couchbase
    pattern:
      comm: beam.smp
      arg: couchbase
      port: 8091
    conf:
      server: http://localhost:8091

If authentication is enabled, you need to edit dragent.yaml file to connect with Couchbase. See Example 1.

Replace <username> and <password> with appropriate values and update the dragent.yaml file.

app_checks:
  - name: couchbase
    pattern:
      comm: beam.smp
      arg: couchbase
      port: 8091
    conf:
      server: http://localhost:8091
      user: <username>
      password: <password>
      # The following block is optional and required only if the 'path' and
      # 'port' need to be set to non-default values specified here
      cbstats:
        port: 11210
        path: /opt/couchbase/bin/cbstats

Additionally, you can configure query_monitoring_url to get query monitoring stats. This is available from Couchbase version 4.5. See Query Monitoring for more detail.

app_checks:
  - name: couchbase
    pattern:
      comm: beam.smp
      arg: couchbase
      port: 8091
    conf:
      server: http://localhost:8091
      query_monitoring_url: http://localhost:8093

See Couchbase Metrics.

Elasticsearch is an open-source, distributed, document storage and search engine that stores and retrieves data structures in near real-time.Elasticsearch represents data in the form of structured JSON documents, and makes full-text search accessible via RESTful API and web clients for languages like PHP, Python, and Ruby. It’s also elastic in the sense that it’s easy to scale horizontally—simply add more nodes to distribute the load. If Elasticsearch is installed on your environment, the Sysdig agent willautomatically connect in most of the cases. See Default Configuration.

The Sysdig Agent automatically collects default metrics. You can also edit the configuration to collect Primary Shard stats.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Elasticsearch is ready to expose metrics without any special configuration.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's dragent.default.yaml uses the following code to connect with Elasticsearch and collect basic metrics.

app_checks:
  - name: elasticsearch
    check_module: elastic
    pattern:
      port: 9200
      comm: java
    conf:
      url: http://localhost:9200

For more metrics, you may need to change the elasticsearch default setting in dragent.yaml:

app_checks:
  - name: elasticsearch
    check_module: elastic
    pattern:
      port: 9200
      comm: java
    conf:
      url: http://localhost:9200
      pshard_stats : true

Enable pshard_stats to monitor the following additional metrics:

With the default settings and the pshard setting, the total available metrics are listed here: Elasticsearch Metrics.

etcdis a distributed key-value store that provides a reliable way to store data across a cluster of machines. If etcd is installed on your environment, the Sysdig agent will automatically connect. If you are using ectd older than version 2, you may need to edit the default entries to connect. See Default Configuration.

The Sysdig Agent automatically collects all metrics.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

etcd will automatically expose all metrics. You do not need to add anything to the etcd instance.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

The default agent configuration for etcd will look for the application on localhost, port 2379. No customization is required.

By default, Sysdig's dragent.default.yaml uses the following code to connect with etcd and collect all metrics.

app_checks:
  - name: etcd
    pattern:
      comm: etcd
    conf:
      url: "http://localhost:2379" 

etcd (before version 2) does not listen on localhost, so the Sysdig agent will not connect to it automatically. In such case, you may need edit the dragent.yaml file with the hostname and port. See Example 1.

Alternatively, you can add the option -bind-addr 0.0.0.0:4001 to the etcd command line to allow the agent to connect.

You can use {hostname} and {port} as a tokens in the conf: section. This is the recommended setting for Kubernetes customers.

app_checks:
  - name: etcd 
    pattern: 
      comm: etcd 
    conf: 
      url: "http://{hostname}:{port}"

Alternatively you can specify the real hostname and port.

app_checks:
  - name: etcd 
    pattern: 
      comm: etcd 
    conf: 
      url: "http://my_hostname:4000"  #etcd service listening on port 4000

If encryption is used, add the appropriate SSL/TLS entries. Provide correct path of SSL/TLS key and certificates used in etcd configuration in fields ssl_keyfile, ssl_certfile, ssl_ca_certs.

app_checks:
  - name: etcd
    pattern:
      comm: etcd
    conf:
      url: "https://localhost:PORT"
      ssl_keyfile:  /etc/etcd/peer.key  # Path to key file
      ssl_certfile: /etc/etcd/peer.crt  # Path to SSL certificate
      ssl_ca_certs: /etc/etcd/ca.crt    # Path to CA certificate
      ssl_cert_validation: True

See etcd Metrics.

Fluentd is an open source data collector, which allows unifying data collection and consumption to better use and understand data. Fluentd structures data as JSON as much as possible, to unify all facets of processing log data: collecting, filtering, buffering, and outputting logs across multiple sources and destinations. If Fluentd is installed on your environment, the Sysdig agent will automatically connect. See Default Configuration. The Sysdig agent automatically collects default metrics.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Fluentd can be installed as a package (.deb, .rpm, etc) depending on the OS flavor, or it can be deployed in a Docker container. Fluentd installation is documented here. For the examples on this page, a .deb package installation is used.

After installing Fluentd, add following lines in fluentd.conf :

<source>
  @type monitor_agent
  bind 0.0.0.0
  port 24220
</source>

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig'sdragent.default.yaml uses the following code to connect with Fluentd and collect default metrics.

(If you use a non-standard port for monitor_agent , you can configure it as usual in the agent config file dragent.yaml.)

  - name: fluentd
    pattern:
      comm: fluentd
    conf:
      monitor_agent_url: http://localhost:24220/api/plugins.json

To generate the metric data, it is necessary to generate some logs through an application. In the following example, HTTP is used. (For more information, see Life of a Fluentd event.)

Execute following command on in the Fluentd environment:

$ curl -i -X POST -d 'json={"action":"login","user":2}' http://localhost:8888/test.cycle

Expected output: (Note: Here the status code is 200 OK, as HTTP traffic is successfully generated; it will vary per application.)

HTTP/1.1 200 OK
Content-type: text/plain
Connection: Keep-Alive
Content-length: 0

Metrics Available

See fluentd Metrics.

Golang expvaris the standard interface designed to instrument and expose custom metrics from a Go program via HTTP. In addition to custom metrics, it also exports some metrics out-of-the-box, such as command line arguments, allocation stats, heap stats, and garbage collection metrics.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

You will need to create a custom entry in the user settings config file for your Go application, due to the difficulty in determining if an application is written in Go by looking at process names or arguments. Be sure your app has expvars enabled, which means importing the expvar module and having an HTTP server started from inside your app, as follows:

import (
    ...
    "net/http"
    "expvar"
    ...
)

// If your application has no http server running for the DefaultServeMux,
// you'll have to have a http server running for expvar to use, for example
// by adding the following to your init function
func init() {
    go http.ServeAndListen(":8080", nil)
}

// You can also expose variables that are specific to your application
// See http://golang.org/pkg/expvar/ for more information

var (
    exp_points_processed = expvar.NewInt("points_processed")
)

func processPoints(p RawPoints) {
    points_processed, err := parsePoints(p)
    exp_points_processed.Add(points_processed)
    ...
}

See also the following blog entry: How to instrument Go code with custom expvar metrics.

Sysdig Agent Configuration

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

No default configuration for Go is provided in the Sysdig agent dragent.default.yaml file. You must edit the agent config file as described in Example 1.

Add the following code sample to dragent.yaml to collect Go metrics.

app_checks:
  - name: go-expvar
    check_module: go_expvar
    pattern:
          comm: go-expvar
    conf:
      expvar_url: "http://localhost:8080/debug/vars" # automatically match url using the listening port
      # Add custom metrics if you want
      metrics:
        - path: system.numberOfSeconds
          type: gauge # gauge or rate
          alias: go_expvar.system.numberOfSeconds
        - path: system.lastLoad
          type: gauge
          alias: go_expvar.system.lastLoad
        - path: system.numberOfLoginsPerUser/.* # You can use / to get inside the map and use .* to match any record inside
          type: gauge
        - path: system.allLoad/.*
          type: gauge

See Go Metrics.

HAProxy provides a high-availability load balancer and proxy server for TCP- and HTTP-based applications which spreads requests across multiple servers.

The Sysdig agent automatically collects haproxy metrics. You can also edit the agent configuration file to collect additional metrics.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

The stats feature must be enabled on your HAProxy instance. This can be done by adding the following entry to the HAProxy configuration file /etc/haproxy/haproxy.cfg

listen stats
  bind :1936
  mode http
  stats enable
  stats hide-version
  stats realm Haproxy\ Statistics
  stats uri /haproxy_stats
  stats auth stats:stats

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's dragent.default.yaml uses the following code to connect with HAProxy and collect haproxy metrics:

app_checks:
  - name: haproxy
    pattern:
      comm: haproxy
      port: 1936
    conf:
      username: stats
      password: stats
      url: http://localhost:1936/
      collect_aggregates_only: True
    log_errors: false

You can get a few additional status metrics by editing the configuration in dragent.yaml,as in the following examples.

Enable the collect_status_metrics flag to collect the metrics haproxy.count_per_status, and haproxy.backend_hosts.

app_checks:
  - name: haproxy
    pattern:
      comm: haproxy
      port: 1936
    conf:
      username: stats
      password: stats
      url: http://localhost:1936/haproxy_stats
      collect_aggregates_only: True
      collect_status_metrics: True
    log_errors: false

Example 2 Collect Status Metrics Per Host

Enable:

app_checks:
  - name: haproxy
    pattern:
      comm: haproxy
      port: 1936
    conf:
      username: stats
      password: stats
      url: http://localhost:1936/haproxy_stats
      collect_aggregates_only: True
      collect_status_metrics: True
      collect_status_metrics_by_host: True
      tag_service_check_by_host: True
    log_errors: false

If you’ve configured HAProxy to report statistics to a UNIX socket, you can set the url in dragent.yaml to the socket’s path (e.g., unix:///var/run/haproxy.sock).

Edit your HAProxy configuration file ( /etc/haproxy/haproxy.cfg ) to add the following lines to the global section:

global
    [snip]
       stats socket /run/haproxy/admin.sock mode 660 level admin
       stats timeout 30s
    [snip]

Add the socket URL from the HAProxy config to the dragent.yaml file:

app_checks:
      - name: haproxy
        pattern:
          comm: haproxy
        conf:
          url: unix:///run/haproxy/admin.sock
        log_errors: True

Metrics Available

See HAProxy Metrics.

The HTTP check monitors HTTP-based applications for URL availability.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

You do not need to configure anything on HTTP-based applications for the Sysdig agent to connect.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

No default entry is present in the dragent.default.yaml for the HTTP check. You need to add an entry in dragent.yaml as shown in following examples.

First you must identify the process pattern (comm:). It must match an actively running process for the HTTP check to work. Sysdig recommends the process be the one that is serving the URL being checked.

If the URL is is remote from the agent, the user should use a process that is always running, such as "systemd".

Confirm the "comm" value using the following command:

cat /proc/1/comm

Add the following entry to the dragent.yaml file and modify the 'name:''comm:' and 'url:' parameters as needed:

app_checks:
  - name: EXAMPLE_WEBSITE
    check_module: http_check
    pattern:
      comm:  systemd
    conf:
      url: https://www.MYEXAMPLE.com

There are multiple configuration options available with the HTTP check. A full list is provided in the table following Example 2. These keys should be listed under the conf: section of the configuration in Example 1.

app_checks:
  - name: EXAMPLE_WEBSITE
    check_module: http_check
    pattern:
      comm:  systemd
    conf:
      url: https://www.MYEXAMPLE.com
      # timeout: 1
      #  method: get
      #  data:
      #    <KEY>: <VALUE>
      #  content_match: '<REGEX>''
      #  reverse_content_match: false
      #  username: <USERNAME>
      #  ntlm_domain: <DOMAIN>
      #  password: <PASSWORD>
      #  client_cert: /opt/client.crt
      #  client_key: /opt/client.key
      #  http_response_status_code: (1|2|3)\d\d
      #  include_content: false
      #  collect_response_time: true
      #  disable_ssl_validation: true
      #  ignore_ssl_warning: false
      #  ca_certs: /etc/ssl/certs/ca-certificates.crt
      #  check_certificate_expiration: true
      #  days_warning: <THRESHOLD_DAYS>
      #  check_hostname: true
      #  ssl_server_name: <HOSTNAME>
      #  headers:
      #    Host: alternative.host.example.com
      #    X-Auth-Token: <AUTH_TOKEN>
      #  skip_proxy: false
      #  allow_redirects: true
      #  include_default_headers: true
      #  tags:
      #    - <KEY_1>:<VALUE_1>
      #    - <KEY_2>:<VALUE_2>

HTTP metrics concern response time and SSL certificate expiry information.

See HTTP Metrics.

http.can_connect:

Returns DOWN when any of the following occur:

Segmentation of the http.can_connect can be done by URL.

http.ssl_cert:

The check returns:

To disable this check, set check_certificate_expiration to false.

Jenkins is an open-source automation server which helps to automate part of the software development process, permitting continuous integration and facilitating the technical aspects of continuous delivery. It supports version control tools (such as Subversion, Git, Mercurial, etc), can execute Apache Ant, Apache Maven and SBT-based projects, and allows shell scripts and Windows batch commands. If Jenkins is installed on your environment, the Sysdig agent will automatically connect and collect all Jenkins metrics. See Default Configuration.

This page describes the default configuration settings, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Requires the standard Jenkins server setup with one or more Jenkins Jobs running on it.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's dragent.default.yaml uses the following code to connect with Jenkins and collect basic metrics.

  - name: jenkins
    pattern:
      comm: java
      port: 50000
    conf:
      name: default
      jenkins_home: /var/lib/jenkins #this depends on your environment

The following metrics will be available only after running one or more Jenkins jobs. They handle queue size, job duration, and job waiting time.

See Jenkins Metrics.

Lighttpd is a secure, fast, compliant, and very flexible web server that has been optimized for high-performance environments. It has a very low memory footprint compared to other web servers and takes care of CPU load. Its advanced feature set (FastCGI, CGI, Auth, Output Compression, URL Rewriting, and many more) make Lighttpd the perfect web server software for every server that suffers load problems. If Lighttpd is installed on your environment, the Sysdig agent will automatically connect. See Default Configuration. The Sysdig agent automatically collects the default metrics.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

For Lighttpd, the status page must be enabled. Add mod_status in the /etc/lighttpd/lighttpd.conf config file:

server.modules = ( ..., "mod_status", ... )

Then configure an endpoint for it. If (for security purposes) you want to open the status page only to users from the local network, it can be done by adding the following lines in the /etc/lighttpd/lighttpd.conf file :

$HTTP["remoteip"] == "127.0.0.1/8" {
    status.status-url = "/server-status" 
  }

If you want an endpoint to be open for remote users based on authentication, then the mod_auth module should be enabled in the /etc/lighttpd/lighttpd.conf config file:

server.modules = ( ..., "mod_auth", ... )

Then you can add the auth.require parameter in the /etc/lighttpd/lighttpd.conf config file:

auth.require = ( "/server-status" => ( "method"  => ... , "realm"   => ... , "require" => ... ) ) 

For more information on the auth.require parameter, see the Lighttpd documentation..

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's dragent.default.yaml uses the following code to connect with Lighttpd and collect basic metrics.

app_checks:
  - name: lighttpd
    pattern:
      comm: lighttpd
    conf:
      lighttpd_status_url: "http://localhost:{port}/server-status?auto"
    log_errors: false

These metrics are supported for Lighttpd version 1.x.x only. Lighttpd version 2.x.x is being built and is NOT ready for use as of this publication.

See Lighttpd Metrics.

Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from the results of database calls, API calls, or page rendering. If Memcached is installed on your environment, the Sysdig agent will automatically connect. See Default Configuration. The Sysdig agent automatically collects basic metrics. You can also edit the configuration to collect additional metrics related to items and slabs.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Memcached will automatically expose all metrics. You do not need to add anything on Memcached instance.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's dragent.default.yaml uses the following code to connect with Memcached and collect basic metrics:

app_checks:
  - name: memcached
    check_module: mcache
    pattern:
      comm: memcached
    conf:
      url: localhost
      port: "{port}"

Additional metrics can be collected by editing Sysdig's configuration file dragent.yaml. If SASL is enabled, authentication parameters must be added to dragent.yaml.

memcache.items.* and memcache.slabs.* can be collected by setting flags in the options section, as follows . Either value can be set to false if you do not want to collect metrics from them.

app_checks:
  - name: memcached
    check_module: mcache
    pattern:
      comm: memcached
    conf:
      url: localhost
      port: "{port}"
    options:
      items: true       # Default is false
      slabs: true       # Default is false

SASL authentication can be enabled with Memcached (see instructions here). If enabled, credentials must be provided against username and password fields as shown in Example 2.

app_checks:
  - name: memcached
    check_module: mcache
    pattern:
      comm: memcached
    conf:
      url: localhost
      port: "{port}"
      username: <username> 
      # Some memcached version will support <username>@<hostname>.
      # If memcached is installed as a container, hostname of memcached container will be used as username
      password: <password>

See Memcached Metrics.

Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elasticsearch) with APIs for resource management and scheduling across entire datacenter and cloud environments. The Mesos metrics are divided into master and agent. Marathon is a production-grade container orchestration platform for Apache Mesos.

If Mesos and Marathon are installed in your environment, the Sysdig agent will automatically connect and start collecting metrics.You may need to edit the default entries to add a custom configuration if the default does not work. See Default Configuration.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Both Mesos and Marathon will automatically expose all metrics. You do not need to add anything to the Mesos/Marathon instance.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

The Sysdig agent has different entries for mesos-master, mesos-slave and marathon in its configuration file. Default entries are present in Sysdig's dragent.default.yaml file and collect all metrics for Mesos. For Marathon, it collects basic metrics. You may need add configuration to collect additional metrics.

In the URLs for mesos-master and mesos-slave, {mesos_url} will be replaced with either the hostname of the auto-detected mesos master/slave (if auto-detection is enabled), or with an explicit value from mesos_state_uri otherwise.

In the URLs for marathon, {marathon_url} will be replaced with the hostname of the first configured/discovered Marathon framework.

For all Mesos and Marathon apps, {auth_token} will either be blank or an auto-generated token obtained via the /acs/api/v1/auth/login endpoint.

Mesos Agent

Marathon

Example 1

Enable the flag full_metrics to collect all metrics for marathon.

The following additional metrics are collected with this configuration:

Metrics Available

See Mesos Master Metrics.

See Mesos Agent Metrics.

See Marathon Metrics.

MongoDB is an open-source database management system (DBMS) that uses a document-oriented database model which supports various forms of data. If MongoDB is installed in your environment, the Sysdig agent will automatically connect and collect basic metrics (if authentication is not used). You may need to edit the default entries to connect and collect additional metrics. See Default Configuration.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Create a read-only user for the Sysdig agent.

# Authenticate as the admin user.
use admin
db.auth("admin", "<YOUR_MONGODB_ADMIN_PASSWORD>")

# On MongoDB 2.x, use the addUser command.
db.addUser("sysdig-cloud", "sysdig-cloud-password", true)

# On MongoDB 3.x or higher, use the createUser command.
db.createUser({
  "user":"sysdig-cloud",
  "pwd": "sysdig-cloud-password",
  "roles" : [
    {role: 'read', db: 'admin' },
    {role: 'clusterMonitor', db: 'admin'},
    {role: 'read', db: 'local' }
  ]
})

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's dragent.default.yaml uses the following code to connect with MongoDB.

app_checks:
  - name: mongodb
    check_module: mongo
    pattern:
      comm: mongod
    conf:
      server: "mongodb://localhost:{port}/admin"

The default MongoDB entry should work for without modification if authentication is not configured. If you have enabled password authentication, the entry will need to be changed.

Some metrics are not available by default. Additional configuration needs to be provided to collect them as shown in following examples.

Replace <username> and <password> with actual username and password.

app_checks:
  - name: mongodb
    check_module: mongo
    pattern:
      comm: mongod
    conf:
      server: mongodb://<username>:<password>@localhost:{port}/admin
      replica_check: true

Some metrics are not collected by default. These can be collected by adding additional_metrics section in the dragent.yaml file under the app_checks mongodb configuration.

Available options are:

collection - Metrics of the specified collections

metrics.commands - Use of database commands

tcmalloc - TCMalloc memory allocator

top - Usage statistics for each collection

app_checks:
  - name: mongodb
    check_module: mongo
    pattern:
      comm: mongod
    conf:
      server: mongodb://<username>:<password>@localhost:{port}/admin
      replica_check: true
      additional_metrics:
        - collection       
        - metrics.commands
        - tcmalloc
        - top

List of metrics with respective entries in dragent.yaml:

See MongoDB Metrics.

MySQL is the world’s most popular open-source database. With its proven performance, reliability, and ease-of-use, MySQL has become the leading database choice for web-based applications, used by high profile web properties including Facebook, Twitter, YouTube. Additionally, it is an extremely popular choice as embedded database, distributed by thousands of ISVs and OEMs.

The MySQL AppCheck is supported for following MySQL versions.

If the Sysdig agent is installed as a Process:

If the Sysdig agent is installed as a Docker container:

The Docker container of the Sysdig agent has Python 2.7 installed. If it is installed, respective versions against Python 2.7 will be supported.

The following environments have been tested and are supported. Tests environments include both the Host/Process and Docker environment.

A user must be created on MySQL so the Sysdig agent can collect metrics. To configure credentials, run the following commands on your server, replacing the sysdig-clouc-password parameter.

MySQL version-specific commands to create a user are provided below.

# MySQL 5.6 and earlier
CREATE USER 'sysdig-cloud'@'127.0.0.1' IDENTIFIED BY 'sysdig-cloud-password';
GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'sysdig-cloud'@'127.0.0.1' WITH MAX_USER_CONNECTIONS 5;

## OR ##

# MySQL 5.7 and 8
CREATE USER 'sysdig-cloud'@'127.0.0.1' IDENTIFIED BY 'sysdig-cloud-password' WITH MAX_USER_CONNECTIONS 5;
GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'sysdig-cloud'@'127.0.0.1';

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

There is no default configuration for MySQL, as a unique user and password are required for metrics polling.

Add the entry for MySQL into dragent.yaml , updating the user and pass field credentials.

app_checks:
  - name: mysql
    pattern:
      comm: mysqld
    conf:
      server: 127.0.0.1
      user: sysdig-cloud
      pass: sysdig-cloud-password

See MySQL Metrics.

NGINX is open source software for web serving, reverse proxying, caching, load balancing, media streaming, and more. It started out as a web server designed for maximum performance and stability. In addition to its HTTP server capabilities, NGINX can also function as a proxy server for email (IMAP, POP3, and SMTP) and a reverse proxy and load balancer for HTTP, TCP, and UDP servers.

NGINX Plus isa software load balancer, web server, and content cache built on top of open source NGINX. NGINX Plus has exclusive enterprise‑grade features beyond what's available in the open source offering, including session persistence, configuration via API, and active health checks.

The Sysdig agent has a default configuration to collect metrics for open-source NGINX, provided that you have the HTTP stub status module enabled. NGINX exposes basic metrics about server activity on a simple status page with this status module. If NGINX Plus is installed, a wide range of metrics are available with the NGINX Plus API.

This page describes the setup steps for NGINX/NGINX Plus, the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and sample results in the Sysdig Monitor UI.

This section describes the configuration required on the NGINX server.

The Sysdig agent will not collect metrics until the required endpoint is added to the NGINX configuration, per one of the following methods:

Configuration examples of each are provided below

The ngx_http_stub_status_module provides access to basic status information. It is compiled by default on most distributions. If not, it should be enabled with the --with-http_stub_status_module configuration parameter.

When NGINX Plus is configured, the Plus API can be enabled by adding /api endpoint in the NGINX configuration file as follows.

The default NGINX configuration file is present at /etc/nginx/nginx.conf or /etc/nginx/conf.d/default.conf.

# HTTP context                                                                            
server {
...
  # Enable NGINX Plus API
  location /api {
    api write=on;
    allow all;
  }
...
}

With the default configuration, only NGINX metrics will be available once the ngx_http_stub_status_module is configured.

app_checks:
  - name: nginx
    check_module: nginx
    pattern:
      exe: "nginx: worker process"
    conf:
      nginx_status_url: "http://localhost:{port}/nginx_status"
    log_errors: true

With this example only NGINX Plus Metrics will be available.

app_checks:
  - name: nginx
    check_module: nginx
    pattern:
      exe: "nginx: worker process"
    conf:
      nginx_plus_api_url: "http://localhost:{port}/api"
      use_plus_api: true
      user: admin
      password: admin
    log_errors: true

This is special case where NGINX open-source and NGINX PLUS are installed on same host but in different containers. With this configuration, respective metrics will be available for NGINX and NGINX Plus containers.

app_checks:
  - name: nginx
    check_module: nginx
    pattern:
      exe: "nginx: worker process"
    conf:
      nginx_plus_api_url: "http://localhost:{port}/api"
      nginx_status_url: "http://localhost:{port}/nginx_status"
      use_plus_api: true
      user: admin
      password: admin
    log_errors: true

List of Metrics

See NGINX Metrics.

See NGINX Plus Metrics.

NTP stands for Network Time Protocol. It is used to synchronize the time on your Linux system with a centralized NTP server. A local NTP server on the network can be synchronized with an external timing source to keep all the servers in your organization in-sync with an accurate time.

If the NTP check is enabled in the Sysdig agent, it reports the time offset of the local agent from an NTP server .

This page describes how to edit the configuration to collect information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's dragent.default.yaml does not provide any configuration for NTP.

Add the configuration in Example 1 to the dragent.yaml file to enable NTP checks.

- name: ntp
    interval: 60
    pattern:
      comm: systemd
    conf:
      host: us.pool.ntp.org
      offset_threshold: 60

ntp.offset, the time difference between the local clock and the NTP reference clock, is the primary NTP metric.

See also NTP Metrics.

Service Checks

ntp.in_sync:

Returns CRITICAL if the NTP offset is greater than the threshold specified in dragent.yaml, otherwise OK.

PgBouncer is a lightweight connection pooler for PostgreSQL. If PgBouncer is installed on your environment, you may need to edit the Sysdig agent configuration file to connect. See PGBouncer#Default Configuration.

This page describes the configuration settings, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

PgBouncer does not ship with a default stats user configuration. To configure it, you need to add a user allowed to access PgBouncer stats. Do so by adding following line in pgbouncer.ini. The default file location is /etc/pgbouncer/pgbouncer.ini

stats_users = sysdig_cloud

For the same user you need the following entry in userlist.txt. The default file location is /etc/pgbouncer/userlist.txt

"sysdig_cloud" "sysdig_cloud_password"

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

No default configuration is present in Sysdig's dragent.default.yaml file for PgBouncer, as it requires a unique username and password. You must add a custom entry in dragent.yaml as follows:

app_checks:
  - name: pgbouncer
    pattern:
      comm: pgbouncer
    conf:
      host: localhost # set if the bind ip is different
      port: 6432      # set if the port is not the default
      username: sysdig_cloud
      password: sysdig_cloud_password #replace with appropriate password

See PGBouncer Metrics.

PHP-FPM (FastCGI Process Manager) is an alternative PHP FastCGI implementation, with some additional features useful for sites of any size, especially busier sites. If PHP-FPM is installed on your environment, the Sysdig agent will automatically connect. You may need to edit the default entries to connect if PHP-FPM has a custom setting in its config file. See Sysdig Agent Configuration.

The Sysdig agent automatically collects all metrics with default configuration.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

This check has a default configuration that should suit most use cases. If it does not work for you, verify that you have added these lines to your php-fpm.conf file. The default location is /etc/

pm.status_path = /status
ping.path = /ping

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's dragent.default.yaml uses the following code to connect with PHP-FPM and collect all metrics:

app_checks:
  - name: php-fpm
    check_module: php_fpm
    retry: false
    pattern:
      exe: "php-fpm: master process"

If you have a configuration other than those for PHP-FPM in php-fpm.conf, you can edit the Sysdig agent configuration in dragent.yaml, as shown in Example 1.

Replace the values of status_url and ping_url below with the values set against pm.status_path and ping.path respectively in your php-fpm.conf:

app_checks:
  - name: php-fpm
    check_module: php_fpm
    pattern:
      exe: "php-fpm: master process"
    conf:
      status_url: /mystatus
      ping_url: /myping
      ping_reply: mypingreply

See PHP-FPM Metrics.

PostgreSQL is a powerful, open-source, object-relational database system that has earned a strong reputation for reliability, feature robustness, and performance.

If PostgreSQL is installed in your environment, the Sysdig agent will automatically connect in most cases. In some conditions, you may need to create specific user for Sysdig and edit the default entries to connect.

See Default Configuration. The Sysdig agent automatically collects all metrics with the default configuration when correct credentials are provided.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

PostgreSQL will be auto-discovered and the agent will connect through the Unix socket using the Default Configuration with the postgres default user. If this does not work, you can create a user for Sysdig Monitor and give it enough permissions to read Postgres stats. To do this, execute the following example statements on your server:

create user sysdig-cloud with password 'password';
grant SELECT ON pg_stat_database to sysdig_cloud;

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's default.dragent.yaml uses the following code to connect with Postgres.

app_checks:
  - name: postgres
    pattern:
      comm: postgres
      port: 5432
    conf:
      unix_sock: "/var/run/postgresql/.s.PGSQL.{port}"
      username: postgres

If a special user for Sysdig is created, then update dragent.yaml file with PostgreSQL#Expanded Example 1.

Update the username and password created for the Sysdig agent in the respective fields, as follows:

app_checks:
  - name: postgres
    pattern:
      comm: postgres
      port: 5432
    conf:
      username: sysdig-cloud
      password: password

See PostgreSQL Metrics.

The default PostgreSQL dashboard includes combined metrics and individual metrics in an overview page.

You can also view individual metric charts from a drop-down menu in an Explore view.

RabbitMQ is an open-source message-broker software (sometimes called message-oriented middleware) that implements Advanced Message Queuing Protocol (AMQP). The RabbitMQ server is written in the Erlang language and is built on Open Telecom Platform framework for clustering and fail-over. Client libraries to interface with the broker are available in all major programming languages. If RabbitMQ is installed on your environment, the Sysdig agent will automatically connect. See RabbitMQ#Default Configuration.

The Sysdig agent automatically collects all metrics with the default configuration. You may need to edit the dragent.yaml file if a metrics limit is reached.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Enable the RabbitMQ management plugin. See RabbitMQ’s documentation to enable it.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's dragent.default.yaml uses the following code to connect with RabbitMQ and collect all metrics.

app_checks:
  - name: rabbitmq
    pattern:
      port: 15672
    conf:
      rabbitmq_api_url: "http://localhost:15672/api/"
      rabbitmq_user: guest
      rabbitmq_pass: guest

The RabbitMQ app check tracks various entities, such as exchanges, queues and nodes. Each of these entities has its maximum limits. If the limit is reached, metrics can be controlled by editing the dragent.yaml file, as in the following examples.

When a maximum limit is exceeded, the app check will log an info message:

This message is suppressed by a configuration parameter, logging_interval.

Its default value is 300 seconds. This can be altered by specifying a different value in dragent.yaml.

app_checks:
  - name: rabbitmq
    pattern:
      port: 15672
    conf:
      rabbitmq_api_url: "http://localhost:15672/api/"
      rabbitmq_user: guest
      rabbitmq_pass: guest
      logging_interval: 10 # Value in seconds. Default is 300

Each of the tracked RabbitMQ entities has its maximum limits. Default limits are as follows:

The metrics for these entities are tagged. If any of these entities are present but no transactions have occurred for them, the metrics are still reported with 0 values, though without tags. Therefore, when segmenting these metrics, the tags will show as unset in the Sysdig Monitor Explore view. However, all such entities are still counted against the maximum limits. In such a scenario, you can specify the entity names for which you want to collect metrics in the dragent.yaml file.

app_checks:
  - name: rabbitmq
    pattern:
      port: 15672
    conf:
      rabbitmq_api_url: "http://localhost:15672/api/"
      rabbitmq_user: guest
      rabbitmq_pass: guest
      tags: ["queues:<queuename>"]
      nodes:
        - rabbit@localhost
        - rabbit2@domain
      nodes_regexes:
        - bla.*
      queues:
        - queue1
        - queue2
      queues_regexes:
        - thisqueue-.*
        - another_\d+queue
      exchanges:
        - exchange1
        - exchange2
      exchanges_regexes:
        - exchange*

Optional tags can be applied to every emitted metric, service check, and/or event.

app_checks:
  - name: rabbitmq
    pattern:
      port: 15672
    conf:
      rabbitmq_api_url: "http://localhost:15672/api/"
      rabbitmq_user: guest
      rabbitmq_pass: guest
      tags: ["some_tag:some_value"]

See RabbitMQ Metrics.

Redis is an open-source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. If Redis is installed in your environment, the Sysdig agent will automatically connect in most cases. You may need to edit the default entries to get additional metrics. See Default Configuration.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Redis will automatically expose all metrics. You do not need to configure anything in the Redis instance.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's dragent.default.yaml uses the following code to connect with Redis and collect basic metrics:

app_checks:
  - name: redis
    check_module: redisdb
    pattern:
      comm: redis-server
    conf:
      host: 127.0.0.1
      port: "{port}"

Some additional metrics can be collected by editing the configuration file as shown in following examples. The options shown in Example 2 are relevant if Redis requires authentication or if a Unix socket is used.

The following example entry results in the metric redis.key.length in the Sysdig Monitor UI, displaying the length of specific keys (segmented by: key). To enable, provide the key names in dragent.yaml as follows.

Note that length is 0 (zero) for keys that have a type other than list, set, hash, or sorted set. Keys can be expressed as patterns; see https://redis.io/commands/keys.

Sample entry in dragent.yaml:

app_checks:
  - name: redis
    check_module: redisdb
    pattern:
      comm: redis-server
    conf:
      host: 127.0.0.1
      port: "{port}"
      keys:
        - "list_1"
        - "list_9*"

app_checks:
  - name: redis
    check_module: redisdb
    pattern:
      comm: redis-server
    conf:
      host: 127.0.0.1
      port: "{port}"
      # unix_socket_path: /var/run/redis/redis.sock # can be used in lieu of host/port
      # password: mypassword                                            # if your Redis requires auth

You can also collect the INFO COMMANDSTATS result as metrics (redis.command.*). This works with Redis >= 2.6

Sample implementation:

app_checks:
  - name: redis
    check_module: redisdb
    pattern:
      comm: redis-server
    conf:
      host: 127.0.0.1
      port: "{port}"
      command_stats: true

See RedisDB Metrics.

Riak CS does not ship with a default configuration because it needs at least access_id and access_secret to work. Add the riakcs entry in your dragent.yaml with those unique parameters:

  - name: riakcs
    pattern:
      comm: beam.smp
      port: 8080
    conf:
      access_id: "my_access_id"
      access_secret: "my_access_secret"
      #is_secure: false 
      s3_root: s3.amazonaws.dev

To get statistics, app_checks also needs permissions to read the riak-cs bucket.

Simple Network Management Protocol (SNMP) is an application-layer protocol used to manage and monitor network devices and their functions. The Sysdig agent can connect to network devices and collect metrics using SNMP.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Simple Network Management Protocol (SNMP) is an Internet Standard protocol for collecting and configuring information about devices in the networks. The network devices include physical devices like switches, routers, servers etc.

SNMP has three primary versions ( SNMPv1, SNMPv2c and SNMPv3) and SNMPv2c is most widely used.

SNMP allows device vendors to expose management data in the form of variables on managed systems organized in a management information base (MIB), which describe the system status and configuration. The devices can be queried as well as configured remotely using these variables. Certain MIBs are generic and supported by the majority of the device vendors. Additionally, each vendor can have their own private/enterprise MIBs for vendor-specific information.

SNMP MIB is a collection of objects uniquely identified by an Object Identifier (OID). OIDs are represented in the form of x.0, where x is the name of object in the MIB definition.

For example, suppose one wanted to identify an instance of the variable sysDescr 

The object class for sysDescr is:             

iso org dod internet mgmt mib system sysDescr       
 1   3   6     1      2    1    1       1

Hence, the object type, x, would be 1.3.6.1.2.1.1.1 

To monitor the servers with the Sysdig agent, the SNMP agent must be installed on the servers to query the system information.

For Ubuntu-based servers, use the following commands to install the SNMP Daemon:

$sudo apt-get update
$sudo apt-get install snmpd

Next, configure this SNMP agent to respond to queries from the SNMP manager by updating the configuration file located at /etc/snmp/snmpd.conf

Below are the important fields that must be configured:

snmpd.conf

# Listen for connections on all interfaces (both IPv4 *and* IPv6)
agentAddress udp:161,udp6:[::1]:161

## ACCESS CONTROL
## system + hrSystem groups only
view systemonly included .1.3.6.1.2.1.1
view systemonly included .1.3.6.1.2.1.25.1
view systemonly included .1.3.6.1.2.1.31.1
view systemonly included .1.3.6.1.2.1.2.2.1.1

# Default access to basic system info
rocommunity public default -V systemonly
# rocommunity6 is for IPv6
rocommunity6 public default -V systemonly

After making changes to the config file, restart the snmpd service using:

$sudo service snmpd restart

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

No default configuration is present for SNMP check.

app_checks:
  - name: snmp
    pattern:
      comm: python
      arg: /opt/draios/bin/sdchecks
    interval: 30
    conf:
      mibs_folder: /usr/share/mibs/ietf/
      ip_address: 52.53.158.103
      port: 161
      community_string: public

      # Only required for snmp v1, will default to 2
      # snmp_version: 2 

      # Optional tags can be set with each metric
      tags:
         - vendor:EMC
         - array:VNX5300
         - location:front

      metrics:
        - OID: 1.3.6.1.2.1.25.2.3.1.5
          name: hrStorageSize
        - OID: 1.3.6.1.2.1.1.7
          name: sysServices
        - MIB: TCP-MIB
          symbol: tcpActiveOpens
        - MIB: UDP-MIB
          symbol: udpInDatagrams
        - MIB: IP-MIB
          table: ipSystemStatsTable
          symbols:
            - ipSystemStatsInReceives
          metric_tags:
            - tag: ipversion
              index: 1        # specify which index you want to read the tag value from
        - MIB: IF-MIB
          table: ifTable
          symbols:
            - ifInOctets
            - ifOutOctets
          metric_tags:
            - tag: interface
              column: ifDescr  # specify which column to read the tag value from

The Sysdig agent allows you to monitor the SNMP counters and gauge of your choice. For each device, specify the metrics that you want to monitor in the metrics subsection using one of the following methods:

The SNMP check does not have default metrics. All metrics mentioned in dragent.yaml file will be seen with snmp.* prefix/

Supervisor daemon is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems., The Supervisor check monitors the uptime, status, and number of processes running under Supervisord.

No default configuration is provided for the Supervisor check; you must provide the configuration in the dragent.yaml file for the Sysdig agent to collect the data provided by Supervisor.

This page describes the setup steps required on Supervisor, how to edit the Sysdig agent configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

The Sysdig agent can collect data from Supervisor via HTTP server or UNIX socket. The agent collects the same data regardless of the configured collection method.

Un-comment the following or add them if they are not present in /etc/supervisor/supervisord.conf

[inet_http_server]
port=localhost:9001
username=user  # optional
password=pass  # optional
...


[supervisorctl]
serverurl=unix:///tmp/supervisor.sock
...

[unix_http_server]
file=/tmp/supervisor.sock
chmod=777 # make sure chmod is set so that non-root users can read the socket.
...

[program:foo]
command=/bin/cat

The programs controlled by Supervisor are given by different [program] sections in the configuration. Each program you want to manage by Supervisor must be specified in the Supervisor configuration file, with its supported options in the [program]section. See Supervisor's sample.conf file for details.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's dragent.default.yaml does not have any configuration to connect the agent with Supervisor. Edit dragent.yaml following the Examples given to connect with Supervisor and collect supervisor.* metrics.

  - name: supervisord
    pattern:
      comm: supervisord
    conf:
      socket: "unix:///tmp/supervisor.sock"
      

- name: supervisord
  pattern:
    comm: supervisord
  conf:
    host: localhost
    port: 9001
# user: user # Optional. Required only if a username is configured.
# pass: pass # Optional. Required only if a password is configured.

See also Supervisord Metrics.

Returns CRITICAL if the Sysdig agent cannot connect to the HTTP server or UNIX socket configured, otherwise OK.

You can monitor the status of your custom application's port using the TCP check. This check will routinely connect to the designated port and send Sysdig Monitor a simple on/off metric and response time.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Any application listening on a TCP port can be monitored with tcp_check.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

No default configuration is provided in the default settings file; you must add the entries in Example 1 to the user settings config file dragent.yaml.

 - name: tcp_check
    check_module: tcp_check
    pattern:
      comm: httpd
      arg: DFOREGROUND
    conf:
      port: 80
      collect_response_time: true

This example shows monitoring a TCP check on an Apache process running on the host on port 80.

comm: is the command for running the Apache server on port 80.

If you want the response time for your port, meaning the amount of time the process takes to accept the connection, you can add the collect_response_time: true parameter under the conf: section and the additional metric network.tcp.response_time will appear in the Metrics list.

See also TCP Metrics.

tcp.can_connect :

DOWN if the agent cannot connect to the configured host and port, otherwise UP.

Varnish HTTP Cache is a web application accelerator, also known as a "caching HTTP reverse proxy." You install it in front of any server that speaks HTTP and configure it to cache the contents. If Varnish is installed on your environment, the Sysdig agent will automatically connect.See Varnish#Default Configuration.

The Sysdig Agent automatically collects all metrics. You can also edit the configuration to emit service checks for the back end.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Varnish will automatically expose all metrics. You do not need to add anything to the Varnish instance.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

By default, Sysdig's dragent.default.yaml uses the following code to connect with Varnish and collect all metrics.

app_checks:
  - name: varnish
    interval: 15
    pattern:
      comm: varnishd
    conf:
      varnishstat: /usr/bin/varnishstat

Optionally, if you want to submit service checks for the health of each back end, you can configure varnishadm and edit dragent.yaml as in Example 1.

When varnishadm is configured, the Sysdig agent requires privileges to execute the binary with root privileges. Add the following to your /etc/sudoers file:

sysdig-agent ALL=(ALL) NOPASSWD:/usr/bin/varnishadm

Then edit dragent.yaml as follows. Note: If you have configured varnishadm and your secret file is NOT /etc/varnish/secret, you can comment out secretfile.

app_checks:
  - name: varnish
    interval: 15
    pattern:
      comm: varnishd
    conf:
      varnishstat: /usr/bin/varnishstat
      varnishadm: /usr/bin/varnishadm
      secretfile: /etc/varnish/secret   

This example will enable following service check.

varnish.backend_healthy: The agent submits a service check for each Varnish backend, tagging each with backend:<backend_name>.

See Varnish Metrics.

Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elasticsearch) with API’s for resource management and scheduling across entire datacenter and cloud environments. Marathon is a production-grade container orchestration platform for Apache Mesos.

If Mesos and Marathon are installed in your environment, the Sysdig agent will automatically connect and start collecting metrics.You may need to edit the default entries to add a custom configuration if the default does not work. See Default Configuration.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Both Mesos and Marathon will automatically expose all metrics. You do not need to add anything to the Mesos/Marathon instance.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

The Sysdig agent has different entries for mesos-master, mesos-slave and marathon in its configuration file. Default entries are present in Sysdig's dragent.default.yaml file and collect all metrics for Mesos. For Marathon, it collects basic metrics. You may need add configuration to collect additional metrics.

In the urls for mesos-master and mesos-slave, {mesos_url} will be replaced with either the hostname of the autodetected mesos master/slave if autodetection is enabled or an explicit value from mesos_state_uri otherwise.

In the urls for marathon, {marathon_url} will be replaced with the hostname of the first configured/discovered marathon framework.

For all mesos and marathon apps, {auth_token} will either be blank or an autogenerated token obtained via the /acs/api/v1/auth/login endpoint.

app_checks:
  - name: mesos-master
    check_module: mesos_master
    interval: 30
    pattern:
      comm: mesos-master
    conf:
      url: "{mesos_url}:{port}"
      auth_token: "{auth_token}"
      mesos_creds: "{mesos_creds}"

app_checks:
  - name: mesos-slave
    check_module: mesos_slave
    pattern:
      comm: mesos-slave
    interval: 30
    conf:
      url: "{mesos_url}:{port}"
      auth_token: "{auth_token}"
      mesos_creds: "{mesos_creds}"
      # Name of individual tasks to monitor, if needed
      # tasks:
        # - mongo
        # - cassandra

app_checks:
  - name: marathon
    check_module: marathon
    interval: 30
    pattern:
      arg: mesosphere.marathon.Main
    conf:
      url: "{marathon_url}"
      auth_token: "{auth_token}"
      marathon_creds: "{marathon_creds}"

Enable the flag full_metrics to collect all metrics for marathon.

Following additional metrics collected with this configuration:

marathon.cpus

marathon.disk

marathon.instances marathon.mem

app_checks:
  - name: marathon
    check_module: marathon
    interval: 30
    pattern:
      arg: mesosphere.marathon.Main
    conf:
      url: "{marathon_url}"
      auth_token: "{auth_token}"
      marathon_creds: "{marathon_creds}"
      full_metrics: true

Riak CS does not ship with a default configuration because it needs at least access_id and access_secret to work. Add the riakcs entry in your dragent.yaml with those unique parameters:

  - name: riakcs
    pattern:
      comm: beam.smp
      port: 8080
    conf:
      access_id: "my_access_id"
      access_secret: "my_access_secret"
      #is_secure: false 
      s3_root: s3.amazonaws.dev

To get statistics, app_checks also needs permissions to read the riak-cs bucket.

MemSQL is a distributed, highly-scalable SQL database that can run anywhere. If MemSQL is installed on your environment, you may need to edit Sysdig agent's configuration file to connect it with Sysdig. See Default Configuration.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

MemSQL will automatically expose all metrics. You do not need to add anything on MemSQL instance.

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

No default configuration is present in Sysdig's dragent.default.yaml for MemSQL. You must add a custom entry in dragent.yaml as follows.

Update authentication parameters and insert the configuration in dragent.yaml file, as in this example:

app_checks:
  - name : memsql
    check_module : memsql
    pattern:
      comm: memsqld
      arg: master
    conf:
      server : 127.0.0.1
      port : 3306
      user : <username>
      pass : <password> #(optional) if set

See **MemSQL Metrics**.

<This will effective after merging of PR 806 and 1022 >

<Amy comment: I can't find PR 806 and 1022; where are they and what's the status?

Sudhakar - Both these PRs are under review and will be visible once approved by Anoop/Naveen. Currently, Andy Dalton has proposed a solution and we are in process of validating the same.>

Oracle Database 12c is a next-generation database providing a new multi-tenant architecture on top of a fast, scalable, reliable, and secure database platform. If OracleDB is installed on your environment, you need to install and integrate Oracle instant client with Sydig.

This page describes the installation steps for Oracle Instant Client, configuration settings, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

The Oracle app check requires Oracle Instant Client libraries. When Oracle DB is installed as a container, it is necessary to install these dependencies on the host separately to make them available to the Sysdig agent.

When Oracle DB is installed as a process, these dependencies will be installed with Oracle. In that case, you can provide the path of the installation directory to the Sysdig agent by editing dragent.yaml.

Download Oracle Instant Client from the official Oracle website. You must download and install the packages below. An Oracle login is required.

Once Oracle DB is ready, run the following commands to create a user:

alter session set "_ORACLE_SCRIPT"=true; 
CREATE USER <username> IDENTIFIED BY <password>; 
GRANT CONNECT TO <username>; 
GRANT SELECT ON gv_$sysmetric TO <username>; 
GRANT SELECT ON sys.dba_data_files TO <username>;

After installing Oracle instant Client, you need to link it with Sysdig agent.

If you are installing Sysdig agent as a docker container, a volume must to be mounted in the container with the directory where the Oracle Instant Client is present.

Examples:

Configure Oracle Libraries for Sysdig Agent

Link Oracle Instant Client libraries with ldconfig so that they will be available to the Sysdig Agent at runtime. If the Sysdig agent is installed inside the Docker container, run ldconfig commands inside the container.

Example:

echo /u01/app/oracle/product/12.1.0.2/db_1/lib/ > /etc/ld.so.conf.d/oracle.conf
ldconfig

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

There is no default configuration for OracleDB, as a unique user and password are required for metrics polling. You must add the entry for OracleDB into dragent.yaml.

app_checks:
  - name: oracle
    check_module: oracle
    pattern:
      comm: "tnslsnr"
    conf:
      server: localhost
      user: <username>
      password: <password>
      service_name: cdb1
      envs:
        ORACLE_HOME: /u01/app/oracle/product/12.1.0.2/db_1
    log_errors: true

Result in the Monitor UI

Support background: https://sysdig.atlassian.net/wiki/spaces/EN/pages/118119/App+checks#Appchecks-TokuMX