How to send Sensu events to InfluxDB

influxdb_flat_logoSensu event handlers are for taking action on events (produced by check results), such as sending an email alert, or storing metrics in InfluxDB. There are several types of handlers: pipe, tcp, udp, transport, and set.

TCP and UDP handlers send the event data to a remote socket. It is advisable to use UDP handler when you have several dozen hosts to monitor. With the UDP protocol, the load is practically non-existent on the Sensu server. This is what we are going to use in this article.

sensu_flat_logoSo we are going to see how to send metrics from Sensu into InfluxDB. To do that the Sensu handler will send all the data collected by Sensu into the InfluxDB database.


InfluxDB Prerequisite

For those who haven’t heard about it yet, InfluxDB is a time series, metrics, and analytics database. Time series databases are designed to address the problem of storing data resulting from successive measurements made over a period of time. It’s designed to support horizontal as well as vertical scaling and, best of all, it’s not written in Java — it’s written in Go.

To manage InfluxDB, we have two choices :
– Web Admin Interface (port 8083)
– Command Line Interface [API] (port 8086)

Like any good DevOps, we are going to do the job via CLI :), but you could connect to the web UI to see how it looks: http://influxdb-host-name:8083

So to start CLI, just execute the influx command on the InfluxDB server. It should look like this :

$ influx
Connected to http://localhost:8086 version 0.9
InfluxDB shell 0.9
>

For interacting with InfluxDB, the query language used is InfluxQL. It’s a SQL-like query language, so for those who are familiar with SQL, it should be easy.

When InfluxDB is first set up, it does not contain any actual databases beside the internal one, so we need to create a database which we will ultimately use to store our metrics.

To do that use the CREATE DATABASE <db-name> InfluxQL statement, where <db-name> is the name of the database you wish to create.

> CREATE DATABASE sensu
>

Then, we need to enable and configure the UDP section in the InfluxDB’s configuration file. Also, in order to have better performance and to avoid dropping metrics, there is some lines to uncomment and this will allow InfluxDB to enable its buffer. See below the UDP section :

[[udp]]
  enabled = true
   bind-address = ":8090"
   database = "sensu"
  # retention-policy = ""

  # These next lines control how batching works. You should have this enabled
  # otherwise you could get dropped metrics or poor performance. Batching
  # will buffer points in memory if you have many coming in.

   batch-size = 1000 # will flush if this many points get buffered
   batch-pending = 5 # number of batches that may be pending in memory
   batch-timeout = "1s" # will flush at least this often even if we haven't hit buffer limit
   read-buffer = 0 # UDP Read buffer size, 0 means OS default. UDP listener will fail if set above OS max.

Finally, restart the InfluxDB service and we should be ready to receive metrics.
So now let’s configure the Sensu handler.

Sensu Configuration

Requirements

The influxdb rubygem needs to be installed in Sensu’s Ruby environment in order to handle writing the data. Note that Sensu vendors their own embedded ruby, so having it available in the system Ruby won’t work.
E.g. /opt/sensu/embedded/bin/gem install influxdb


There is only one new file to create in your Sensu configuration directory, and this is the handler script :

/etc/sensu/extensions/influxdb_line_protocol.rb

#!/usr/bin/env ruby

require "sensu/extension"

module Sensu
    module Extension
        class InfluxDBLineProtocol < Mutator
            def name
                "influxdb_line_protocol"
            end

            def description
                "returns check output formatted for InfluxDB's line protocol"
            end

            def run(event)
                host = event[:client][:name]
                ip = event[:client][:address]
                metric = event[:check][:name]
                output = event[:check][:output]
            
                data = []
                output.split("\n").each do |result|
                    m = result.split
                    next unless m.count == 3
                    key = m[0].split('.', 2)[1]
                    key.gsub!('.', '_')
                    value = m[1].to_f
                    time = m[2].ljust(19, '0')
                    data << "#{key},host=#{host},ip=#{ip},metric=#{metric} value=#{value} #{time}"
                end

                yield data.join("\n"), 0
            end
        end
    end
end

What this script will do is collect all the data from Sensu output checks, parse them, and inject them into the InfluxDB database.

Next we have to define the above handler by editing the following JSON file :

/etc/sensu/conf.d/handlers.json

{
    "handlers": {
        "default": {
            "type": "set",
            "handlers": ["logstash", "influxdb"]
        },
        "logstash": {
            "type": "pipe",
            "command": "/etc/sensu/handlers/notification/logstash.rb"
        },
        "influxdb": {
            "type": "udp",
            "mutator": "influxdb_line_protocol",
            "socket": {
                       "host": "dockercompose_influxdb_1",
                       "port": 8090
            }
        }
}

Note : To fill this file, you can refer to the Sensu documentation : https://sensuapp.org/docs/latest/handlers

As you can see, I have another handler already defined (logstash). By doing that, I can define a set of handlers, it’s a way to send the same event data to multiple backends.

And finally, configure checks with the InfluxDB handler. In my case all my checks are defined with the handler ‘default’, so that all my checks results are sent to Logstash AND InfluxDB.

{
  "checks": {
    "load_metrics": {
       "type": "metric",
       "command": "/opt/sensu/embedded/bin/ruby /etc/sensu/plugins/linux/load-metrics.rb --scheme `grep name /etc/sensu/conf.d/client.json | cut -d\\\" -f4`",
       "interval": 60,
       "subscribers": ["linux"],
       "handlers": ["default"]
    },
...

Now restart sensu-server and you should see several metrics in the sensu database. To see the data, run the following query on the sensu database via the web admin interface :

select * from /.*/ order by time desc limit 3

influxdb web admin

Normally, nothing should appears in the sensu log. But to track the event sent to InfluxDB, you can launch this command from the InfluxDB server :

nc -u -l 8090

Display Data on Grafana

Set InfluxDB as Datasource

In this step, we will add our InfluxDB database as a source in Grafana.

To add the data source, open the side menu by clicking the Grafana icon in the top header. In the side menu, click Data Sources. Click on the Add New link in the top header to bring up the data source definition screen.

grafana datasource

Populate this screen using your settings and test the connection to your database.

Build a Dashboard

Finally, you are ready to build your own dashboard with beautiful graphs.

grafana graph

grafana request

Conclusion

In this article, we have seen how to inject data via a Sensu handler into an InfluxDB database, collect by Sensu engine and display this data in pretty graphs in Grafana.

We have proved that the stack of these 3 systems — Sensu, InfluxDB and Grafana — work together correctly and are ready to be put into a production environment.

Leave a Reply

7 Comments on "How to send Sensu events to InfluxDB"

avatar
Sort by:   newest | oldest | most voted
thapakazi
Guest

Nice Read,

…looking forward this since long ago
Finally found the missing puzzle.

Thanks a ton.

trackback

[…] How to Send Sensu Events to InfluxDB […]

Chaim Keren-Tzion
Guest
Hi Adel, I have recently installed InfluxDB on my Sensu server and followed these instructions but no-go. I don’t have any errors in the logs and they show that the handler and extension are successfully loaded but nothing is written to the ‘sensu’ InfluxDB database. My settings are the same as above with the exception that I set ‘ “host”: “localhost”, ‘ in the influxdb “handlers” config. When I stop the InfluxDB service and run ‘nc -u -l 8090’ I don’t see any traffic. Also ‘tcpdump -nvv udp dst port 8090’ shows no traffic. I am able to manually write… Read more »
Chaim Keren-Tzion
Guest

Please ignore my post. I had mistakenly tested the procedure with a check that does not produce metrics.

Chinnu
Guest

hi for me Code that sends from sensu to logstash (currently not working)

you help me?

Chinnu
Guest
#! /usr/bin/env ruby # # handler-logstash # # DESCRIPTION: # Designed to take sensu events, transform them into logstah JSON events # and ship them to a redis server for logstash to index. This also # generates a tag with either ‘sensu-ALERT’ or ‘sensu-RECOVERY’ so that # searching inside of logstash can be a little easier. # # OUTPUT: # plain text # # PLATFORMS: # Linux # # DEPENDENCIES: # gem: sensu-plugin # gem: diplomat # # USAGE: # # NOTES: # Heavily inspried (er, copied from) the GELF Handler written by # Joe Miller. # # LICENSE: #… Read more »
vikas
Guest

Hi Adel,
I am getting below error
[I] 2017-02-08T12:45:07Z Failed to parse points: unable to parse ‘{“id”:”2d1a4edd-7cea-4316-9c22-2317c1a6ba18″,”client”:{“name”:”sensu-client”,”address”:”192.168.0.5″,”subscriptions”:[“Web_server”,”client:sensu-client”],”version”:”0.26.5″,”timestamp”:1486557896},”check”:{“type”:”metric”,”command”:”/etc/sensu/plugins/check-apache.rb”,”interval”:60,”handlers”:[“influxdb”],”subscribers”:[“ALL”],”name”:”apache_check”,”issued”:1486557906,”executed”:1486557906,”duration”:0.101,”output”:”OK – Apache daemon is runningn”,”status”:0,”history”:[“0″,”0″,”0″,”0″,”0″,”0″,”0″,”0″,”0″,”0″,”0″,”0″,”0″,”0″,”0″,”0″,”0″,”0″,”0″,”0″,”0″],”total_state_change”:0},”occurrences”:1,”action”:”create”,”timestamp”:1486557907}’: missing tag value service=udp

wpDiscuz