Monitoring openHAB using icinga2

I replied to another thread with the post below, which was discussing how to create rules to monitor when a sensor stops reporting.

It got a bit of interest so I was asked to start another thread. Below is a series of config I use for monitoring various parts of my openHAB system using icinga2, MQTT and mqttwarn.

I haven’t put this in the tutorials section of the WIKI since it is by no means complete, there are loads of pieces missing in order to set this up from scratch so I don’t think it has much value unless it is fleshed out a lot, and unfortunately I don’t have the time to do this currently.

Anyway, have a read and post your thoughts/comments. Interested to hear if anyone else is doing this sort of thing and how.

Original post…

I have done most of my openHAB integration with icinga2 by adding MQTT bindings to the sensor/battery/state items I want to monitor and then implementing rules and checks via mqttwarn.

E.g. in my mqttwarn.ini I have an icinga2 target configured;

[config:icinga2]
host      = 'https://icinga2-host'
port      = 5665
username  = 'icinga2-api-username'
password  = 'icinga2-api-password'
targets   = {
                      # host    service
    'check-service' : [ 'host', 'service', 'mqttwarn' ],
    }

Then I have a series of subscriptions (also in mqttwarn.ini);

# various MQTT services/clients
[/clients/+]
targets  = icinga2:check-service
format   = icinga2_lwt_publish()

# zwave node alive/dead
[/network/+/+/zwave]
targets  = icinga2:check-service
format   = icinga2_network_zwave()

# battery reports for various sensors
[/sensor/+/+/battery]
targets  = icinga2:check-service
format   = icinga2_sensor_battery()

Then I have a series of functions which are being referenced in the targets above, e.g. icinga2_sensor_temp();

def icinga2_lwt_publish(data, srv):
    # /clients/+
    # /homie/+/$online
    payload  = data['payload']
    parts    = data['topic'].split('/')
    host     = parts[2]

    if payload == '1' or payload == 'true':
        status = 0
        output = "OK: {0} is online".format(host)
    else:
        status = 2
        output = "CRITICAL: {0} is offline".format(host)

    icinga2_payload = {
        'exit_status'     : status,
        'plugin_output'   : output,
        'service'         : "{0}.home!mqtt-lwt".format(host),
        }

    return json.dumps(icinga2_payload)

def icinga2_network_zwave(data, srv):
    payload  = data['payload']

    if payload == 'alive':
        status = 0
        code = 'OK'
    elif payload == 'dead':
        status = 2
        code = 'CRITICAL'
    else:
        status = 3
        code = 'UNKNOWN'

    # /network/location/type/zwave
    parts    = data['topic'].split('/')
    location = parts[2]
    type     = parts[3]
    host     = "{0}-{1}".format(location, type)

    icinga2_payload = {
        'exit_status'     : status,
        'plugin_output'   : "{0}: zwave node is {1}".format(code, payload),
        'service'         : "{0}!zwave".format(host),
        }

    return json.dumps(icinga2_payload)

def icinga2_sensor_battery(data, srv):
    batt_warn = 20
    batt_crit = 10

    battery   = int(float(data['payload']))
    status    = 0
    code      = 'OK'

    if battery <= batt_warn:
        status = 1
        code   = 'WARNING'

    if battery <= batt_crit:
        status = 2
        code   = 'CRITICAL'

    # /sensor/location/type/battery
    parts    = data['topic'].split('/')
    location = parts[2]
    type     = parts[3]
    host     = "{0}-{1}".format(location, type)

    icinga2_payload = {
        'exit_status'     : status,
        'plugin_output'   : "BATT {0}: {1}%".format(code, battery),
        'service'         : "{0}!battery".format(host),
        'performance_data': [ "battery={0}%;{1};{2}".format(battery, batt_warn, batt_crit) ],
        }

    return json.dumps(icinga2_payload)

So once you establish the required payload for an icinga2 check it becomes pretty easy to add.

You then have to define the host/service/checkcommand in your icinga2 config and away you go. I won’t go into the icinga2 config as that is a whole new kettle of fish!

But the key bits from this are mqttwarn. Using this tool it is very easy to take data from MQTT topics and transform it into something useful for another service. Without having to touch any openHAB config. I already had all my sensor data being published to MQTT topics in the hope I would one day be able to use it - and along came icinga2!

The real beauty of icinga2 is the ability to configure how and where you are notified. For example by default a service won’t send a notification until it has failed 5 checks (checks are typically executed every minute but this is of course configurable).

I have my system setup to publish alerts to Pushover (again via mqttwarn) and also direct to a private Slack channel;

Here is a screenshot of my hallway pir sensor (a host) with 4 services being checked by icinga2;

5 Likes

I was thinking of implementing something like this too. Already using MQTT and was aware of mqttwarn.

Till now I was making rules to check different items like: is there free space for backup storage? is the temperature too high? But this is not practical in openhab and it is much better to have independent machine to monitor openhab items (I use it as hub for everything at home including mp3 tracks counting, running external programs for backup, playing music, computer performances logging etc.).

Few questions…

Is it possible to monitor when was the last time the mqtt topic was updated…and if it was too long ago reset it to unknown and send warninig?

I have most sensors reporting directly to mqtt but can you give example about checking item state from rest api? like from http://serverip:8080/rest/items/ServerCPU_temp/state

Does anyone tried icinga2 on pi3?

That is kind of the whole point of the passive checks in icinga2. The monitoring system doesn’t actively check things but expects to receive a passive check periodcally (i.e. an update to an MQTT topic). You configure the service in icinga2 with a check interval and if it doesn’t receive an update in that time it will generate a warning.

No I don’t use the REST API for any monitoring, It is all MQTT based using mqttwarn.

I finished my setup. It works NICE!

Seems like it is better to have monitoring logic outside openhab. It will work (send notifications) even if openhab is down…of course only for items that are available outside openhab like in mqtt.

I do export some items to mqtt to make them available in icinga anyway.

PS. Onewire binding seems to work strange. Seems like unchanged values are not published to MQTT from Openhab and some sensors are not updated for an hour or so. If I apply heating they change instantly…hm…I wanted to monitor if some sensor is down

Hi!

I would also like to monitor a few openhab things with Icinga2.
Actually I got some notificiation rules running, when a thing is offline oder some item states a are „unexpected“.

The Icinga-agent is already installed on my openhab-lxc to send some basic checks.

As i read you are using mqtt to send states to Icinga - anyone there, who did it already without mqtt with the agent?