New Add-on bundle for Prometheus health Metrics

prometheus
metrics
Tags: #<Tag:0x00007f014bab6be0> #<Tag:0x00007f014bab6a50>

(Steve Cliffe) #18

I’m pretty sure I did replace the file, shutdown, cleared the cache and restarted.

Noticing the “2.4.0-SNAPSHOT” in the filename I’ve now updated to the latest OH snapshot (I was running 2.3) and have a different problem:

2018-08-16 13:26:09.030 [WARN ] [org.apache.felix.fileinstall        ] - Error while starting bundle: 
file:/packages/OpenHAB2/addons/com.kuguhome.openhab.prometheusmetrics-2.4.0-SNAPSHOT.jar
org.osgi.framework.BundleException: Could not resolve module: 
com.kuguhome.openhab.prometheusmetrics [225]
Unresolved requirement: Import-Package: io.swagger.annotations; version="1.5.7.SNAPSHOT"

The runtime bundled version is 1.5.8


(friesenkiwi) #19

I am running OH 2.3 release as well, the bundle is not tested with the latest openHAB snapshot, sorry. :frowning: Probably, the io.swagger.annotation version has changed lately


(friesenkiwi) #20

@sjcliffe does it work for you now with the 2.3 stable release?


(friesenkiwi) #21

@diyha you mentioned (Presence with Android after using Iphone) you would like to see some “stats” from your openHAB instance. I think this bundle will do the thing for you.
In detail:

  • what bindings are installed :white_check_mark:
  • what bindings were uninstalled after they were installed :white_check_mark: (you record the installed bundles over time in prometheus and look at them, can even issue PromQL queries to filter bundles for specific properties, e.g. changed from installed to uninstalled)
  • how many things a binding serves :x:(this is not directly implemented yet, but you will get a list of all things)
  • how many exceptions are thrown by a binding shortly after being installed (may indicate poor docs and poor parm descriptions) :white_check_mark: (the metrics bundle counts the number of log messages by level)
  • how many exceptions are thrown by a binding in a given time :white_check_mark: (see above)
  • how many exceptions are thrown by platform (may indicate popular platform issue) :white_check_mark: (see above and additionally the events per source are counted, this will also be revealing)
  • how often OH reboots :white_check_mark: not directly counted but can be inferred from resetting event counters
  • how often rule engine throws errors (may indicate poor rue language VM and tooling) :white_check_mark: (counted in the logs and event bus)
  • etc :white_check_mark: (number of inbox entries, bundle and thing state, thread/pool size and counts, jvm/process/memory insights)

Let me know if you have any questions, suggestions or feature requests or if the bundle helps you in any way.


(DIY HA) #22

Thanks man! :+1:
Will definitely look into this.


(dalibor) #23

Hello, is there potential/initiative to get this as a plugin in (some of) the store?
Would love to get statistics on what is openhab doing inside prometheus


(Rich Koshak) #24

Since this relies on so many external services I don’t see how this could be made into a binding or add-on. It could probably be added to openHABian though.

Setting this up is still on my todo list but I don’t have a lot of time to work on additions to my system these days. I’m fighting a bit of bit rot at the moment that is taking a lot of my time (RPi that controls my garage has gone wonky, Wunderground is shutting off it’s free API, etc).


(friesenkiwi) #25

Yes, there is an initiative to team up with Deutsche Telekom and get the metrics into Eclipse Smart Home core. But this may take a little while.
Meanwhile, following the instructions above and in the readme over at GitHub, it is already possible to use our bundle. Just drop the .jar files into your addons directory.
The bundle only relies on the prometheus library bundles, which you can download and also just drop into the directory, it is not relying on any external service!


(Martin Tamke) #26

Hello @friesenkiwi - is the working jar still available on your github? the link of your above is deceased - https://github.com/KuguHome/openhab-prometheus-metrics/releases/download/v0.9/com.kuguhome.openhab.prometheusmetrics-2.4.0-SNAPSHOT.1.jar

I would like to test and feeback on your really promising approach.


(Rick) #27

have a look at https://github.com/KuguHome/openhab-prometheus-metrics/releases/


(Rick) #28

I finally got it up & running :slight_smile: As it was kind of a challenge, I’m posting the complete instruction here:

To get this cool dashboard in Grafana you need to:

  1. Have a working Prometheus setup. (See https://prometheus.io/docs/prometheus/latest/getting_started/).

  2. Have a working Grafana setup. (See InfluxDB+Grafana persistence and graphing).

  3. Download http://central.maven.org/maven2/io/prometheus/simpleclient/0.4.0/simpleclient-0.4.0.jar and place it in the addons directory. Monitor the Openhab log file and wait until the jar has successfully registered in Openhab.

2018-12-15 03:15:44.636 [INFO ] [org.apache.felix.fileinstall        ] - Installing bundle io.prometheus.simpleclient / 0.4.0
2018-12-15 03:15:44.881 [INFO ] [org.apache.felix.fileinstall        ] - Started bundle: file:/usr/share/openhab2/addons/simpleclient-0.4.0.jar
  1. Download http://central.maven.org/maven2/io/prometheus/simpleclient_common/0.4.0/simpleclient_common-0.4.0.jar and place it in the addons directory. Monitor the Openhab log file and wait until the jar has successfully registered in Openhab.
2018-12-15 03:16:14.901 [INFO ] [org.apache.felix.fileinstall        ] - Installing bundle io.prometheus.simpleclient_common / 0.4.0
2018-12-15 03:16:15.296 [INFO ] [org.apache.felix.fileinstall        ] - Started bundle: file:/usr/share/openhab2/addons/simpleclient_common-0.4.0.jar
  1. Download http://repo1.maven.org/maven2/io/prometheus/simpleclient_hotspot/0.4.0/simpleclient_hotspot-0.4.0.jar and place it in the addons directory. Monitor the Openhab log file and wait until the jar has successfully registered in Openhab.
2018-12-15 03:17:56.501 [INFO ] [org.apache.felix.fileinstall        ] - Installing bundle io.prometheus.simpleclient_hotspot / 0.4.0
2018-12-15 03:17:56.613 [INFO ] [org.apache.felix.fileinstall        ] - Started bundle: file:/usr/share/openhab2/addons/simpleclient_hotspot-0.4.0.jar
  1. Download the latest jar from https://github.com/KuguHome/openhab-prometheus-metrics/releases/ and place it in the addons directory. Monitor the Openhab log file and wait until the jar has successfully registered in Openhab.
2018-12-15 03:18:27.292 [INFO ] [vice.internal.HttpServiceFactoryImpl] - Binding bundle: [com.kuguhome.openhab.prometheusmetrics_2.4.0.201808140810 [312]] to http service
2018-12-15 03:18:27.402 [INFO ] [rg.ops4j.pax.web.utils.ClassPathUtil] - Ignoring bundle scan for /META-INF/services javax.servlet.ServletContainerInitializer.
2018-12-15 03:18:27.407 [INFO ] [ce.jetty.internal.HttpServiceContext] - registering context DefaultHttpContext [bundle=com.kuguhome.openhab.prometheusmetrics_2.4.0.201808140810 [312], contextID=default], with context-name:
2018-12-15 03:18:27.419 [INFO ] [.jetty.server.handler.ContextHandler] - Started HttpServiceContext{httpContext=DefaultHttpContext [bundle=com.kuguhome.openhab.prometheusmetrics_2.4.0.201808140810 [312], contextID=default]}
2018-12-15 03:18:27.481 [INFO ] [org.apache.felix.fileinstall        ] - Started bundle: file:/usr/share/openhab2/addons/com.kuguhome.openhab.prometheusmetrics-2.4.0-SNAPSHOT.jar
  1. If loading of the bindings went ok, you should be able to open http://openhabserver:8080/rest/metrics/prometheus and you will see a page like https://github.com/KuguHome/openhab-prometheus-metrics/wiki/Example-scrape . If an error occured in the previous steps, you’ll see a 404 error.

  2. Now add a job in Prometheus. I installed it today so I’m by no means an expert. In prometheus.yaml, add the following under scrape_configs:

  - job_name: 'openhab_metrics'
    scrape_interval: 10s
    scheme: http
    metrics_path: /rest/metrics/prometheus
    static_configs:
    - targets:
      - 'openhabserver:8080'
  1. After a Prometheus service restart, you can browse to http://prometheusserver:9090/targets, and the scraper is up:
## [openhab_metrics (1/1 up)](http://prometheusserver:9090/targets#job-openhab_metrics)

|Endpoint|State|Labels|Last Scrape|Scrape Duration|Error
|http://openhabserver:8080/rest/metrics/prometheus|UP|instance="openhabserver:8080" job="openhab_metrics"|8.967s ago|78.19ms|
  1. Now download the Grafana dashboard from https://github.com/KuguHome/openhab-prometheus-metrics/wiki/Grafana-source and replace the data source “kugu-prometheus” to the correct data source. For me, it was just “Prometheus”.

  2. Import the dashboard in Grafana. Grafana shouldn’t complain about missing data sources.

I’m really impressed by the amount of insight you get when using this dashboard. Thanks, @friesenkiwi !


(Simon Tennant) #29

Great instructions.

Is anyone aware of a similar Prometheus exporter to scrape item values. It would be awesome to be able to expose values like power consumption or lumens or or water flow… to Prometheus’ calculation functions. I did see one approach that could work via exporting values first to MQTT.

https://github.com/hikhvar/mqtt2prometheus looks like it comes somewhat close but since Prometheus need to know whether each metric is a gauge or a counter it would require that each metric is individually configured.


(Rick) #30

You should be able to go via InfluxDB -> Prometheus (https://www.influxdata.com/blog/influxdb-now-supports-prometheus-remote-read-write-natively/) or indeed the MQTT way.


(friesenkiwi) #31

Oh, wow I’m very sorry I didn’t reply in time, somehow I didn’t get or notice the forum mail notifications :frowning:

Thanks @ricadelic for the detailed instructions. Actually it should be no problem to place all the .jar files into the addons folder at the same time or at a random order. @mdnx the link https://github.com/KuguHome/openhab-prometheus-metrics/releases/ as @ricadelic posted it is the right place to look. The bundle is currently running in several field systems for us in beta state, development is somewhat paused but we are very interested in feedback!

@imaginator I am not aware of a similar approach for a direct item Prometheus exporter.From our point of view, the InfluxDB or any other persistence plugin would be the “correct” way to get the item values (going MQTT->Prometheus seems one step too much for me). We instead wanted to get values/metrics, which were (until now) hidden. Also, to our mind, Prometheus is more of a health metrics monitoring tool and less of a general value scraper. But you can certainly use it that way. It also shouldn’t be too much of a problem, to write an “Exposable” along the lines of https://github.com/KuguHome/openhab-prometheus-metrics/blob/master/com.kuguhome.openhab.prometheusmetrics/src/main/java/com/kuguhome/openhab/prometheusmetrics/exposable/OpenHABThingStateMetric.java which queries the item registry instead of the thing registry and outputs their values.
If it would be “switchable” within the bundle, we could include it, if you do a PR on GitHub.


(Rich Koshak) #32

I haven’t set this this up myself yet, but my understand is that Prometheus users Grafana as the display. So there shouldn’t be anything stopping you from adding graphs of stuff saved to InfluxDB through Persistence to the same Graphana dashboard.


(friesenkiwi) #33

Grafana can display all kinds of data sources, including MySQL, Prometheus or InfluxDB, you can mix them on the same board or even on the same graph.


(Simon Tennant) #34

My thinking on Prometheus is less about the graphing and more about the Prometheus query functions. I’d love to be able to detect water leaks by looking for sustained low values. Or predict energy consumption for the rest of the month. Or hook into the Prometheus alert manager … Or… :grinning:


(friesenkiwi) #35

That’s actually a nice idea. As I said, it should be pretty easy to implement.


(Simon Tennant) #36

…next level awesome would be the ability to include PromQL in rules.

If prometheus(sum(count_over_time(basement-humidity[60m])), then...

(DIY HA) #37

Very interesting suggestion :+1::+1: