Grafana: stop to manage your dashboards manually via UI! (dashboad auto generation, part 2)

Tutorials & Examples Solutions
,
1

Hello all!

Intro

This is the second part of article Grafana: stop to manage your dashboards manually via UI! (dashboad auto generation, part 1) .

In part 1 we used official tool for managing Grafana dashboards: Grafonnet, whihc makes you ready dashboard from jsonnet templating system. This method is relative easy to use and create dashboards, but has some limitations:

  • Not all code is reusable really.
  • Limited loops, you can only generate targets, but not other items.
  • No connection with Openhab internals, you can’t loop over groups → you need to manage items lists manually.

Okay, what we can do, if we have really many items in our setup? Let’s use Python!

We can use general scripting language to build our dashboards using one of Grafana libraries. Let’s use grafanalib, it provides interface not more complex than Grafonnet.

For openhab we also can choose one of existing bindings, let’s use python-openhab.

Installation

Now prepare your setup with modules:

pip3 install python-openhab grafanalib

Dashboard

Create your dashboard.py file with basic init: imports, OpenHAB object and upload function:

# Include python-openhab
from openhab import OpenHAB
# Include all from grafanalib.core with alias grafana
import grafanalib.core as grafana
# And InfluxDB from submodule
from grafanalib.influxdb import InfluxDBTarget
# Generator for JSON
from grafanalib._gen import DashboardEncoder
# And some common functions for upload
import json
import requests
from os import getenv

# This function uploads dashboard, reading env:
# GRAFANA_URL, GRAFANA_USERNAME, GRAFANA_PASSWORD
def upload_to_grafana(dashboard):
    json = json.dumps(
        {
            "dashboard": dashboard.to_json_data(),
            "overwrite": True
        }, sort_keys=True, indent=2, cls=DashboardEncoder)

    server = getenv("GRAFANA_URL")
    username = getenv("GRAFANA_USERNAME")
    password = getenv("GRAFANA_PASSWORD")

    headers = {
        'Content-Type': 'application/json'}
    r = requests.post(
        f"{server}/api/dashboards/db",
        data=json,
        headers=headers,
        auth=requests.auth.HTTPBasicAuth(username, password)
    )
    print(f"{r.status_code} - {r.content}")

# Create OpenHAB object with your URL from OPENHAB_URL env
openhab = OpenHAB(getenv("OPENHAB_URL"))

Simple dashboard

Now we can define our first simple one:

# Create dashboard for 1 item (temperature):
dashboard = grafana.Dashboard(
    title="Test 1",
    uid='openhab_test_1',
    tags=['openhab'],
    timezone='browser',
    time=grafana.Time('now-24h', 'now'),
    rows=[
        grafana.Row(panels=[
            grafana.Graph(
                title="Temperature",
                dataSource='openhab_home',
                lineWidth=1,
                yAxes=grafana.single_y_axis(min=None, format='°C'),
                targets=[
                    InfluxDBTarget(
                        query=f'SELECT mean("value") FROM "ext_climate_temperature" WHERE $timeFilter GROUP BY time($__interval) fill(previous)',
                        alias='Temperature',
                    ),
                ],
            ),
        ]),
    ],
).auto_panel_ids() # Call this to have valid dashboard JSON

# Upload it
upload_to_grafana(dashboard)

This will create dashboard with one graph on it, reading values from one static item.

Screenshot_20210927_202617
Screenshot_20210927_2026171819Ă—383 28.6 KB

Dashboard with double y-axes

Also, this approach hard to google, but solution is simple: you need to define 2 y-axes and add override block to pin our items to it (same as we did in part 1):

# Create dashboard for 2 items (temperature and humidity with 2 axis):
dashboard = grafana.Dashboard(
    title="Test 2",
    uid='openhab_test_2',
    tags=['openhab'],
    timezone='browser',
    time=grafana.Time('now-24h', 'now'),
    rows=[
        grafana.Row(panels=[
            grafana.Graph(
                title="Temperature",
                dataSource='openhab_home',
                # Define 2 axes for our items
                yAxes=grafana.YAxes(
                    left=grafana.YAxis(min=None, format='°C'),
                    right=grafana.YAxis(min=None, format='%'),
                ),
                # Important: Force graph's colors and Y-Axis
                seriesOverrides=[
                    {
                        'alias': "Temperature",
                        'color': '#bf1b00',
                        'yaxis': 1,
                    },
                    {
                        'alias': "Humidity",
                        'color': '#65c5db',
                        'yaxis': 2,
                    },
                ],
                targets=[
                    InfluxDBTarget(
                        query=f'SELECT mean("value") FROM "ext_climate_temperature" WHERE $timeFilter GROUP BY time($__interval) fill(previous)',
                        alias='Temperature',
                    ),
                    InfluxDBTarget(
                        query=f'SELECT mean("value") FROM "ext_climate_humidity" WHERE $timeFilter GROUP BY time($__interval) fill(previous)',
                        alias='Humidity',
                    ),
                ],
            ),
        ]),
    ],
).auto_panel_ids()  # Call this to have valid dashboard JSON

# Upload it
upload_to_grafana(dashboard)

This will generate double-axes and double-metrics dashboard.

Screenshot_20210927_202941
Screenshot_20210927_2029411819Ă—383 48 KB

Real automation

“Okay, this is same as Jsonnet-way, but where is real automation, Peter?” - you may ask. Let’s do it.

Read Openhab items group and create list of targets (one per item). I have special group named g_battery_level whohc holds all items, who reports some battery persentage, we can create summary graph:

# Read items group and create dashboard with contents
dashboard_targets = [] # Final array with targets for all items
group = openhab.get_item('g_battery_level') # Request group from OPenHAB API
# Iterate group
for v in group.members.values():
    # Get some item info
    item_info = openhab.get_item_raw(v.name)
    # Get item display name
    item_label = item_info['label']
    # ... create target!
    dashboard_targets.append(
        InfluxDBTarget(
            query=f'SELECT mean("value") FROM "{v.name}" WHERE $timeFilter GROUP BY time($__interval) fill(previous)',
            alias=item_label,
        ),
    )

# Now just create and upload dashboard with result targets attached
dashboard = grafana.Dashboard(
    title="Test 3",
    uid='openhab_test_3',
    tags=['openhab'],
    timezone='browser',
    time=grafana.Time('now-24h', 'now'),
    rows=[
        grafana.Row(panels=[
            grafana.Graph(
                title="Battery level",
                dataSource='openhab_home',
                lineWidth=1,
                fill=False, # Do not fill graph
                yAxes=grafana.single_y_axis(min=None, format='%'),
                targets=dashboard_targets,
            ),
        ]),
    ],
).auto_panel_ids()  # Call this to have valid dashboard JSON

# Upload it
upload_to_grafana(dashboard)

This will create dashboard with all items in one graph.

Screenshot_20210927_203604
Screenshot_20210927_2036041819Ă—383 41 KB

Dashborad with latests status

Sometimes it is usefull to have a look on current status. Lets disaply nice summary of all batteries in our setup.

We wiil request same group, but target will use last funcrion instead of mean and we will remove WHERE (defining time-filtering). As result, our dashboard will show latest values always, ignoring Grafana date selector.

# Read items group and create dashboard with contents
dashboard_targets = []  # Final array with targets for all items
group = openhab.get_item('g_battery_level')  # Request group from OPenHAB API
# Iterate group
for v in group.members.values():
    # Get some item info
    item_info = openhab.get_item_raw(v.name)
    # Get item display name
    item_label = item_info['label']
    # ... create target!
    dashboard_targets.append(
        InfluxDBTarget(
            query=f'SELECT last("value") FROM "{v.name}" ORDER BY time DESC LIMIT 1 SLIMIT 1',
            alias=item_label,
        ),
    )

# Now just create and upload dashboard with result targets attached
dashboard = grafana.Dashboard(
    title="Test 4",
    uid='openhab_test_4',
    tags=['openhab'],
    timezone='browser',
    time=grafana.Time('now-24h', 'now'),
    rows=[
        grafana.Row(panels=[
            grafana.GaugePanel( # Panel with indicators
                title="Battery level",
                dataSource='openhab_home',
                format='%',
                # Define colors for values
                thresholds=[
                    {'color': 'red', 'value': 0},
                    {'color': 'yellow', 'value': 20},
                    {'color': 'green', 'value': 40},
                ],
                targets=dashboard_targets,
            ),
        ]),
    ],
).auto_panel_ids()  # Call this to have valid dashboard JSON

# Upload it
upload_to_grafana(dashboard)

Now what we have!

Screenshot_20210927_203827
Screenshot_20210927_2038271819Ă—383 85.1 KB

Outro

To finally use this, just call something like:

export GRAFANA_URL=http://grafana.local:3000
export GRAFANA_USERNAME=admin
export GRAFANA_PASSWORD=password
export OPENHAB_URL=http://openhab.local

python3 dashboad.py

This will generate dashboards for you and upload them!

Hope this may help you to avoid any manual work in future, you can by-easy maintainn dashboards with hundreds of items, have modern configuration-as-code approach and share your findings.

You dont need to change your dashboards in 50 places anymore, if you added new Zigbee device, for example, just manage device groups properly in Openhab as usial, and you could just call script again, and all your dashboards will be updated in one second.

Example for this lession: dashboard.py.txt (6.4 KB)

My setup: @petrows / saltstack-config/dashboards

Screenshot_20210927_075944
Screenshot_20210927_0759441812Ă—730 136 KB

Previous part: Grafana: stop to manage your dashboards manually via UI! (dashboad auto generation, part 1)

1 Like