Extract data from Stiebel Eltron ISG using REGEX, InfluxDB and Grafana - Tutorial

[ThierryS (Thierry Stucker)]

Dear all,

I have been inspired by many posts and also I could not find a straight forward solution to my need, which was to extract data from my heat pump (Stiebel Eltron ISG), to store the data in a database and be able to render them into graphs. Please note that you can use this type of process to extract and record data from other devices displaying web pages or also allowing API calls such as sonnenBatterie.

I am using a solution that combines openHAB to extract the data, InfluxDB to store them and Grafana to display them into graphs.

I wanted to give back to the community what I’ve learned hoping that it will be useful to somebody. This is a step-by-step guide to install the entire suite of packages and configuration.

As this is my first post and one of my first project, it contains certainly some errors and I am glad to receive feedback.

Here is a screenshot of what the results look like:

Install openHAB, InfluxDB, Grafana to read, record Stiebel Eltron ISG data and present them graphically

Installation and configuration of openHAB on a Raspberry Pi 3 (RPI)

The principle is that I am using openHAB to perform “web scrapping” of the ISG page through which you can read the data of your Stiebel Heat Pump. For that I am using REGEX (see later notes on the fact that REGEX expressions may depend on the operating system, i.e. in my case Raspian (linux))

It is important to make your ISG “password less” (no user name and no password), meaning that when you type the IP address of your ISG you should be able to see the web page and the actual data directly. I couldn’t find a way to include credentials in a REGEX call. But as the device is on my network I thought it is not a major security issue. I’ll improve this when I will find or someone suggets a solution. If you are not comfortable making your ISG “pasword less” this guide does not work. From memory, you go in the ISG: Profile / Safety and blank out User Name and Password. If this is not the case, REGEX will obviously return a null data as there is nothing to read.

How does it work?

• home.items: I am creating an item for each data I want to retrieve from the ISG page. At this stage, each item is in a string format. For example OutTempStr
• This item item is followed by a second item which, will be the equivalent in number format e.g. OutTempNum
• As you can see you have for a pair of two items for each data element, which are named xxxStr for string and xxxNum for number; examples are: OutTempStr, OutTempNum
• The conversion string to number occures in home.rules
• These variables are then used to record their historical data values in InfluxDB and Grafana reads the data in InfluxDB to represent them graphically.

Also note that I will be using the following in this guide. Obvioulsy adjust with the actual IP adresses of your own devices:

• 192.168.1.125 refers to the IP address of the Raspberry PI
• 192.168.1.132 refers to the IP address of the ISG page (Stiebel)

Initial installation

Refer to the openHABian installation procedure at https://docs.openhab.org/installation/. Please note that this guide was developped in September 2017 and menus, options may change over time.

openHAB initial configuration

• Once openHAB has been installed on the RPI, obtain the IP address from the RPI, it is visible after logging. For the remaining part of this guide I will use the following IP for illustration purposes: 192.168.1.125
• Go to http://192.168.1.125:8080
• Select the “Expert” package (it takes approx. 2 mins)
• Go to Putty and open an SSH connection to the RPI using the 192.168.1.125 IP address
• User is “openhabian” and password is still the default one: “openhabian”
• Once logged in, type sudo openhabian-config
• Reboot the RPI by typing sudo reboot
• Once rebooted, from the menu, select “Update”
• Once done, select “Upgrade Systems” from the menu
• Go to “Apply Improvements” and perform each improvements listed in the sub-menu, some time you will need to type exit when the process seems to hang
• Exit “openhabian-config”
• Once back in the command line, type passwd .This will change the password for openhabian
• Once completed, type, sudo smbpasswd openhabian . This will change the smb password for openhabian
• Type sudo timedatectl set-timezone for example Europe/Zurich (note type timedatectl list-timezones to get the available time zones)
• Type timedatectl to check the set timezone
• Type sudo reboot

OpenHAB setup of Items, Rules, Sitemap

• Go to the openHAB web interface, i.e. http://192.168.1.125:8080
• Go to “PAPER UI”
• In the menu (top left corner) select “Add-ons”
• Go to “BINDINGS” and install “HTTP Bindings”
• log:tail can be accessed by typing http://192.168.1.125:9001 check that the binding has been installed and refresh the page

Caching the ISG pages

• Go into the RPI by accessing SSH
• Type sudo nano /etc/openhab2/services/http.cfg
• Copy paste the two http links (see examples at the end of this topic)
• This caches the two pages and avoids calls to the ISG for each items value retrievals

Creating Items

• Type sudo nano /etc/openhab2/items/home.items
• Copy paste the items list (see examples at the end of this topic)

Creating Rules

• Type sudo nano /etc/openhab2/rules/home.rules
• Copy paste the rules list (see examples at the end of this topic)

Creating Sitemap

• Type sudo nano /etc/openhab2/sitemaps/home.sitemap
• Copy paste the sitemap list (see examples at the end of this topic)
• Setting the default indications for “CLASSIC UI” and “BASIC UI”. This will tell openHAB, which files to look for Items, Rules and Sitemap
• Go to “PAPER UI” and in the top left corner menu “Configuration” and “Services”
• Go to “Basic UI” and select “CONFIGURE”
• Under Default Sitemap type “home” and SAVE
• Go to “Classic UI” and select “CONFIGURE”
• Under Default Sitemap type “home” and SAVE
• Go to http://192.168.1.125:8080 and the items configured should appear

Influxdb and Grafana installation and configuration

Installation

• SSH into the RPI
• Type sudo openhabian-config
• Select from the menu: “Optional Components”
• Select “Grafana” in the sub-menu and install it
• This will install both Influxdb (timeseries database) and Grafana (graphical representations from data store into Influxdb)

Influxdb set-up

• First install the openHAB Influxdb persistence
• Go to http://192.168.1.125:8080 and then “PAPER UI”, “Add-ons”, “PERSISTENCE” and install “Influxdb Persistence”
• Verify correct installation in the log (http://192.168.1.125:9001). If it fails uninstall and reinstall the persistence
• Go into RPI using SSH and type sudo service influxdb start

Let’s create the Influxdb database, users and grant rights

• Type influx
• Type create database stiebel, “stiebel” is the name of the database to create, you can name it as you want but you will need to provide the exact same same in the later instructions and set-up
• Type show databases to verify that the database “stiebel” has been created
• Now we are going to create the users and assign priviledges
• Type create user admin with password 'admin' with all privileges
• Type create user openhab with password 'openhab'
• Type show users to verify that all users got created with the adequate access rights
• Type grant write on stiebel to openhab, this grants write privilege on “stiebel” database to “openhab” user
• Type create retention policy "default" on "stiebel" duration inf replication 1 default
• Type show retention policies on stiebel
• The Influxdb database and its users are now created. Type exit to return to the linux command line

Influxdb settings and connection to openHAB

• Type sudo nano /etc/influxdb/influxdb.conf
• Uncomment the following lines in the http section
  [http] enabled = true
  bind-address = “:8086”
  auth-enabled = false
• Restart the InfluxDB service: sudo systemctl restart influxdb.service
• Edit the Influxdb configuration file sudo nano /etc/openhab2/services/influxdb.cfg
• Uncomment and complete the following lines:
  url=http://localhost:8086
  user=openhab
  password=openhab
  db=stiebel
  retentionPolicy=default
• Create the persistence file sudo nano /etc/openhab2/persistence/influxdb.persist
• Create the persistence policy (see examples at the end of this topic):
• openHAB starts now to write data in the stiebel database
• Verify that records are being written to the database and type as follows:
  type influx
  type use stiebel
  type show measurements select one of the variable that is likely to have changed recently and to have recorded some transactions e.g. OutTempNum
  type select * from OutTempNum
• At this stage Influxdb is fully set up and openHAB writes into the database, using the variables at the frequency defined in the persistence policy influxdb.persist

Grafana set-up

• Grafana can be accessed at http://192.168.1.125:3000 using the default login user id: “admin”, password: “admin”
• Once logged, select “add data source” and perform the following set up:
• In the “Config” section, go to the “Type” line and select “Inlfuxdb” from the drop down list
• In the “Http settings” section, in the Url line insert http://localhost:8086 note that Access should be proxy
• In the “Http Auth” section, click on “Basic Auth” to enable it
• In the “Basic Auth Details” section, input User: “openhab” with Password: “openhab”
• In the “InfluxDB Details” section, input Database: “stiebel”, User: “openhab” with Password: “openhab”
• Finally enter “Stiebel” at the top of the page in the line Name
• Click on “Save & Test” at the bottom of the page and it should indicate “Success, Data source is working” as confirmation
• Grafana is now able to read in the Influx database
  From the top left menu, select “Dashboards” and then “+ New” and insert charts

The installation is now completed !

Examples

http.cfg

In this example, the IP address 192.168.1.132 represents the ISG device’s address.

ISG10.url=http://192.168.1.132/?s=1,0
ISG10.updateInterval=60000

ISG11.url=http://192.168.1.132/?s=1,1
ISG11.updateInterval=300000

home.items

Please note as I could not find a way to add credentials to the REGEX request, ISG should be “password less”. If you are not be comfortable having your ISG without credentials, then this procedure does not work. In my case, it is fine and I will improve it when I’ll find a solution or someone suggests one. To make ISG “password less”, go to “Profile” then “Safety” and blank out “User name” and “password”

Please also note that I found that REGEX works differently depending on the operating system. The following expressions work on a Raspberri Pi with raspian. It is highly likely that different operating systems will require to adjust the REGEX expressions. I could not find good enough manual to explain why REGEX is expecting different expressions, especially when or not to use the “\”. But i can confirm that the following examples work on my configuration.

Group gAll
Group gMeteo (gAll)
Group gStiebel (gAll)

//Meteo
String OutTempStr "Temperature exterieure [%s°C]"   { http="<[ISG10:60000:REGEX(.*<td class=\"key\">OUTSIDE TEMPERATURE</td>.*?<td class=\"value\">(-?[\\d]+,[\\d]).*?</td>.*)]" }
Number OutTempNum "Temperature exterieure [%.1f°C]" <temperature> (gMeteo,gAll)

//Stiebel Sensors
String HeatPumpDepStr   "PAC temp depart [%s°C]"    { http="<[ISG10:60000:REGEX(.*<td class=\"key\">ACTUAL FLOW TEMPERATURE WP</td>.*?<td class=\"value\">(-?[\\d]+,[\\d]).*?</td>.*)]" }
Number HeatPumpDepNum   "PAC temp depart [%.1f°C]"  <boiler_viessmann> (gStiebel,gAll)

String HeatPumpRetStr   "PAC temp retour [%s°C]"    { http="<[ISG10:60000:REGEX(.*<td class=\"key\">ACTUAL RETURN TEMPERATURE</td>.*?<td class=\"value\">(-?[\\d]+,[\\d]).*?</td>.*)]" }
Number HeatPumpRetNum   "PAC temp retour [%.1f°C]"  <boiler_viessmann> (gStiebel,gAll)

String HeatPumpPressStr "PAC pression [%s bar]"     { http="<[ISG10:60000:REGEX(.*<td class=\"key\">PRESSURE HTG CIRC</td>.*?<td class=\"value\">([\\d]+,[\\d]+).bar.*?</td>.*)]" }
Number HeatPumpPressNum "PAC pression [%.2f bar]"   <boiler_viessmann> (gStiebel,gAll)

String HeatPumpVolStr   "PAC debit [%s l/min]"      { http="<[ISG10:60000:REGEX(.*<td class=\"key\">FLOW RATE</td>.*?<td class=\"value\">([\\d]+,[\\d]).*?</td>.*)]" }
Number HeatPumpVolNum   "PAC debit [%.1f l/min]"    <boiler_viessmann> (gStiebel,gAll)

String HeatPumpDailyStr "PAC energie [%s kWh]"      { http="<[ISG11:300000:REGEX(.*<td class=\"key\">COMPRESSOR HEATING DAY</td>.*?<td class=\"value\">([\\d]+,[\\d]+).*?</td>.*)]" }
Number HeatPumpDailyNum "PAC energie [%.3f kWh]"    <boiler_viessmann> (gStiebel,gAll)

String HeatPumpOutputHPStr "PAC Output HP [%s pc]"   { http="<[ISG10:60000:REGEX(.*<td class=\"key\">OUTPUT HP</td>.*?<td class=\"value\">([\\d]+).*?</td>.*)]" }
Number HeatPumpOutputHPNum "PAC Output HP [%d %%]" <boiler_viessmann> (gStiebel,gAll)

String HeatPumpIntRateStr "PAC Int Pump Rate [%s %%]" { http="<[ISG10:60000:REGEX(.*<td class=\"key round-leftbottom\">INT PUMP RATE</td>.*?<td class=\"value round-rightbottom\">([\\d]+).*?</td>.*)]" }
Number HeatPumpIntRateNum "PAC Int Pump Rate [%d %%]" <boiler_viessmann> (gStiebel,gAll)

home.rules

//Outside Temperature conversion String to Number
rule "OutTemp String to Number conversion"
when
    Item OutTempStr changed
then
    OutTempNum.postUpdate(Float::parseFloat(String::format("%s",OutTempStr.state).replace(',','.')));
end


///Heat Pump Actual Flow Temperature conversion String to Number
rule "HeatPumpDep String to Number conversion"
when
    Item HeatPumpDepStr changed
then
    HeatPumpDepNum.postUpdate(Float::parseFloat(String::format("%s",HeatPumpDepStr.state).replace(',','.')));
end


///Heat Pump Actual Return Temperature conversion String to Number
rule "HeatPumpRet String to Number conversion"
when
    Item HeatPumpRetStr changed
then
    HeatPumpRetNum.postUpdate(Float::parseFloat(String::format("%s",HeatPumpRetStr.state).replace(',','.')));
end


///Heat Pump Pressure HTG CIRC conversion String to Number
rule "HeatPumpPress String to Number conversion"
when
    Item HeatPumpPressStr changed
then
    HeatPumpPressNum.postUpdate(Float::parseFloat(String::format("%s",HeatPumpPressStr.state).replace(',','.')));
end


///Heat Pump Flow Rate conversion String to Number
rule "HeatPumpVol String to Number conversion"
when
    Item HeatPumpVolStr changed
then
    HeatPumpVolNum.postUpdate(Float::parseFloat(String::format("%s",HeatPumpVolStr.state).replace(',','.')));
end


//Heat Pump Compressor Heating Day String to Number
rule "HeatPumpDaily String to Number conversion"
when
    Item HeatPumpDailyStr changed
then
    HeatPumpDailyNum.postUpdate(Float::parseFloat(String::format("%s",HeatPumpDailyStr.state).replace(',','.')));
end


///Heat Pump Output HP conversion String to Number
rule "HeatPumpOutputHP String to Number conversion"
when
    Item HeatPumpOutputHPStr changed
then
    HeatPumpOutputHPNum.postUpdate(Float::parseFloat(String::format("%s",HeatPumpOutputHPStr.state)));
end


///Heat Pump Int Rate conversion String to Number
rule "HeatPumpIntRate String to Number conversion"
when
    Item HeatPumpIntRateStr changed
then
    HeatPumpIntRateNum.postUpdate(Float::parseFloat(String::format("%s",HeatPumpIntRateStr.state)));
end

home.sitemap

sitemap home label="House"
{
        Frame label="Meteo"
        {
                Text item=OutTempNum
        }
        Frame label="Stiebel Eltron"
        {
                Text item=HeatPumpDepNum
                Text item=HeatPumpRetNum
                Text item=HeatPumpPressNum
                Text item=HeatPumpVolNum
                Text item=HeatPumpDailyNum
                Text item=HeatPumpOutputHPNum
                Text item=HeatPumpIntRateNum
        }
}

influxdb.persist

Strategies {
    everyHour     :  "0 0 * * * ?"
    everyDay      :  "0 0 0 * * ?"
    everyMinute   :  "0 * * * * ?"
    every5Min     :  "0 */5 * * * ?"
    every5Sec     :  "*/5 * * * * ?"

// If no strategy is specified for an item entrey below, the default list will be used
default = everyMinute
}

Items{
    OutTempNum                         : strategy = everyMinute
    HeatPumpDepNum, HeatPumpRetNum     : strategy = everyMinute
    HeatPumpPressNum, HeatPumpVolNum   : strategy = everyMinute
    HeatPumpOutputHPNum                : strategy = everyMinute
    HeatPumpIntRateNum                 : strategy = everyMinute
    HeatPumpDailyNum                   : strategy = every5Min
}

Revision history

• December 2017: original post
• August 2018
  . included improvement formating recommendations from rlkoshak
  . included improvement from Thomas Graph to “cache” the Stiebel web pages, reducing the calls to the ISG pages

[rlkoshak (Rich Koshak)]

@ThierryS

Download the latest Openhab2 image

Change to latest openHABian image. There are many other ways to download and install openHAB and this line could be confusing to some users. Even I had to read down to see your username/password to know for sure what exactly you meant.

It’s a minor quibble, but the proper spelling is “openHAB” with the lowercase “o” in all cases. It doesn’t matter, but this is such an excellent and thorough writeup that using “OpenHAB” keeps jumping out of the page at me.

@ThierryS

Grafana set-up

You missed bolding this one.

Note that you can create headings using #

# Heating 1

Heading 1

## Heading 2 

Heading 2

etc.

@ThierryS

Items example

Please please please use code fences.

```
code goes here
```

When/if you go back to add code fences, double check your quotes. The form will automatically convert " " to " " (i.e smart quotes) which OH cannot handle causing copy and paste into OH config files to not work.

Excellent writeup. Thanks for contributing!

[GrafT (Thomas Graf)]

Hi,

thank you for sharing your work.

[ThierryS (Thierry Stucker)]

Many thanks Rich, it took me some time to come back and improve the guide. Thanks to your excellent suggestions, it should be now easier to read and the parameters example are now correctly represented.

Thanks a lot for your suggestions,

Thierry Stucker

[ThierryS (Thierry Stucker)]

A big thank you Tom for suggesting to “cache” the http calls to the ISG web pages !! I have improved the instruction including your recommendation

[shorty707]

Please Note that all ISG Web can have a Modbus Addon for free.
The Former expensive addon from Stiebel is now made a free addon.

Maybe more reliable?

[Stefan83]

yes, but ISGweb can’t have Modbus and EMI simultaneously.
(EMI for connecting with photo voltaic)

[GrafT (Thomas Graf)]

Hi, there is an Openhab Binding for reading ISG over Modbus: Stiebel Eltron ISG - Bindings | openHAB
It’s way more easy to read the data now!

[shorty707]

very cool.

• does it distiguish between different WP Systems? (WP3, WP3i, WP System) ?

• does it distinguish between Heatpump and HEatpump with integrated ventilation?

• There is more data available via modbus.

• Is it just not in the documentation but the channel is available when I install the binding?

Cheers

edit: now you should buy a Zehnder Q350 with LAN Module waiting for a binding there since years

[rene1879 (rene)]

seems so, but the binding is not listed in OH3.

[robd (anonym)]

First install Modbus (TCP Slave) binding, there you can add the Stiebel Eltron ISG

[rene1879 (rene)]

Modbus is installed. But where ‘there’ I can add Stiebel Eltron ISG binding then?

[robd (anonym)]

I hope I can remember it correctly:
Add thing -> choose binding modbud -> select modbus:heatpump (Stiebel Eltron ISG)

[rene1879 (rene)]

not really.

[robd (anonym)]

yes, install Modbus first.
Then add item where you select Modbus as its binding, then you get a list where you can select Stiebel Eltron