Tutorial for simple Nefit Easy setup


(Ron) #1

OK, tutorial might be overstating it a little, but at least it is a short overview of steps to setup the prerequisites and some rules to process the API data…

Nefit Easy central heating solution

Nefit is a Dutch brand of the Robert Bosch Group that develops and produces Central Heating and Hot Water systems, running on natural gas.
There is currently no binding available (@marcel_verpalen did a first alpha version some time ago, but development ‘stalled’) and I am not yet ready to develop a binding myself, so I resorted to existing solutions and OH rules to build a temporary solution.
I figured this might be useful for others, I decided to create a short post describing the steps and OH setup to make the ‘magic’ happen.

Nefit uses the Nefit Easy label for their domestic smart heating solution.
It consists of a smart thermostat and an online system, accessible via an Android and iOS App, and a non-published API. When you buy the Nefit Easy thermostat you get an account to access the API.
Like most API’s, there are limitations to the number of connections and/or requests you can make.
Robert Klep did a great job reverse engineering the API and created the node.js based easy-server tool that runs as a service and keeps a single connection open for all requests.
Via simple HTTP GET commands to this service all kinds of information can be retrieved and - although I did not yet implement it - it is also possible to change settings like temperature setpoint.

Note: Nefit Easy is a different platform and not compatible with Bosch Smart Home system.

Setup the easy-server

Not much to it, as can be seen in the README on Github.
The basic steps are (alternatively you can use a prepared Docker container):

  1. Install node.js (version 6 or higher, I used 10 TLS).
  2. Run npm i nefit-easy-http-server -g to install the easy-server globally on the system.
  3. Set 3 environment variables to hold the login info (the same info used in the App):
    NEFIT_SERIAL_NUMBER=
    NEFIT_ACCESS_KEY=
    NEFIT_PASSWORD=
  4. Start the service with easy-server --port 4000 (default is 3000, but Grafana is already using that in my setup).
  5. Test if it works with the command curl http:/localhost:4000/api/status (or use your hostname/IP address).
  6. If all works well, you can make the service auto-start.

Define the items

You should define a few items to hold the information retrieved from the Nefit API.
I have selected a few, but there are other items to choose from.

Number:Temperature      NE_Temperature      "Kamertemperatuur [%.1f °C]"        <temperature>   (gRestore)
Switch                  NE_HotWaterActive   "Warmwater"                                         (gHistory)
Number:Temperature      NE_SetPoint         "Ingestelde temperatuur [%.1f °C]"  <heating>       (gPersist,gTemp,gLivingroom)
Number:Pressure         NE_Pressure         "CV waterdruk [%.1f bar]"                           (gHourly,gRestire,gRestore)
// Indicates whether the burner is active
Switch                  NE_CvActive         "Nefit CV ketel"                    <fire>          (gRestore)
// Helper item to retry the call to easy-server after 2 minutes
Switch                  NE_GasRetry         {expire="2m,command=OFF"}

Create the rules

There are a few rules defined to get the data from the Nefit Easy API (via easy-server).
The first rule runs every 30 seconds to get the latest status for the items defined above. And although the Nefit API has limits, using the easy-server as intermediate system, works flawless at that request rate for more than a month now.

// Nefit Easy Server URLs
val String nefitApi = "http://localhost:4000/api/"
val String nefitBridge = "http://localhost:4000/bridge/"

//----------------------------------------------------------------------------------------
// Get the Nefit CH/HW system status every 30 seconds (from the easy-server daemon).
// See https://github.com/robertklep/nefit-easy-http-server.
// The API command '/api/status' returns the following JSON result:
// {    "user mode":"clock", "clock program":"auto","in house status":"ok",
//      "in house temp":21.2,
//      "hot water active":false,
//      "boiler indicator":"off","control":"room",
//      "temp override duration":0,
//      "current switchpoint":8,
//      "ps active":false,"powersave mode":false,
//      "fp active":false, "fireplace mode":false,
//      "temp override":true,
//      "holiday mode":false,
//      "boiler block":null, "boiler lock":null, "boiler maintenance":null,
//      "temp setpoint":20.5,"temp override temp setpoint":20.5,"temp manual setpoint":20,
//      "hed enabled":null, "hed device at home":null,
//      "outdoor temp":3, "outdoor source type":"virtual"
// }
// The info retrieved is store in the following Items:
//  NE_Temperature      - The current measured room temperature
//  NE_SetPoint         - The set temperature
//  NE_Pressure         - The pressure in the central heating pipes (in bar)
//  NE_CvActive         - Switch indicating the CV is active
//  NE_BurnerActive     - Switch indicating the Central Heating is active
//  NE_HotWaterActive   - Switch indicating the Hot Water is active
//----------------------------------------------------------------------------------------
rule "Nefit Easy status"
    when
        Time cron "0/30 * * * * ?"
    then
        val String tag = "Nefit.Status.Change"

        // Get Nefit Easy temperature and setpoint info in JSON format from the easy-server API.
        var String json = sendHttpGetRequest(nefitApi+"status")
        if (json == NULL || json == "")
            logWarn(tag, "Empty API Status response, retrying")
        else {
            // Extract the relevant parameters ('in house temp' and 'temp setpoint') from the JSON response message.
            NE_Temperature.postUpdate(transform("JSONPATH", "$.['in house temp']", json))
            NE_SetPoint.postUpdate(transform("JSONPATH", "$.['temp setpoint']", json))
        }

        json = sendHttpGetRequest(nefitBridge+"system/appliance/systemPressure")
        if (json == NULL || json == "")
            logWarn(tag, "Empty API Pressure response, retrying")
        else
            // Format: {"id":"/system/appliance/systemPressure","type":"floatValue","recordable":0,"writeable":0,"value":1.9,"unitOfMeasure":"bar","minValue":0,"maxValue":25}
            // Extract the relevant data (the 'value' parameter) from the JSON response message
            NE_Pressure.postUpdate(transform("JSONPATH", "$.value", json))

        json = sendHttpGetRequest(nefitBridge+"ecus/rrc/uiStatus")
        if (json == NULL || json == "")
            logWarn(tag, "Empty API UI response, retrying")
        else {
            val vBurner = transform("JSONPATH", "$.value.BAI", json)
            // Extract the relevant data from the JSON-formatted response message
            switch (vBurner) {
                case "CH": {
                    // Cantral heating active
                    if (NE_CvActive.state == NULL || NE_CvActive.state != ON) NE_CvActive.sendCommand(ON)
                    if (NE_BurnerActive.state == NULL || NE_BurnerActive.state != ON) NE_BurnerActive.sendCommand(ON)
                    if (NE_HotWaterActive.state == NULL || NE_HotWaterActive.state != OFF) NE_HotWaterActive.sendCommand(OFF)
                }
                case "HW": {
                    // Hot water tap active
                    if (NE_CvActive.state == NULL || NE_CvActive.state != ON) NE_CvActive.sendCommand(ON)
                    if (NE_HotWaterActive.state == NULL || NE_HotWaterActive.state != ON) NE_HotWaterActive.sendCommand(ON)
                    if (NE_BurnerActive.state == NULL || NE_BurnerActive.state != OFF) NE_BurnerActive.sendCommand(OFF)
                }
                case "No": {
                    // Hot water tap and Central heating inactive
                    if (NE_CvActive.state == NULL || NE_CvActive.state != OFF) NE_CvActive.sendCommand(OFF)
                    if (NE_HotWaterActive.state == NULL || NE_HotWaterActive.state != OFF) NE_HotWaterActive.sendCommand(OFF)
                    if (NE_BurnerActive.state == NULL || NE_BurnerActive.state != OFF) NE_BurnerActive.sendCommand(OFF)
                }
                default: {
                    // Just to be sure, we catch unknow values
                    logWarn(tag, "Unknown JSON response, turning off Nefit switch status")
                    if (NE_CvActive.state == NULL || NE_CvActive.state != OFF) NE_CvActive.sendCommand(OFF)
                    if (NE_HotWaterActive.state == NULL || NE_HotWaterActive.state != OFF) NE_HotWaterActive.sendCommand(OFF)
                    if (NE_BurnerActive.state == NULL || NE_BurnerActive.state != OFF) NE_BurnerActive.sendCommand(OFF)
                }
            }
        }
    end

The next rule gets the daily gas consumption. It returns historic data, so the last data to retrieve is yesterday’s totals. Therefor I run this rule just after midnight to get the data from the day before and persist it in InfluxDB with a timestamp of 1 second before midnight of the previous day.

//----------------------------------------------------------------------------------------
// Retrieve yesterdays natural gas usage for Hot Water (HW) and Central Heating (CH).
// The Nefit API returns the gas usage in kWh, not in cubic meter (as measured by the utility
// company). So we must convert this to m3. The Nefit Easy app uses a conversion factor
// of 0.12307692F (kWh * 0.12307692f = m³) for natural gas, which means a caloric value
// of 29 MJ/m³ (for Dutch households, this value is rather low, and a value of 35.17 MJ/m³
// is more accurate. More information can be found on the Gasunie website).
//
// Format of the JSON messages received (32 entries per page):
//  {"id":"/ecus/rrc/recordings/gasusage","type":"recordings","recordable":0,"writeable":0,
//    "value":[
//      {"d":"08-01-2019","hw":18.3,"ch":85.9,"T":73},
//      {"d":"09-01-2019","hw":20.5,"ch":83.1,"T":53},
//      {"d":"10-01-2019","hw":2.1,"ch":91.9,"T":37},
//      {"d":"11-01-2019","hw":9.5,"ch":80.5,"T":70},
//      {"d":"12-01-2019","hw":17.9,"ch":66,"T":68},
//         ...
//      {"d":"255-256-65535","hw":6553.5,"ch":6553.5,"T":-1}
//    ]
//  }
//
// For conversion of kWh to m3 natutal gas:
//  1 kWh equals 3.6 MJ
//  1 m3 = 35.17 MJ/3.6 MJ = 9.7694 kWh
//  1 kWh = 0.102365 m3
// Since Nefit is a Dutch brand of the Robert Bosch Group, the conversion is hardcoded.
//----------------------------------------------------------------------------------------
rule "Retrieve yesterdays gas usage for water and heating (after midnight)"
    when
        Time cron "0 2 0 * * ?" or
        Item NE_GasRetry changed to OFF
    then
        val String tag = "Nefit.Gas.Usage"
        val String influxDb = "openhab_db"                      // The Influx Database to store measurements
        val String influxUrl = "http://localhost:8886/"         // The InfluxDB daemon hostname and port#
        val String seriesHW = "NE_GasUseHW"                     // The measurement name to store the Hot Water gas usage
        val String seriesCH = "NE_GasUseCH"                     // The measurement name to stire the Central Heating gas usage

        // Get number of gas usage pages in JSON format
        var String json = sendHttpGetRequest(nefitBridge+"ecus/rrc/recordings/gasusagePointer")
        if (json == NULL || json == "") {
            logWarn(tag, "Empty gas usage pointer response, retrying")
            NE_GasRetry.sendCommand(ON)                         //Set retry switch (expires in 1m and turns to OFF)
        }
        else {
            // JSON format: {"id":"/ecus/rrc/recordings/gasusagePointer","type":"floatValue","recordable":0,"writeable":0,"value":1166,"unitOfMeasure":"","minValue":1,"maxValue":6400}
            val vPage = Math::round(Math::ceil((Float::parseFloat(transform("JSONPATH", "$.value", json)) - 1) / 32))

            // Get the last page for yesterdays data
            json = sendHttpGetRequest(nefitBridge+"ecus/rrc/recordings/gasusage?page=" + vPage.toString)
            if (json == NULL || json == "") {
                logWarn(tag, "Empty gas usage data page {} response, retrying", vPage.toString)
                NE_GasRetry.sendCommand(ON)
            }
            else {
                // Walk through JSON Array and find yesterday's entry.
                var int arrayLen = Integer::parseInt(transform("JSONPATH", "$.value.length()", json))
                // logError(tag, "Array length = {}", arrayLen)
                var String yesterday = now.minusDays(1).toString("dd-MM-yyyy")
                for (var i = 0; i < arrayLen; i++) {
                    if (yesterday == transform("JSONPATH","$.value["+i+"].d", json)) {
                        val String hotWater = (Float::parseFloat(transform("JSONPATH","$.value["+i+"].hw", json)) * 0.102365).toString
                        val String centralHeating = (Float::parseFloat(transform("JSONPATH","$.value["+i+"].ch", json)) * 0.102365).toString
                        val String timeStamp = now.withTimeAtStartOfDay.minusSeconds(1).millis.toString
                        logDebug(tag,"Hotwater: {}, CentralHeating: {}, Timestamp: {}", hotWater, centralHeating, timeStamp)
                        var String cmd = "/bin/sh@@-c@@/usr/bin/curl -s -X POST "+influxUrl+"write?db="+influxDb+"\\&precision=ms --data-binary '"+seriesHW+" value="+hotWater+" "+timeStamp +"'"
                        logDebug(tag, "Cmdline={}", cmd)
                        executeCommandLine(cmd, 2500)
                        cmd = "/bin/sh@@-c@@/usr/bin/curl -s -X POST "+influxUrl+"write?db="+influxDb+"\\&precision=ms --data-binary '"+seriesCH+" value="+centralHeating+" "+timeStamp+"'"
                        logDebug(tag, "Cmdline={}", cmd)
                        executeCommandLine(cmd, 2500)
                        return;
                    }
                }
            }
        }
    end

I also created a small Python script (my very first Python script…) to retreive all historic data from the API and store it in InfluxDB. But that’s for another topic.
BTW, our heater held data for the last 3+ years (when we bought the Nefit Easy thermostat).

Example sitemap entries

To make the example complete, a few sitemap entries for Basic UI.

Frame label="Nefit CV ketel" {
    Switch item=NE_CvActive mappings=[ON="Aan", OFF="Uit"]
    Text item=NE_SetPoint
    Text item=NE_Temperature
    Text item=NE_Pressure
    Switch item=NE_BurnerActive mappings=[ON="Aan", OFF="Uit"] visibility=[NE_BurnerActive==ON]
    Switch item=NE_HotWaterActive mappings=[ON="Aan", OFF="Uit"] visibility=[NE_HotWaterActive==ON]
}

Hope this is usefull for others with a Nefit Easy system. Also, since I am still learning to make best use of OH every day, any advice on improving on this is welcome.

UPDATE: And this is what our simple Basic UI setup looks like:


Will there be a "Bosch Smart Home" binding?
(Stefan) #2

Thanks! I’ll give it a try as soon as I’ve setup Openhab. We just moved so I have to start from scratch.


(Ron) #3

Thanks @vzorglub, I didn’t pay attention to the riight section to place this post…


(marcel) #4

You can use this rule to set the temperature

rule "POST SetTemp Change"
when
        Item NE_SetPoint received command
then
    var baseurl = "http://127.0.0.1:4000/bridge"
    var endpoint1 = "/heatingCircuits/hc1/temperatureRoomManual"
    var contenttype = "application/json"
    var Number setpointNE = (NE_SetPoint.state as QuantityType<Number>).doubleValue
    var POSTrequest = '{"value":' + setpointNE + '}'
    var output = sendHttpPostRequest(baseurl+endpoint1, contenttype, POSTrequest)
    logInfo("Nefit Setpoint: ", output);
    var endpoint2 = "/heatingCircuits/hc1/manualTempOverride/status"
    output = sendHttpPostRequest(baseurl+endpoint2, contenttype, '{ "value":"on" }')
    logDebug("Nefit Setpoint:", output);
    var endpoint3 = "/heatingCircuits/hc1/manualTempOverride/temperature"
    output = sendHttpPostRequest(baseurl+endpoint3, contenttype, POSTrequest)
    logDebug("Nefit Setpoint:", output);
end

(Ron) #5

Thanks.

And I just noticed a small calculation error in the rule to get yesterdays gas usage. Fixed in post #1.


(Ron) #6

And for those that want to get all historic gas usage data and store it in InfluxDB, the little Python script I created (please be gentle, it is my very first attempt at anything Python…:wink:):

'''
This simple python script reads the historic gas consumption data from the Nefit CV,
using Robert Klep's easy-server (see https://github.com/robertklep/nefit-easy-http-server).

This script assumes the easy-server is running and ready to accept calls.
The data returned specifies energy used for central heating and hot water in kWh units.
To translate that to cubic meters, which is the unit of measurement used on the utility
meters in The Netherlands, the estimated caloric value of 35.17 MJ per m3 is applied.
Since 1 kWh = 3.6 MJ, this results in a conversion ratio from kWh to m3 of 0.102365.

Run this script with the following command:
    python3 nefit-history.py INFLUXDATA

The extracted data is written to a text file in the Influx Line Protocol format and
with a record structure similar to what openHAB applies:
    <measurement> value=<value> <timestamp>
Separate measurements are created for hot water and central heating and no tags, to keep
it compatible with other OH persisted items.

To get the data into InfluxDB run the following command:
    influx -port PORT -username USER -password PASSWORD -host localhost -import -path=./INFLUXDATA -precision=s
'''

import sys
import os
import time
import math
import requests
import json

################################################################################
# START OF CUSTOM CONFIGURATION
################################################################################

easyServer = "http://localhost:4000"    # URL of the easy-server
influxDB = "openhab_db"                 # Influx Database name
CHmeasurement = "NE_GasUseCH"           # Influx measurement name for Central Heating gas usage data
HWmeasurement = "NE_GasUseHW"           # Influx measurement name for Hot Water gas usage data
caloricValue = 0.102365                 # From kWh to m3 (1 kWh = 3,6 MJ, 1 m3 = 35.17 MJ in Netherlands)

################################################################################
# END OF CUSTOM CONFIGURATION
################################################################################

# Holds the values read from the pages with HW and CH values
nefit_list = []

#-------------------------------------------------------------------------------
# Get number of gas usage history pages from easy-server.
#
#   JSON format of message expected:
#     {"id":"/ecus/rrc/recordings/gasusagePointer",
#      "type":"floatValue",
#      "recordable":0,"writeable":0,
#      "value":1166,
#      "unitOfMeasure":"",
#      "minValue":1,"maxValue":6400
#     }
# Returns the total number of pages to read and parse.
#-------------------------------------------------------------------------------
def get_pages(pagesURL):
    # We're interested in the 'value' parameter which indicates the total #entries
    p = requests.get(url = pagesURL)
    data = p.json()
    entries = data['value']
    pages = math.ceil(entries/32)
    print("Number of entries = %d,  the number of pages = %d" % (entries, pages))
    return pages

#-------------------------------------------------------------------------------
# Parse the passed array of JSON value data and save it in a list.
# A date entry of '255-256-65535' indicates the end of the list.
#-------------------------------------------------------------------------------
def process_page(data, nefit_list):
    # We're interested in the 'value' array which holds the HW and CH entries.
    for item in data:
        nefit_details = {"date":None, "hotwater":None, "centralheating":None}
        if item['d'] == "255-256-65535":
            break
        pattern = '%d-%m-%Y,%H:%M:%S'
        nefit_details['date'] = str(time.mktime(time.strptime(item['d']+",23:59:59", pattern)))[:-2]
        nefit_details['hotwater'] = float(item['hw']) * caloricValue
        nefit_details['centralheating'] = float(item['ch']) * caloricValue
        nefit_list.append(nefit_details)
    return

#-------------------------------------------------------------------------------
# Retrieve pages of JSON data and pass them to process_page to extract the data.
# JSON format of messages expected:
#    { "id":"/ecus/rrc/recordings/gasusage",
#      "type":"recordings",
#      "recordable":0,"writeable":0,
#      "value":[
#        {"d":"08-01-2019","hw":18.3,"ch":85.9,"T":73},
#        {"d":"09-01-2019","hw":20.5,"ch":83.1,"T":53},
#        {"d":"10-01-2019","hw":2.1,"ch":91.9,"T":37},
#           ...
#        {"d":"255-256-65535","hw":6553.5,"ch":6553.5,"T":-1}
#      ]
#    }
#-------------------------------------------------------------------------------
def process_pages(nefit_list):
    # Get the total number pages to walk through
    pages = get_pages(easyServer + "/bridge/ecus/rrc/recordings/gasusagePointer")
    for page in range(pages):
        print("Processing page %s  \r" % str(page+1), end='')
        p = requests.get(url = easyServer + "/bridge/ecus/rrc/recordings/gasusage?page=" + str(page+1))
        data = p.json()
        # We're interested in the 'value' array which holds the HW and CH entries.
        process_page(data['value'], nefit_list)
    return

################################################################################
# Start Here
################################################################################

if len(sys.argv) != 2:
    print("Usage: %s <outputfile>" % sys.argv[0])
    exit(1)

process_pages(nefit_list)

# Write data in Influx Line Protocol (DML) to output file
# Can be imported afterwards with command (Linux example):
#   $ influx -port 'xxxx' -username xxxx -password xxxx -host xxxx -import -path=./output.txt -precision=s

try:
    outfile = open(sys.argv[1], "w")
except Exception as e:
    print("%s" % e, file=sys.stderr)
    exit(3)

print("Writing Influx Line Protocol File %s" % sys.argv[1])

print("# DML", file=outfile)
print("# CONTEXT-DATABASE: " + influxDB, file=outfile)
print("# CONTEXT-RETENTION-POLICY: autogen", file=outfile)
print("", file=outfile)

for item in nefit_list:
    print(CHmeasurement + " value=%.3f %s" % (item['centralheating'], item['date']), file=outfile)
    print(HWmeasurement + " value=%.3f %s" % (item['hotwater'], item['date']), file=outfile)

From then on, the daily rule presented in post #1 will keep adding to this.