Setting OH2 for KNX, a Windows install

Tutorials & Examples openHAB Stories
1

WORK IN PROGRESS
This topic is still being redacted - pls do not comment yet (I did not find yet how to have it as unpublished draft)

Scope and Objectives

Introduction

My OpenHab 1/KNX installation becoming old fashioned, notably on the side of graphical UI, I wholeheartedly welcomed OpenHab version 2 and its new features. Nevertheless, the new concepts and behaviors of the OH2 with KNX addon puzzled me. The various posts and tutorials were not always explicit about which version (1 or 2?) and their configuration files. Some useful tricks apply to OH1 and OH2, but not always. I spent a lot of time to read the manual, community threads, source code, to get some sense of what is required and what could work. I thought that clearer steps would help, which is why it is now my turn to feed back my experience, even though I still have a very imperfect view. It will help to remind important steps, and improve their description over time when facing some unavoidable quirky features of both systems.

Scope

This Topic aims at detailing steps to install and configure OH2 with v1 KNX binding on a Windows platform. Syntax and location of configuration files will relate only to these versions. Graphical UI (such as HabPanel and HomeBuilder) and manual installation is considered, although I tend to give preference to manual installation. The objective is to give an up-to-date, step-by-step cooking recipe to set up such systems. Be warned, despite all the efforts of the developers, the KNX binding has still a serious issue (read further).

Starting with a Windows platform does not preclude a later installation on Raspberry PI, which is within scope too. However, setting up a diskless, headless platform requires system thinking beyond the current scope, such as network disk servers, big data management, failure handling, …

Who am I?

Trained in electronics and computing science, with some years of UNIX sysadmin, I developed code for operating systems and secure systems in the 90’s. A certified KNX installer, I dream of smart homes since the 2000’s.Not afraid of sophistication, I pursue low-maintenance, high-reliability systems. Home is a big lab, with more than 120 KNX devices operating on three lines, and four times as much group addresses. Lights, heating, fans, stores, door openers, weather sensors, energy meters, … are all part of the KNX system. OH1 brought me then monitoring of energy consumption, crepuscular lighting to wake up, up/down of stores according to sun position and inside/outside temperatures, and some logic functions difficult to do with classical stateless logic. I intend to record as many data as possible with a view to train AI systems to predict and anticipate user behaviors.

What it is not

This Topic is not a tutorial on how to design and set up a KNX system. There is a minimum of training required to understand the KNX concepts and design a proper configuration, which is available from the KNX association as well as in many KNX manuals and webinars floating around the web.

It is also not a replacement for the OH documentation and tutorials. As time evolves, the OH documentation is likely to be more up-to-date than this Topic.

Intended audience

This topic is written for an intended audience of KNX-savvy persons, with an interest for IT solutions to complement KNX systems. The tricks and step-by-step approach will hopefully interest many more OH2 installers.

A brief view on my Windows setup

The Windows platform is an all-purpose computer, with several development tools, and UNIX-like environment (cygwin, gvim, …). The installed software of interest for OpenHAB is:

  • Windows 7, 64-bit version, SP1 (fully reinstalled in November 2017)
  • full JDK 64-bit install. My latest is Java version “1.8.0_152” (Java™ SE Runtime Environment (build 1.8.0_152-b16), Java HotSpot™ 64-Bit Server VM (build 25.152-b16, mixed mode)). Note to self: prefer the 32-bit architecture on ARM hardware, such as Raspberry PI.
  • MySQL server, 64-bit version. My latest is 5.7. Not necessary, MySQL Workbench is a useful companion.
  • PowerShell 5.1. As of OH2.2, a requirement to update OpenHAB by running the PowerShell script:
cd <openHAB installation directory>
. .\runtime\bin\update.ps1 Update-openHAB -OHVersion 2.2.0
  • obviously, OpenHAB. My latest is 2.2.0, upgraded on 18/12/17 by the update.ps1 script.
  • obvious as well, KNX binding. My latest is 1.11.0
    Either of the following Karaf instructions tells you which binding you run:
feature:list | grep knx
bundle:list -s | grep knx
config:list "(service.pid=org.openhab.knx)"

Note: there is a KNX binding 2.x in preparation, with promising features such as using knxproj (the KNX project describing a given installation), but it seems that development on this binding stopped in May 2017,
according to the last github commit (by @kgoderis). Pity, as re-using knxproj or the export file from the web-service exporter would avoid double configuration work. This is not a stable version, and installation is not guaranteed. I did not try it. It is therefore a decision to continue with KNX v1

  • Git (optional), only if you want to build OpenHab from the sources yourself.
  • openSSL (optional), to generate server certificate to authentify the identity of the MySQL database server towards JDBC.
  • Eclipse. Although installed for Java development and for Smarthome, I failed to get it re-compile the sources. Possible a configuration error on setting workspaces. As I’m confortable with vi for editing, Eclipse is for later…
  • WireShark (optional), to solve communication issues between OH and the KNX interface, by using the KNX tunnel mode for KNX messages inspection.

A brief view on my KNX setup

A sophisticated installation, of 120 KNX devices operating on three lines, and four times as much 3-level group addresses. Carefully designed to compartment parts of the installation while guaranteeing minimal functionality in case of failure of devices on any line. The main line (1.0) is connected to the two other lines (1.1 and 1.2) through line couplers with filter tables enabled. These filter tables keep the overall KNX traffic low by keeping local messages local to a given line. Priority KNX messages belong to 15.x.x group addresses.

Note: This latter design might need revision, or filter tables disabled, if overall traffic is to be exposed to OpenHAB.

The KNX system is accessed through a KNX-IP interface (not router) of GIRA (IP Schnittstelle 2168). KNX PA 1.0.3, 192.168.0.3, it gets known as GA 15.15.254 on the KNX bus, as the default 15.15.255 was taken by ETS. The fact that this device is an interface and not a router requires to only use the TUNNEL mode.

Note: Most comments in the community threads rather apply to the KNX-IP router version (GIRA KNX IP Router 2167). This latter model performs both TUNNEL and ROUTER modes, but is said to be better in ROUTER mode. I did not test that device.

There is no such thing as a KNX Thing

Since OH2, a Thing concept has been introduced to separate the hardware layer from the virtual configuration modelled through Items.

This has created a lot of confusion for me. What would really be a KNX Thing? I wish I had someone telling me that there is no need for a KNX Thing, just continue with your Items mapped to the KNX world and everything will be fine!

For those interested in a longer read, reflecting on the KNX philosophy of designing systems, there is an evident parallel and it clarified my own Thing-king.

  • KNX has a physical architecture, called topology, with a physical address (often referred to as ‘PA’) for each KNX device. This would be openHab’s Thing.

  • KNX has also a functional architecture, which is based on group addresses (often referred to as ‘GA’). Group addresses have logic or semantic meaning, connecting sensors (eg a push button) and actuators (eg a switching actuator) through the use of messages. Each KNX device exposes a number of connection points (openHab’s channel) which are assigned to one or many group addresses. A device sends a telegram in the KNX network and every other device in the KNX network will receive this message and decide if it needs to act upon it. Therefore, the functional view is much like a publish-subscribe model, where the publishers and subscribers are statically configured.

Note: a good KNX design is to have direct commands and a feedback loop on the state (a listening GA) which sends the state that the actuator implemented. My KNX design has command and listening GAs.

  • Telegrams transport information in the KNX network, in the format specified by the Data Point Type (DPT). A given Group Address can only be assigned one DPT. KNX device manufacturers usually specify the DPT of the connection points. It is a good idea to know the DPT to avoid misreading of value, as a 4-byte value can be a long or float, signed or unsigned number. Command and state are therefore embodied in these connection points and telegrams, and cater for the functional implementation of the system. These are similar to openHab’s Items. A careful design of the KNX flags ensures a good symbiosis with OpenHab’s Items, avoiding for example the can’t read group address message.

Does it matter? Jein. Yes and no. KNX made, in its own way, an effort to virtualise a physical system into functions and message passing ahead of its time. If designed properly, you should not have to know which KNX device (Thing) offer which connection point (Channel) to perform a function. It is hidden. Virtualised to the logical and functional plane.

You should not have to re-introduce the physical KNX topology into openHab, and it is why I consider a wise design choice of the KNX binding not to have exposed KNX Thing(s). In short, there should not be any such thing as a KNX Thing.

KNX has however limitations. Designed to be robust, safe and to last many years without maintenance, it lacks the reactivity of the Internet world in terms of product development. For example, visualisation devices, when they exist, are limited and expensive. Past events are also difficult to be taken into account, as the KNX network is stateless by design. This is where I see openHab as a “system of systems” complementing KNX.

It is however still important for me to compartment both systems, and ensure minimal functionality of the KNX system even in the case of openHab or related hardware failure. This design decision will most likely show up in my openHab design implicitly. Please have it in mind when reading this Topic.

Installing OpenHab 2 with KNX Add-On v1

Enough reading, show me the code!

Standard OH2 installation

  1. Starting with downloading OH2.1, and the 2.1 add-ons (note: a newer version is out now!), it was enough to follow the standard install process. Coming from OH1, I selected the expert mode. I created a launch CMD file to make sure that the ENV variables are correctly set.
@SET JAVA_HOME=C:\Program Files\Java\jdk1.8.0_152
@export JAVA_HOME
@cd /d <openHab install directory>
start "OpenHAB server" /HIGH .\start.bat

  1. Once installed, start OpenHab with this CMD file.

  2. It is now time to install the KNX add-on through the UI interface: open http://localhost:8080 in a web browser.

  3. Go to Paper UI, select Add-ons, then bindings. Install KNX. Wait a while. In my case, the rotating icon did not stop. I had to stop Karaf (enter ‘logout’), wait a while, restart after editing the knx configuration file (see next step). This explains sometimes why I prefer to manually edit configuration file instead of blindly trusting UI to do the job.

  4. It is still not clear whether <openhab install directory>/conf/services/knx.cfg needs to exist before or after install. In any case, stop Karaf. Wait a while. Then, edit the file with a DOS/Unix-compatible editor (that is, one able to deal with CR instead of CR-LF). While I use gvim, you might want to unix2dos the configuration file and edit with notepad.

In line with the KNX installation described before, my knx.cfg file looks like this:

ip=192.168.0.3
#busaddr=15.15.255
ignorelocalevents=true
type=TUNNEL

I also set in knx.cfg the “localIp=192.168.0.<XX>” to my Windows machine, as there are several network interfaces coming from other software (VPN, VirtualBox). Despite this, I still get the Karaf message that several network interfaces are detected. Not a problem, yet it’s nagging me (I like clean systems).

After starting OpenHab, let’s check in Karaf:

config:list "(service.pid=org.openhab.knx)"
Pid:            org.openhab.knx
BundleLocation: mvn:org.openhab.binding/org.openhab.binding.knx/1.11.0
Properties:
   busaddr = 15.15.254
   ignorelocalevents = true
   ip = 192.168.0.3
   localIp = 192.168.0.<XX>
   service.pid = org.openhab.knx
   type = TUNNEL

I observed some strange behaviors with OH2.1 which were partially solved after upgrading to OH2.2, WireShark helped me here a little, some report that it is due to WireShark setting internet interface in promiscuous mode. It is difficult to tell what solved which issues at this point.

A few trials and errors also taught me that Karaf can be whimsical with configuration parameters of its own, disregarding modifications to the cfg files. Pay attention if you modify parameters through Karaf. Read this Tutorial about basic Karaf/OpenHab commands, or the Karaf manual.

Some further Karaf/openHab examples:

config:property-delete -p org.openhab.knx busaddr

Changing properties

config:edit org.openhab.knx
config:property-list
config:property-set busaddr 15.15.255
config:update

or to cancel your edits:

config:cancel

Generate the log of what’s going with the KNX binding:

log:set TRACE org.openhab.binding.knx 
log:set TRACE tuwien.auto.calimero

Do not forget to reset the log settings after solving the issues:

log:set DEFAULT org.openhab.binding.knx 
log:set DEFAULT tuwien.auto.calimero

Cygwin unix tools for Windows do the job for me of editing and monitoring log files. It is not uncommon to have several windows open with continuous display of logs.

tail - f <xxx>.log

Alternatively, you achieve the same in Karaf with:

log:tail

I used a minimal items file with one switch and one number, to test the new functionalities. Not so great as binding.knx.internal.bus.KNXBinding sends to the KNX bus commands twice. Events from OH (such as ON commands) as well as data coming from the KNX bus (such as brightness or temperature data value coming from sensors) are always sent once more than needed. This behavior is rather annoying: if OH is somewhat slower, a quick ON-OFF sequence will end up with ON1 - OFF - ON2, and the light stays ON (unwanted). It also unnecessarily activates triggers in the KNX system on receipt of sensor data.

My items file looks like:

Switch Lampe_Cave_Bureau		"Bureau"			{ knx="5/3/51" }	/* { knx="5/3/51+<7/3/51" } */
Number Weather_Brightness 		"Clarte [%.1f Lux]"		<sun_clouds>    { knx="<14/5/0" }
Number electricityTotalActivePower "Active Power [%.1f WH]" (gElectricityData) {autoupdate="false", knx="<13.010:8/6/10"}
  • 5/3/51 is the KNX GA to command my office light. 7/3/51 is the feedback loop from the actuator, what I called before a listening GA, giving the actual state of the light. I edited out the listening GA for test purposes.
  • 14/5/0 is the KNX GA to get the state from a brightness sensor. To poll periodically its value, it gets the < in knx="<14/5/0". It does not prevent the annoying behavior of openHab to re-issue a command to the KNX bus with the same value, even with the knx.cfg’s ignorelocalevents=true. To solve this echoing of command, it is enough to specify autoupdate=“false”, but it has also some side effects.
  • 8/6/10 is the KNX GA to get the state of an energy meter. As before, it uses <. The 13.010 in “<13.010:8/6/10” refers to the datapoint type of this number in KNX. It is essential here to map this 4-byte value to a Long integer, and not let openHab interpret it as a 4-byte float value.

I described the echoing behavior in this thread, also reported in other threads, such as this one, and this one. For the latter one, I did not experience (yet) the infinite loop described and the fixes mentioned by @lewie (that is, using the autoupdate openHab parameter - more on this later) helped solve the echo -to some extent-.

I did do a minimal sitemap to test that values were showing, by manually editing a sitemap file. It’s a rather straightforward exercise. My sitemap looks like this:

sitemap tangissart label="Tangissart"
{
    Frame label="Demo" {
        Switch item=Lampe_Cave_Bureau icon="big_bulb"
        Text item=Weather_Temperature valuecolor=[>25="orange",>15="green",<=15="blue"]
    }
	Frame label="Weather" {
		Text item=Weather_Temperature 
		Text item=Weather_Brightness  
	}
}

What is most important to me at this point:

  • commands and states behave as expected. Carefully scrutinise the log tails to check for any WARN or ERROR messages. Solve these.

  • listen to the KNX bus with ETS and check that the behavior is as expected. Messages are sent once, are ack when they should, are read correctly. Look for repetitive telegrams (sign of misconfiguration or error, either on the KNX or on the openHab side).

Once satisfied, I also installed the Astro and NTP bindings. Straightforward through Paper UI. They were both needed to re-create my older OH1/KNX configuration, for sun position and precise timing. Make sure that the things configuration file contains the astro things with your own location. The <openhab install directory>\conf\xxx.things is an easy one.

astro:sun:home  [ geolocation="50.606290, 4.522321, 124", interval=1800 ] {
	Channels:
        Type rangeEvent : rise#event [
            offset=-10
        ]
}

astro:moon:home [ geolocation="50.606290, 4.522321, 124", interval=1800 ]

ntp:ntp:home  [ hostname="0.pool.ntp.org", refreshInterval=60, refreshNtp=30 ]

Note: the offset is only needed to have an event at sunrise 10 minutes before actual sunrise.

Configuration files

There are only few files to modify, all of them in <openhab install directory>\conf. Those of interest to us are in items, sitemaps and services directories at this point in time.

Manual configuration, or not?

It’s not only a matter of taste, unfortunately. With UI, configuration is a few clicks away. Error messages, are however not always properly handled, and when things go wrong, it’s difficult to know what happened. This is why I usually prefer manual configuration files. Modifications are visible, error messages tend to be consistent all things being equal. Manual configuration files also tend to be easier to be scripted. Don’t be afraid to edit files!

Note: keep a version of the original file (for example by copying it, COPY <xxx> <xxx>.orig).

KNX Thing

The things configuration file is the least interesting for KNX. Don’t even bother. KNX functions are much better dealt with in the items configuration file without such a thing.

KNX Items

Mapping KNX commands and states to openHab Items is generally easy and the syntax of the binding well explained in the manual. Reading sources is also possible (expert level required).

Some easy rules:

  • one main GA (for command) + 0 or many listening GA (for state). Main GA goes first.
  • < to periodically poll the value.
  • datapoint type to cast values (format xx.xxx: just before a GA). It might even be required for some commands, notably in Dimmer and Rollershutter.
  • number and string format to keep visualisation under control.

autoupdate=“false” or autoupdate=“true” to solve the nagging and infamous echo of commands to the KNX bus?

The most nagging part of the OH2 setup is the fact that the KNX binding echoes commands and states back to the KNX bus. This is not yet solved, and is being discussed across many threads. Frustration is building up, both from the users and the developers sides as there does not seem to be an easy solution, despite all the efforts and good will of many talented developers, as it touches the architectural design of OH2.

The following discussions give some insights:

@lewie recommends therefore to play with the autoupdate=“false” parameter (actually a binding, implemented or not by the various add-ons) whenever possible. It has however noticeable side effects.

Remember that “autoupdate=“false” is a special instruction which keeps the current state of the item, even if a command has been received. This way, the Item is unchanged unless you explicitly post an update to the item.”. It is typically used to implement push button with a switch, for example to always send OFF to switch off all lights and keeping the UI to show an always-OFF switch. More details in this thread.

It has also a side effect on the persistence strategies, as explained next.

My advice: use the parameter with good sense, and remember its side effects. And be patient for a solution if you really need to use dimmer and rollershutter…

Choosing persistence

It costs many times more to acquire data than to retain data. (adapted)

Even without thinking of the need for big data to train deep learning systems in Artificial Intelligence, time series of energy production and consumption, temperatures, weather data, lighting scenario, … are invaluable to increase user comfort over time.

Looking beyond solutions to restore values after restart, MySQL offered various advantages in my view: long-standing existence and maturity, cross-platform, free and professional versions, flexible. Reading a lot about the quirks of MySQL, it appeared safer to connect to it through JDBC. Having installed MySQL with the right Java Connector, it was enough to install it from Paper UI, after

  • in the database server shell, create a database for openHab:
CREATE DATABASE OpenHAB;
(alternatively, CREATE DATABASE OpenHAB CHARACTER SET utf8 COLLATE utf8_general_ci;)
CREATE USER 'openhab'@'localhost' IDENTIFIED BY 'YOURPASSWORD';
GRANT ALL PRIVILEGES ON OpenHAB.* TO 'openhab'@'localhost';

Note: although you use MySQL, you installed JDBC persistence, and not MySQL persistence. It has implications for further configuration. In short, when asked about the database, its name is ‘jdbc’ and not ‘mysql’.

  • having provided a configuration file in <openhab install directory>/conf/services/jdbc.cfg.
url=jdbc:mysql://localhost:3306/openhab?useSSL=false
user=openhab
password=YOURPASSWORD

Note: to avoid MySQL complaining that the server identity is not authentified (a requirement since MySQL5.5.45), useSSL=false avoid MySQL complaining to the log file about the lack of server certificate. OpenSSL to generate a self-signed certificate and its installation in the database server solve the matter.

  • in <openhab install directory>/conf/persistence/jdbc.persist, enter your persistence strategies:
Strategies {
        EveryMinute: "0 * * * * ?"
        Every5Minutes: "0 */5 * * * ?"
        EveryHour: "0 0 * * * ?"
        EveryDay: "0 0 0 * * ?"
        // if no strategy is specified for an Item entry below, the default list will be used
        default = everyChange
}

Items {
	// everyChange: persist the state whenever its state has changed
	// everyUpdate: persist the state whenever its state has been updated, even if it did not change
	gPersist* : strategy = everyUpdate, EveryHour 
}

Notes:
1. the difference between everyChange and everyUpdate
2. the use of design pattern to configure persistence, here a group which is added in the Items configuration file to the Items to be persisted. Have a look at the excellent Tutorials on Design Patterns by @rlkoshak, notably the Group Based Persistence for what concerns this section.
3. the modification to the Items configuration file to add gPersist as a group to Items to be persisted:

Number electricityTotalActivePower "Active Power [%.1f WH]" (gElectricityData,gPersist) { knx="<13.010:8/6/10"}

With this configuration ready, it is wise to restart openHab.

In the MySQL database shell, several tables will now appear. A Items table, containing the description of the Item being persisted, and a number. This number identifies the table which stores time and value related to this Item, under the Item<000number> table. If not, there is no persistence, and it might be time to review Items with an autoupdate parameter and the persistence strategy, in particular whether you persist everyChange or everyUpdate).

Charting

Habpanel is great.
repeat not mysql, but through jdbc. what it means. avoid mysql config (including later in charts, use jdbc as ref)

4 Likes