The state of the Z-Wave binding

Development
1

I’ve been trying to debug some things in the Z-Wave binding the last couple of days, and it seems to be “stagnating”. It doesn’t have a lot of activity, and the POM isn’t even updated to the latest version (still pointing to 5.0.0 instead of 5.1.0-SNAPSHOT). Spotless isn’t in effect, and it generally “differs” from much of the other OH code I’ve seen.

So, my question is: Is it “falling behind”? If so, why? Is there a lack of manpower or interest/will behind this?

For me, the binding is very essential to OH, it’s what runs “the core” of my Things. I know about the new ZWave-JS binding, but I’d really prefer not to have to run another thing with updates, bugs etc. in addition to OH. It’s also a waste of hardware resources to do anything using JS instead of Java. So, I’d really hate if what we see is “the beginning of the end” of the Z-Wave binding, with a slow decline into being completely obsolete, without support for any modern hardware etc.

I’m not claiming that this is happening, I’m asking if it is - I see some “signs” that worry me slightly. But, if this is the situation, is there anything we can do to turn this around? If so, what?

2

I can’t answer your basic question and some of what I might say could be wrong.

I wasn’t around at the very beginning but speculate it was some sort of German / English collaboration to add needed technologies (ZW and Zigbee) to the early OH spin-off from a German Telecom. I think the agreement was that those bindings would stay under the English developer control. He had connections with Silabs (the technology owner) and developed many separate tools and had other clients besides OH.

I started with OH in 2017 to monitor the wattage draw on a portable generator using open-source software. Over the years my network and knowledge grew due to the developer and others on the forum. Around the Covid year(s) the developer moved to New Zealand and with his job and now has limited time. To give back for all the help over the years I do the best I can to support OH ZW on the forum (including some guides) and had a few PRs merged (notably the 700/800 series devices).

Outside of OH, Silabs “unloaded” (probably not accurate word) the ZW technology to the ZW Alliance. Also, HA (which has more financial flexibility for paid development than the OH foundation) has turned the Zwave-js application into a better tool (IMO) and has paid the fees for access to the ZW Alliance information on the Silabs technology.

The main gaps with the OH ZW binding (besides the nuisances like spotless and less consistency with other add-ons) are that S2 security and Long-Range inclusion (nodes 256+) are not included. Both require major refactoring. Also, ZUI has incorporated tools from the Silabs Simplicity Studio (like connection health, route setting, nvm backup, etc.) directly in the application. However, IMO LR is overrated, S0 security is ok and Simplicity Studio is free, so I still think the OH ZW binding is an option for some. Full Disclosure: I mostly use ZUI (via MQTT) but have a few OH ZW nodes on an old Rpi3b to answer questions.

3

I have no idea of what the challenges are with S2 and LR, nor am I that interested for my personal use - I don’t want to use security simply because I would never dream of using Z-Wave for anything remotely “sensitive” anyway, and the coverage is so bad that I have trouble reaching nodes that are in the next building, while there are > 100 m to the next neighbor, so it’s just not “a thing” for me to encrypt the traffic. If somebody want to hide in the bushes outside with a Z-Wave receiver to figure out which lights are on and off, be my guest. It won’t be long before our dog and/or horses have found the individual anyway.

When it comes to LR, it sounds like something that could be useful, but I’m never the one to jump on the latest hype, and I’ll only consider things when others have tried it out for a long time and confirms that it actually delivers as promised.

For me at least, I don’t see the need to move to ZUI in any foreseeable future. That doesn’t mean that I can’t use it to make a backup of the device, for example, but I don’t want to rely on another system running 24/7 for things to work. I’m sure the Z-Wave binding can never “match” ZUI in features, but if it can do what most people need, most people won’t have to take on the additional burden of running yet another system. When it comes to actual numbers (how many have what needs), I don’t know, of course. Maybe S2 is important for a lot of people, I can’t know.

You don’t need to use Simplicity Studio if you use the Z-Wave binding. I can imagine that I will use ZUI if I need to delete nodes or some other “maintenance” in the future, because it means that I don’t have to move the stick from the RPi to a computer to do it. I’ve installed ZUI in a docker container on the RPi, that I have shut down by default. I can thus start it at will to do the kind of things I would use Simplicity Studio for - and still let the Z-Wave binding manage normal operations.

What has been bothering me for a long time with the Z-Wave binding is what I believe is some concurrency bugs, and the hopelessness of having to submit updates to devices to the database and then wait for the next OH release to use them. This can take months, and you don’t even get to test what you have done - so the “feedback loop” is moving at glacial speed and is close to useless.

What I’ve been dabbling with the last couple of days is to try to find a way to “sideload” XMLs for devices from a local folder. I’ve met some challenges, but it can absolutely be done. The biggest hurdle is the state of the code, that versions etc. isn’t updated means that dependencies won’t resolve, the spotless issues leads to wasted time and frustration, but most importantly: I’m not sure if it’s worth my time doing it. If the activity is almost none, what is the chance that a big PR will ever be merged? I’m tired of doing work that is just left to rot in some PR that never gets looked at, so I’m not about to try to do that if the state of the binding/repo is as sad as it appears.

I believe that the ability to add device files locally, both to test them before submitting them to the database, and simply to get things working without waiting for months, would make a huge difference to the usefulness of the binding. It would to me at least.

If I could identify and solve some of the concurrency issues as well, I think the binding could serve my needs for many years to come.

Whatever deals/arrangements exists around the binding is thus very important here, as it decides whether it’s worth doing any work on it or not.

1 Like
4

Lets add @chris to this discussion..

2 Likes
5

There is no reason for someone with your skills to not be able to test locally. I do this all the time. I frequently provide a file for other forum posters, and I wrote a guide for it. The easiest is using the compressed file editor. You can add new, modify existing or both. Worst case, post your issue on the forum and I’ll see what I can do.

6

I can do it if I want it enough, but I think it’s way too difficult as it is. The goal here wasn’t to enable me to test locally, it was to make it easier for everybody, so that you know what you submit to the database - and to bridge the time gap between when you get that “not-yet-supported” device and when it’s part of the next OH version.

If I were to only do it for myself, I would find a simpler solution. Also, I generally don’t upgrade my production system if there’s not a good reason to, because I just hate regressions/things that break, so the whole idea of having to wait for a new version just doesn’t work that well. I’m still running 4.2.3 :wink:

7

Unfortunately I’ve not had a lot of time to put in to OH over the past year or two, and this will likely continue for another 18 months or so. Since moving from the UK to NZ, I’m still working for the same company in the UK, but am now working on a project that requires me to be in the UK for 2 weeks a month, and often also in the US, so I spend 3 or 4 days every month flying, and only 1-2 weeks at home in NZ (if I’m lucky).

Anyway, sob stories aside…

There are ways to do this locally, but way back when the OH2 binding was written, it was required by maintainers that the XML files were in the JAR to (in theory) ensure that the code and XML were compatible. This was at the time at least some rational behind this since there was a lot of development in the binding, and it was quite likely for new features in an XML not to be backward compatible with an old binding. I do agree that this is now a PITA though and XML device definitions is something I probably wish I’d avoided completely!

I’ve not looked at LR, but S2 is definitely not simple. Is it required - that’s personal preference but it’s certainly considered a shortfall in the binding, but one that is not easy to resolve (someone was looking at it in the past). Even my implementation of S1 took quite a long time.

Personally I’m also running OH4 still - being away from home most of the time means I don’t want to break things for the other half :slight_smile: . You can run the snapshot in most cases - that might not work with OH5 was the Java version changed.

1 Like
8

“Life outside OH” is what it is, for everybody, the question is if others could step in and do things here and there while you’re unavailable. By the time you’re “back”, there could be a lot to catch up, and potentially few users still using it.

I’ve tried to find ways to do this locally in the past (3+ years ago) where I didn’t succeed. I know that @apella12 has written guides now, which is great, but it’s still “less than ideal” as I see it. The 4.3.x snapshot might very well not work with 4.1, 4.2 etc. Also, if I have to modify the JAR manually, any patch version upgrade would overwrite it. I even asked for a way to do this before submitting a new device to the database some years ago, and the answer I got then was simply:

..which is technically correct as of today, since I asked for supplying them outside the JAR, but the only option I was presented with then was to configure a dev environment and basically modify and build my own JAR. Since I now have gone through all the trouble of getting a working OH dev environment, it wouldn’t be to much hassle (although it would still take a lot of time to make a build for 4.2.3 since it means hours of rebuilding core++ the way my setup is), but at the time, I didn’t have this, and I considered it “outside the scope” of what I was willing to do (and there is a history to this too, that I will spare you).

The result of this is that I concluded that I must basically only buy devices that are already in the database, or accept that there could be a lot of struggle (potentially several iterations of submit to database → wait for and test snapshot → modify and submit..) and time to get them to work. I think that is problematic. I can’t imagine that I’m the only one that has concluded that this is the situation, whether it’s technically correct or not.

When it comes to the maintainer requirement, I must just say that I’m not opposed to the database and the definitions being packed with the binding. It’s very convenient for all the devices that’s there, and it avoids a lot of unnecessary trouble for people if the device is there. All I’m saying is that there should be a way, other than editing the JAR, around this, to make it possible to test your configuration before submitting it to the database, and to “bridge the gap” when there’s no version available with the device in question. If they were/are opposed to that, I don’t think they see the full picture.

It definitely won’t work if the snapshot is built with a newer Java version than the one you’re running. Even if it would in theory work without any conflicts, Java simply refuses to run bytecode created by newer versions, unless it has been compiled “for a previous version”. So, getting any 5.x snapshot running of 4.x will only work if you’re actually running 4.x with Java 21, and that’s not an option if you’re on a 32-bit platform, and I also think it will cause problems with the Graal add-ons.

But, I don’t think this thread should evolve into only being about the ThingType definitions - my intention was to discuss the broader picture. Are there anybody that can review and merge in the binding repo besides yourself @chris ?

9

If you accept that OH will continue to require XMLs in the jar, the ZW DB has an option to download an XML that will work in any version of OH as soon as you are finished creating it (but you will have to edit your jar). Also note that the snapshot (with the latest ZW DB update) will only be in the latest pre-release OH version (currently 5.1) and have all the dependencies inherent in that version. Database updates are not backported, and with the current time limitations noted above, are not included in patch releases. The ZW DB in your 4.2.3 ZW jar will only have devices in the DB as of June 2024.

OTOH the ZUI (ZWave-js UI) does have a folder to add devices for testing. You have to create your own JSON following the build rules and include needed templates. However, the drawback is the PR approval can take months (I have done about dozen), since the urgency for inclusion is low.

10

You pretty much lay out the case for why this doesn’t work. I know that you can download the XML from the database, I’ve done that, but the clue is that there’s no way to get it into OH. Which is what I wanted to address. I know that it’s not possible (without hacking the JAR), and all I’m saying is that I wanted to try to change that. Not by “messing with” what is provided in the JAR, but to allow files in some defined folder to be parsed in addition (preferably with higher priority than those in the JAR in case you want to override one - although I haven’t (yet) found a way to control priority among ThingTypeProviders so I don’t know if that’s possible).

Making it possible to read the XML files from a second location shouldn’t be a big problem, however when looking at the details, it’s a bit more of a problem, because a lot of the classes needed are “encapsulated away” so that you can’t use them. This could be solved either by modifying the encapsulation in Core (but then it wouldn’t be backwards compatible) or by doing a copy of the required classes. But, those are details. What I wanted to say is that I’m contemplating attempting doing something like this, but it’s pointless to undertake something like this if it won’t be accepted anyway.

11

I’m genuinely curious about your motivation here.
Is the need to run an additional component like ZUI the main reason you’re willing to invest in extending the native Z-Wave binding?

Or are there other factors that matter just as much ?

I’m not asking to convince you to switch, I’m asking because understanding this is important for improving the zwavejs binding.

1 Like
12

Well I agree it would be a good addition, but I’m just an old guy with limited skills just trying to help other folks deal with the “as is” to the best of my ability. Good luck

13

My motivation? I just think a Java solution is a lot more desirable. For those who prefer to run ZUI, that’s perfectly fine with me, but I don’t like to patch together a lot of different solutions using “glue and tape”. To me, it’s inefficient and fragile (not talking about this particular solution, just setting up different systems in that way generally).

In addition, this is JS, depending on JS libraries (aka NPM or similar). That in itself is a world I really want to avoid, with constant updates and deprecations that break stuff for no good reason. It can be a full-time job to run such systems without actually achieving anything, just to “keep it working”. The more interdependencies and complications, the less desirable. On top of that, JS is extremely low performance, not to mention single-threaded. I generally avoid software written in Python and JS if at all possible.

So, to start with I’m “stacked against” using ZUI I guess you could say. In addition, I got the impression that integrating it with openHABian was quite out of the question. If it were maintained that way, and interoperability with JS library versions etc. were handled by somebody else, I might be more inclined to reluctantly use such a solution. But, that’s far from the case, as far as I can understand:

So, my thinking is that for those that want the extra ZUI can offer, or that don’t mind running and maintaining separate systems, the “ZUI binding” is fine - for the rest, I think it would be good if the Z-Wave binding could serve them well.

2 Likes
14

I know that I can sometimes come off as criticizing when I have no such intention, and this is the case here too. Just for the record, I haven’t intended to criticize anybody for anything here, I’m merely asking: Is the situation somewhat like I perceive it - and if so - what can be done about it?

1 Like
15

Well, that quoted statement was primarily on Docker, much less so on ZW-JS so if anyone submitted a ZW-JS pull request to openHABian to work without Docker, I might give it a chance.
But we already have Zigbee2MQTT and the ZWay binding in openHABian and I’m reluctant to provide and maintain 3 different solutions for the same need.

And I’m with you on this. @chris’ Java solution is the most integrated one and definitely the preferred option so I’d love to see that getting at least the maintenance to be on par with the others when it comes to devices support. S2 etc. ain’t needed.

2 Likes
16

I’ve started to upgrade to OH5. I decided to move to a Raspberry Pi 5 for the OH server, and I also wanted to move my ZWave items to a newer controller. Currently I have a Huzbzb stick. For the upgrade, I went with a Zooz 800 lr stick. I also decided to try out the new zwave-js-ui binding. I decided to install zwave-js-ui on an RPI4. I tried the bare metal instal, but couldn’t get the zwave logs to use a time zone other than GMT. The I installed it in Docker, which I’ve never used before, but finally got it going. It turns out I was making it harder that it needed to be. The time zone issue was resolved in the Docker install.

The biggest benefits I’m hoping to get are 1. being able to remove dead nodes, and 2. being able to have a test OH system that can also see the ZWave items since they are now available from the ZWave-js-u server.

17

That has always been “possible” using Simplicity Studio or “Z-Wave PC Controller 4.78” which I think is a precursor. I prefer that, because it’s less hassle. But, that means moving the stick to a different computer, which is why I’ve installed ZUI with Docker on my RPi - but disabled Docker so that it doesn’t run on startup. When I want to remove nodes, I can just stop the OH Z-Wave binding, start Docker and ZUI, remove node(s), shut it down again and restart the Z-Wave binding. Yeah, it’s a bit of hassle, but at least I don’t have to move the stick that way.

Unless you have dead nodes to remove on a very regular basis, it’s not really required that you can do that “live” IMO, but of course it doesn’t hurt.

18

I’ve implemented part of my “plan”. I’ve created this add-on:

It’s already on the community marketplace, and I’ve successfully tested it on my 4.2.3 installation. I expect it to work down to 4.1.0, which is the case for another add-on I created in the same “build environment”. It’s built against 4.3.0, using Java 17. It also works with the latest 5.1 snapshot.

This means that it should already be possible to use this to provide device descriptions for older OH versions that won’t see any database updates. Files can be downloaded using the database and put in conf/thingtypes and it should just work.

There are two caveats, that I’ve already mentioned above:

  • There is no priority/rank solution, so it will be “random” which version of a specific ThingType is used if the same ThingTypeUID is provided using this add-on and another source (like a binding). I can make a solution for this, but it would require changes in Core (so I’d appreciate hints as to whether this is worth doing/would be merged).
  • The ZWave binding only reads the ThingTypes during startup, and the cache is static, so it doesn’t even help to restart the bundle. The whole JVM must be restarted for changes to register in the ZWave binding, but the changes in OH in general are registered as soon as a file is added/deleted/modified. I can change this, but it would require changes to the ZWave binding (so I’d appreciate hints as to whether this is worth doing/would be merged)

I don’t know if there are other bindings with a similar “challenge”, but if there are; this add-on should be able to serve them as well. The add-on is in no way specific to the ZWave add-on.

19

I just tried to look at the JSON export from the database, to potentially support that as well - but it seems to be “broken” to me - what I get isn’t a valid JSON. Is this something that was once started but never completed, or is it some way to interpret/use this information?