Z-Wave database maintenance

,
1

One of the big issues with maintenance of the Z-Wave binding is maintaining the device database. While the database isn’t needed to actually use a device (with some exceptions!), without the database, it’s not possible to configure a device since the configuration data is only provided in the manufacturers user manual… So, for OH1, the main purpose of the database is to provide the configuration, and also to add options to work around ‘bugs’ (or features!) in devices! For OH2 though, there will be more information to allow automatic bindings…

We now have over 230 devices in the database, and there are a number of requests each week to add more devices. Maintaining entries in the database is taking considerable time as it is often left to a couple of people to do (hence the backlog). This possibly also puts people of openHAB as they want to get their devices added, and it takes time with the limited number of people adding devices…

Unfortunately, this ‘issue’ is about to get twice as bad as the database in OH2 is a very different format (to match the requirements of OH2/ESH). I’ve written a converter to convert files from OH1 to OH2, but this still means maintaining two systems. This will inevitably end up with errors, and just be a bit of a nightmare to produce and maintain :frowning:

The other issue with editing the database, and one of the reasons it gets left to a few people to do, is that it’s all XML based. While this isn’t exactly difficult to edit, it’s not quick and easy by any means, and it in itself is prone to errors, and difficult to ensure correct formatting.

I think it would be good to have something more user friendly, and maybe meets the following requirements -:

  1. Simple to edit so anyone can update when theres errors. This in itself should improve the quality of the data.
  2. Provide a nice WYSIWYG editor to allow editing the HTML descriptions.
  3. A single system able to produce files for both OH1 and OH2 database formats.
  4. Be able to link to user manuals so we have a reference!

So… Over the Christmas break (while watching horizontal rain and gale force winds in South Wales!) I’ve put together something to do this. It started off as something for myself to make it easier to edit the file, but I think it would be beneficial if it was available more generally and I’d welcome any feedback on this… It provides a list of devices which is searchable, then a summary page once a device is selected which provides all the information on a device (config, associations etc), and then you can add and edit this information…

Below are a couple of screenshots… The live version if you want to have a look is here - it’s still in development, although most of the general data device information is done. (if you want to play with the editor, you’d need to register - I’ll likely rest everything shortly, so don’t worry about messing anything up!).

Comments welcome…

Cheers
Chris

Device list…

Pasted image1418×670 62.8 KB

Device configuration page (part of!)…

Pasted image1412×780 84.5 KB

Association group editor (part of!)…

Pasted image1254×828 44.4 KB

5 Likes
2

This looks like it will become an invaluable tool Chris - well done once again.

Couple of minor things I noticed when having a quick play;

  1. Can you sort the search constraint combos (for manufacturer and device class) - makes it a lot easier to quickly find what you are looking for.

  2. Can you display the device description in the results grid? Most people won’t know what the device code means and the description should make it a lot easier to parse.

  3. Would it be possible to group all instances for a single device (for different versions) in the search results? So you only see a single entry for Fibaro FGD211 for example. When clicking you then either show all versions in the detail view, or prompt the user to select which version they are interested in? Means you could remove the version min/max columns from the search results to make way for the description as well.

  4. Noticed some of the longer parameter descriptions have been truncated - is this something to do with your migration tool perhaps?

I can imagine this will become the go-to resource for device manuals and configuration references. Do you envisage this being bundled with HABMIN2?

Awesome stuff - hope you had a merrry Xmas and all the best for 2016!
Ben

1 Like
3

Hey Ben,
Thanks for the thoughts - yes, I’ll take a look at both 1 and 2…

This is something I looked at, but things start to get complicated with the database so I went with this (for now at least).

Yes - I’ve tried to keep things short… There’s (some) method in my madness - my concept is that names and labels should be kept short, along with values of options, so that they all fit nicely (without wrapping) in the various OH2 UIs. I’ve tried to add a short label, a short description, then a longer overview (which can be nicely formatted HTML). When I then generate the database XML files, I’ll use a combination of these to generate the files in a consistent way…

When I import the database before going live I guess there’s 2 options -:

  1. Remove the restriction and then put something in the UI to prompt people to shrink it.
  2. Truncate and also put the same thing into the overview.

Either way, it will require some manual tidying, but one of the ideas is that if it’s easy enough to do, in a nice editor, then we might get a better quality database as well. At the moment, there are a lot of long option values that are unnecessary (IMHO anyway :smile:).

Christmas was quiet (other than gale force winds, horizontal rain and too much food :smile:… And happy new year also. - looks like we won’t be down under again until later in the year (prob Dec)…

Cheers
Chris

4

Yep simpler is always better IMO as well. But some of these parameters are pretty complex in terms of how they work, and often benefit from examples showing how to configure them - think of the AeonLabs MultiSensor with the various bits enabling different sensor reporting - an example in that goes a long way to explaining how it works.

I am sure you will come up with something clever and very user friendly!

Shame about your trip down under, keep in touch, would be good to put a face to the name one day.

5

Yes - one thing I’ve also added, but not tested, is a user comments system - the idea being that people could add comments ‘outside’ of the database. I also thought about adding an OH1 items config section - for OH2 this is needed and I have ideas on this but it’s not all in there yet…

I’m definitely open to ideas - I don’t wanna make it too difficult or it’ll never get done, so I want to concentrate on the core database features first…

For sure - we will likely be down your way as my sister wants to South with us but it needs to be in the school hols… (my ‘kiwi experience’ for 2015 was a trip to the RWC final - absolutely awesome! :smile:)

6

Are you kidding?! I was there as well!! How good was that.

7

Damn - you should have mentioned that you were over this way… It was a great day though :smile:

8

Yeah I should have - was over for work and had zero time to do anything bar work and make it along to the games. Definitely a bucket list item ticked being there to watch that :).

9

Hi Chris,

That looks awesome. It will certainly make it easier to update the database.

My thoughts are similar to Ben in that a description of the item would be useful.

Thanks for the help yesterday too! Learning much, so sing out if there is anything that I can help with!

Cheers

Andrew

10

Awesome work @Chris. I would second the idea that this could someday end up as a part of HABMIN2.

I guess the part that I wonder about is how to hook it all together to get the community working on keeping it up to date. Could you hook it into Github, and provide change comments, which could then go back to a commit for you to check out and accept?

Happy New Year! Sorry to hear about your interesting yet blustery holiday weather.

11

Great work Chris! Is the idea to eventually have the database exist entirely outside the codebase for OH2? This would be pretty slick with a button in Habmin that says “import latest database” or such. People can then edit away without having to touch binding code to use and test their creations and new devices.

12

This is definately a concern, and why I raise it here. The point of course is that currently, most of the updates to the database are performed by a small number of people, and most of the requests are from a wider community, so it shouldn’t be too bad (I hope). I would like to think that if a simple GUI tool were available, people would use it rather than write XML…

I would of course still look to export to Github, but I’m not sure it can be hooked directly to Github - it would more be an export function…

No - ESH requires the database to be within the JAR (although it might be possible to write something that allowed it to be outside, but it’s not how it is currently implemented)…

My concern with having the database outside of he JAR is quality control and version control - I’d be concerned that people would edit the database, and potentially screw something up, and the interaction with the Java side could get very messy. Currently the database includes options to make it work correctly with certain devices that have bugs (sorry - lets call them implementation features :smile:) and if you set these wrong, we end up with something not working.

My personal preference is for people to contribute directly in a simple way… I can understand the rational behind wanting an external database, but as a maintainer, the uncontrolled nature makes me worry I’ll have a lot more headaches with support :frowning:

13

It is trade-offs, for certain, and part of the overall struggle @kai and team are going through with the core. How do you get more people involved to keep things current, but control the overall quality, feel, and vision of the product. I liken it to how Linus maintains the core kernel. Still not sure how he keeps up.

Maybe a channel thing where there is an experimental z-wave database, and a more hardened “production” one that has all the controls? It seems the problem will get worse instead of better with the rate of new products coming. The database maintenance site you created is certainly a step that will help either way.

14

Interesting idea. However, at the moment, people can compile their own JAR, and edit the XML, test and submit. Generally, the people who can do this are also the ones who are comfortable with editing the XML in the first place, so I’m not 100% sure it will really help?

Yep - that’s my view - try and make the whole process easier, and remove the whole XML thing from the users sight… Make it accessible so anyone can edit the database - provide people the ‘power’ to update the settings etc, and hopefully we’ll get a nicer database…

One of the problems with the current database is editing XML is prone to errors - or at least, formatting it up is not simple… It’s like writing HTML in a text editor - you can do it, but it’s gonna take a few attempts to get it right, and more to make it presentable… I want to put all this editing into a nice form, with a nice editor for editing the HTML bits…

15

Just a quick update on this - especially with respect to OH2 binding… I think the database is now mostly complete. It now allows uploading the node XML files from openHAB - this gets parsed and a list of endpoints and items/channels is generated (see below). This is then used to automatically generate the openHAB2 database XML file, and also the openHAB1 file (just for the configuration and association groups).

There is one last thing I need to add and that is the command class configuration options. These are options that we add to the command class to fix bugs/features of devices (ie to work around non standard implementations). Once this is done I will likely scrub the database completely and reload it from the current database so that I can start to migrate this toward OH2…

Again, if anyone wants to have a play, I’d welcome any feedback…

More information can be found here and the device list is here.

Cheers
Chris

Pasted image1414×1098 85.7 KB

1 Like
16

[quote=“chris, post:1, topic:5609, full:true”]
(if you want to play with the editor, you’d need to register[/quote]

Tried to register a couple of hours ago, no activation link was sent (also nothing in my spam folder).
Tried to register a second time now, still no activation link.

Could you have a look on your CMS?

17

Will do - I did get an email to say that you’d registered a few hours back - maybe it’s not sending externally…

It shows both your users are there. I see others have activated so I’m guessing it is sending (at least some times)…

I activated the first one… Will need to work out what’s up though, so thanks for the info…

18

I found a mail relay issue - I’ve now (hopefully) fixed this and have resent outstanding emails (but not the ones where I’ve manually authorised the accounts)…

19

Yes, thanks, mail from second registration was received some minutes ago.

20

I consider that the database is reasonably stable and I’d really like to start getting data into it so that we can get the OH2 database into shape. Thanks to everyone who has had a play - especially @xsnrg.

The system provides the following functions -:

  • Simple editing of parameters, associations and other data
  • Uploading of files associated with a device so we have a reference of user manuals etc
  • Logging of changes, and a ‘review’ process that allows some level of quality control
  • Upload of XML files from the nodeX.xml files to populate most of the devices detailed information
  • Upload of OH1 database files so that we can add files that are created manually
  • Copying of association and configuration parameters from one device to another new device
  • Export to both OH1 and OH2 database formats

I’ve updated the information on how to use the system here. This should (I hope) provide plenty of information on how to use the database, but please feel free to ask any questions.

Note that you will need to register in order to edit the database, and also email me to add you to the editors list. I appreciate this is a bit of a pain, but it is needed to make sure rogues don’t screw things up (although the database is backed up hourly!).

At the moment we still need to manually take the output and add it to the binding…

There are a lot of outstanding requests for database additions, so I’d really appreciate it if people can help support openHAB by adding devices as much as possible - it should now be relatively easy to add entries using the data in the pepper database, or manufacturers manuals.

http://www.cd-jackson.com/index.php/zwave/zwave-device-database/zwave-device-list

Any questions, problems, or suggestions, please ask :smile:

Cheers
Chris

1 Like