The Great ReBuild of 2019 - Best Practice, ontology and to fix what isn't broken

So, this will potentially be a wide discussion. Or it might die at first post. I guess we’ll see! :smile:

I joined the OpenHAB train by v.2 and currently have a setup containing most well known standards and a few lesser known ones. It all runs on a RPi3 and works pretty well. It has, however, grown quite organically and what started with structure and order is today… less structured. Both in and outside of OpenHAB itself (i.e. I have packages that has become redundant and then only partly removed, services running that I under utilize etc). After some deliberation and because I apparently enjoy complicating my life unnecessarily, I’ve embarked on rebuilding everything. The technical platform will be the same; i.e. RPi3 running OpenHABian with whatever bindings I actually use. I will however endeavor to be more best practice oriented.

For now, I am maintaining a single SD-card which, when it is setup with basic configuration - including security, backup and DR capability - will be backed up in entirety. So will all relevant hardware (z-wave stick and other configuration containing devices). Documentation will also be kept. I will then keep to a schedule of automatic backup/restore of content and configurations, as well as manual mirrors for the same as previous stated. I will use a dev > test/staging > prod setup to maintain stability. So. My first questions:

  1. Do you have any general input on do’s/don’t’s for maintaining the environment and how to develop my environment?

  2. Ontology; it seems that this is a golden opportunity to implement Eclipse SmartHome Ontology. Admittedly it is not quite as well documented as one would wish for (or it may be that I am less than stellar in finding it), but although it may not be quite as mature as everything else it still feels like too well aligned with improving structure not to take into account. Not too many topics on it in the community though, and in those few I’ve found most questions have really gone unanswered. I have one big one at this point: I miss several fairly obvious location types (most specifically room types; office, nursery, guestroom spring to mind). When I study the documentation it doesn’t really seem correct to simply add the on my own - is that correctly interpreted? Or can I just go ahead and do so? If not, how do I go about suggesting such additions?

  3. Are there other compilations of “best practice” oriented white papers out there? I’ve seen some of @rlkoshak excellent best practice oriented forum posts - are there other sources that should be considered and that could be recommended?

  4. Are there any technical or architectural improvements you would suggest? I’ve glanced at docker but haven’t really seriously considered it - should I? Anything else?

  5. Is there any official way that new versions of OH are announced that is “subscribable”? I want to stay away from potentially breaking updates so I would like to have full insight in when the stable release is updated. :slight_smile:

I’ll rest there for now. Feel free to pitch in with any thoughts or ideas you’ve had yourself - tried or not - that you feel I should consider!

Cheers!

Petter

On 2) - never heard of people to use it
I think it is key to understand that OH implies no taxonomy whatsoever - i.e. it doesn’t follow, mandate or just prefer a specific representation of the physical world and the rules logic to apply to it. That’s blessing and sometimes curse, but I’d stick with that and make up my own mind on all of these.
On 3) - none I heard of. Yes of course you’ll get a couple of hits such as blogs if you google.
But I’m convinced the best place to find the most, most comprehensive and most up to date knowledge on best OH practices are this forum and the official docs (worth re-reading after some time as they keep improving, too).
On 4) - well. There’s major discussion going on ATM what to recommend - the outcome will probably be pretty much along the lines what you’ve defined for yourself already. And the most important recommendation will be to use what you know and prefer - which is what you already do.
On 5) - you can watch the build repositories for either release, milestone or even snapshot builds. The first two list all breaking changes in their readme that developers are aware of.

Thanks for taking the time, Markus; I appreciate it!

On 2) Well, yeah; still this is a breaking discipline. Automating may have happened in more enterprise scale situations before, but the vast invasion of homes and premises is currently ongoing. I would expect that systemic standard will go from being of fringe interest to becoming increasingly important. With the emergence of HABot, at least, the Eclipse SH Ontology seems to have found some foothold. I do like the idea of adhering to a standardized framework… but should I read your input as Brick/Ecl.SH.Ont having been marginalized (again)…?

On the rest) Thanks a bunch. I live in the documentation, actually, and reread atleasr on major releases. And most definitely as I started this project. :slight_smile: By the way, I believe it was your writing of thinking of the complete chain in creating continuity - and not just stare blindly on the SD card/data (in the Amanda for OpenHABian readme) - which had me including the config carrying hardware such as the z-stick. So special thanks for that! ^^

//Petter

I would interested to hear how others maintain a separate development environment. Currently I also run on the PI, and develop on it as well. Due to the hardware integration nature, unless the rules are abstract enough, it is not easy to test on a separate hardware that doesn’t have coordinators such ZWave and Zigbee. I am also backing up the PI using these commands:

Backup: 
sudo dd bs=4M if=/dev/sdb of=~/backups/raspberry-pi-images/raspbian-2018-10-29-weather.img

Restore:
sudo dd bs=4MB if=~/backups/raspberry-pi-images/raspbian-2018-10-29-weather.img of=/dev/sdb conv=fsync

It works but it is not ideal. Whenever my PI is off for more than 30’, some of my Zigbee sensors (Xiaomi motion sensor) will go offline, and require re-pairing. They sometimes got into weird state too.

I know you’re talking about architecture level, but at a lower level, I would consider whether continuing to use xtend rule is the wise choice now. There are a number of limitations with xtend rules including the inability to test them.

If rules are abstract enough, we can test the logic without needing the hardware. In addition, common home automation action could be generalize into reusable modules. I now create my rules in Python, which provides a lot more freedom than xtend.

Also sort of related to everything else, I’ve been pondering on how the devices have been represented. If you look at the devices and the rules today, there are similarity between them and the relational database in the development world. We tend to tightly bind our rules to the specific items. But perhaps we should think of higher level constructs, something like the Object Relation Mapping (ORM) in the development world.

A possible way to model a HA system is by the physical layout. A Room object can contains various device objects such a Light, MotionSensor, Temperature, and so on. This allows creating a central manager that control/query various aspect of one or more rooms, and allow a layer of abstraction on top of the flat items. When time allowed, I plan to re-factor my current Python light rules, and see if modelling like above works well enough. It bugs me today when the complex light rules aren’t testable.

  1. Try to automate the backups if you can. Because of hardware limitations, you will be somewhat limited in what you can do with three environments because, for example, you only have the one Zwave controller. Be sure to take that into account. Don’t over proscribe your development process. Make sure that all the steps you are imposing on yourself actually add value. With a good backup and restore strategy you can recover from a lot of mistakes in no time at all. Ask yourself, are you really gaining anything by having three simultaneous deployments? I’m not saying you are not. I just want you and other users to think about what value the steps add value to you and you are not just doing it because it’s a “best practice.”

  2. Yannick has started docs here. I can’t answer your specific questions. I’ve not played with HABot and the ontology yet. But it is admitted to be an initial release so there will be lots of stuff like this that hasn’t been figured out yet.

  3. The Design Patterns are good for Rules development. The Tutorials and Solutions section is intended to be this as well. But to a large degree we try not to over define what is and is not a best practice. Every home automation system is different. Even trying to come up with a naming convention recommendation was a controversial and extended argument. Just this week there is a multi-hundred post thread about whether it makes sense to add a couple of sentences to the intro document recommending a hardware platform. We have lots of users with strong and orthogonal opinions on just about everything. So you won’t find much. When we can come up with a consensus, that pretty much always ends up in the Docs (for example, the Item naming convention consensus is in the Item configuration docs https://www.openhab.org/docs/configuration/items.html#name).

  4. I see no reason to use Docker on an RPi unless you have other compelling reasons to use Docker (e.g. the rest of your environment is already Dockerized). A straight openHABian will give you the most common deployment of OH. Depending on other problems you might experience, you might consider:

  • hosting persistence DBs and Grafana (if you are using them) on a different machine
  • moving the file system to an external SSD or HDD to mitigate SD card wear out
  • minimizing writes to the SD card to mitigate SD card wear out

But in general, what you propose is a pretty standard install.

  1. There is an announcement on this forum in the Announcements category and one or more articles posted to the openHAB Blog. I’m pretty sure you can set the Announcements category to “watching” (button at the bottom of the page I think) and you will get an email for all new posts.

I don’t think so. It has only been part of OH for a couple of months now and it is primarily used to drive HABot which not many users are using yet. There isn’t enough of a user base yet to figure out where it’s going, if anywhere.

I personally do not maintain a separate dev deployment. But if I were to do so, it isn’t the Rules that need to be abstracted, it’s the Items and they already are abstract enough. All you will need to come up with is a way to exercise your Items and Things independent of the hardware since they won’t actually be connected to the devices. Thankfully this isn’t so hard through the REST API or by creating test Rules.

Then all your Rules and sitemaps and such can be developed and tested and when you move it to prod you just need to create the link to the Things (if they don’t already exist) to connect the behaviors to the devices.

Unfortunately development or testing that involves adding new hardware or figuring out a problem with a specific piece of hardware is not going to be possible in dev. You have to have the hardware to test the hardware and the only way to have the hardware is to buy two of them, take the hardware away from prod, or do that development and testing in prod.

Explain. There are plenty of ways to test Xtend Rules. I’m not necessarily arguing against the recommendation (though unless one wants to move to JSR223 I do think the recommendation is premature) but saying one cannot test Xtend Rules is not a correct statement.

It’s the Items that need to be abstract, and they are. The Rules don’t care that your LivingRoom_Lamp is connected to a Zwave outlet, an MQTT Sonoff, is a Proxy Item, or is an Unbound/Virtual Item. The Rule will do what you tell it to do on that Item regardless. You as the tester just need to look at your sitemap or watch events.log and openhab.log to see whether the Item changes states as expected when the Rules run. One can even write testing Rules to cause Items to change state that triggers Rules, wait a bit, then check that all the Items are in the expected states once the Rules are done.

Only if one is not following the Design Patterns and other coding best practices documented here on the forum and elsewhere. Rule can and should be generic. If Rules ARE tightly bound to specific Items then that is a code smell and there might be a better approach.

I find it woks far better to organize your HA system based on function rather than location. When using function (e.g. lighting) you increase the likelihood that similar code will be co-located, increase the chances for shared code, and increase opportunities for code reuse.

What have you tried? This sort of Rule should be very testable. But note that “there ain’t no such thing as a free lunch”. You have to write your tests just like with any programming platform.

1 Like

I do use groups a lot; it is the most sensible way to group related items. But you can’t really get away from knowing specific items. For example, a motion sensor is associated with a light switch and a timer. There is a direct connection between them. One way to achieve this is to have the same prefix such a FF_Foyer_.

When I meant higher level of abstraction, I refer to the ability to take certain action without needing to know the specific item name. Obviously the relationship still have to be defined either through the naming pattern or the tag mechanism. However, by organizing based on components, we can say something like this in the rule:

  • When a motion sensor is triggered, ask the manager to turn on the room’s light and set the timer.
  • When a timer expires, get the room it’s in, and turn off the light there.

It also allows asking questions like: is the room occupied? Here the implementation could be based on the switch state, or the motion sensor state, or a combination of both. The key thing is that the rules need not know the names of the items in each room.

I never tried to test the xTend rules. I find that my hands are tied due to the lack of component support. What I’ve been doing with the Jython rules is to have as little code in the rules themselves as possible. All business logic code goes into the module/library. These can then be tested using Python’s JUnit equivalent. There are different levels of testings here: test logic that is not tied to OH objects at all, and test logic that directly interacts with the OH items. For the latter, the JSR232 allows the ability to create/remove items directly, which I don’t think is available to xTend.

In any case, what I am describing here is regular software engineering principles. It’s difficult to achieve with the regular user bases since there are such a diverse background. At the same time, it is painful to see everyone doing the same thing again and again. As HA matures, IMO, we will eventually converge into reusable components that reuse the same logic. All the user need to do is to define the items, and the configuration values. For example, with the physical layout above, the logic should be common enough to handle many usage scenarios.

I also think this is the trend for devices such as Amazon Alexa and Google Home. It is perfectly doable to create a rule engine. They can implement something like this: “OK Google, when XYZ motion sensor is triggered, turn on this room’s light for x minutes”. And then provide management interface such as “OK Google, read me all the rules”, “OK Google, change this xyz rule to …”

Why don’t you use Amanda ?
If you have 2 systems (e.g. prod + dev), you can set it up to run on just one of them to be the backup host (which has access to the storage), the other being a backup client.

All of this describes rules DSL best practices. In my personal setup I think I have one rule, maybe two, that directly references an item by name. All the rest use Member of triggers and triggeringItem to reference the item (or construct the name of an associated item).

Have you looked at HABot? That shirt is thing is what the new ontology tragging is supposed to help support.

This is where I have a problem with your original statement. You said one cannot test Rules DSL Rules. That tells all the future readers of this forum that testi g them is impossible. This is a flat out false and missleading statement and why I said something.

Had you actually said “I don’t like the limitations of Xtend which makes it harder for me to build and test rules the way I want to” I’d have had no problems.

Far too often people say “X is impossible” when what they mean is “I don’t like how X works” and new users and less technical users end up becoming missinformed and mislead.

That is one of the main drivers behind the next gen rules engine which will replace Rules DSL as the default one it matures.

Blockquote
I do use groups a lot; it is the most sensible way to group related items. But you can’t really get away from knowing specific items. For example, a motion sensor is associated with a light switch and a timer. There is a direct connection between them. One way to achieve this is to have the same prefix such a FF_Foyer_.

Blockquote
When I meant higher level of abstraction, I refer to the ability to take certain action without needing to know the specific item name. Obviously the relationship still have to be defined either through the naming pattern or the tag mechanism.

You are more or less making the case of standardized ontology; I tend to agree with you, which is one of the reasons I am considering getting in early on the Eclipse adaptation of Brick. The challenge being that, as the discussion shows, it is still seriously lacking in many fairly basic aspects. All part of being an early adopter, I assume. We’ll see if I try to get onboard. Also +1 on Markus comment to use Amanda for automatic backup. That along with the consoles ability to capture a config snapshot prior to change has been enough to stay stable so far.

Thanks for a long and detailed answer, I really appreciate it. You may have me reconsidering my development setup. My current live setup has managed well with nightly backups and the occasional manual backup prior to larger changes. Of course, I will rebuild on a fresh SD-card rather than trying to sort our the old mess. ^^

Getting an external HDD is a real good idea! That would certainly decrease the risk for SD card degradation. On the other hand, quality backup and easy restore can mitigate that risk fairly well - and a new SD card (if push comes to shove) is not expensive… while an extra HDD both increases complexity and physical footprint. Still, I think I will go down that path.

A big thank you to all of you for your answers!