Marketplace MQTT Event Bus

Tutorials & Examples Solutions
41

Accepted proof of QOS2 working, would be to still get a message, for an event that took place while the node (remote openhab in my case) was offline, once it went back online. Simply getting a message proves MQTT publishing & subscribing works, but can it still work with a flaky network.

Test 1: Here’s how Ive been proving QOS0 works: System “setup” (keeping it vague), event triggered on remote openhab instance, event received on my main openhab instance. Piece of cake.

Test 2: QOS2 with crappy/unreliable network: First test the system works via QOS0. Then change all “nodes” along the way to QOS2 (publisher, subscriber, openhab…). Then unplug the remote openhabs network cable. Then trigger an event on the remote openhab. Wait 10 minutes. Then plug it back in. If QOS2 is successfully setup, you will recieve the event on the main openhab instance. If its only setup for QOS0, then youll never get it because it did its job: fire and forget. So, I did get this to work great with “MQTT Persistent Service”, but since youre telling me thats going away I need a new solution. I’m going to try your solution (event bus), and also test that I can get QOS2, and I will report back.

I’m going to be perfectly clear: most people wont give a crap or have the need to do what I’m trying to do :slight_smile: So for that reason I won’t take it personally if you dont respond! ha

My use case and why I REQUIRE QOS2: I’m guaranteed the delivery of the message. Think the security context. Again, I have temperature sensors that report every 5 minutes that I dont care about. I can miss every other message and I wont care. But my door sensor I need to know exactly when it was opened, and what time it closed.

You are probably 100% correct that openhab is event based and MOST people will never have a need for what I’m trying to do. End-goal: I’d like some kind of dashboard (either openhab or grafana pointing to my influx data on openhab), with all the “regular” use cases (most of this is working great now), but ALSO, a way to drill into my remote openhab instances and look at all the door activity: maybe a grafana table that shows all the “activity”; the times the door opened and closed. I hope that use-case makes sense so I don’t sound like Im nuts :slight_smile:

Regarding the whole state thing, and if you get a flood of MQTT messages all at once, out of order, once the network connection resumes - youre 100% correct… that’s why I was wondering about trying to add the “event time”, along with the event. (Typically, or in all cases, openhab says, oh, event happened? Cool. Ill add that to my “log”, along with the current time. It happens auto-magically. So I get it, again, my use-case, and idea aren’t something people commonly do.) The end result would be a dashboard (like normal people), and the ability to drill in and get the reporting for the door: Im positive Ill run into some more issues, specifically like the ones you mentioned - but I also hope Ill figure them all out.

Ie, if I have “normal” openhab events along with its auto-populated event times, that go into my influx db, I’m good. The reporting (say grafana) runs a query, generates the graph or whatever, all is great. In addition to that, I use MQTT to pass along remote openhab events, along with MY manually/programatically generated timestamps, they end up in the influx db as well. Then grafana runs a query (doesnt care where the event timestamp came from: auto vs manual), draws a chart, all is good. It also generates the activity reports.

Regarding the retention feature of MQTT: that won’t make my “door activity” reporting 100% accurate, and I already know I have an unreliable connection between the publisher and the broker, so yea, sucks for me :slight_smile: And obviously retention is perfect for last know state…but not all the previous opens and closes that happened when it was offline.

I used to use Wink. I have one at each house (remote location), theyd all tie into my Wink account/cloud. Wink works, mostly. It has ZERO customization. Then I found out about this. Besides the lack of customization, a major issue with the Wink hubs/bridges in these crappy network areas are I’d miss many events when a door would open, and the network had a blip - so I’m trying this system to fix that issue. (I use Tmobile hotspots for the Wink to connect to the “Internet”, and into the Wink cloud from there so I’m just asking for trouble - as opposed to a hard wired internet connection. But I pay $5/month for these hotspots. These remote locations move often to different houses so I can’t setup a permanent connection - its construction and I’m temporarily monitoring that the heat is still working in the winter so pipes dont freeze and no one breaks in, or when workers come and go, which is why I MUST get the door alerts.)

Anyways, hope that answers some questions. I certainly have some homework to do.

Quick questions:

  • How does one know if a binding or add-on is a ver1 binding, and will soon be deprecated? Is that the orange “v1” tag, like here: MQTT - Persistence Services | openHAB ?
  • Given my “odd” use-case, without wasting your time with an answer, do you (or anyone else) know if it will even be possible to pass a timestamp with my event? (So even if I get a flood of events, likely out of order, I’ll apply the timestamp to “true-up” the data & events)
  • Am I going down the wrong path?! If my use-case makes sense, should I be using something else? (I’m still using openhab in my house for “regular” things, and it works perfect and is awesome. Its just the remote instances I’m trying to get “just-right”) (Any of this relevant? HomeAssistant MQTT Components - Bindings | openHAB or MQTT Homie - Bindings | openHAB )
  • With your Event Bus method: Do you happen to know for sure, with a 10min network outage, will the event still get sent once the network connection resumes? (If not, no biggie, Ill test & let the group know)

Thank you!!!

42

If you have control of your remote client, and it knows the time, you can add a timestamp payload field to its messages.

If you want to add a timestamp to the worldwide MQTT specification, that is rather beyond the scope of this forum.

I suspect so. Neither openHAB nor MQTT are suitable for mission critical systems like air traffic control, they are deliberately designed to cope gracefully with a flaky world. Not to remove the flakiness.

I’m not being funny here - just for example, I cringe everytime someone creates an openHAB “burglar alarm”. Yes, it’s better than nothing … but purpose made systems are cheap and the designers have thought of a hundred things that you haven’t.

If you absolutely must know exactly when a door was opened despite external disasters, the only way I can see is a robust independent battery-backed system mounted at the door itself. It need only keep a timestamped events log, that you can interrogate remotely when your primary systems are feeling better.

Actually that sounds rather like a burglar alarm …

43

Mission critical, like a burglar system 100% uptime, no. But I do know I get network blips so I do want to account for that which the MQTT qos2 specification appears to be able to handle just fine, and my testing proves it. (All the other things that may happen I’m fine with - Wink probably gave me 75% of my alerts, closer to 90%+ would be fantastic)

So pass the time field ( I saw you can easily pass a comma separated list of variables with MQTT persistence service but that’s going away), does the event bus have a way to do this? Or would I just pass it Via another channel and hold the info in a new item field?

Btw I just started messing with openhab a few weeks ago, but come from a sw dev background so I’m still trying to wrap my head around items, and having one for every variable, and then get this insane list in the gui. But then again, maybe I should stop starring at Paper and only look at the configs! (?) Or put in a feature request to group the items in paper … My items list seems abundant this early in the game, but I guess it’s designed this way to be able to then use & display variables. But at my rate I’ll have thousands in no time - maybe that’s normal… Or maybe I’m doing it all wrong!!!

You guys are great for your help & I do appreciate the humor :slight_smile:

44

Well, as I trust I’ve shown here, you can’t get there from here. openHAB can not and without massive changes to the core and overall architecture will never be able to handle this use case.

Your best approach is to record everything locally and when connectivity is restored sync the databases or configure your Grafana instance to query the remote instance database. You can report the time that the door entered it’s current state in a separate Item and you can publish that Item to your event bus too and show it on your sitemap. That gets your display. But even if you have a persistent connection you will never be able to tell OH that “this event actually occurred 10 minutes ago.” Only the remote OH will know that.

And that’s the part you can’t have. It can’t be done with OH as it exists.

What you can have an InfluxDB local to each OH instance and record everything locally when they occur. Then you can sync the databases or point your Grafana at both databases. But you cannot maintain the date time with the events across the two instances. You have to manage that outside of OH.

I still think this is an XY Problem. You are trying to solve the problem in a way that doesn’t actually solve your problem and is suboptimal. You need to look outside of OH to solve this.

But that isn’t to say that OH could not support that clear-session flag. But you need to file an issue to get that supported.

Lots of ways. That’s probably the easiest. Also, if you look at the URL the name of the add-on will often end in a 1. You can run a bundle:list in the Karaf console and the version will be 1.xx.

But keep in mind, that URL is to a Persistence add-on, not a Binding. There are six types of add-ons and the best way to tell depends on which type you are looking at.

No, not in a way that would be meaningful to OH. Your best bet is to record everything locally, publish the most recent event and the time using the event bus with the retained flag, and configure your Grafana to use both instances of InfluxDB or synchronizing the two instances of InfluxDB when connectivity is restored.

I think yes. MQTT is not going to solve this for you, at least not given how OH works.

Those two links are not relevant. They are cases of two MQTT topic standards (i.e. a standard in a way to use MQTT) that allows OH to automatically discover devices and create Things for you. They will do nothing for your use case.

I’ve never tested an outage for that long. Outages for a few dozen seconds work. And I can see the connection retries.

If it doesn’t work, I can add code that when the connection to the broker is restored it blasts out the current state of all the Items. That would be easy to implement.

But OH doesn’t.

It can do anything you code it to do. It’s all right there. You can add the date time to the one message which would just be adding now.toString to the publishMQTT calls. Or you can do it without changing code by using a separate DateTime Item. When the door opens on the remote instance, use a timestamp-update profile or a Rule to populate the corresponding DateTime Item. Then the Event Bus will publish it like any other configured Item.

But… the local instance of OH will still treat the events as if they occurred when the message was received. So your InfluxDB will never be correct. You will need to handle that outside of OH.

The MQTT does, or can be made to accurately cause the two OH instances to accurately reflect the current state of all the synced Items. It cannot be used to accurately reflect the past states of the two instances.

PaperUI is only intended for use as an administration tool. It is also being wholly replaced. For actual monitoring and use of your home automation you need to create a Sitemap or HAPPanel or play around with HABot. You have full customization control over these UIs.

That’s not unusual. Items are where you model your home automation. This is where you give zwave:saldkfjslkdf:node12:switch meaning (e.g. FrontroomLamp).

Since you have a SW background, I strongly recommend not getting started with Rules DSL. Use Scripted Automation instead and code Rules in Python. This gives you other options as well. For example, you can store and access data on Items using Item Metadata. This can eliminate a bunch of Items whose sole purpose is configuration information (e.g. During the day set this light to 50%, but at night set it to 75% which can be encoded as Item metadata on that Item or represented as two additional Items).

See [beta testers wanted!] Jython addon w/ helper libraries (requires OH 2.5.x) to get started.

Also, if you do use Scripted Automation, you can just clone the branch of the helper library where my PR is (see OP), copy the eventbus stull over to your $OH_CONF/automation folder and add two variables to configuration.py and you have the event bus. And Scott is working furiously to make that sequence of steps much simpler in the future.

And I look forward to the results of your test re QOS2 and a 10 minute loss of connectivity to the broker. If it doesn’t work, I’ll add the code to blast out the current state of everything when the broker connection is restored. But I’ll probably only add it to the Python code.

45

It’s an administrator’s interface.
It’s not supposed to look pretty or hide things.
There’s a big choice of proper user-facing UIs

There is scope for making it much “nicer” but it’s just a toolbox that needs repainting and tidying up. Effort better spent elsewhere, but you can of course take the job on :wink:

What you might like is to invent a handshaking protocol. That can carried over any transport you like, including MQTT.
Your client must be able to store and manage a list of events.
It tries repeatedly to send the oldest event to the remote system.
It does not give up and move to the next until a positive acknowledgement is received from the remote.
QOS2 is not exactly that because of the broker.

46

Paper UI is no more…

47

I already gave up on Wink, and have a good amount of time into openHAB already. My home with openHAB is working great now and I’m positive I’ll be able to satisfy all my HOME requirements. I just have to get this last remaining piece working (remote instance) and I’ll be all set :slight_smile: Well, until I get a touch tv in my kitchen showing a “family dashboard” :slight_smile: (I dream big!) I really hate having multiple systems so thats my motivation for the remote part. openHAB really is awesome - where it lacks (my use-case) it makes up for in the ability to add code/functions/scripting… so for that reason I’ll try to figure this out, I think I will. Unfortunately I’m not sure what my “flaky network” really means - 5 second blips or 1 minute blips…not sure I’ll be able to easily find out. If its 5 seconds sounds like Richs OP method will solve my issue.

Anyways, you guys provided some excellent information. I wish my task was easy, but it’s clearly not. It never is for me! I’ll have to become much more familiar with openHAB. I guess I was especially looking for clarity so I didn’t re-invent the wheel. Time-pending, I have to read up on scripting, figure out this event bus, test, see if I can update influxdb with scripting or an API… Id prefer not to touch the openhab code since thatll be a learning-curve in itself to learn the sw design in order to even make a code update (plus I may lose this with updates)… Anyways, I’ll report back :slight_smile:

Thanks again!!

48

Ok, just reread the thread, again, since that’s step 1 in my schaboggle :slight_smile:

So based on the OPs method, you need a generic mqtt thing to send/publish events to the broker. Ok. And the instructions beyond that are pretty simple.

I got mqtt publishing & subscribing (basically syncing across 2 insurance) using ONLY a broker thing and a generic mqtt thing, on each instance. The generic thing had channels that posted to the topics, and received on the other end that recieved them & posted to items. Ok. (Is this an eventbus the way you mean?)

My question is: what’s the difference between that, and Richs OP “eventbus method” that appears the same, but also has rules?

Ie, what’s the difference between OPs eventbus method with and without the rules, since the generic mqtt thing subscribes and publishes without the rules just fine? Do the rules give you more granularity & control? Or is this something to do with the binding deprecation you mentioned? And what’s your script/helper for, the code for your rules?

(Apologies in advance. I know I can figure it out… But the first 3x I did I used deprecated bindings UGH! It’s 1 thing to reinvent the wheel, it’s another to ride a horse…and Im all covered in horse meneur at this point!)

Thanks!

49

Hey this is my first time setting this up and I am not so savvy with Openhab, I tired the method explained above but it didn’t work.
My first goal is to get my Openhab event bus to show up under a topic in MQTT so I can read the outputs in NodeRed.
My second goal is to change my physical things into mqtt switches and dimmers.

I am wondering how did you achieve this using generic MQTT?

50

I have a need for 2 zwave usb sticks at 2 locations. Is it more stable to have 2 instances of OH connected via MQTT 2.5 event bus, or have the 2nd zwave usb stick via a SERNET/SOCAT type connection?

51

Without the Rule how do you define what Item’s updates and/or commands get published to the event bus? On the other side, how do you subscribe to the messages and update/command the corresponding local Items? That’s what the Rules do. You add the Items to the right Groups and the Rules publish based on the Group membership. For the subscription, the Rule determines what Item an update/command belongs to based on parsing the topic the message comes from.

To achieve this without Rules, you would need to create and configure Channels and Items to correspond with each Item you want to sync with the event bus. That is far more work over all than just adjusting an Item’s Group membership.

And it is possible to implement this without Group membership and just publish everything (at least in Scripted Automation, I don’t think this is possible in Rules DSL) but I think it’s better to give the users control over what gets published to the event bus.

The OP uses generic MQTT. I don’t understand the question. What errors are you seeing? What debug steps have you taken so far?

I’ve seen some report ser2net/socat is unreliable and unstable and others have reported that it is rock solid. I think YMMV. Either are an option.

52

I think the simple answer is via the channel linking and “FOLLOW” profile, instead of the “Default” profile. (See below)

I only have <10 data points to sync up, but here’s my configuration (Method Generic w/thing & items):

  • MQTT Broker Thing & Generic MQTT Thing on “Main” & “Remote” side.
  • Remote side: Generic MQTT Thing: Create channel(s) inside this thing, eg. “temp” & CmdTopic=property1/temp1. These channel(s) links to an actual existing Item (thats linked to a real temp sensor), but using the “FOLLOW” profile. This is important!
  • Main side: Generic MQTT Thing: Create channel(s) for each sensor or data point, eg. “temp” & StateTopic=property1/temp1. Create an item for each channel as normal, eg “temp”, maintaining the data type, number in this case, with the Default profile. Use this new item as normal.

My current topics:

  • property1/temp1 QOS:0
  • property1/humidity1 QOS:0
  • property1/door QOS:2
  • property1/doorbattery QOS:0
  • property1/motion1 QOS:2

This works & this alone (no rules, no scripting, nothing else except obviously a mosquitto server) sends updates from the remote sensors across the pipe to my main openhab instance. But I’m already guilty of using deprecated methods and “painting myself in a corner” it seems. So what’s the difference between what I did and the “Event Bus” method with rules? Is my way using the “Event bus”? Is your method just a different approach where yours gives you more granularity and control via scripting? Am I doing it wrong? I found this https://www.openhab.org/v2.3/addons/bindings/mqtt1/ (I see its for an older ver), but it does point out there are different methods such as “Item binding” & “Eventbus binding” - so maybe I’m doing it the deprecated way? (The MQTT Persistence Service was the easiest method I tried so far - too bad its going away )

Thanks again!

53

You have to individually set up the Channels and the Follow Profile for each and every Item you want to publish and you have to individually set up the Channels for each and every Item you want to subscribe and sync on the other side. That’s just standard MQTT configuration/integration like you would do with anything (e.g. Tasmota).

If you only have a few Items this isn’t that big of a deal. If you have hundreds or thousands of Items that approach is completely unmanageable. That’s what the Event Bus was created to address in MQTT v1 and that’s what the above was created to address. With a few lines of code, code you never even have to look at if you use Scripted Automation, you can get hundreds of Items publishing and subscribing.

I would say no. It’s not an event bus because you are essentially setting up the MQTT stuff individually on an Item by Item basis.

It’s not the degree of control. Both approaches have the same amount of control. But the event bus approach scales. Your approach does not. You may never need to scale, but for those that do, your approach is not suitable.

There really is no such thing as “wrong” if it works functionally. There are approaches that are harder or easier to understand, or take more or less work, or are easier or harder to maintain. But that doesn’t make them wrong, maybe just less good in certain circumstances.

No, if you are using the v2 binding you are not using the deprecated way. The code in the OP is the replacement for the “Eventbus binding” at that link because the v2 MQTT binding doesn’t support that natively for policy and technical reasons (i.e. the way the MQTT v1 event bus configuration works is not allowed in V2 bindings).

Easy doesn’t mean fit for purpose. But honestly, it was not much more easy to use than the OP above. You have to set up and configure the broker connection for both. You have to identify each Item that gets published to the event bus.

But with the MQTT Persistence it’s one way. You are on your own on the subscriber side and have to configure each and every Item you want to mirror with it’s work MQTT binding config.

With the OP above it’s two way. It handles both the publishing of updates and commands as well as what’s necessary to subscribe and update/command Items based on the published stuff. On the subscriber side you only need that 11(ish) lines of code (or import the MQTT Event Bus from the Helper Library) and create the Items (no links, no binding configs) and you are done.

MQTT Persistence only addresses half of the problem.

54

Crystal clear. Thank you Rich.

I’ll report back when I start digging into your approach & testing.

55

Trying to get this implemented, but I’m positive I’m missing something fundamental with the JSR223 initial setup (I never used this before. I’m not missing a binding am I?!) I’m on openhab ver openHAB 2.5.0-1 (Release Build) if that’s relevant. I found this in configuration.py.example: “Requirements MQTT 2.5 M1 or later Binding installed.” - I installed the MQTT binding inside the PaperUI, the binding that comes with it. Am I supposed to do something different? And I don’t see “M1” or whatever in my version, is that the issue?

Is a config file needed, like openHAB-conf\automation\lib\python\configuration.py ?

My Current error: (Rule is getting kicked off, but then I get the string split error - missing import? But not sure where I’d add that)

  • 2020-01-20 21:08:21.638 [INFO ] [ome.model.script.SubscribeMQTT.rules] - Subscribe update to the event bus
  • 2020-01-20 21:08:21.642 [ERROR] [ntime.internal.engine.RuleEngineImpl] - Rule ‘Subscribe for commands and updates from the event bus’: ‘split’ is not a member of ‘org.eclipse.smarthome.core.thing.events.ChannelTriggeredEvent’; line 8, column 20, length 24
My Setup Steps for Reference: (Probably wrong!)
  • Install (all openhab instances):
    ** Prerequisites: Install MQTT Broker binding & Install “Rule Engine (Experimental)” binding
    ** Download scripts: from [https://community.openhab.org/t/mqtt-2-5-event-bus/7693], click [https://github.com/openhab-scripters/openhab-helper-libraries/pull/257 GitHub pull request link]->Commits tab->"<>" symbol to “Browse the repository at this point in history”->Clone or Download zip link
    ** Implement scripts:
    *** Create: /etc/openhab2/automation/jsr223/ folder (if not already exists)
    *** Copy \Community\MQTT Eventbus\automation\jsr223\python -> /etc/openhab2/automation/jsr223/
    ** Add Broker Thing: Set IP/Hostname, QOS:2, ClientId: [openhabX], openhabian_mqtt / password
  • Install - Publisher:
    ** Create Group item in /etc/openhab2/items/groups.items: Group:String PubItems
    ** Add Items to this group (All items in this group will get published to an MQTT topic): eg.
    Contact Door “Door [%s]” (PubItems) {channel=“zwave:device:415c59ca:node3:sensor_door”}
  • Install - Subscriber:
    ** MQTT Broker: Add “Publish Trigger” Channel: ID:openhab1, label:subscription to openhab2, MQTT Topic:openhab2/out/#, Seperator character:#
    ** Create all Items that will be synched…
  • Restart openhab service: sudo systemctl restart openhab2
56

Maybe. Did you follow the instructions at JSR223 Scripting | openHAB, in particular installing the Experimental Next Gen Rules Engine (available under the Misc tab in PaperUI. Once you’ve done that I recommend following the instructions at Helper Libraries for openHAB Scripted Automation — openHAB Helper Libraries documentation or even better at [beta testers wanted!] Jython addon w/ helper libraries (requires OH 2.5.x).

This is required to set up Scripted Automation so you can run Rules in Python. As you can see with the last link, the steps to install it are getting easier and soon it will just require installing the add-on through PaperUI and in OH 3 it will just come installed by default.

openHAB has versions. You have the release versions (e.g. 2.5.1), testing or milestone releases (e.g. 2.5 M6 or milestone 6), and snapshots (e.g. 3.0-SNAPSHOT). That sentence means you need to run a version of OH, or at least a version of the MQTT binding that as released after the 2.5 M1 release which happened in February of last year IIRC.

Absolutely if you are using MQTT Eventbus by rkoshak · Pull Request #257 · openhab-scripters/openhab-helper-libraries · GitHub. It’s all in the docscrings (i.e. the comments between """ at the top of the file and in the Rule). In this case it says:

    Called when a member of the Group defined by eb_out_gr in configuration.py
    receives a command or an update. When the Rule triggers it publishes the
    state or command to the MQTT topic
    [eb_name]/out/[item name]/[state|command] where:
        - eb_name: defined in configurations, the name of this OH instance
        - item name: the name of the Item that triggerd the Rule
        - state: when the Rule was triggered by an update
        - command: when the Rule was triggered by a command.

When you first installed the Scripted Automation and the helper libraries per the instructions in the links above, one of the steps was to create this configuration.py file.

You missed the steps to install Python and the helper libraries.

57

I just updated to: 2.5.1-2 via the openhabian-config tool (01:Update, 02:Upgrade System, 40->41:Release version). I’m not sure if “2.5.1-2” is newer than 2.5 M1 release. Regardless, am I good now or do I need to further update based on the “Requirements MQTT 2.5 M1 or later Binding installed”? I’m also not clear if a snapshot is like alpha or beta (Id like to avoid alpha or beta at this point and I’d guess it is). I’ve been struggling the entire time I started using openhab finding documentation, or documentation that’s relevant given the quick evolution of the software (I find info for out-of-date versions that are no longer relevant all the time).

58

Since 2.5 has been released, any 2.5 release will be newer than any 2.5 Milestone. There is no reason to have a “testing” version for the already released version number.

OH 2.5.1-2 was released a few weeks ago. OH 2.5 M1 was released sometime in February of 2019. You are fine.

SNAPSHOT is literally everything that has been merged into the baseline in the past 24 hours. They are completely untested. And the 3.0 SNAPSHOTS are completely unusable right now anyway.

But pay attention to the repo you are using and the version number. Versions progress from

  • X.X SNAPSHOT: new version almost every day
  • X.X Milestone: about once a month when there are no known major breaking changes, a X.X SNAPSHOT will become a new Milestone.
  • X.X: about once every six months after a brief beta period, a SNAPSHOT will become the new point release.
59

You’re getting peoples’ hopes up for things that are not going to happen! The Jython bundle is never going to be available for installation through Paper UI in OH 2.5.x. As for OH 3.0, you’re making definitive statements about things that have not been finalized. There is a possibility that the new rule engine and Jython will be core features installed by default. It’s also possible that one or both will be optional. These things are currently being discussed and have not been finalized.

60

I FINALLY got the Jython hello world script to log. Since this took me a long time to figure out, I wrote up a guide: JSR223 Helper Libraries/ Jython Support Install Guide

Back to testing the event bus…