Node-RED as Alternative Rule Engine

[rgerrans]

@dakipro

What would you guys suggest for notification system

I’m overdue to add my notification system to the tutorial. I use Node-Red for the logic and email-based notifications, and OpenHAB for other notifications (via broadcast to app and voice to audio sink). For the OH notifications, I do it by defining notification string variable items in OH, populating them in Node-RED and then using OH rules to trigger the notifications when the variables are updated.

Items:

String  vNotification
String vSpeech

OH Rules:

rule "Send Notification"
when
	Item vNotification received command
then
	sendBroadcastNotification(vNotification.state.toString)
end

rule "Send Speech"
when
	Item vSpeech received update
then
    say(vSpeech.state.toString)
end

I’ll try to update my tutorial with all the details this week.

[rgerrans]

Updated Finally carved out some time to add a Notifications section to the main flow and comment on how I integrated with MQTT:

MQTT

I have fully migrated my deployment over to MQTT using the built in MQTT nodes (MQTT Archives - Node RED Programming Guide) You just need to replace any input / ouput nodes in the below example with MQTT nodes. The one item of note is you can’t (or at least I haven’t figured out how to) request a status from the MQTT broker like you can in the OpenHAB nodes. Instead you have to store any state information in a Node-RED flow or global variable like I defined below.

Notifications

While Node-RED has some addon Nodes to help support different types of notifications, I wanted to still be able to leverage some of the built in functionality of OpenHAB as well. This required setting up necessary OpenHAB items and rules to interfact into OpenHAB’s functionality.

Broadcast Notifications
Through the OpenHAB Cloud Connector http://docs.openhab.org/addons/io/openhabcloud/readme.html you are able to use the mobile apps to both access your system and receive broadcast notifications http://www.myopenhab.org/. To integrate this into Node-RED requires an item to carry the notification information and a rule to trigger the notification.

Item Definition:

String  vNotification  "Notification [%s]"`

Notifications.rules:

rule "Send Notification"
when
	Item vNotification received update
then
	sendBroadcastNotification(vNotification.state.toString)
end

And finally a Node-RED flow to send the notification. Here I used a change node to create my text message:

You can also use the template node to include other payloads or topics into your notification message:

If you want to add a time stamp to your message I was able to put together the following to create a plain language time stamp (i.e. 3:32 pm)

For cut-and-paste purposes:

msg.payload =  msg.payload + new Date().toLocaleTimeString('en-US',{timeZoneName: 'short'});
return msg;

Audio & Voice
OpenHAB has the ability to play audio files and convert text-to-speech: http://docs.openhab.org/configuration/multimedia.html. To take advantage of this integration we follow a similar approach as we did for the Broadcast Notifications above using a combination of a custom item and a rule combined with a Node-RED flow.

First to trigger a text-to-voice (assuming you have already setup everything in OpenHAB as per above).

Item Definition:

String vSpeech

Speech.rules:

rule “Send Speech”
when
Item vSpeech received update
then
say(vSpeech.state.toString)
end

And finally a NodeRed flow to tie it together.

I went a step further with mine to control the volume levels during and after sending speech (turning it up while the speech is broadcast and turning it back down). So I created a sub-flow (https://github.com/node-red/node-red/wiki/Design:-subflows

Similarly for sending an Audio file saved to your conf/sounds directory:

Item Definition:

String vSound

Sound.rules:

rule "Send Sound"
when
	Item vSound received update
then
    playSound(vSound.state.toString)
end

And Node-RED flow:

Email
For email you can choose to use the SendMail binding http://docs.openhab.org/addons/actions/mail/readme.html or alternatively you can do it directly from Node-RED using the email node https://flows.nodered.org/node/node-red-node-email. The one trick I did apply here is I didn’t want to configure each node instance with my smtp server information and I wanted the ability to turn on and off the ability to send email (so I didn’t spam my wife when I was testing) so instead created an email subflow and a control flow.

Email Subflow:

Control Flow:

[dakipro (dalibor)]

Thank you for the notifications flow, I like the usage of sub-flows to make things more reusable. I am thinking on extending it to have different priority levels sent with the message and then in the subflow have the logic that will determine what will be sent, where and when. (some messages I want to see immediately on the phone, some maybe once a day on email, some should be announced etc).

You said that “I have fully migrated my deployment over to MQTT using the built in MQTT nodes” - is this just to be independent of specific binding and have it all via “pure” mqtt, or do you have some other motives as well?

[rgerrans]

@dakipro

I am thinking on extending it to have different priority levels sent with the message

You can add your own subtopic to the payload to make that easy enough to do msg.payload.priority

@dakipro

is this just to be independent of specific binding and have it all via “pure” mqtt, or do you have some other motives as well?

Part of it is to channel my inner-geek just to do it. It is more the idea of moving to a tiered approach with mqtt being the central bus. When I started I suspected that using an mqtt broker might be likely a fraction faster and less load then monitoring a REST API like the openhab nodes do, plus I was looking into migrating to ZWay (didn’t support locks very well) or OpenZWave (doesn’t work with my Cooper dimmers) so wanted a more flexible approach.

[rswennen (Rohnny Swennen)]

@rgerrans thanks a lot for this tutorial, installed node red yesterday and I totally get it that this is the type of tools OH is missing.

But this is more than just a GUI to the OH rules, it’s a complete new rules engine, so you process everything outside of OH. You indicate you moved all MQTT away from OH to node red. But node red has nodes for a lot of other interfaces as well, like serial, global cache.

What is in your view the setup to go for, migrate other bindings to node red as well when there is a good node available or you think MQTT will be all ?
I can imagine when moving other bindings to node red as well but then what role does OH play ? Just an inventory of items? Just curious as, as excited as I am to start using node red as rule engine, I’m also concerned that in the end I’ll end up with a patchwork of systems that’s going to break every time some component upgrades (it’s already a challenge to keep OH operational and stable with every upgrade).

[rgerrans]

Adding Node-RED definitely increases the overall complexity, but also increases the overall flexibility.

When I look at OH I think of it as three layers: the bindings (the interface layer); the rules engine (the application layer); and the GUI (the presentation layer).

By moving to MQTT to sit on top of the interface layer, I made it so I can isolate the OH bindings as interfaces and expand the interface layer through other programs. A specific example in my case is that I have an NX584 alarm system. The current OH binding didn’t quite meet my needs so I was able to use a 3rd party interface to map the communication to MQTT (I did it through Node-RED temporarily but will make some tweaks to do it direct to MQTT).

With Node-RED, I’ve pretty much replaced the application layer from OH and also using it as an additional interface layer by replacing a number of the bindings (I use a WeatherUnderground node to replace my WU binding, a timer node to replace Astro, and a harmony hub node to replace harmony binding) and then feed the outputs back into MQTT. I figure going direct through these nodes should improve performance (though I’m sure it’s a pretty minuscule improvement)

Now I’m looking at some MQTT GUI interface tools to potentially replace the OH SItemaps/App which will give me the ability to not have to map back into OH items that I control outside of it (like the alarm system, harmony, etc.) and give me more flexibility:

Need for a cross platform, customizable ui

I was in the same situation as TS some time ago, when I started. But then I found, that there are many nice products on the market, which you just need to find. So if you need cross-platform customizable UI without any programming I know at least two options. They both provide responsive feedback due to MQTT nature(no REST stuff). All you need is to set up MQTT event bus binding in OpenHAB and … CommandFusion iViewer - for this you will need MQTT Plug-in, which I provided in it’s group. Aft…

If all you are looking for is a more visual rules interface, it sounds like this is an interesting alternative:

Flows Builder - a visual designer for the new rules engine

Hi, I’ve been playing with the “experimental” rules engine lately, to see if I could migrate most of my DSL rules to it (they are pretty simple). It’s working OK so far, and I think it has a lot of potential, especially with the JSR 223 scripting support, but the Paper UI interface to design them was IMHO not great - lots of clicks, dialog boxes, unneeded animations and so on. So I went ahead and forked HABPanel to make my own IDE … it’s still good old AngularJS 1.x, not the fancy new …

[Artyom_Syomushkin]

@rgerrans

and also using it as an additional interface layer by replacing a number of the bindings (I use a WeatherUnderground node to replace my WU binding

Be careful of WU node in NodeRed - I was pretty happy initially when moved from Yahoo Weather binding to WU Node-Red node. But after some time it started to crash my Node-Red runtime, and I got really dissapointed. So I returned to OH weather binding. That’s why I keep staying with original concept - Openhab for bindings and Node-Red only for rules.

[rgerrans]

@Artyom_Syomushkin

But after some time it started to crash my Node-Red runtime, and I got really dissapointed

I had an issue of a runaway WU process. I reset my api key and no issues since.

[Artyom_Syomushkin]

BTW I started using Node-Red for rules at the moment when there were no other alternatives. But now I’m seriously evaluating Codesys with MQTT library. This is a serious “industrial” software, targeting automation applications. At some point it will replace Node-Red in my home.

[rgerrans]

Looks like overkill for what I need to do but will be interested in an update once you deploy it just to see what the overhead / reliability looks like.

Thanks

[Artyom_Syomushkin]

Reliability looks already great, as I have one RPi running Codesys autonomously for half a year without any issues.

[rswennen (Rohnny Swennen)]

@rgerrans have you ever looked into node red in combination with the OH homekit binding ? There is a separate node red home kit binding that seems to support all homekit types (outlet, fan, garage door etc.)

node-red-contrib-homekit

https://swifting.io/blog/2017/04/05/39-homekit-devices-simulator-using-node-red-raspberry-pi/

I’m tempted to look into using node red to try to support other types of devices.
One way would be to drop the OH homekit binding and use the node red one but that would mean I would need to redefine all lights again in node red and manually map them, not the smartest way IMO (the bulk of the items are lights which work well in OH - homekit).

So wondering whether you have any experience? Would it be possible to have the IOS device connect to 2 separate homekit accessories (the openhab bridge accessory in parallel with the accessories defined in node red?). This way one could leave all lights and thermostats in OH and create non supported accessories like fans or outlets in node red and link them though to whoever they are represented in OH (dimmer, switch, contact etc …)

[rgerrans]

@rswennen

have you ever looked into node red in combination with the OH homekit binding ?

Unfortunately I can’t help you there. I’m an Android guy so don’t know much about the capabilities / limitations of Homekit. Sorry.

If there is no added advantage of going direct from Node-RED to Homekit I wouldn’t move them over. My original setup was Node-RED as my rules engine and OH as the communication bus and UI interface. Once I jumped on the MQTT bus (as per my earlier post) I’ve started doing some direct connections from Node-RED or just to MQTT from other systems but for some items it’s as much because I can vs because I needed to.

[Joe_Lawrence]

I’ve had this happen a few times, ended up using NSSM (Non-Sucking Service Manager), to run Node-Red as a service, and re-start on crash. I also get notices when Node-Red “bounces”. Haven’t had a string of rapid fire “bounces” yet, although NSSM has a decent way of handling multiple crashes by extending the time between each restart until it works. You can still stop the service and run Node Red in a “Dos box” for trouble shooting purposes.

[shawnmix]

@rgerrans - so I’ve just stumbled through this and great timing. I’m thinking about overhauling some things, and what better way than to uproot my rules!!

Anyhow, I have a few questions perhaps you can indulge. I’ve only been playing for about an hour but I wanted to ask about some specific cases:

• Is there a way to input lines that would have normally been done with logInfo("rules.automation", "Some message Here"? I use this frequently to have a way to “indicate” in my logs somewhere that something happened due to my rules processing. I sometimes use this later as well for troubleshooting and identifying what actions happened when.
• Is there a way to send 2 commands? Or perhaps more specifically a command an update? I have a common scenario in my switches (I believe we run the same type of rules with Case statements and the 1.2, 1.3, 1.4, etc for multiple taps). Specifically in mine, I always reset to 2.1 to enable “convenience” 1 touch off always in certain scenarios with mixed switches and to ensure that if the last scene run is run again, it will be picked up properly.
• Can I use multiple inputs to a switch? In a scenario where I might have a When statement with a couple devices to monitor (in my scenario, some alarm system modes) where I need to evaluate the rule any time ANY of those items changes?
• Lastly I know someone mentioned the HomeKit NodeRED stuff in here. I installed it on a whim and was tinkering around. But it seems that it is providing a whole lot more information and frankly I’m not a coder so all the JSON and other stuff is making my head spin. Could be the late hour! But I’m wondering what would be the basics to “translate” other Nodes into OH Nodes for proper control. I’m only investigating this further as the regular HomeKit integration currently isn’t working, but this one seems to at least present the items flawlessly for me.

[rgerrans]

@shawnmix

I’m thinking about overhauling some things, and what better way than to uproot my rules!!

Best cure for boredom, blowing up your system and scrambling to get it back up and running before the WAF gets to low

@shawnmix

Is there a way to input lines that would have normally been done with logInfo(“rules.automation”, “Some message Here”?

I could think of a couple different approaches.

• One would be to create a logging rule, and then pass your log message to a variable along the lines of what I did here: Broadcast Notifications
• You can also use the Debug nodes to view it in real time
• Finally, what I did for times that I want to capture events when I’m not running debug is build a node-red log file using the following subflow:
  

Flow:

[{"id":"5642ea48.78faa4","type":"file","z":"1535d28d.7fd35d","name":"Log","filename":"/opt/node-red.log","appendNewline":true,"createDir":true,"overwriteFile":"false","x":699,"y":839,"wires":[]},{"id":"f2a13fb8.41799","type":"function","z":"1535d28d.7fd35d","name":"Timestamp","func":"msg.payload =  new Date().toLocaleTimeString('en-US',{month: '2-digit', day: '2-digit', year: '2-digit'}) + ' - ' + msg.payload;\nreturn msg;","outputs":1,"noerr":0,"x":483.4686508178711,"y":839.6844158172607,"wires":[["5642ea48.78faa4"]]}]

@shawnmix

Is there a way to send 2 commands? Or perhaps more specifically a command an update?

Not 100% sure I know what you are trying to do here but I assume send an initial state to your scene switch and then right after send a second state? If so, then I do that with a Trigger node. I fork the flow off my previous node with one going direct to the OH item and then run a parallel flow with a short delay on the Trigger node and the 2nd message you want sent as the output of the Trigger node. Something like this:

@shawnmix

Can I use multiple inputs to a switch?

Yes, I do this quite a bit. The key I found if you need to do specific action after based on which input caused the event is to track/change the topic of your message to indicate the device (I use mqtt so convert the topic to simple descriptions so I can input it into a notification message). For example here I have my three door alarms going to a single Switch Node:

@shawnmix

all the JSON and other stuff is making my head spin. Could be the late hour! But I’m wondering what would be the basics to “translate” other Nodes into OH Nodes for proper control.

I haven’t written code in decades, so it took me a while to wrap my head around the JSON. Fortunately, Node-RED gives you a JSON node that will convert the JSON string into a message object which breaks apart the different components so you can use the message format msg.payload. to evaluate. For example I use a Nest node as below:

Which gives me a message object that looks like this:

So in my change node I pull the status from the “away” data object like this::

Hopefully all that info helps, let me know any specific clarifications or if I missed what you were asking

[shawnmix]

@rgerrans

I could think of a couple different approaches.

So this one is a bit tricky. I think the actual solution needs to be a new type of node, that is intended specifically for the log function. Or perhaps it needs to be added as another type of payload or msg (forgive my use of the terms). It should be Send Command, Post Update, and Log {type(info, debug, etc)}. Unfortunately using the item and trying to trigger a rule off this to post the logging won’t work. I tried this for something else and you run into issues when stuff fires off too fast. Honestly, it’s not extremely necessary, but I do like to use it as a way to easily identify in my logs when certain rules are running as mentioned for various purposes. Logging files or from the debug inside the NR interface would be of no help as it needs to appear in OpenHAB.log or Events.log like so (output from frontail):

2017-09-26 19:47:46.004 [vent.ItemStateChangedEvent] - master_bathroom_scene changed from 1.0 to 2.0
2017-09-26 19:47:46.030 [ome.event.ItemCommandEvent] - Item 'left_vanity_switch' received command OFF
==> /var/log/openhab2/openhab.log <==
2017-09-26 19:47:46.035 [INFO ] [rthome.model.script.rules.automation] - Master Bathroom - All OFF rule executed (1 Tap DOWN)
==> /var/log/openhab2/events.log <==
2017-09-26 19:47:46.051 [ome.event.ItemCommandEvent] - Item 'right_vanity_switch' received command OFF
2017-09-26 19:47:46.065 [ome.event.ItemCommandEvent] - Item 'master_bathroom_dimmer' received command 0
2017-09-26 19:47:46.082 [vent.ItemStateChangedEvent] - master_bathroom_dimmer changed from 100 to 0
2017-09-26 19:47:46.083 [GroupItemStateChangedEvent] - g_master_bathroom changed from ON to OFF through master_bathroom_dimmer
2017-09-26 19:47:46.088 [vent.ItemStateChangedEvent] - master_bathroom_scene changed from 2.0 to 2.1

@rgerrans

Not 100% sure I know what you are trying to do here but I assume send an initial state to your scene switch and then right after send a second state?

Probably helps to provide a sample rule:

rule "Office Dimmer Scene"
when
  Item office_scene changed
then
{
  switch(office_scene.state){
    case 2.1 : {}
    case 1.3 : {
      sendCommand(office_dimmer, 100)
        logInfo("rules.automation", "Office Dimmer set to 100 - (2 Taps UP)")
      postUpdate(office_scene, 2.1)
    }
    case 2.3 : {
      sendCommand(office_dimmer, 1)
        logInfo("rules.automation", "Office Dimmer set to 1 - (2 Taps DOWN)")
      postUpdate(office_scene, 2.1)
    }
  }
}
end

========> Side note - you can see the use of the logInfo part I spoke about in the first section
So what I’m doing is if I see the 2 taps up, I turn the dimmer to 100%, if I see the 2 taps down, I turn the dimmer to 1%. My understanding of rules processing, first case match goes to end so I put the 2.1 as a catch to “expedite” the rules. Though it may just be completely unnecessary.

My goal here though, is to sendCommand to the dimmer, log an info event, and THEN, send a postUpdate to the scene state to reset to 2.1. I’ll spare you the boring explanation of why 2.1, but the point is if I walk into the office and want to use the Dim scene, then later turn the light to 100% using OH instead without touching the switch, then try to use the switch scene (taps down) again - it won’t work if the scene hasn’t been reset. Funny way our switches work - if you change the actual light item (or channel), it won’t affect the scene channel/item.

Should this just be as I’ve designed now?

@rgerrans

Yes, I do this quite a bit. The key I found if you need to do specific action after based on which input caused the event is to track/change the topic of your message to indicate the device

Ok, I see so if you need to evaluate against WHICH device triggered the rule to fire, then you need to do some JSON manipulation to call it out? Or will this come from the msg.payload? Again, samples probably help but I think you have the right idea here already:

rule "Presence Based Automation"
when
  Item virtual_user1 changed or
  Item virtual_user2 changed or
  Item virtual_user3 changed
then

@rgerrans

Hopefully all that info helps, let me know any specific clarifications or if I missed what you were asking

Ok, I think that does help. I’m still trying to wrap my head around and play around with the translation from one node to the next, and more specifically understanding the outputs. I’m having to put in a bunch of the debug pieces to see the output and start to piece it together. My main reason to want to understand this is that part someone suggested about HomeKit could be a FANTASTIC solution to a few of the items I have that don’t respond as well with the OH items - specifically the Thermostats and some other non-generic items. If you manage to get that working, please do share!! But I don’t have high hopes as I believe I saw you indicate you’re on Android

I imagine it should look something like this:

But I need to figure out the way to translate the msg.payload from OH format to the necessary HomeKit format, and then back again into OH. Once I get that, this will be CLUTCH!

PS - Thanks for all the effort in putting together this little tutorial! This is fantastic!!

[shawnmix]

Oh and one more side question - organization. Is it advised to create a new “Flow” for each rule? Or can/should I put multiple similar rules into one “Flow”? I use the quotes on the term flow as I really see them as just tabs here, but I know it’s likely more than that as I see Flows have an ID, and I saw in some instances you seem to have included the flows you mentioned in a one liner type of format - presumably an export type capability?

[rgerrans]

@shawnmix

I think the actual solution needs to be a new type of node, that is intended specifically for the log function.

That is most likely not possible, unless they exposed the logInfo to the REST api since that is what the OH nodes are tied to.

@shawnmix

Unfortunately using the item and trying to trigger a rule off this to post the logging won’t work. I tried this for something else and you run into issues when stuff fires off too fast.

I’m assuming you mean that it will inject the log message into OH logs (which I just tested and works fine) but that the message gets logged before the device triggers? If so you could always put a slight delay in front of your log message in Node-RED to make sure it comes out later though if you are only trying to figure out if your rule fired or not I’m not sure why the sequence in your log file is of issue. As far as the need to be in a single file, personally I actually break my OH into separate log files already for mqtt and zwave so I can turn on debug w/o over saturating my main log file. I then added the node-red logging and I just look at time stamps across the various files to figure out timing. A little more work on my part but not that big of a deal.

@shawnmix

Should this just be as I’ve designed now?

That looks about right from what I see you trying to do (assuming your items are all defined correctly). I don’t see you changing the msg.payload back to 2.1 before sending to the Office Reset unless you are doing it in the outbound node. I think you are good for not triggering a circular loop when changing the office scene back to 2.1 (which it looks like you trapped in your rule and left off your case switch in Node-RED so should be fine).

To add your logging to this, what I would do is just add another OH node to each change node that sends a message to a virtual logging string and then have a rule that fires off that string to insert the message into your OH log. So it would look something like this (remember I use mqtt not OH nodes now but same idea):

With the appropriate items defined and a rule like this:

rule "Logging"
when
	Item vLogMsg changed
then
    logInfo(vLogTopic.state.toString, vLogMsg.state.toString)
end

You could also attach your “Office Reset Scene” node (with the appropriate payload as per above) after the node that changes the vLogMsg item to get the sequence to follow as well.

@shawnmix

see so if you need to evaluate against WHICH device triggered the rule to fire, then you need to do some JSON manipulation to call it out?

Actually all you have to do using the OH Nodes is to use the msg.item in your switch rules to pull the item name.(not the item label) and then if you want to use a simpler name just change the topic or payload based on mapping the msg.item name (let me know if you want an example).

@shawnmix

If you manage to get that working, please do share!! But I don’t have high hopes as I believe I saw you indicate you’re on Android

Correct, Android guy here so no need for HomeKit. I found for some of the other items I deal with I can find interfaces either direct to mqtt (hence my switch over), direct through Node-RED or now I’ve figured out how to interface directly into node.js so can get to them if someone wrote an interface there but no one wrote a wrapper for Node-RED yet. Sorry I can’t be of help there.

@shawnmix

Is it advised to create a new “Flow” for each rule? Or can/should I put multiple similar rules into one “Flow”?

Personally, I group rules by “Flow” my flows tend to tie to specific physical rooms or groups of flows.

@shawnmix

I saw in some instances you seem to have included the flows you mentioned in a one liner type of format - presumably an export type capability?

Most of my flows (also not sure why the use that word interchangable between a linked group of nodes and the tabs. Here I mean the groups of nodes) are isolated and I have multiple per tab. For this tutorial and most my responses I mock them up for simplicity to screen shot them. Here’s an example of what one of my tabs looks like if that helps to clarify:

[bruxi (Jens Bruckmann)]

The more I work with Node-RED the more I love it. It’s much more intuitive and better to debug than the native rule engine with its very special language.

I often used the functions changedSince oder historicState, like in this example:

                        if (! PlayerPaused.changedSince(now.minusMinutes(30), "influxdb") {
                            sendCommand(HomeEntertainmentMainSwitch,OFF)
                            sendCommand(BeamerSwitch,OFF)
                            sendCommand(LivingroomRenderer,OFF)
                            logInfo("Entertainment","Player on Pause for more than 30 minutes, powering off")
                    }

Is there a way to copy such a functionality in Node-RED?

Another question is, how can I e.g. count members of a group having a specific state like with these statements in xTend:

                if(gIndoorMotion.members.filter(m | m.changedSince(now.minusMinutes(presence_timeout), "influxdb")).size == 0) {
                    motion_occured = false
                    logInfo("PresenceCheck", "No motion occured during the last " + presence_timeout + " minutes")
            }

If those functions could be integrated in Node-RED I would start migrating all my rules in a moment.
If that is not possible I have the idea of using “proxy” items in Node-RED to set to then afterwards trigger a rule in xTend, use the functions in question and then to set another “proxy” item to a specific value to be handled in Node-RED again. But that would not be very comfortable and my intention is to replace xTend totally.