Support to distinguish between human- and machine "agent" induced events

HI all,
I recently made my blinds-switches “smart” by writing a rule to open all my ground-floor blinds when I switched the first one near the stairs (only doing this in some time range in the morning and vice versa when shutting in the evening)…

Unfortunately, when intending to operate the blinds individually, the “smart” blinds control operates them all.

My use cases where I want to operate them individually are:

  • rules establishing illumination scenes
  • automated presence simulation during absence (on vacations) switching individual lights and operating blinds

So I look for a possibility to get the information whether a human or a machine interaction triggered an event.

Looking for this support I explicitly got aware of the Multi Agent Systems and Ambient assisted living topics and read an interesting paper (these ideas are actually 10 years old…)
In a Multi Agent System with some human agent it would be definitely good to discriminate the humand interaction from the machine action because

  • the human interaction will always be asynchronous (e.g. confirmation of user-requests)
  • the human user must be in the center of the interaction concept to feel comfortable with the system (i.e. he must feel to be in control of the system and not being controlled by the system) as he/she needs to buy the devices

From what I read about OpenHAB development, rules are centered on actions taken due to some input.
So concluding: ESH or OpenHAB should have a central concept to distinguish between human user agent intended events and machine agent triggered ones. Because the first ones might be subject to learning natural behavior and could be “augmented” with additional smart behavior.

Pushing the blinds switch in the morning, I’d like to be asked e.g. through my smart phone whether I want to open all blinds at once because I did so all the time in the past weeks.

I’m aware of this related discussion which yields some (rather complicated and error prone) workarounds. I actually hope to start a more general discussion on advanced user interaction concepts with ESH/OpenHAB.

Any suggestion to the concrete topic or the more conceptual/visionary ones are highly welcome!

Best regards
Henning

Warning, below is a stream of consciousness dump of what I think it would take to approach this problem given the current state of automation technology.

OH and ESH are strictly event based and there is no technical way to distinguish between a human generated event verses an automatically generated event for the vast majority of technologies.

Without those “rather complicated and error prone” workarounds there simply is no way for these devices to report any distinction between a user initiated action and a system initiated action.

As such the best one can do is infer the source of the event but that isn’t always straight forward either. For example, a number of devices are semi-autonomous (e.g. thermostats). These semi-autonomous devices can make changes based on their own logic or programming. If you are lucky they will report to OH when they make an action but I know of no device that will tell OH the difference between an action it decided to take verses an action that it was directed to take by a human. These sorts of distinctions are simply not built in to the APIs and protocols that these devices use to communicate.

So we are left with:

  • for dumb devices like switches, if they generate an event and OH didn’t originate that event, assume it was a human
  • for dumb devices like switches that don’t generate events when manually triggered, poll the device for changes or set up some other sensor to detect they were triggered (e.g. power consumption module, light sensor, etc) and use that to determine when the device was triggered and then if the event to trigger the device didn’t originate from OH assume it was was a human
  • for smart devices that can take actions on their own, either treat all actions they take autonomously as if they were human generated, or treat none of them as if they were generated by human interaction.

That is about as far as you can get with the state of technology today. If you go 100% DIY then you can add in the ability to report a distinction between human generated and not.

Given that the automation technologies are not up to the task, what can OH do to support this? The vast array of technologies and the varied behaviors (e.g. some devices ack commands, others do not, some report state changes, others don’t, etc) a single approach to detecting and tracking human verses automated events is probably impossible. But if we narrow the problem space to:

  • only allow devices that report state changes that occurred when manually changed
  • only allow “smart” devices if their smarts are disabled and all decisions and actions are initiated from OH

then it should be possible, with some modifications to the core, add a parameter to an Item indicating what the source of the last command and update was (i.e. sitemap, rule, or binding). If the source is from a binding then we are halfway to knowing whether the event was human generated or not. Events from the sitemap are by definition human initiated. Events from a rule are by definition computer initiated. Events from the binding…?

In addition to knowing the source of the event you also need to know whether or not a rule or the sitemap was the cause of the event.

Think of each action as a transaction. There is “something” that initiates an event. This can kick off any number of additional events (e.g. flicking one switch which causes events to turn on a bunch of switches). ALL of the events that resulted from that initial event should be considered the same, i.e. human initiated or computer initiated, even if the bulk of them are caused by a rule that is running in response to that initial event from the sitemap.

The more I think about it the less feasible it seems given the current state of the technology and the OH architecture.

So what if we turn the problem on its side. Rather than attempting to distinguish between human and computer generated events one implemented some learning based on the HA’s state. For example, periodically (every minute?) just save all the states then use trending, markov chaining, bayesian nets, neural networks, etc to “learn” how the house is configured at any given moment and to anticipate future desired states based on past states. It could even “ask” before making the changes.

It doesn’t even necessarily have to keep track of the states periodically and instead just store the events as a time series stream of events.

This is something I could see fitting in with the current state of technology and OH architecture. It could be implemented as a binding or external to OH and externally stimulate OH through the REST API or MQTT messaging.

With this approach it doesn’t have to distinguish between human and rule initiated because the distinction doesn’t matter as much in this approach.

I don’t really see the problem here. In industrial automaton controllers you have clear scope of variables: inputs, outputs, internal variables. Controller never writes to inputs, so, when they change it is quite clear who was initiator. In event-based automation events are variables. There can be input events - generated outside of the system( by user for example), internal events - generated by rules engine, output events - used to control system outputs. What we have in described situation in OH is total brake of the don’t-write-to-inputs rule. E.g controller(OH rules engine) writes(generates) to user input(user event). This should never happen. User events should be generated only externally and not by any rule. And if you want to control blinds by user input and by rule engine, then blinds control should not anymore be exposed to user, but be hidden by rule(aka become internal variable) and user should interact with them only via rule, not directly.
I know that this creates more programming work, but so it is solved in automation.
Another point about intelligent I/O devices - this topic is one of the reasons why I’m strong supporter of centralised control and as much as possible dummy I/O devices. In this case you can be 100% sure if you get input change - it is initiated by human or external conditions change, and not by internal logic of some “intelligent” thermostat

If you have a solution I would love to see it written up as a Design Pattern. I’m certain the community would greatly benefit from seeing how to code a “don’t-write-to-inputs” using OH.

If you provide me example of a do-not-write-to-inputs problem to be solved by such pattern I can try. I guess topicstarter problem is a bit too complex for this purpose.

Well, you said you “don’t really see a problem here” and that “if you want to control blinds by user input and by rule engine, then blinds control should not anymore be exposed to user, but be hidden by rule” that you either already have written something like this in OH or have an approach for a way to do it.

If not, then I would contend my original point stands, given the state of HA technology and the OH architecture, distinguishing between human and computer initiated events is not trivial and not always possible.

Yes, I use such approach in my system. You just wanted examples, right? It’s not about topicstarter problem, but about blinds, lights and conflicts.
1st one:
I have non-intelligent blinds controller which does not allow positioning and accepts simple commands like up, down and stop. It is a rollershutter item in OH and visible on Sitemaps. Now I created a rule, which based on events from motor commands calculates working time in either direction and thus position of the blinds. And another rule, which allows to position the blinds to desired position. What I do with items? The original rollershutter item(the one which binded to motor) dissapears from sitemaps, as now it is controlled solely by the rules. Instead I create new virtual rollershutter item which only binded to these rules, but now it accepts up, down,stop and position commands from user. It appears in sitemaps. Also it’s current position state updated by the rule. So user now has no ability to control blinds directly and thus no conflict "who pushed that motor?"
2nd one:
Control of the hall light by motion sensor and override with switch. Trivial task, but if you leave switch control as it is - connected to the light directly, you might probably come in situation, that rule overrides switch input and light goes off when you don’t want it to. So in this case you must know who triggered the light - user or rule.
To overcome this again - direct light control switch item dissapears from sitemaps. Saying that I also mean, that physical switch, which normally assotiated with this light, for example input on module, must also be disassociated from the light output. Probably this might become a problem if your I/O module input controls it’s output directly. Then you create a rule which in addition to motion sensor has input for manual light switch item. The logic in rule ensures that when this switch is in OFF or ON position, motion sensor is overrided. Only in position AUTO rule works on motion sensor. This item goes to sitemap and binds to your physical switch. And rule output controls the light item

Two remarks from my side:

  1. openHAB is not input/output based, but rather uses a datapoint concept (that was inspired by KNX), so there is a mismatch here. But yet, “virtual” items can be used to split a datapoint into “manually controlled” and “rule controlled”.

  2. Under the assumption that we do not have any external automation logic, i.e. everything that comes from a binding can be regarded as a manual operation, a flag on the events should suffice for most use cases that @Artyom_Syomushkin describes. I see the value in this as this can be a very valuable information for machine learning algorithms (or simpler cases like a replay-based presence simulation). The good news is that events in ESH already have a suitable property called “source”. This is currently set to the channelUID, if the event has been received from a binding and it is set to “org.eclipse.smarthome.core.autoupdate” if a state update occurred through the auto-update after receiving a command. For the rules, I could imagine that it should be set to the rule uid that issues the event (this is not yet the case). What we might need as well is a prefix to more clearly define which source string is a channeUID and which is a rule UID. Is this something along the lines of your thoughts @Artyom_Syomushkin?

Just a bit of 2 cents from my side as well. Home Automation is somehow distinct from industrial automation in my opinion. For one in the investment side as from failsafe side. Most of the users of openHAB don’t have it running in HA-clusters or other failsafe environments like the industry needs. I for myself run it on a Raspberry Pi - on a SC Card; not that failsafe at all. On the other side I do think that automation is in a way a luxury, that I use to make life a bit easier at home. At the same time I do expect my diverse technology, gadgets and all the other stuff to function decentralised without the help of a central server logic.
That being said, I don’t think it would be a good idea to place openHAB in the middle as a single instance or interpreter of all interactions (be them human or machine induced). In that mindset, also my KNX installation for example is also strictly non-centralised and I can use all actuators manually and not solely via logics (except heating actuators, which react of course on temperature).

What I think would suit openHAB (and KNX …), is of course a way to distinguish, whether actions are indeed rule-based or human based. But where to end and where to start. is my heating example rule-based? or rather a third kind “machine based”? and are there more categories like “temperature based”, “sun based”, “time based”, … if you start in this direction, it surely gets like “Enterprise”-style complex and everybody has that special case, like he needs to distinguish between “sun based” and “temperature based” movement of his blinds! :wink:

Long story short: from my perspective, it would be nice to have some context for actions, but in the end, my whole home should work without central logics also.

Yes. What I really would love is a writeup example along the lines of this. Explain the problem, explain the solution, provide a code example. Something with enough information where they could apply the technique to their own problems.

However, based on your description what you are really doing is applying the Proxy Item Design Pattern with a careful selection of devices that allow you to disable or separate the physical switch from the actuator.

There are not many consumer grade devices that will allow that which is what leads me to the conclusion that this is not quite so easy to accomplish in practice for most openHAB users.

I agree. We can do a lot with such a flag. My whole Dead Man’s Switch Design Pattern, the most complex and least reliable of all the Design Patterns I’ve come up with thus far and the source of the OP’s original post, no longer becomes necessary.

I don’t think anyone is proposing any change to OH that would prevent this. However, in some situations having the ability to implement some or all of one’s automation in the way described would be very useful.

I could provide my solution, designed in Node-Red, but I believe you don’t like external to OpenHab rules engine. But anyway…

So great, no need for extra tutorial. You had to propose this already in your first post.

Well, I’m in impression of opposite - there are not so many consumer-grade devices which can control output autonomously based on own input. These kind of “intelligent” devices should be more expensive than simpler ones. Perhaps you talk about Thermostats, as example, right? Anyway I’m quite sure one can disable such behaiviour - at least I was told that flush-type fibaro devices can do that.

In terms of this topic decentralized system makes solving these problems very, very complicated. On the other hand making central controller redundant is not as complex as before, thanks to latest clusterization technologies. I think we are just few months away from such implementation.

I have no problem with Node Red and have encouraged people to post tutorials on using it with OH in the past.

Actually what I’m referring to are devices that let you separate the physical control from that actuator. For example, a zwave switch that will report to OH that a user has toggled the switch but not turn on/off the power. These are rare. In fact, in my experience, finding a switch that reports to OH that it has been manually toggled at all can be a challenge. Without that this solution is challenging if not impossible.

So as @rlkoshak wrote above, your first assumption might not work with merged I/O units. So this means this approach either.
But basically I’m not sure if we need to complicate the HA system more, rather than teaching people to correct programming or making some limitations. OpenHab is quite mature application, providing many wonderful solutions, but at the same time with adding all these ChannelUIDs it becoming not very user friendly. And this takes many potential users going away, thus I believe it’s time to return to roots and start thinking how we simplify things. As I and @rlkoshak in his patterns wrote, usage of intermediate items(or events) should solve this problem, so I don’t see the reasons of providing additional property to event itself.
Also think about definition of “Event” - is it by the way specified anywhere in OH documentation? I know couple of other softwares using Event type of messaging and noone of them provides you the source of Event. E.g. it’s a Feature of event - it has no source. And I think it’s more or less everywhere so. Don’t mix events and messages - message will have source and destination. Event has no source, destination can be provided, but not as must.

As far as I know, but not tested - all Fibaro flush switches should allow doing that. Fibaro RGBW controller can do it 100% and is suitable for many things, including switching the power.

OK, that is all well and good. But there are hundreds and thousands of consumer grade home automation devices that OH supports. Not all, and perhaps not even most will do that. I know from direct experience, of my eight zwave actuators from various vendors (GE, Linear, Aeotec, etc.) exactly 0 will allow one to separate the “button” from the actuator. Supporting that feature may be the exception rather than the rule.

But @kai’s suggestion could make identifying whether the source of the event was from a rule or a person easier to achieve in some (not all) circumstances.