Rule "Item changed" with rapid Item updates

Hey everybody,

I’m a little stuck in one of my rules. I’ve got an Item (Distance from a mmWave Sensor) that updates very quickly (multiple times a second) that I am subscribing to the changed event in a rule. Now I only want to react to “relevant changes” (Distance is less than 3m or above 3m i.e.). Now I’ve written a rule that does that but its reaction is very slow. I’ve got the feeling that the script has to work through a lot of event updates and that is why the reaction is “so poor”. It takes 4-6s to react to my presence in an area of interest (<3m / >3m). I tried using an if statement with a time measurement ( now > lastDistCheck.plusSeconds(1)==false ), I’ve tried a ReentrantLock mechanism, I’ve tried thread.sleep. Still the reaction is rather slow.
Can I somehow stop an Item from sending updates while I process and ask it to resume after I’m done processing somehow?
Thank you all for your input and help.

Kind regards

Are you executing something that takes a long time to complete?
You could use a mutex, or even a simple flag e.g. if (!processing) { processing = true; do_processing(); processing=false }

There is no option to ignore updates to an Item.
Maybe it’s not the best option to use the raw data in openHAB at all.

You could use a profile to round the incoming value, so that the item is not changing as frequently as it’s currently doing.

No actually the “processing” is quite simple, just turn on two lamps and save the old/initial state for lamps. So it’s not the “processing” that’s causing the lag. Thank you for your answer!

I had feared that. I don’t want to start “hardcoding” distances into the esp Firmware/Flash… Thanks

Hmmm that sounds interesting… I’ll have to explore that route…
Right now I’m using a Everything Presence One with BETA Firmware over MQTT.

Profiles only allow for strings as values as far as I can understand :confused:

show us the rule

You can parse the string to a number within a javascript transformation…

The following example should divide by 10, to transform the value from mm to cm.
Same way you can e.g. transform to meters and round the value to get only 1 decimal number, what would actually result in less frequent state changes.

(function(i) {
    var num = Number.parseFloat(i);
    num /= 10;
    return num;
})(input)

Note, I’ve moved this to a more appropriate category. Tutorials and Solutions are for posting tutorials, not asking for one.

All of these are going to only make it slower or do nothing at all.

As mentioned already, if you have a rule that triggers by an Item update or change, that rule is going to run every time the Item updates or charges. But each rule only gets one thread so if the rule is already running when triggered, those triggers get queued up and worked off in serial and in order.

Thus, adding sleeps is only going to make the rule take longer to process all these events. A ReentrantLock will do nothing because the rule is already locked.

That doesn’t really address the problem. The rule is already going to block so it will only be processing one event at a time anyway. Adding a lock like this is essentially a noop.

Ultimately, OH isn’t really going to be able to process events that occur that fast. It’s not designed for that kind of data rate. When you have a data rate like this, it’s always going to be better to push this sort of processing towards the device instead of trying to manage it in OH. But there are some things you can do.

You don’t have to hard code a difference so much as hard code a threshold. Only report the value if the distance is above a certain relatively small threshold. Even if you can thin the updates sent to OH down to one or two per second that would probably be enough.

Not entirely accurate. Indeed the value that comes into the profile is going to be a String and the value returned by the profile will be a String. But that String will get transformed back to a Number once it gets to the Item.

However, all that transformation to and from a String might add additional latency which could cause issues. And since the only way to achieve this filtering of events at the profile level is through a SCRIPT transform you are really just moving the delay from the rule to the transform. But that may be good enough if the range of changes to the Item you care about are far enough apart. The biggest thing you need to watch out for is that you don’t end up with a large backlog of events being processed which will delay the delivery of the value to the Item.

You shouldn’t need a transform to convert from mm to cm. That’s what UoM is for.


OK, now for the ideas.

Ignore small changes in the rule

One idea is to keep everything in the rule and reject any change that isn’t different enough from the last accepted state. In JS Scripting it would looks something like:

var THRESHOLD = 0.01; // choose some appropriate value
var lastValue = cache.private.get("last", () => -1);
var delta = Math.abs(parseFloat(event.itemState.toString()) - lastValue);
if(delta >= THRESHOLD) {
  cache.private.put("last", lastValue);
  // process the event
}

That calculates the difference between the current value and the previous value and ignores it if the difference isn’t large enough. But from the sounds of things I’m not sure this would work because it’s the overhead in triggering the rule that is causing problems.

Don’t Send Small Changes from the Device

Ideally this would be implemented in the device. Any of these other approaches will be brittle and not work all the time. OH simply was not built to handle updates from a single source at that rate. The values should be filtered before they get to OH.

Filter Out Small Changes in a transform Profile

The same code could be used in a profile with minor modifications.

(function(i) {
  var THRESHOLD = 0.01; // choose some appropriate value
  var lastValue = cache.private.get("last", () => -1);
  var delta = Math.abs(parseFloat(event.itemState.toString()) - lastValue);
  if(delta >= THRESHOLD) {
    cache.private.put("last", lastValue);
    return i;
  }  
  else {
    return null;
  }
})(input)

You’ll still pay the same startup and execution penalty and I think the same queueing but because this is closer to the device it might be enough to avoid that queue of rule triggers from building up with less resources and therefore by avoiding that work avoid the delays.

Obviously your programming language of choice can be used. The logic would be the same.

Use a Rule Language with Less of a Initial Run Penalty

If you are using Rules DSL or JS Scripting it might be worth trying with jRuby, Groovy, or Java Rule. Rules DSL and JS Scripting currently can have quite a large time on the first run of a given rule. If it takes 10-20 seconds for the rule to process that first event, and you have five events per second, that means you’ll have a queue of 50-100 rule triggers to work off before you catch up with now. If the events keep coming in at the same rate, you may never catch up. Using a language with less of a start up penalty can keep that queue from building up or building up so large.

Ensure you have lots of free resources

No Swap should be in use. System load should be < 1, possibly even < 0.5. Either of these will cause a delay in rule execution.

Use a timestamp instead of a threshold

This is less ideal because what you want to do is filter out the noise and smooth out the events, not necessarily convert an event driven stream to a polling stream. If a polling stream were desired, that would best be implemented in the device anyway.

But the overall approach is the same. Save the last timestamp in the cache and only if enough time has passed process the next event, ignoring all the rest.

Use a timer

This one is probably overly complicated but you might be able to get some use out of a timer.

First of all, this approach won’t work well with JS Scripting. Multi-threaded access is not allowed in JS Scripting so you’ll never have a case where a Timer is running and the rule is running at the same time. But I understand the other rules languages do not have this limitation so it should work better there.

In Rules DSL

var timer = privateCache.get("timer");
if(timer === null) {
    privateCache.put("timer", createTimer(now, [ |
        // process the event
        privateCache.put("timer", null)
    ])
}

In this case when the rule triggers it spins off a Timer to process the event. If the rule triggers again and the Timer exists that event is ignored until the Timer completes.

I’m actually not convinced that this approach will work that well in any case because the evidence points to the problem being that queue of rule triggers. But in cases where “process the event” does take a long time, this approach could be useful.

One advantage this approach has is that pulling a value from the privateCache should take less to process than calculating the delta or some other more complicated checks.

Whats about a cron based rule to Check every few seconds the value?

That’s another approach. It would be less responsive but it would completely eliminate the queueing up of triggers to the rule.

It will always process the latest state, but it might also miss a ton of stuff that happens between the polling period.

In general you are right, however in this case the purpose was to reduce the frequency of item changes.

E.g. if the value send from the device is 10mm and would change to 11mm, by using UoM to convert mm to cm the item would change and a rule would be triggered.

If you do this within a transformation profile AND round the value as .0f already in the profile, the change from 10mm to 11mm would NOT result in an item change, as both values would be transformed to 1cm. Therefor any further processing / rule would be triggered less often.

Another idea:

In your rule trigger, simply accumulate the data into a buffer.

Then in a separate cron rule (e.g. every second), run and analyse that buffer and process it however you like. In this manner you still get to see all the data even though you’re only checking once every second.

You can use a ConcurrentHashMap and/or locks to avoid concurrency issues.

Wow. Now that’s what I call a comprehensive Answer.
Thank you for that. My feelings and thoughts were going along the same lines but of course your level of understanding regarding the OH ecosystem go far beyond mine. I’ll have to think about the options and I’ll come back with my findings…
Thank you again.

And that is exactly what my thoughts were too. I don’t need the intermittent states, just the first and last of an update as I won’t be beaming between places in my flat (unfortunately) :wink: Thank’s for the hint.

Hmmm also a good approach… I’ll have to let the ideas sit with me for a little. And I’ll get back with my choice and outcome. Thank you all!

Another thing that came to my attention: you wrote at some point that you’re using Mqtt? I would definitely try to reduce the amount of messages sent by the device when you are using mqtt. Otherwise you might also have adverse effects on other devices that are otherwise unrelated (only share the same messaging infrastructure). Also make sure that your mqtt persistence strategy does not write based on the message count but time interval based. Otherwise you would add another load to the whole system for constant file system IO.

As described the current rule is already pretty light weight. I’m thinking the primary issue is the overhead to trigger and start the rule, not run the rule itself. If that’s the case I’m not sure this fixes the problem and the rule triggers will still build up.

Depending on the language in use this may not be necessary. In all cases, the rule is already thread safe, meaning that you can’t have two instances of the same rule running at the same time. In JS Scripting, because of a limitation in GraalVM with the way we implement JS, the Timer is also thread safe so you can never have a case where the rule and the Timer, or two Timers from the same rule are running at the same time.

In languages that do not have this locking/limitation of the timers through, some sort of locking queue structure is required (you’d probably want to use something ordered that can be used like a Queue instead of a Map though).