Rules run slow when persistence has high latency

  • Platform information:
    • Hardware: RPi3
    • OS: HypriotOS image
    • Docker image: openhab:2.5.1-armhf-debian

I run persistence service (influxDB) on remote machine to reduce the number of writes to flash card of raspberry for prolonging card life. The connection to the persistence can sometimes have high latency and can even disappear for short periods of time (connection provided by LTE router).

I experience that sometimes my rules are executed after a huge delay (e.g. half to several minutes after the trigger is supposed to occur) while in most of the times they run quickly.

Can it be cased by the remote persistence setup?
Do I do something completly wrong?

Can someone who understands how rules engine works under the hood comment whether the rule execution can be postponed by the long running persistence value updates?

I would make a guess this is more to do with events bus than rules.

I don’t think so but you can enable org.openhab.persistence.influxb debugging to find out

Yes: you run Docker on some exotic OS on a box to have 1 GB of RAM only.
If I was to guess your system is busy with paging at times (see output of iostat 1).
Why don’t you move to openHABian?

OpenHAB provides Docker image for armhf architecture, thus I can conclude that running in docker on RPi is intended use case.
HypriotOS is a best choice for it as it’s purpose is out of-the-box docker runtime on RPi.
I don’t think that Docker adds considerable overhead as Docker is very lightweight.

HypriotOS is more convenient to me then OpenHABian, but that is a different topic.

iostat does not show any read/writes to the flash card.
Also I mount userdata/logs to the tmpfs to prolong flash card life. Logs take <40Mb, so it does not consume much of available 1Gb memory.

Thus In my setup I should not see any intensive flash card writes/reads at all.

Docker shows that OpenHAB container uses about 300Mb of memory. That’s OK.
I noticed that openHAB container always has 200 - 250 threads created (although they wait something asynchronously as CPU usage is 1-3%). That looks suspicious as they theoretically can get unblocked and waken up near simultaneously…

I’ll do something to monitor peaks in flash card writes/reads to check you hypothesis about paging.

@rossko57, thank you. Is there any way to assess health of the event bus?
Does it have bandwidth? Can it be cluttered? How can I detect that?

And consider the following.
In the rule code I do if(PeoplePresent.state == ON) {...}.
Am I right that item state access actually does a lookup to the default persistence (thus can take long time if the persistence latency is high)? If it so, the high latency persistence can significantly slow down the rules that extract the item states.

Not that I know of. Traffic in events.log should give a fair indication, but is not complete.

I am not clear about the interaction between persistence and events bus -but from previously people doing things like persisting all Items every second and noticing an impact in the form of delayed actions, there does seem to be something. Might just be contention for i/o resources rather than any direct link.

No. Items are objects in memory, you’re just fetching a property.
You can run opnHAB with no persistence configured at all.

Containers target automated frequent deployments mostly on cloud servers which is quite the opposite use case to an HA server. Mind you there’s Windows for ARM, too. Mind you there’s ARM boxes to have more than 1 GB.
You are of course free to conclude whatever you like but that does not make your conclusion right let alone a good idea.

Which is untypically small, usual deployments see Java use 500-600MB.
openHABian and other optimized deployments use an increased -Xms Java parameter to avoid garbage collection kick your compiled rules off memory which results in heavy delays on some rules to execute. See if that gets you anywhere.