HABApp: Handling Long Startups?

So, I’m not 100% sure I can describe this right, but I’ll try.

Reboots of my RPi4 with ZRam and a SSD drive take about 10 minutes before everything is loaded up and ready to go. I think it mostly has to do with syncing disks as it is several minutes before I can ping the host, and then a while after that (>200 things and ~1700 items) for MainUI to be ready.

I just noticed, after a reboot, that a HABApp rule was not firing. I thought it had to do with the group value issue I reported as it was returning a state of NULL when OH was reporting it off.

After typing a question for the forums, I realized that I probably should restart the habapp.service first to make sure. The rule immediately ran correctly, reporting the group state as ON.

Thus it appears habapp.service is starting up before persistence restores all the values, I think.

What would a good/best practice be to sequence these two events? Nobody here likes rules not running…

@Spaceman_Spiff

Hmm, I haven’t had any startup issues yet so we’ll have to investigate

Can you restart and see if the HABApp service is correctly running.
If so, can you take a look at the logs?

enabled DEBUG level logging. Here is the log coming up after a fresh reboot, with errors.

2021-05-13 10:32:59.320 [INFO ] [HABApp                              ] - HABApp Version 0.30.0
2021-05-13 10:32:59.320 [DEBUG] [HABApp.files.watcher                ] - Adding watcher for /etc/openhab/habapp with <FileEndingFilter ending: .yml>
2021-05-13 10:32:59.322 [DEBUG] [HABApp                              ] - Added event listener for "HABApp.Files" (type RequestFileUnloadEvent)
2021-05-13 10:32:59.323 [DEBUG] [HABApp                              ] - Added event listener for "HABApp.Files" (type RequestFileLoadEvent)
2021-05-13 10:32:59.323 [INFO ] [HABApp.mqtt.connection              ] - MQTT disabled
2021-05-13 10:32:59.323 [DEBUG] [HABApp.openhab.plugin               ] - Starting setup
2021-05-13 10:32:59.324 [DEBUG] [HABApp.openhab.plugin               ] - Setup LoadAllOpenhabItems complete
2021-05-13 10:32:59.324 [DEBUG] [HABApp.openhab.ping                 ] - Ping stopped
2021-05-13 10:32:59.324 [DEBUG] [HABApp                              ] - Added event listener for "V_HABApp_Ping" (type ItemStateEvent)
2021-05-13 10:32:59.324 [DEBUG] [HABApp.openhab.plugin               ] - Setup PingOpenhab complete
2021-05-13 10:32:59.325 [DEBUG] [HABApp.openhab.plugin               ] - Setup ThingOverview complete
2021-05-13 10:32:59.325 [INFO ] [HABApp.openhab.thing                ] - Config folder does not exist - textual thing config disabled!
2021-05-13 10:32:59.325 [DEBUG] [HABApp.openhab.plugin               ] - Setup ManualThingConfig complete
2021-05-13 10:32:59.325 [INFO ] [HABApp.RuleParameters               ] - Parameter files disabled: Folder /etc/openhab/habapp/params does not exist!
2021-05-13 10:32:59.326 [DEBUG] [HABApp.files.watcher                ] - Adding recursive watcher for /etc/openhab/habapp/rules with <FileEndingFilter ending: .py>
2021-05-13 10:32:59.367 [DEBUG] [HABApp.openhab.connection           ] - Trying to connect to OpenHAB ...
2021-05-13 10:32:59.436 [WARN ] [HABApp.openhab.connection           ] - "Cannot connect to host localhost:8080 ssl:default [Connect call failed ('127.0.0.1', 8080)]" (<class 'aiohttp.client_exceptions.ClientConnectorError'>)
2021-05-13 10:32:59.436 [INFO ] [HABApp.openhab.connection           ] - ... offline!
2021-05-13 10:33:00.440 [DEBUG] [HABApp.openhab.connection           ] - Trying to connect to OpenHAB ...
2021-05-13 10:33:00.448 [WARN ] [HABApp.openhab.connection           ] - "Cannot connect to host localhost:8080 ssl:default [Connect call failed ('::1', 8080, 0, 0)]" (<class 'aiohttp.client_exceptions.ClientConnectorError'>)
2021-05-13 10:33:00.448 [INFO ] [HABApp.openhab.connection           ] - ... offline!
2021-05-13 10:33:02.451 [DEBUG] [HABApp.openhab.connection           ] - Trying to connect to OpenHAB ...
2021-05-13 10:33:02.464 [WARN ] [HABApp.openhab.connection           ] - "Cannot connect to host localhost:8080 ssl:default [Connect call failed ('127.0.0.1', 8080)]" (<class 'aiohttp.client_exceptions.ClientConnectorError'>)
2021-05-13 10:33:02.465 [INFO ] [HABApp.openhab.connection           ] - ... offline!
2021-05-13 10:33:06.469 [DEBUG] [HABApp.openhab.connection           ] - Trying to connect to OpenHAB ...
2021-05-13 10:33:06.479 [WARN ] [HABApp.openhab.connection           ] - "Cannot connect to host localhost:8080 ssl:default [Connect call failed ('::1', 8080, 0, 0)]" (<class 'aiohttp.client_exceptions.ClientConnectorError'>)
2021-05-13 10:33:06.479 [INFO ] [HABApp.openhab.connection           ] - ... offline!
2021-05-13 10:33:14.482 [DEBUG] [HABApp.openhab.connection           ] - Trying to connect to OpenHAB ...
2021-05-13 10:33:14.496 [WARN ] [HABApp.openhab.connection           ] - "Cannot connect to host localhost:8080 ssl:default [Connect call failed ('127.0.0.1', 8080)]" (<class 'aiohttp.client_exceptions.ClientConnectorError'>)
2021-05-13 10:33:14.496 [INFO ] [HABApp.openhab.connection           ] - ... offline!
2021-05-13 10:33:30.499 [DEBUG] [HABApp.openhab.connection           ] - Trying to connect to OpenHAB ...
2021-05-13 10:33:32.746 [INFO ] [HABApp.openhab.connection           ] - Connected to OpenHAB instance 3485852a-7831-427e-bd09-a0bce7c0e619
2021-05-13 10:33:32.746 [INFO ] [HABApp.openhab.connection           ] - OpenHAB version 3.1.0.M4 (Milestone Build)
2021-05-13 10:33:33.993 [INFO ] [HABApp.openhab.items                ] - Updated 1662 Items
2021-05-13 10:33:34.027 [WARN ] [HABApp.openhab.connection           ] - Status 404 for GET http://localhost:8080/rest/things/
2021-05-13 10:33:34.027 [DEBUG] [HABApp.openhab.connection           ] - <ClientResponse(http://localhost:8080/rest/things/) [404 Not Found]>
2021-05-13 10:33:34.027 [DEBUG] [HABApp.openhab.connection           ] - <CIMultiDictProxy('Allow': 'DELETE,POST,GET,PUT,OPTIONS,HEAD', 'Date': 'Thu, 13 May 2021 14:33:34 GMT', 'Content-Length': '0', 'Server': 'Jetty(9.4.38.v20210224)')>
2021-05-13 10:33:34.028 [ERROR] [HABApp.openhab.connection           ] - Traceback (most recent call last):
2021-05-13 10:33:34.028 [ERROR] [HABApp.openhab.connection           ] -   File "/opt/habapp/lib/python3.7/site-packages/HABApp/openhab/connection_handler/func_async.py", line 90, in async_get_things
2021-05-13 10:33:34.029 [ERROR] [HABApp.openhab.connection           ] -     return load_json(await resp.text(encoding='utf-8'))
2021-05-13 10:33:34.029 [ERROR] [HABApp.openhab.connection           ] - ValueError: Expected object or value
2021-05-13 10:33:35.766 [DEBUG] [HABApp.openhab.ping                 ] - Ping started

After things settle, if I restart, these errors are not there.

Quite possible, startup is something of a chaos of parallel activity.
In particular, if you default restore-on-startup for all 1700 Items, that will take a while.

1 Like

yeah, I suppose I should try to clean that up… so many things don’t work properly without an initial value, however. Yes, I know I should handle this better in the rule code, but that is one reason I’m trying to move to HABApp…

HABApp should wait until things and items have been restored.
The messages while it is trying to connect can be ignored (hence warning).

The only issue I see in the log is that it doesn’t seem to load the things completely.
I guess I’ll take a deeper look there and have to improve the logic.

However this shouldn’t prevent your rules from runing!
Do you get follow up errors when habapp is actually loading the rules?
That’s the interesting part and unfortunately it’s missing in the log.

2021-05-13 10:32:59.325 [INFO ] [HABApp.RuleParameters               ] - Parameter files disabled: Folder /etc/openhab/habapp/params does not exist!
2021-05-13 10:32:59.325 [INFO ] [HABApp.openhab.thing                ] - Config folder does not exist - textual thing config disabled!

Do you really don’t have a folder for parameter and textual thing config created?

Wouldn’t it be nice to have some kind of startup parameter that sets the delay for the first openHAB access. So it is possible to set it depensing on systemrequirements.

I don’t get errors, but HABApp might think the state of an item is NULL when it will restore to ON later, so the rule won’t run correctly.

Do I need them? Not sure what they do… params says optional in the config file…

There is the -s command line argument but it also gets applied when restarting HABApp manually.

You should see in the HABApp_events.log that the item then changes from None to ON and after that the rule should run correctly.

Or do you you want to do something based on the state when you load the rule but the value is still none?
There is always the option to use get_value which allows you to specify a default in case the value is None:

if MyItem.get_value(default_value=5) == 5:
    do_something()

instead of

if MyItem == 5:
    do_something()

If you want to use them you need them. :wink:
I was thinking maybe it’s a file issue and the rule files are not there, that’s why I was asking.

I think it’s more a case of not doing something when Item state is still NULL (because persistence has not yet got round to restore)

1 Like

This is not my experience. The value never changes over 40 minutes of running the rule, until I restart HABApp.

[2021-05-14 08:44:59,688] [                   MyRule]     WARN | None
[2021-05-14 08:44:59,688] [                   MyRule]     WARN | False
[2021-05-14 08:44:59,688] [                   MyRule]     WARN | False

[2021-05-14 08:57:14,542] [                   MyRule]     WARN | None
[2021-05-14 08:57:14,542] [                   MyRule]     WARN | False
[2021-05-14 08:57:14,542] [                   MyRule]     WARN | False

[2021-05-14 09:24:43,243] [                   MyRule]     WARN | None
[2021-05-14 09:24:43,243] [                   MyRule]     WARN | False
[2021-05-14 09:24:43,244] [                   MyRule]     WARN | False

sudo systemctl restart habapp.service

[2021-05-14 09:25:43,288] [                   MyRule]     WARN | OFF
[2021-05-14 09:25:43,289] [                   MyRule]     WARN | False
[2021-05-14 09:25:43,289] [                   MyRule]     WARN | True

Lots of good tips in this thread… my problem is my free time comes in bursts, and I never seem to have enough time to do everything the way I want… and once I’m close to what I think is a solution I learn a better way to have done something in the first place. LOL.

I think Parameters might be one of those things. Will look at that more closely this weekend.

Everything about HABApp seems extremely well thought out and implemented, which is why I intend to migrate to it. I just want to make sure there is not a big ‘gotcha’ before I start moving everything. Please do not think I am being critical!

Also, changed the title from ‘Handling Log Startups’ to ‘Handling Long Startups’ as that was clearly a misleading typo that I missed!

1 Like

This is interesting. I’ve never been quite convinced whether restore-on-startup generates an Item Changed event, or not, or even only sometimes. Does HABapp rely on that to get a new Item state?

Now that I think of it, the item in this example is a group item…which probably makes a difference.

It certainly does.
Note, I’m pretty sure that restore-on-startup never restores a Group Item state (by design).
Group state - if any - is calculated from aggregation of member states.
I’m not very convinced restoring members will trigger an aggregation.
If not, it’ll just sit there NULL until one of the members gets a regular update.

Yes - it does. I have mapdb running and at least there it’s working and generating the events correctly.
On the other hand my openhab is currently running since 86 days so maybe I’m wrong :smiley:

That’s my understanding, too.

Sincerest apologies for the fire drill, then!

I have been trying to not send a command to a group repetitively so I don’t clog up the logs, and thus check its state before doing so, but clearly checking for NULL is the answer.

Will incorporate this knowledge into my rule rewriting, and hopefully everything will be much more reliable!

1 Like

No problem at all :slight_smile:
Another option is to run a startup function which will send the command once when the rule is loaded.
That way you always have a defined state.

I think it’s something like this (If you only have one item, otherwise it makes sense to do it in a function):

self.run.soon(MyItem.oh_send_command, 'ON')

That’s what I prefer because then I always have a defined state and don’t have to do endless None checks. :smiley:
Also reloading the rule while openHAB is still running will always result in the same behavior.

Thanks again Spaceman…

I was just thinking about this again… and it probably doesn’t really matter because all of the better practices above make sense, but…

I realized that it was unlikely that the group item who’s state I was querying changed between the reboot and when I restarted the habapp service, and that turns out to be the case.

Running the script with code to get the last_change value for the group item returns:

[2021-05-14 14:25:00,369] [                   MyRule]     WARN | 05/14/2021, 09:25:39
[2021-05-14 14:25:00,369] [                   MyRule]     WARN | OFF
[2021-05-14 14:25:00,370] [                   MyRule]     WARN | False
[2021-05-14 14:25:00,370] [                   MyRule]     WARN | True

there are no log entries for the group item between 9:24 and 9:26

So I’m not really sure what is happening. Just thought I’d report.

I don’t really want to derail the thread from the post above, but while at it, I still can’t reliably get a group state.

This code below never logs the text based on either of the == ‘ON’ or == ‘OFF’ conditions.

Using 0.30.0

from HABApp.openhab.items import GroupItem, SwitchItem
import logging
import time
import datetime
time.sleep(1)
log = logging.getLogger('MyRule')
group_item = GroupItem.get_item("gTestGroup")
switch_member_item = SwitchItem.get_item("T_TestSwitchY")
switch_item = SwitchItem.get_item("T_TestSwitchY")
log.warning("Getting Group State")
log.warning(group_item.get_value())
log.warning("And Member Item")
log.warning(switch_member_item.get_value())
if group_item.get_value() == 'ON':
    log.warning("Hooray! Was ON!")
if group_item.get_value() == 'OFF':
    log.warning("Boo! Was OFF!")
log.warning("Setting Group State To On")
group_item.oh_send_command("ON")
time.sleep(.5)
log.warning("Getting Group State")
log.warning(group_item.get_value())
log.warning("And Member Item")
log.warning(switch_member_item.get_value())
if group_item.get_value() == 'ON':
    log.warning("Hooray! Was ON!")
if group_item.get_value() == 'OFF':
    log.warning("Boo! Was OFF!")
switch_member_item.oh_send_command("ON")
time.sleep(.5)
log.warning("Getting Group State")
log.warning(group_item.get_value())
log.warning("And Member Item")
log.warning(switch_member_item.get_value())
if group_item.get_value() == 'ON':
    log.warning("Hooray! Was ON!")
if group_item.get_value() == 'OFF':
    log.warning("Boo! Was OFF!")
[2021-05-14 18:10:14,084] [                   MyRule]     WARN | Getting Group State
[2021-05-14 18:10:14,084] [                   MyRule]     WARN | OFF
[2021-05-14 18:10:14,085] [                   MyRule]     WARN | And Member Item
[2021-05-14 18:10:14,085] [                   MyRule]     WARN | ON
[2021-05-14 18:10:14,085] [                   MyRule]     WARN | Setting Group State To On
[2021-05-14 18:10:14,587] [                   MyRule]     WARN | Getting Group State
[2021-05-14 18:10:14,587] [                   MyRule]     WARN | ON
[2021-05-14 18:10:14,587] [                   MyRule]     WARN | And Member Item
[2021-05-14 18:10:14,587] [                   MyRule]     WARN | ON
[2021-05-14 18:10:15,089] [                   MyRule]     WARN | Getting Group State
[2021-05-14 18:10:15,089] [                   MyRule]     WARN | ON
[2021-05-14 18:10:15,090] [                   MyRule]     WARN | And Member Item
[2021-05-14 18:10:15,090] [                   MyRule]     WARN | ON
[2021-05-14 18:10:52,287] [                   MyRule]     WARN | Getting Group State
[2021-05-14 18:10:52,287] [                   MyRule]     WARN | OFF
[2021-05-14 18:10:52,287] [                   MyRule]     WARN | And Member Item
[2021-05-14 18:10:52,288] [                   MyRule]     WARN | ON
[2021-05-14 18:10:52,288] [                   MyRule]     WARN | Setting Group State To On
[2021-05-14 18:10:52,789] [                   MyRule]     WARN | Getting Group State
[2021-05-14 18:10:52,790] [                   MyRule]     WARN | ON
[2021-05-14 18:10:52,790] [                   MyRule]     WARN | And Member Item
[2021-05-14 18:10:52,791] [                   MyRule]     WARN | ON
[2021-05-14 18:10:53,292] [                   MyRule]     WARN | Getting Group State
[2021-05-14 18:10:53,293] [                   MyRule]     WARN | ON
[2021-05-14 18:10:53,293] [                   MyRule]     WARN | And Member Item
[2021-05-14 18:10:53,293] [                   MyRule]     WARN | ON

What am I missing?

What’s the Group aggregation function? Are there other members, what state are they in?