OH3 Lost all Things

Lars_Reidelbach · January 19, 2021, 8:12am

Hi,

I use OpenHAB3 now 3 weeks working fine. Yesterday I see problem that nothing is changed through OpenHAB. When I look at the config there is no things anymore. All items are still there but without the things nothing can be changed.
In the logs I can see the error messages:

Example:

2021-01-19 08:58:34.017 [ERROR] [internal.handler.ScriptActionHandler] - Script execution of rule with UID ‘Rule-System-Start-Komplett’ failed: val telegramAction = getActions(“telegram”,“telegram:telegramBot:5eb3137cf6”) telegramAction. ___ sendTelegram(“OpenHAB neu gestartet!”)
The method sendTelegram(String) is undefined for the type ThingActions; line 1, column 93, length 12

Has someone an idea why this is happend ?

Bildschirmfoto von 2021-01-19 09-08-04

best regards,
Lars

HaKuNa · January 19, 2021, 8:41am

Could you find anything in openhab.log/event.log around the time you guess the configuration disappeared ?
On which Platform are you running OH ?
Where are your configuration files (things/items/…) stored (created via GUI or stored in files locally/NAS/…) ?
A little bit more information about your setup and logfiles are required if you expect help here.

Lars_Reidelbach · January 19, 2021, 10:01am

All Things and Items are created over GUI. Nothing over files.
I see nothing in the logfiles for errors

Release = Raspbian GNU/Linux 10 (buster)
Kernel = Linux 5.4.83-v7+
Platform = Raspberry Pi 3 Model B Plus Rev 1.3
CPU Usage = 0% avg over 4 cpu(s) (4 core(s) x 1 socket(s))
CPU Load = 1m: 0.16, 5m: 0.03, 15m: 0.01
Memory = Free: 0.08GB (9%), Used: 0.86GB (91%), Total: 0.95GB
Swap = Free: 2.39GB (100%), Used: 0.00GB (0%), Total: 2.39GB
Root = Free: 22.54GB (83%), Used: 4.50GB (17%), Total: 28.23GB
Updates = 0 apt updates available.
Sessions = 1 session(s)
Processes = 114 running processes of 32768 maximum processes

Verwendet an Software:
openHAB 3.0.0
openHABian v1.6.2

HaKuNa · January 19, 2021, 10:18am

not only errors are relevant, so please share your logs

Lars_Reidelbach · January 19, 2021, 10:26am

Ok here some files.

openhab.log (2.0 KB) events.log (24.9 KB)
openhab-yesterday.log (48.9 KB) events-yesterday.log (37.2 KB)

HaKuNa · January 19, 2021, 10:48am

hmm, strange.

As an example during startup the items changed to 51, but was not successful

2021-01-19 08:58:34.182 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'LampeFlurUGLinks_Farbtemperatur' changed from NULL to 51
2021-01-19 09:02:25.567 [INFO ] [openhab.event.ItemCommandEvent      ] - Item 'LampeFlurUGLinks_Farbtemperatur' received command 100
2021-01-19 09:02:25.576 [INFO ] [penhab.event.ItemStatePredictedEvent] - Item 'LampeFlurUGLinks_Farbtemperatur' predicted to become 51
2021-01-19 09:02:32.068 [INFO ] [openhab.event.ItemCommandEvent      ] - Item 'LampeFlurUGLinks_Farbtemperatur' received command 0
2021-01-19 09:02:32.089 [INFO ] [penhab.event.ItemStatePredictedEvent] - Item 'LampeFlurUGLinks_Farbtemperatur' predicted to become 51
2021-01-19 09:02:34.725 [INFO ] [openhab.event.ItemCommandEvent      ] - Item 'LampeFlurUGLinks_Farbtemperatur' received command 71
2021-01-19 09:02:34.739 [INFO ] [penhab.event.ItemStatePredictedEvent] - Item 'LampeFlurUGLinks_Farbtemperatur' predicted to become 51

In the “old” log there we can see the expected behaviour (received -> prediced -> changed)

2020-12-29 11:02:56.858 [INFO ] [openhab.event.ItemCommandEvent      ] - Item 'LampeFlurUGLinks_Farbtemperatur' received command 54.0
2020-12-29 11:02:56.900 [INFO ] [penhab.event.ItemStatePredictedEvent] - Item 'LampeFlurUGLinks_Farbtemperatur' predicted to become 54.0
2020-12-29 11:02:56.936 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'LampeFlurUGLinks_Farbtemperatur' changed from NULL to 54.0
2020-12-29 11:02:58.151 [INFO ] [openhab.event.ItemCommandEvent      ] - Item 'LampeFlurUGLinks_Farbtemperatur' received command 10.0
2020-12-29 11:02:58.159 [INFO ] [penhab.event.ItemStatePredictedEvent] - Item 'LampeFlurUGLinks_Farbtemperatur' predicted to become 10.0
2020-12-29 11:02:58.182 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'LampeFlurUGLinks_Farbtemperatur' changed from 54.0 to 10.0
2020-12-29 11:02:59.944 [INFO ] [openhab.event.ItemCommandEvent      ] - Item 'LampeFlurUGLinks_Farbtemperatur' received command 100.0
2020-12-29 11:02:59.953 [INFO ] [penhab.event.ItemStatePredictedEvent] - Item 'LampeFlurUGLinks_Farbtemperatur' predicted to become 100.0
2020-12-29 11:02:59.972 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'LampeFlurUGLinks_Farbtemperatur' changed from 10.0 to 100.0
2020-12-29 11:03:02.518 [INFO ] [openhab.event.ItemCommandEvent      ] - Item 'LampeFlurUGLinks_Farbtemperatur' received command 29.0
2020-12-29 11:03:02.527 [INFO ] [penhab.event.ItemStatePredictedEvent] - Item 'LampeFlurUGLinks_Farbtemperatur' predicted to become 29.0
2020-12-29 11:03:02.545 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'LampeFlurUGLinks_Farbtemperatur' changed from 100.0 to 29.0

And what happened between 2020-12-29 and 2021-01-19.
Was the openhab instance stopped, or the whole PI ?

I think you have enabled ZRAM and the config was never written to disk.

Lars_Reidelbach · January 19, 2021, 11:50am

Yes you are right ZRAM is activ. I will search a topic and look how I can disable it.

I restore my backup and all things now ok. I try to deativate zram and look at the system.

HaKuNa · January 19, 2021, 12:27pm

It was just an assumption, and deactivate ZRAM is just a workaround.
You should check the forum if there are already reported similar issues with the same openhabian build.

The ZRAM and SD-card thing on the PI was a reason for me to use a different (more stable) hardware.
But this is just a personal preference.

Lars_Reidelbach · January 19, 2021, 12:34pm

What you use for a hardware? Could be interesting.

HaKuNa · January 19, 2021, 12:38pm

Minix Neo J50C-4 Mini
Intel Pentium Silver J5005, 4GB, eMMC

Running CentOS8, installed all the required SW “manually”.
I have no need for an all-in-one solution like openhabian, again, just a personal preference.

J-N-K · January 19, 2021, 5:54pm

@maintainers There are several reports in the Facebool group with the same problem: out of nothing things disappear, nothing in the logs. Any idea how to debug that? Is this a core issue or is this an openhabian issue?

pfink · January 19, 2021, 6:47pm

For a start, I think it would be good to provide a dump of the json db to see whether it’s corrupted somehow: /var/lib/openhab/jsondb

There are also backups under /var/lib/openhab/jsondb/backup, restoring them could may be at least a workaround until the bug is found and fixed.

Also, I would give a try stopping openHAB and deleting the cache with openhab-cli clean-cache and starting it again. This helps sometimes when weird things have happened.

J-N-K · January 19, 2021, 6:49pm

At least in Facebook the things database is reported with 0 bytes.

pfink · January 19, 2021, 6:54pm

Ok, in this case deleting cache won’t work, then you definitely need to restore the backup while openHAB is turned off.

Regarding debugging: It could be checked whether multiple instances of openHAB are running which are trying to write to the same file (just an idea…). Thread dumps may also help finding the cause (anyhow, those would have to be taken directly after the issue occured, before openHAB has been restarted).

Disclaimer: I’m not a core maintainer, so I’m just thowing out some ideas. Personally, I use file based configuration so I don’t have this issue.

pfink · January 19, 2021, 6:59pm

Do users who have this issue upgraded their openHAB(bian) or does it also occur with fresh installations?

james_2_roberts · January 19, 2021, 7:09pm

There are a couple of threads relating to a similar issues, perhaps related to this (using dsl rules in OH3 UI) ?

OH3 Thing.json suddenly empty - #3 by DaDen
Dispatching event to subscriber 'org.openhab.core.internal.items.ItemUpdater...takes more than 5000ms. And Thing are deleted

I suffered from it at the weekend, thankfully I’ve been taking regular backups so was able to restore only losing a few hours work (I’m now taking multiple backups when I’m making changes).

Cheers
James

mstormi · January 19, 2021, 7:10pm

I myself have been seeing the jsondb file be reset to 0 bytes at times in the past.
I never found out but this is unrelated to openHABian and ZRAM.
If it was ZRAM related it could only show up on power outage and reboots:
if there was a failure in saving files that changed during operations with ZRAM enabled, they would get reset to their last checkpoint save’s version (file version at system/ZRAM startup). Assuming you properly shutdown/reboot your machine once in a while that should not be 0 bytes then but show at least the last checkpointed version (unless that has been 0 bytes ever).

pfink · January 19, 2021, 7:12pm

Dirty quick fix idea: Restoring backup automatically when json db is 0 bytes and write a warning/error to the openhab.log when it happens?

Anyhow, it would be really good if someone manages to take the thread/karaf dump when this happens:

Do NOT stop openHAB
Run openhab-cli console on CLI (default password: habopen)
Run dev:dump-create (this could take a while)
Find the dump in /usr/share/openhab2/runtime/ and download it from your openHAB server
Open a GitHub issue in core: https://github.com/openhab/openhab-core/issues/new
Attach the dump to the created GitHub issue

digitaldan · January 19, 2021, 9:05pm

This sounds like it is related to Things-configuration is totally gone in OH3 after a while · Issue #2100 · openhab/openhab-core · GitHub ? If so this has been fixed in the nightlies

biologist · January 19, 2021, 9:11pm

It was me providing this bug-report and wondered if I’m the only one facing this problem. But yes, indeed, can confirm that with a recent nightbuild, the problem is gone. In fact previously the CPU-load on one core was very high. Since this issue was fixed, my Raspi4b can relax now