OH3 Lost all Things

@maintainers There are several reports in the Facebool group with the same problem: out of nothing things disappear, nothing in the logs. Any idea how to debug that? Is this a core issue or is this an openhabian issue?

For a start, I think it would be good to provide a dump of the json db to see whether it’s corrupted somehow: /var/lib/openhab/jsondb

There are also backups under /var/lib/openhab/jsondb/backup, restoring them could may be at least a workaround until the bug is found and fixed.

Also, I would give a try stopping openHAB and deleting the cache with openhab-cli clean-cache and starting it again. This helps sometimes when weird things have happened.

At least in Facebook the things database is reported with 0 bytes.

Ok, in this case deleting cache won’t work, then you definitely need to restore the backup while openHAB is turned off.

Regarding debugging: It could be checked whether multiple instances of openHAB are running which are trying to write to the same file (just an idea…). Thread dumps may also help finding the cause (anyhow, those would have to be taken directly after the issue occured, before openHAB has been restarted).

Disclaimer: I’m not a core maintainer, so I’m just thowing out some ideas. Personally, I use file based configuration so I don’t have this issue.

1 Like

Do users who have this issue upgraded their openHAB(bian) or does it also occur with fresh installations?

There are a couple of threads relating to a similar issues, perhaps related to this (using dsl rules in OH3 UI) ?

I suffered from it at the weekend, thankfully I’ve been taking regular backups so was able to restore only losing a few hours work (I’m now taking multiple backups when I’m making changes).

Cheers
James

I myself have been seeing the jsondb file be reset to 0 bytes at times in the past.
I never found out but this is unrelated to openHABian and ZRAM.
If it was ZRAM related it could only show up on power outage and reboots:
if there was a failure in saving files that changed during operations with ZRAM enabled, they would get reset to their last checkpoint save’s version (file version at system/ZRAM startup). Assuming you properly shutdown/reboot your machine once in a while that should not be 0 bytes then but show at least the last checkpointed version (unless that has been 0 bytes ever).

Dirty quick fix idea: Restoring backup automatically when json db is 0 bytes and write a warning/error to the openhab.log when it happens?

Anyhow, it would be really good if someone manages to take the thread/karaf dump when this happens:

  1. Do NOT stop openHAB
  2. Run openhab-cli console on CLI (default password: habopen)
  3. Run dev:dump-create (this could take a while)
  4. Find the dump in /usr/share/openhab2/runtime/ and download it from your openHAB server
  5. Open a GitHub issue in core: https://github.com/openhab/openhab-core/issues/new
  6. Attach the dump to the created GitHub issue
3 Likes

This sounds like it is related to Things-configuration is totally gone in OH3 after a while · Issue #2100 · openhab/openhab-core · GitHub ? If so this has been fixed in the nightlies

It was me providing this bug-report and wondered if I’m the only one facing this problem. But yes, indeed, can confirm that with a recent nightbuild, the problem is gone. In fact previously the CPU-load on one core was very high. Since this issue was fixed, my Raspi4b can relax now :slight_smile:

1 Like

Occasionally my RPi hangs and gets rebooted by watchdog. On one occasion when it restarted I was left with a zero byte hueEmulationUsers.json which I had to manually restore from the backup directory.

Given that more important files such as the things DB seem to get updated without making any changes in the UI, it seems something like a power cut or a crash could leave OH unable to restart.

Perhaps the suggestion by @pfink of automatically restoring zero byte files is actually a good idea since it might make OH more resilient to unexpected reboots.

Not related to the topic, but Pi4 can boot from USB now.

this is not related to the topic :wink:

Yeah, would be interesting which technology openHAB uses for the json db, whether it is some library or an own implementation. Maybe the current solution is just not designed to protect against data loss when something unexpected like power loss or memory shortage happens.

IIRC @chris has built this. Chris do you read?

If I remember correctly it uses gson to convert the data to json. The source is available if someone wants to look a it - it is reasonably simple.

FWIW, just had this on startup. 0 bytes now unfortunately apparently is not considered a ‘corruption’.

2021-01-24 15:15:36.779 [ERROR] [re.storage.json.internal.JsonStorage] - Error reading JsonDB from /var/lib/openhab/jsondb/org.openhab.core.thing.Thing.json. Cause com.google.gson.stream.MalformedJsonException: Expected name at line 73660 column 10 path $.modbus:data:3db5c802.value.channels[10].acceptedItemType.
2021-01-24 15:15:36.782 [INFO ] [re.storage.json.internal.JsonStorage] - Json storage file at '/var/lib/openhab/jsondb/org.openhab.core.thing.Thing.json' seems to be corrupt - checking for a backup.
2021-01-24 15:15:37.905 [INFO ] [re.storage.json.internal.JsonStorage] - Json storage file at '/var/lib/openhab/jsondb/backup/1611493585440--org.openhab.core.thing.Thing.json' is used (backup 1).

I’ve raised an issue for a zero byte file not being considered as corrupted.

I had the ‘All things gone’ issue three Times now. Since update to 3.0.1 havent had agian. Fingers crossed. Im running x86_64 NUC not a Pi. Just fyi

After 12 days again no things anymore. So my changes with ZRAM are not relevant and I will activate it again.