OH3 Lost all Things

Occasionally my RPi hangs and gets rebooted by watchdog. On one occasion when it restarted I was left with a zero byte hueEmulationUsers.json which I had to manually restore from the backup directory.

Given that more important files such as the things DB seem to get updated without making any changes in the UI, it seems something like a power cut or a crash could leave OH unable to restart.

Perhaps the suggestion by @pfink of automatically restoring zero byte files is actually a good idea since it might make OH more resilient to unexpected reboots.

Not related to the topic, but Pi4 can boot from USB now.

this is not related to the topic :wink:

Yeah, would be interesting which technology openHAB uses for the json db, whether it is some library or an own implementation. Maybe the current solution is just not designed to protect against data loss when something unexpected like power loss or memory shortage happens.

IIRC @chris has built this. Chris do you read?

If I remember correctly it uses gson to convert the data to json. The source is available if someone wants to look a it - it is reasonably simple.

FWIW, just had this on startup. 0 bytes now unfortunately apparently is not considered a ā€˜corruptionā€™.

2021-01-24 15:15:36.779 [ERROR] [re.storage.json.internal.JsonStorage] - Error reading JsonDB from /var/lib/openhab/jsondb/org.openhab.core.thing.Thing.json. Cause com.google.gson.stream.MalformedJsonException: Expected name at line 73660 column 10 path $.modbus:data:3db5c802.value.channels[10].acceptedItemType.
2021-01-24 15:15:36.782 [INFO ] [re.storage.json.internal.JsonStorage] - Json storage file at '/var/lib/openhab/jsondb/org.openhab.core.thing.Thing.json' seems to be corrupt - checking for a backup.
2021-01-24 15:15:37.905 [INFO ] [re.storage.json.internal.JsonStorage] - Json storage file at '/var/lib/openhab/jsondb/backup/1611493585440--org.openhab.core.thing.Thing.json' is used (backup 1).

Iā€™ve raised an issue for a zero byte file not being considered as corrupted.

I had the ā€˜All things goneā€™ issue three Times now. Since update to 3.0.1 havent had agian. Fingers crossed. Im running x86_64 NUC not a Pi. Just fyi

After 12 days again no things anymore. So my changes with ZRAM are not relevant and I will activate it again.

ok, I am on 3.1.0 release and lost all things today. restoring the things file from yesterday didnā€™t help. any ideas?

Feedback: issue because of full volume. Unfortunately had to redo most, backups were overwritten

1 Like

Uaaah. after so much pain with OH3 migration i spent nights to get it work finallyā€¦same issue here, after a simple reboot all things ā€œnot in fileā€ are gone. no backup so iā€™m fĀ§$% up.

RPI 4 8GB - OH3.2M1

start to hate OH3 a lot as it seems a more generic problem. Any chance i can have access to old json files whatever to bring it back without the pain to rerun everything?

Thereā€™s backup files of the JSON things files in /var/lib/openhab/jsondb/backup.
If you have made a config backup itā€™s a zip file you can extract.
Might even work to copy over your 2.x JSON file to 3.x as AFAIK it has not changed in format.

Thanks.
the problem is that the auto-stored backup jsonā€™s with timestamp are with zero byte size. so i already started reentering everything. Its little frustrating when you see this issue has been arround. my mistake i trusted too much and wanted to finalize everything before doing a first GOLD backup. in the end - even if a lot do say GUI is greatā€¦i better prefer the old school txt style as much as possible. these are the ones that still are arroundā€¦

Well Iā€™m surprised to hear the issue is still in there, but frankly yes one could have anticipated that and taken a safer approach.
You can increase the number of jsondb backup files, thereā€™s an option for that in UI.
You can manually copy the current JSON files somewhere at any time. You can take config backups at any time. And you should setup automated system backup before you start configuring.

sure, all worked fine and improved day by dayā€¦so still not finished and a backup was not on the radar. something i suffer a lot now, ā€¦never make same mistake twice! Thanks

Recently I began experimenting with different take towards configuration management. I made an extension to export/import things and items as well as links between them via XML. Procedure is that you can click everything via UI, dump it into XML, reset storage and switch to fully managed config within eye blink.

My goal was to be able to ā€œprocessā€ things/items and let them be processed by standard IT tooling. Present OH textual config is out of scope since its parsing complexity (it is a custom domain specific language) and lack of support for writing it. Existing json files stored by OH are its internal format which is bound to the way how OH works hence it is not easiest one to work with.

If youā€™d be up testing this XML import/export Iā€™ll deploy JARs for you so you could play with it. I know format is far from being compact, but it is a compromise between readability and processing abiltiy.

Sample items:

Sample things: connectorio-addons/items.xml at master Ā· ConnectorIO/connectorio-addons Ā· GitHub

1 Like

well you cannot blame the software (vendor) for the poor server management. But yes, I also had sleepless nights getting all back where it was. I was almost done. Only minor tweaks. Lost a bit track and wanted to do later. Then it ran for a few days and out of a sudden nothing worked. And of course the major backup was on my list but never done. Learned from that.
It works now. Have it on production and test system. Happy with it and keep an eye on the disk size (after increasing it to 500GB shouldnā€™t be an issue anymore).

not sure what you mean by that. Iā€™m on Pi4 with 8GB RAM and a 32GB SD cardā€¦that hopefully will never crash, and if sooā€¦i now have daily backups :slight_smile: