Persistence stores items despite definitions in config file (OH3, later switch to OH4)

ccbeorn · August 29, 2023, 7:33am

Hi there,

apologies for a lengthy introduction…
My first post on this platform, even though I am an enthusiastic OH user since roughly 8y. I use it mainly for a customized alarm system, heating control and some peripherals (wallbox, lights, switches). So far, I could solve my issues and challenges through finding similar / matching topics in this community.
I am greatly thankful for all the contributions of the various experts here!

Now, after using a fairly wobbly OH2 system for many years, I finally started migrating the system to OH3 a couple of months ago (starting from a clean slate).
Even though OH4 was just released, I will stick to OH3 for now.

Now, as for the topic:
Whilst using mainly text files in OH2, I used the GUI in OH3 extensively to install all the various things and items and (for the first time) structured the whole setup using the Model functionality.

Yesterday, I configured persistence, using an SQL database. The weird outcome: once saving the connection information, the system immediately started storing items and their values with everyChange, although no .persist file was existing yet.
I thought I understood the documentation in a way that persistence in OH3 can only be configured using text files (.persist). And searching the GUI for almost an hour, I couldn’t find any context menu or config UI for things or items where a persistence definition could already be set up (despite the main persist config where the database connect information is placed).

And what is even weirder: the behavior (of storing values on everyChange) didn’t change when I actually placed a .persist file in the config directory, using an every5min configuration for the items in question. The persist file was recognized by the system as the openhab.log clearly confirmed. Yet, it didn’t make a difference, the system kept storing on everyChange, which led me to disable one thing (Shelly 3em) because it was flooding the database with values every second for several items.

Content of the .persist file:

Strategies {
    everyMinute :  "0 * * * * ?"
    every5Minutes: "0 0/5 * * * ?"
    everyHour    : "0 0 * * * ?"
    everyDay     : "0 0 0 * * ?"

    // if no strategy is specified for an Item entry below, the default list will be used
    //default = everyChange
    default = every5Minutes
}

/*
 * Each line in this section defines for which Item(s) which strategy(ies) should be applied.
 * You can list single items, use "*" for all items or "groupitem*" for all members of a group
 * Item (excl. the group Item itself).
 */
Items {
    shelly* : strategy = everyMinute
    shellyem38caab561eed31721666236_Leistung : strategy = everyMinute
}

Not sure what else to provide, the persistence connect information is pretty straight forward and certainly without any item/thing based context.

Hope anyone can give me a hint of what I am doing wrong…

Platform information:
- Hardware: Intel Celeron
- OS: Unraid Platform - Linux Debian VM
- Java Runtime Environment: Zulu11.41+23-CA (build 11.0.8+10-LTS)
- openHAB version: openHAB 3.0.1

ubeaut · August 29, 2023, 8:03am

What database?
I use mapdb and rrd4j and those files are called rrd4j.persist and mapdb.persist
Here is the contents of my rrd4j.persist that may help?:

Strategies {
everyMinute  : "0 * * * * ?"
every5Minute  : "0 */5 * ? * *"
every10Minute  : "0 */10 * ? * *"
everyHour : "0 0 * ? * *"
// if no strategy is specified for an item entry below, the default list will be used
default = everyChange
}

Items {
        // persist all items on every change and restore them from the db at startup from mapdb
        FroniusSymoInverter_CurrentSolarYield : strategy = everyChange, every10Minute
        FroniusSymoInverter_SelfConsumption : strategy = everyChange, every10Minute
	LocalSun* : strategy = everyChange, every10Minute
        gTemp* : strategy = everyChange, every10Minute
        gBatterycheck* : strategy = everyChange, every10Minute
        * : strategy = everyChange, everyHour
}

ccbeorn · August 29, 2023, 12:42pm

Database is MySQL.
Thanks für sharing your persist file, but that doesn’t help me at this moment. My issue is that I have to find out what triggered persistence actions before I even placed a .persist file.
Other than that, you can see that my persist file does not look that much different, I use the same principal approach.

rlkoshak · August 29, 2023, 4:05pm

In OH 3 all persistence add-ons have a default strategy which is usually all items saved on every change with restoreOnStartup. This default should be documented in the add-on readme.

Correct, UI support wasn’t added until OH 4.

That sounds like a bug but one that is unlikely to be backported to OH 3 if indeed it is a bug and something fixable.

Does the behavior persist after a restart?

The file is named properly for the add-on? OH will happily load it but only if the name of the file exactly matches the name of the add-on does it get applied.

ubeaut · August 29, 2023, 6:41pm

@rikoshak
UI support wasn’t added until OH 4.

Where in the UI is this persistence support to add the configuration?
I can’t see it anywhere.
Thanks

rlkoshak · August 29, 2023, 7:14pm

Settings → Other Addons → Persistence Addon → Little blue gear icon

All add-on level settings have moved there. There is work to resurface them higher in the UI but for now that’s where you’ll find them.

For persistence add-ons that have connection parameters, this location should be familiar as that’s where you’d configure those as well.

ubeaut · August 29, 2023, 10:37pm

OK found it.
I can only change the logging from the blue gear.
I guess that means I still have to have the persistence file.

rlkoshak · August 29, 2023, 10:41pm

Which add-on?

There should be a big blue “Persistence Configuration” on that page. Click that and you get the persistence configurator.

If it’s not there, there is a problem with the add-on and it needs to be reported.

I’m still using files for now so it’s showing the config read only. If it’s managed, you’ll have all the options to create and apply strategies.

ubeaut · August 29, 2023, 10:49pm

OK…I didn’t think to click and the blue heading…Duh!
I thought that was just the heading. It isn’t really that obvious to us old blind blokes.
I will remove my file based and put it here and see what happens.
I did look in the documentation and I didn’t see anything like this. Picture is worth 1000 words.
Thanks again for point out the obvious.

rlkoshak · August 29, 2023, 10:56pm

One trick is to leave the file, click on the code tab, remove the file, return and create the now editable config, click the code tab and paste. That will give you the same config you have in the file as a place to start.

ubeaut · August 29, 2023, 10:58pm

Yep…that I do know.
What I did notice is after I changed it and restarted I didn’t see the message in the log file that says:
2023-08-30 08:54:41.302 [INFO ] [el.core.internal.ModelRepositoryImpl] - Loading model ‘rrd4j.persist’
And I have the logging set to info.

rlkoshak · August 29, 2023, 11:05pm

The ModelRepository only loads text based configs. You also don’t see it talking about loading blah.rules, foo.items, and bar.thing, because you don’t have those in your system. But if you did have those files, you’d see it report loading those too, each on a separate line.

Everything that is managed gets loaded from the JSONDB, there is minimal parsing required and there is nothing that logs when it loads it (unless something goes wrong or you have trace level logging enabled).

ubeaut · August 29, 2023, 11:12pm

Thanks. I guess when I look at the log file now and watch it start I have to assume the mapdb and rrd4j are working unless I see anything else that looks like errors.

All good I have swapped over to the UI way and will see how that goes. I just have to remember not to edit the files (they have been renamed anyway) and I have to remember to click the blue heading! That one got me.

I am now running no files at all I think?

Wolfgang_S · August 30, 2023, 5:40am

As this happened more often would it be possible/make sense to add something like a link icon in front of the clickable head lines ?

ubeaut · August 30, 2023, 5:47am

Or have it as a drop down option as well?

ubeaut · August 30, 2023, 5:59am

What would be really good is if you go to the persistence tab where you set the default persistence and if you click that it takes you to the configuration instead of searching through the addons and also set the default on that page as well?

ccbeorn · August 30, 2023, 6:09am

Thanks, that explains the initial behavior. And reminds me to put a .persist file first before connecting to the database in runtime.

Understood. Due to the fact that I am on a fairly old version of OH3 (and debian for that matter), I decided yesterday to renew the environment in a separate VM and am currently setting everything up in the new OH4.

Yes, the file is called mysql.persist and the log files confirmed that the file was loaded. Over the minutes, I tried different changes to see if the behavior of saving all values would eventually change.

2023-08-29 00:20:28.658 [INFO ] [el.core.internal.ModelRepositoryImpl] - Loading model 'mysql.persist'
2023-08-29 00:20:28.666 [WARN ] [el.core.internal.ModelRepositoryImpl] - Configuration model 'mysql.persist' has errors, therefore ignoring it: [1,1]: misma
tched input '<EOF>' expecting 'Strategies'

2023-08-29 00:21:52.739 [INFO ] [el.core.internal.ModelRepositoryImpl] - Loading model 'mysql.persist'
2023-08-29 00:23:57.606 [INFO ] [el.core.internal.ModelRepositoryImpl] - Loading model 'mysql.persist'
2023-08-29 00:25:26.426 [INFO ] [el.core.internal.ModelRepositoryImpl] - Loading model 'mysql.persist'
2023-08-29 00:26:15.697 [INFO ] [el.core.internal.ModelRepositoryImpl] - Loading model 'mysql.persist'

Yes, it did. That was also my first assumption after placing the .persist file and even though the log file confirmed that it was found & loaded.

Anyways, thanks a lot for the help. I will report on how it goes with OH4.

rlkoshak · August 30, 2023, 11:57am

There is a PR to rework where and how most of this stuff appears in MainUI already. Something is going to change but I haven’t followed the issue that closely. I do know that it will be moved higher up in the MainUI Settings at a minimum.

ccbeorn:

Yes, the file is called mysql.persist and the log files confirmed that the file was loaded. Over the minutes, I tried different changes to see if the behavior of saving all values would eventually change.

shouldn’t that be jdbc.persist? JDBC - Persistence Services | openHAB

ccbeorn · August 30, 2023, 9:29pm

Hmm, I took the name from my OH2 Setup where it was called mysql.persist. Maybe I did that one wrong.

In OH4, I now kept on using the UI setup, including the persistence. Something seems strange still:

With the persistence addons installed, I got already lots of error messages in the log file, even though I hadn’t entered any connect information in the persistence addon yet.

2023-08-30 22:58:10.639 [WARN ] [persistence.jdbc.internal.JdbcMapper] - JDBC::openConnection: no driver available!
2023-08-30 22:58:10.641 [WARN ] [jdbc.internal.JdbcPersistenceService] - JDBC::store: No connection to database. Cannot persist state '21.4 °C' for item 'Homematic_Heizthermostat_Wohnzimmer_Actual_Temperature (Type=NumberItem, State=21.4 °C, Label=Actual Temperature, Category=Temperature, Tags=[Point], Groups=[Homematic_Heizthermostat_Wohnzimmer])'! Will retry connecting to database when error count:0 equals errReconnectThreshold:0

I then entered the connect information in the persistence addon config and got the following log message, which looks good so far:

2023-08-30 23:03:01.990 [INFO ] [persistence.jdbc.internal.JdbcMapper] - JDBC::openConnection: Driver is available::Yank setupDataSource

With that, I see an (empty) items table in the mysql DB was created, so far so good. But that’s it. Even with the default strategy still at “everyChange”, nothing got written in the database since.
I now added the 5min strategy and defined that as default strategy but nothing seems to happen in the database whatsoever. Shouldn’t every item value be now stored in the database every 5min? Or do I have to define the items first? But then, default Strategy wouldn’t make sense…

configurations: []
cronStrategies:
  - name: everyMinute
    cronExpression: 0 * * ? * *
  - name: everyHour
    cronExpression: 0 0 * * * ?
  - name: everyDay
    cronExpression: 0 0 0 * * ?
  - name: every5Minutes
    cronExpression: 0 0/5 * * * ?
defaultStrategies:
  - every5Minutes
thresholdFilters: []
timeFilters: []
equalsFilters: []
includeFilters: []

rlkoshak · August 30, 2023, 10:14pm

My understanding is that in OH 3 all the separate persistence add-ons that depend on JDBC (basically all the ones that talk SQL which includes MySQL) have been moved to the JDBC add-on. There isn’t a separate page in the docs for MySQL any more which indicates the add-on isn’t there any more.

There is a short section in the docs that talk about migrating: JDBC - Persistence Services | openHAB

Maybe there is something relevant there

Your config looks incomplete. I don’t see any actual configurations, just strategies. This is what mine looks like:

configurations:
  - items:
      - "*"
    strategies:
      - everyMinute
      - everyChange
    filters: []
cronStrategies:
  - name: everyMinute
    cronExpression: 0 */1 * * * ?
defaultStrategies:
  - everyChange
thresholdFilters: []
timeFilters: []
equalsFilters: []
includeFilters: []