Persistence changes in 5.1

[mstormi (Markus Storm)]

Default persistence configuration has changed in 5.1 as announced here.

Please discuss all persistence - migration issues here rather than in the openHAB 5.1 Release discussion thread, thank you.

[sihui]

@tcgerhard

any more obvious way to do this?

openHAB 5.1 Release discussion

I misspoke in how I worded it. I meant that to mean if you wanted to change to the old default behavior from a more constrained config but clearly on rereading it I didn’t say that at all. But if you’ve created a .persist file, you’ve already dealt with it. If you didn’t have a .persist file upgradeTool created a config for you. You are only forced to do anything if you had a .persist file. And if you had a .persist file, all you had to do was remove one line and possibly modify others if y…

[Levin1 (Levin)]

Hi

First, thank you very much for your effort in developping Openhab.

After upgrading to 5.1, I got the following Persistence Warning in “Health Checks” and I’m not sure why:

Problem: Persistence strategy for persistence items only has a restore strategy

This is my config file:

Strategies {
	//every10Minutes: "0 0/10 * * * ?"
	//every15Minutes: "0 0/15 * * * ?"
	//every30Minutes: "0 0/30 * * * ?"
	//default = everyChange
	every4Hours:	"0 0 0/4 ? * * *"
	every8HoursUebergang: "0 0 0/8 ? MAY,JUN,JUL,AUG,SEP *"

}

Filters {
	greater01 : > 0.1
	greater1 : > 1
	greater3 : > 3
}

Items {
	//* : strategy = everyChange, restoreOnStartup
	* : strategy = restoreOnStartup
	persistence_everyChange*: strategy = everyChange
	persistence_everyChange_thr_greater_01* : strategy = everyChange filter = greater01
	persistence_everyChange_thr_gr_1* : strategy = everyChange filter = greater1
	persistence_everyChange_thr_gr_3* : strategy = everyChange filter = greater3
	Heizung_Warmwasser_Solltemperatur : strategy = every4Hours
	Heizung_Warmwasser_heizen : strategy = every4Hours
	Heizung_Kuehlen : strategy = everyChange, every8HoursUebergang
	Heizung_kuehlen_Solltemperatur : strategy = everyChange, every8HoursUebergang
	Heizung_kuhlen_Vorlauftemperatur : strategy = everyChange, every8HoursUebergang
	Heizung_Sommermodus : strategy = everyChange, every8HoursUebergang
	UG_Feuchtigkeit : strategy = everyUpdate
	UG_Temperatur : strategy = everyUpdate
	OG_Gang_Temperatur : strategy = everyUpdate
	EG_Kueche_Temperatur : strategy = everyUpdate
}

Is this warning because of the “* : strategy = restoreOnStartup” ?
Why is this wrong? Should I add “restoreOnStartup” to every Group Item directly?

[Mherwege (Mark Herwege)]

@Levin1

Is this warning because of the “* : strategy = restoreOnStartup” ?
Why is this wrong? Should I add “restoreOnStartup” to every Group Item directly?

Yes, it is. And it is a case hat was somewhat overlooked when developing this check.

your configuration should work as is. However, it will also restore items you do not persist anymore if they are still in the database. That could be unexpected. Putting restoreOnStartup on every line would get rid of the warning and avoid that potential side effect.

There is also the case where you would persist something in a rule, or only once, and then still use restoreOnStartup for it. You would not need any other persist strategy then. I can see that as valid, but it will still be shown as a health warning.

I guess there is a tradeoff here. I am not sure removing the warning is the best thing to do. It would need to be some extra check like: don’t flag if restoreOnStartup on all items and there are other item lines that would effectively persist something.

A warning is just what it is meant to be. It should point to potential issues, avoiding a lot of digging to find the problem, but if you understand why it is there, it may be acceptable.

[jimtng]

@Mherwege

A warning is just what it is meant to be. It should point to potential issues, avoiding a lot of digging to find the problem, but if you understand why it is there, it may be acceptable.

This may not be easy to do, but usually in other software they provide the option to say “don’t warn me again about this” - with the option to reset/restore all warnings. This lets them see the warning and also silence them if they are aware of it.

Is there a way to see what items exist in the persistence database regardless of the current strategy / settings? It would be nice if such UI would also let people clean up the database by deleting them (filter + multiselect etc similar to the items/things/rules listing). If not, that might be a good thing to add - I don’t know, it’s just an idea. Even better if we can clean up the entries that are “older” than a certain point - again not sure if that’s possible / easy to do.

[Mherwege (Mark Herwege)]

@jimtng

This may not be easy to do, but usually in other software they provide the option to say “don’t warn me again about this” - with the option to reset/restore all warnings. This lets them see the warning and also silence them if they are aware of it.

It is a good idea, but may require quite some work to do this. And this should then not be just for the persistence warnings (e.g. think of the semantic warnings). All of them get the list of warnings from an api call, but the server never stores a list of unique ID’s for warnings with statuses. There is no update going from the UI to the server either. So there is a lot of infrastructure to build to achieve this.

@jimtng

Is there a way to see what items exist in the persistence database regardless of the current strategy / settings? It would be nice if such UI would also let people clean up the database by deleting them (filter + multiselect etc similar to the items/things/rules listing). If not, that might be a good thing to add - I don’t know, it’s just an idea. Even better if we can clean up the entries that are “older” than a certain point - again not sure if that’s possible / easy to do.

The API calls exist to get the list of items currently available in a persistence store. That could be exposed in the UI.

However, what can be done with it depends on the underlying persistence service. Currently, only modifiable persistence services would allow changing data in the store from OH side. These are the services that allow updating data at other timestamps than the current time. rrd4j and mapdb are not modifiable, while e.g. influxdb is. Some persistence services have their own OH command line or external tools to manage the DB. Some can be managed using external tools. To be able to create a UI to manage persistence DB content, we probably need to extend the api again and extend the persistence add-ons to make it possible in the first place. I think in the long run, it could be worth it, but it is a major undertaking.

[Nadahar (Nadar)]

If there are warnings that “some people will have to live with”, there absolutely must be a way to mute them. I’m just wondering if it could be made as easy as a persistence configuration option that says “Disable persistence health warnings”, which then prevented the health warning from being generated in the first place?

[jimtng]

IMO persistent (pardon the pun) warnings, even when they can be disabled would be pretty annoying. Disabling the warning would be yet another thing for the user to have to do.

[Mherwege (Mark Herwege)]

@Nadahar

If there are warnings that “some people will have to live with”, there absolutely must be a way to mute them. I’m just wondering if it could be made as easy as a persistence configuration option that says “Disable persistence health warnings”, which then prevented the health warning from being generated in the first place?

It would not require any API change and not be a lot of effort to provide that function in the UI and store it in the browser cache. Storing it on the server would require a new API endpoint. That would be better.

[Nadahar (Nadar)]

@jimtng

IMO persistent (pardon the pun) warnings, even when they can be disabled would be pretty annoying. Disabling the warning would be yet another thing for the user to have to do.

I absolutely agree, I’m just saying that, lacking better solutions, or in the meanwhile, there should be some way to disable them when they aren’t relevant/don’t apply - otherwise you condition people to ignore the health warnings altogether, which is even worse.

There is another situation where this is an annoyance, which I find myself in frequently with test installations: When you don’t want persistence. The warnings don’t seem to quite accept that.

[Nadahar (Nadar)]

@Mherwege

It would not require any API change and not be a lot of effort to provide that function in the UI and store it in the browser cache. Storing it on the server would require a new API endpoint. That would be better.

I haven’t looked at the details, is the warning generated entirely in MainUI based on some “rules” for how the configuration should look? I thought maybe Core did some part of the whole.

Using the browser for settings is quite annoying IMO, it already drives me crazy that the theme settings are stored in the browser. That means that you have to change it for every browser on every device you have - which “feels” like an endless undertaking, where things will “never” be in a consistent state.

[Mherwege (Mark Herwege)]

@Nadahar

I haven’t looked at the details, is the warning generated entirely in MainUI based on some “rules” for how the configuration should look? I thought maybe Core did some part of the whole.

The list of health issues by category (persistence, semantics, broken links…) are provided by individual API endpoints. The UI calls each of them.

As said, ideally the server API should be enhanced to manage health issue statuses, be it on a category level, or an individual message level.

[rlkoshak (Rich Koshak)]

@Mherwege

It is a good idea, but may require quite some work to do this. And this should then not be just for the persistence warnings (e.g. think of the semantic warnings). All of them get the list of warnings from an api call, but the server never stores a list of unique ID’s for warnings with statuses. There is no update going from the UI to the server either. So there is a lot of infrastructure to build to achieve this.

It would be a little bit of a hack, but checking the box could just update the logging config to stop showing the warning in the logs. Each warning would have to have a unique logger but that should be achievable.

Then it’s just saving a flag somewhere (maybe even a cookie?) to not show it in the UI.

That would work with the existing APIs and reduce the infrastructure work required.

[Nadahar (Nadar)]

@rlkoshak

It would be a little bit of a hack, but checking the box could just update the logging config to stop showing the warning in the logs. Each warning would have to have a unique logger but that should be achievable.

I’d call that hack pretty huge

[tcgerhard]

Rather than addressing this in the logger, another approach might be to allow a change to the syntax, somthing allowing a new keyword, “RestoreOnStartupEvenIfItMightBeStale” ('I’ll leave the exact wording to a better wordsmith) which behaves just like RestoreOnStartup, but cues the API to not generate the warning.

[rlkoshak (Rich Koshak)]

@tcgerhard

but cues the API to not generate the warning.

The warning isn’t that the data might be stale. The warning is that, based on the persistence config, it looks to the health checker that you are trying to restoreOnStartup Items that were never saved in the first place.

[tcgerhard]

@rlkoshak

The warning isn’t that the data might be stale. The warning is that, based on the persistence config, it looks to the health checker that you are trying to restoreOnStartup Items that were never saved in the first place.

Thanks for clarifying that, that’s why I wasn’t proposing that specific string as much as pointing out that the user could declare that they understand the implications of their choice. Maybe RestoreOnStartupIfAvailable?

[glen_m]

Not really sure if this is the right place to put this based on the discussions on persistence spread over various threads post 5.1.0 release, however this looks like a starting point.

For context, prior to the 5.1.0 upgrade, I had:

• UI based persistence configuration
• No default persistence strategy
• Explicit configuration for every item where persistence is required
• Zero persistence issues as a result of the upgrade

Looking at the release notes, specifically the following:

To avoid these issues, we now require an explicit configuration for anything to be persisted.

It makes me wonder if the workflow of setting up persistence for items in the UI contributes to people steering away from using ‘explicit’ configurations.

Would it be helpful to be able to select (available) Persistence Strategies for an Item, within the Item screen, as opposed to having to jump across to the Persistence Settings, after you create/edit an item?

The strategies would still be created in the Persistence Settings Screen, but on the Items screen it would be another section with may be something like:

• RRD4J
  • EveryMinute
  • RestoreOnStartup
• InfluxDB
  • EveryChange
  • EverySecond
  • EveryMinute

With the user then can:

• Selecting a checkbox for each of the strategies they require
• Maybe a ‘greyed out’ tickbox showing that strategy is already enabled (or excluded) for that item, either (in order):
  • Via a persistence file config, where they cannot edit/change that in UI
  • Or inherited via group membership, where the user can explicitly enable it for that item (or exclude that item)

The persistence setting screen still has value in this context for both creating the strategies, and also providing a persistence service based view of Items/Groups configured against it.

I’m sure the above result can already be somewhat achieved by creating groups for each Persistence Strategy, and selecting the applicable ones as parents at time of item creation/edit? But this creates a level of (unnecessary?) abstraction, which hides details of which items are persisted when looking at things from the persistence settings screen.

I know its easy enough for me to suggest half-baked ideas like this, with no understanding of the complexity of what something like this would take. But I’m sure a simplified workflow, where a user can manage the persistence within the item screen, would help if having explicit persistence configurations is the stated direction that users should be heading in.

Happy to raise a feature request with further detail, if there is any merit in the above, and this would help avoid related issues in future (albeit the transition away from the default config should only be a one time issue, I presume).

Again, I had no persistence issues as a result of the upgrade, but just more of of an observation….

Cheers

[rlkoshak (Rich Koshak)]

@glen_m

would help if having explicit persistence configurations is the stated direction that users should be heading in.

Do not read too much into that.

Consider this line in a .persist file

*: everyChange, restoreOnStartup

or this managed config:

configurations:
  - items:
      - "*"
    strategies:
      - everyChange
      - restoreOnStartup
    filters: []
aliases: {}
cronStrategies: []
thresholdFilters: []
timeFilters: []
equalsFilters: []
includeFilters: []

That’s an “explicit persistence configuration”. Every Item, no matter when it’s created, gets persisted on everyChange and is restored on startup. Once set, this config never need to change again no matter how many Items were there before nor how many Items are crearted in the future.

Explicit config doesn’t mean each and every Item is mentioned in the persistence config individually. What it means is you must have a configuration in the frist place. The add-ons will not start doing stuff without a configuration like it does now.

I’m intrigued by being able to adjust the persistence config for an Item from the Item’s page but I’m worried about the complexity. We don’t only have the strategies to worry about, there’s also aliases and filters to content with.

[Mherwege (Mark Herwege)]

Instead of changing the persistence configuration from the item screen, what about showing if the item is configured for persistence, with the service and strategy, and provide a link to the persistence configuration screen?