Automatic persistence: detect which types cannot be properly persisted and skip them

[ceedee]

Hi there!

I’m automatically persisting all items which are in a specific group. For all such items I basically run the follwing script:

(function (context) {

	var logger = Java.type('org.slf4j.LoggerFactory').getLogger('org.openhab.model.script.Rules.Persistor');

	Number.isNaN = Number.isNaN || function(value) {     
		return value !== value
	}

	var hasNumber = function(item) {
		return !isNaN(parseFloat(item.state))
	}
	
	var isConvertibleToNumber = function(item) {
		var Transformation = Java.type("org.openhab.core.transform.actions.Transformation")
		var transformedState = Transformation.transform("MAP", "persistor.map", item.state.toString())
		if (!isNaN(transformedState)) {
			return true
		} else {
			logger.error("item " + item.name + " not persitable, state=" + item.state)
			return false
		}
	}
	
	context.persistItemState = function (item) {
			
		if (!hasNumber(item) && !isConvertibleToNumber(item)) {
			logger.error(item.name + " is no number but " + parsedState + " (was: " + item.state + "), skipping!")
		} else {
			var persistence = Java.type("org.openhab.core.persistence.extensions.PersistenceExtensions")
			persistence.persist(item, "influxdb")
			persistence.persist(item, "mapdb")
			persistence.persist(item)
			logger.debug("Item changed: " + item.name + ", persisted value " + item.state + " to InfluxDB, MapDB and default persistence")
		}
	}

})(this)

In my persistor.map I’ve the following entries:

ON=1
OFF=0
OPEN=1
CLOSED=0

The problem I wanted to solve with this is this: Sometimes I see that a number value cannot be persisted in InfluxDB (exception from the persistence layer) because it previously tried to persist a string:

[ERROR] [org.influxdb.impl.BatchProcessor    ] - Batch could not be sent. Data will be lost
org.influxdb.InfluxDBException$FieldTypeConflictException: partial write: field type conflict: input field "value" on measurement "light-brightness-percent" is type float, already exists as type string dropped=2
	at org.influxdb.InfluxDBException.buildExceptionFromErrorMessage(InfluxDBException.java:144) ~[bundleFile:?] 

I assume it was something like NULL or UNDEF. So I decided to do a check before persisting.

I did the mapping check because my number conversion check does not work for switches and contacts although in InfluxDB I see Integer values instead of Strings.

But doing that mapping check is a rather big smell in the code since it couples the inner workings of the persistence layer to my map. My question is if there’s a better way to prevent my initial problem.

Best regards,
ceedee

[rlkoshak (Rich Koshak)]

@ceedee

assume it was something like NULL or UNDEF

OH doesn’t persist NULL and UNDEF unless you’ve somehow forced it to with this rule. The only way this could have happened is if you previously had a String Item that you changed to some other type.

Why are you coding this yourself instead of setting an appropriate strategy?

Unless this recently changed, InfluxDB already transforms ON/OPEN to 1 and OFF/CLOSED to 0.

The better way would be to just use what OH provides instead of reinventing the wheel.

In the mean time, you’ll probably have to go and clean up the database if those String values because calling .persist on those items will forever fail now.

[ceedee]

@rlkoshak

Why are you coding this yourself instead of setting an appropriate strategy?

I’d prefer to adjust persistence where I configure the item, especially with openHAB 3 where I do the configuration via MainUI but then still have to switch to an editor to edit the persistence files for MapDB and InfluxDB. It feels awkward to reference the item name in several places – you always have to keep them in sync. That’s what I tried to avoid.

Let me ask a design question on the persistence API: why is there a persist method if there is no safe way to call it? – Basically that’s why I asked the initial question. I was hoping that there’s a way to ensure proper persistability before calling that method.

[rossko57]

@ceedee

I’m automatically persisting all items which are in a specific group

You know you can specify this group in a xxx.persist file, to persist all members truly automatically?

@ceedee

I was hoping that there’s a way to ensure proper persistability before calling that method.

I don’t think the item.persist method will ever overcome a service error like old data in a different form to existing Item.

[ceedee]

@rossko57

You know you can specify this group in a xxx.persist file, to persist all members truly automatically?

No, I didn’t. Sounds like a great solution to my problem, thanks I’ll give it a try and report here!

@rossko57

I don’t think the item.persist method will ever overcome a service error like old data in a different form to existing Item.

I was talking about something like persistIfSet(...). I’m wondering what the use cases for a persist method are if one cannot really check if the data can be properly persisted. – I won’t need to use the method assuming that the group persisting works, but I’m still curious

[rossko57]

You might review

Design Pattern: Group Based Persistence

See Design Pattern: What is a Design Pattern and How Do I Use Them for an explanation of what a DP is and how to use them. Problem Statement There are several reasons why a home automation enthusiast would want to use openHAB persistence: charting, using historical Item states in rules logic, restore previous values on openHAB restart, detailed analysis, access to the data with external tools, etc. However, different databases are more or less well suited to each of these use cases. Concept Si…

@ceedee

I’m wondering what the use cases for a persist method are if one cannot really check if the data can be properly persisted

Exactly the same as the automated system default and xxx.persist file based strategies I suppose, which don’t do any checking either.

[ceedee]

@rossko57

You might review

I’ll check this, thanks!

Update on the group persistence: seems to work well so far, I’ve deleted my custom rule.

[rlkoshak (Rich Koshak)]

@ceedee

Basically that’s why I asked the initial question. I was hoping that there’s a way to ensure proper persistability before calling that method.

It’s intended to be used in those cases where the user wants to do some checking before a state is persisted, such as skipping an obviously faulty reading from a flakey sensor. I presupposes that you’ve already checked that the state to be saved is one you want to save.

I still would expect it to not save NULL and UNDEF states since there really is no save way to store them for most Item types. But perhaps the methods assumes you know what you are doing.

You initially said you are guessing that NULL or UNDEF was stored. You should confirm whether or not that’s true. I’m not yet convinced that’s what happened

You can test whether the Item is NULL or UNDEF before calling persist, assuming that is the actual cause of the problem. But if the cause is because you’ve removed an old String Item and recreated it as a Number Item (for example) then it’s up to you to clean up the database.

[ceedee]

It only looked like using a .persists-file works properly, but I got again lots of errors in the log. So I decided to do more investigation using the rule again.

I re-enabled the persistor rule but this time I only use it for InfluxDB.

I identified a couple of items which have String content and saw that my checks keep them from being peristed. Now I removed these items from the persist group. – So, errors are reduced now, progress

But now I’ve trouble with persisting light brightness levels. Some work, others dont. I tried to debug for a dimmer item (Hue). So I connected to my InfluxDB:

Using series show I got

...
light-brightness-percent,floor=firstFloor,item=PHLHUELTFfWrPanelWall_Brightness
...

Using select * from "light-brightness-percent" I do not get a measurement for this item.
I checked via Grafana if I can get a measurement, but I couldn’t.

How can I continue investigating?

A question regarding the previous discussion in this thread: You suggested to check how UNDEF / NULL are handled. How can I check if there’s UNDEF / NULL coming to my rule? – But that’s only one problem.

[rossko57]

@ceedee

item=PHLHUELTFfWrPanelWall_Brightness

Seems related -

Help with rule

Underscore in Item name. Coupled with influxdb, you should look at - and the workaround

[ceedee]

I’ve read what you linked. I think, it doesn’t apply to me: I’m using InfluxDB 1.8. (And I’ve not checked the underscore replacement setting, so everthing is forwarded to InfluxDB untouched)

Apart from that I’m seeing measurements arriving for items named in the same way:

> select * from "light-brightness-percent"
1642875988401000000 firstFloor PHLHUELTFfLrIndirectLeft_Color  31,65,0
1642876109068000000 firstFloor PHLHUELTFfLrIndirectLeft_Color  31,65,0
1642877111131000000 firstFloor PHLHUELTFfLrSpot3_Color         86,40,0
1642877111133000000 firstFloor PHLHUELTFfLrSpot2_Color         86,40,0
1642877111135000000 firstFloor PHLHUELTFfLrSpot1_Color         86,40,0
1642877114568000000 firstFloor PHLHUELTFfLrSpot2_Color         86,40,100
1642877114569000000 firstFloor PHLHUELTFfLrSpot1_Color         86,40,100
1642877114570000000 firstFloor PHLHUELTFfLrSpot3_Color         86,40,100
...

What comes to my mind is that I actually just wanted to store the brightness component instead of the HSB-Value. I’m not sure if Grafana can make some sense out of it. But that’s another topic.

PHLHUELTFfWrPanelWall_Brightness is a dimmer for a light with color temperature only, so it should be just a number. Does this point into a direction for a solution?

Still, I’ve a lot of measurements containing just numbers like

> select * from "battery-level-percent"
1643132238756000000 firstFloor  AVMTRMFfBer_BatteryLevel                          100
1643132238757000000 firstFloor  AVMTRMFfWr_BatteryLevel                           100
1643132238758000000 firstFloor  AVMTRMFfBar_BatteryLevel                          100
1643132238764000000 firstFloor  AVMTRMFfLr_BatteryLevel                           60
...

Would it help to post screenshots of the working and not working items?

Update: I tried removing the influxdb metadata from the non-working item. I now got entries in InfluxDB:

> select * from PHLHUELTFfWrPanelWall_Brightness
1643133014642000000 PHLHUELTFfWrPanelWall_Brightness 0
1643133014853000000 PHLHUELTFfWrPanelWall_Brightness 22
1643133018315000000 PHLHUELTFfWrPanelWall_Brightness 22
1643133028366000000 PHLHUELTFfWrPanelWall_Brightness 50

I’ve a wild guess: Could it be that the measurements have to be of the same type? (Would make sense from a charting perspective, although the items are separated into floors etc)

If that’s true, my above question holds: how to just persist the B part of the HSB type.

Best regards,
ceedee