Design Pattern: Group Based Persistence

persistence
designpattern
Tags: #<Tag:0x00007fae06c12210> #<Tag:0x00007fae06c11ec8>

(Rich Koshak) #1

Copying this from the great big Design Pattern thread into its own thread so I can find it more easily. I reference this design pattern more than any other. There are some minor updates including:

  • added an InfluxDB to the mix for charting
  • added a warning to always specify the persistence engine when making calls to persistence methods

Problem Statement

There are several reasons why a home automation enthusiast would want to use openHAB persistence: charting, using historical Item states in rules logic, restore previous values on openHAB restart, and my.openhab integration, detailed analysis, access to the data with external tools, etc. However, different databases are more or less well suited to each of these use cases.

Concept

Since no one persistence engine is best for all use cases, configure more than one and use Groups to allocate which Items get saved to which persistence engine and how.

For example, one can set up MapDB for restoreOnStartup, rrd4j for charting recent data and historical data, my.openhab for IFTTT integration, and MySQL for Items for detailed external analysis.

For each use case, create a Group and only use these groups in the .persist files.

Then allocate your Items to whatever group(s) based on how that specific Item will be used.

#Example
In my current setup I restoreOnStartup on all my Items, I use rrd4j for charting and historical state, I have a couple of Items I expose to IFTTT through my.openhab, and no Items that I analyze or expose the DB outside of OH.

Items:

// Persistence Groups
Group gMyOpenhab // for Items that are exposed to IFTTT
Group gChart     // for Items to persist for charting
Group gHistory   // for Items to preserve their history
//Group gRestore // for Items to restore on startup (currently everything so commented out)
...
// Example Item that I Chart
Number          Weather_Temp_Min        "Minimum Outside Temp [%.0f] °F"        <temperature> (gChart, gWeather_Temp_Chart)

// Example Item that I use historical data
Contact       N_D_Front                        "Front Door [MAP(en.map):%s]"             <frontdoor>  (gHistory, gDoorSensors, gRemindDoorSensors, gAlarmSensors, gHydraSensors)    { mqtt="<[mosquitto:entry_sensors/main/front_door:state:default]" }

// Example Item that I expose to my.openhab NOTE: it is also a member of gHistory
Switch  T_D_Garage1     "Garage Door 1"         <garagedoor> (gHistory, gMyOpenhab, gGarageOpeners)

mapdb.persist file: I only use MapDB for restoreOnStartup and restoreOnStartup applies to all Items

Strategies {
        default = everyUpdate
}

Items {
        // persist all items on every change and restore them from the db at startup
        * : strategy = everyChange, restoreOnStartup
}

NOTE: I don’t know if setting a default strategy is required, I included it anyway, but it is basically meaningless because it is never used in the Items section.

To make restoreOnStartup work we need to persist every change. The ‘*’ resolves to all Items.

rrd4j.persist file: I use rrd4j for historic data. Because of limitations of rrd4j data must be saved at least every minute for charting or else you end up with blank charts and previousState and historicState do not appear to work.

Strategies {
        // for rrd charts, we need a cron strategy
        everyMinute : "0 * * * * ?"

        default = everyChange
}

Items {
        // additionally persist weather info every minute
        gHistory* : strategy = everyUpdate, everyMinute
}

**influxdb.persist file: I use InfluxDB along with Grafana for generating complex and attractive charts. See the InfluxDB+Grafana tutorial for details.

// persistence strategies have a name and a definition and are referred to in the "Items" section
Strategies {
        default = everyChange
}

Items {
        gChart* : strategy = everyUpdate
}

myopenhab.persist file

Strategies {
        default = everyUpdate
}

Items {
        gMyOpenhab* : strategy = everyUpdate
}
Advantages

Advantages

The advantages of this approach include:

  • The persistence behavior for each Item is documented with the Item (i.e. the group membership) keeping as much information about the behavior of the Item in one place
  • Adding new Items or changing how they are persisted simply involves changing that Item’s group membership.
  • Once set up, the .persist files need never be changed unless you decide move from one persistence engine to another
  • Uses the best persist engine for the desired behavior. For example MapDB can save all types of Items but only keeps the last value do it is great for restoreOnStartup but not great for historic data whereas rrd4j is great for charting recent data but can’t save anything that isn’t a numerical value so isn’t so great for restoreOnStartup

A note about usage:

openHAB can have only one default persistence engine. Any calls you make in Rules where you do not specify a persistence engine will use the default. When using this approach for Persistence be careful when and if you change the default persistence engine as that may break some of your Rules.

I avoid problems, I recommend always specifying the persistence engine you want to use in all of your persistence calls. For example:

MyItem.previousState("rrd4j")
MyItem.lastUpdate("mapdb")

Storing item states for later recall
InfluxDB+Grafana persistence and graphing
InfluxDB + Grafana vs. MySQL + rrd4j
OH2: restoreOnStartup not working properly
Design Pattern: Working with Groups in Rules
Low-power Z-Wave switch with two outputs needed
Persistence of nested groups
Keep the item settings during reboot
Last Stage After reboot
Last Stage After reboot
System started rule not executing on build #1034
Create txt File
YAGSA - Yet Another Group Structure Approach
InfluxDB openHAB stores no Data
Push Notifications with subject and variables?
Comprehensive Wunderground using HTTP Binding Example
Thank you all!
Unable to restore state for item
No working trigger functions openHAB 2.1
One things / items file or many?
MQTT persistence worked in 2.0.0 snapshot but not 2.1.0 release
Persitance of items in group
How to store / query Influxdb from OH?
Keep 'when Item received update' rules from running during RestoreOnStartup
CHart displays one item correctly, another one not
Excluding a certain items from being persisted into database
Problem min and max since?
Persisting groups (MariaDB)
Design Pattern Persistance Language Problem
Netatmo temperature and chart
Best persistence for new install?
Problems with Items/Things in Habmin/PaperUI
Sitemap and Persistence Not Working As Expected
Persistence & Graphs - why so complicated?
Wake Up Light
Number value help please
Blank items in UI after restart/reload of OH1
Interested in hardware
Some newb questions I could not find answer for
Items not appearing in myopenhab.org/items despite being correctly listed in my.openhab.org/rest/items
Change rwe switch from "ON" to "1"
Rules - Heating
Does restoreOnStartup require an everyChange/everyUpdate strategy
Developing a persistence addon (not a binding)
My openHab Setup
Presence detection via iPhone's mDNS Entry. How to integrate into OH2?
Persistence: "don't persist" alternative to override default strategy?
Error that I don't understand
Help with Persistence DB config
Openhab2 RPI System Temperature and DS18B20 OneWire Chart with persistence
Help with Persistence DB config
Persistence recommendations
Setting OH2 for KNX, a Windows install
RestoreOnStartup when updating items file
(Ham Wong) #2

HI @rlkoshak

I found you only use every change on mapsdb else all in every update,
may I know it was due to database requirement or just personal favorite?

P.S. I found in my RPI3B, influxDB miss store few of items from time to time until I change to every minute, I haven’t try every update yet.


(Rich Koshak) #3

I use everyChange because I only use it for restoreOnStartup. Given that I only need to save a new value when an Item’s state changes. If I receive an update that doesn’t change the Item’s state, I don’t need to overwrite the value already stored in the DB.

However, if I were using MapDB to also figure out when the last time the Item received an update, whether or not the update caused the Item to change, then I’d use everyUpdate instead.


(Kai S) #4

Thank you @rlkoshak for this post.

Here is my experience:

  • I was looking at 14 different Persistence add-ons that are available to me, with no guidance on what to chose or why
  • I then looked at the user manual under configuration -> persistence - which to this date is a stub http://docs.openhab.org/configuration/persistence.html
  • I then found your post by pure luck … and feel extremely grateful for this contribution.

I’m wonderig if @ThomDietrich (contributor on docs page) would want to add this great content directly into the docs … at least the rrd4j and MapDB items … these 2 side by side (and why) - this is perfect for 95% of new adopters … and for the rest it will be a perfect jumping board into more advanced persistence options (aka the other 11 or 12 …)

Greetings, Kai


(Rich Koshak) #5

There is an issue already opened for this and in all likelihood I’m supposed to be working on it. Thom will concur, I’m pretty good on the forum, not so good on github.

Of course, Thom and I disagree as to whether this DP is even a good idea (he would rather define everything in the .persist file and not mix concerns in the .items files.


( ) #6

Hey guys,
as always I’m trying to provide clarity about the possibilities of openHAB in the docs without mixing my personal preferences in as the one true solution. :angel:

@kai-bcn you are absolutely right, the stub looks kind of sad. I felt the same way about the Transformation stub and started working on it just last week (check it out, click the View button to see the result of the source code).

Next could be the Persistence article but quite frankly I don’t know when I’ll find the time to get to that. It would actually be great if somebody like you @kai-bcn could go through the old wiki article and do some spell checking and grammar improvements (Either in the wiki on GitHub or if that is not your turf, you could also do that here in a private message). Next we could simply move it over to documentation and replace nothing with something. https://github.com/openhab/openhab1-addons/wiki/Persistence

Best! Thomas <-- A mystery solved


(Anders Alfredsson) #7

I just started using this approach to persistence (I find it much easier to maintain as my number of items starts increasing) but I have noticed a small issue. I have a group item Group:Switch:AND(OFF,ON) Presence_Group <present> (Persistence_EveryChange) that contains presence items for me and my wife, which I want persisted (the state of the group as well as the state of the individual items).

However, it only works if I explicitly write the group item name in the .persist file and not as part of the persistence group. I guess this behavior is intentional, to avoid persisting a lot of groups, which most of the time isn’t very useful, but thought I should mention it here in case someone else have the same issue.


(Dome) #8

@rlkoshak, thanks for yet another useful Design Pattern. As a non-engineer who is looking to learn as well as improve my openHAB setup, I appreciate your mix of examples and explanation. I was recently setting up a timeline in HABpanel. When it didn’t display properly, I figured out I had my rrd4j configured incorrectly, and your topic helped me get it straight.


(Will) #9

This is an extremely helpful design pattern, shortened my learning curve dramatically, kudos Rich!


(Chris Colden) #10

quick question. if i have a group of lights, can i just add the group to the gHistory, or do i also need to include each item individually in the group gHistory?


(Rich Koshak) #11

That is a good question. I’m at an airport right now so can’t test it out, but I suspect Persistence does not recurse though subgroups so you would have to add the Items to gHistory individually. But it is worth a test.


(Christoph Weitkamp) #12

I use the nested group approach for persistence in my setup and it works for me.


(Chris Colden) #13

Thanks @rlkoshak and @cweitkamp I will give it a go. I’m hoping it does work, else my items file is going to explode with additional group memberships i’ll have to add.