Migrate your existing persistence data to InfluxDB

rrd4j
influxdb
grafana
Tags: #<Tag:0x00007f212933b340> #<Tag:0x00007f212933b1d8> #<Tag:0x00007f212933af80>

(Christoph Wempe) #1

If you start using InfluxDB to store your data (for example to get great charts with Grafana) you might want to migrate your old data from the “old” persistance service (like rrd4j) to the new database.
Otherwise our new charts would only start from the time you started using InflluxDB.

So here is a little bash script wich reads the available data via REST and imports the data to InfluxDB.

This main version of the script does work only with openHAB 2.0 at the moment!!!

Here is a branch wich is compatible (only) to openHAB 2.1:


Migrate rrd4j data to InfluxDB
InfluxDB+Grafana persistence and graphing
(Rich Koshak) #2

@ThomDietrich, should this be linked to from the OP for InfluxDB+Grafana Tutorial? More people may want to switch if they know they now have an easy way to migrate their data out of whatever they are using now and into InfluxDB.


( ) #3

Good call, I would maybe have missed it.

@christoph_wempe this is a great help. I’ve linked the tool in the InfluxDB+Grafana article: InfluxDB+Grafana persistence and graphing


(Dan) #4

Fantastic. Fear of losing old data had made me stick to rrd4j.

The other thing that stops me moving is concern that InfluxDB databases may spiral to enormous size - I used to have that with db4o before I moved everything to rrd4j. Is there a neat way of purging old records from InfluxDB?

thanks

Dan


(Christoph Wempe) #5

I think you could write a script and use the DELETE-SQL command to delete entries older than X.

But I cannot imagine database size will be a problem.
Then again, I have no experience in this topic. :slight_smile:


(Rich Koshak) #6

You can configure a retention policy that will automatically purge data as it ages.

Also, InfluxDB does a MUCH better job of handling large databases. And in practice I see between 1-6 mb per Item per year which isn’t that big of a deal size wise.

And, unlike rrd4j, InfluxDB had a nice way to interact and delete data as needed through it’s rest API.

And records can be deleted through OH’s rest API also.

Setting up a retention policy and a discussion of database sizes are included in the tutorial link Christoph provided.


(Dan) #7

Awesome - thank you!


(Christoph Wempe) #8

It looks like the script is not working with openhab 2.1. :disappointed:

The Rest interface does now return a single line of data rather than multiple lines.

I will look into it tomorrow. :sleeping:


(Christoph Wempe) #9

The latest version of the script supports switch items.
The previous one did not, because ON/OFF had to be translatet to 1/0.

I also created a branch for openhab 2.1 compatibility.

Unfortunately my RaspberryPi is really overwhelmed by the script to testing it is not as easy as it could be. :slightly_frowning_face:

I will not spent time to create a script that is compatible with 2.0 and 2.1!
This should be easy, but I don’t have a OH2.0 system to test it anymore.


(SmartJanitor) #10

Great CWempe! Thanks for the contribution and the update for oh 2.1. Will certainly use this as a) I enjoy grafana/influx very much and b) I am a data geek. But first I need to get some other stuff going in openhab.


(Erica Bithell) #11

Thanks a lot for the script - with its help I’ve managed to migrate all my data from rrd4j to InfluxDB in OH 2.1. A couple of small changes make the script work as expected:

  1. All of the date formats need to be changed to:
    tenyearsago=`date +"%Y-%m-%dT%H:%M:%S.%3NZ" --date="10 years ago"
    And so on for the others. Note the upper case %3NZ instead of %:z at the end. This might not be the only way to get the formatting working, but it works for me.

  2. Also there is a typo in the curl line for the period one week through to one day ($onwdayago should be $onedayago)

The script output includes pairs of lines with a 100 Continue response followed by 204 No Content, but this doesn’t seem to be important. As far as I can tell, the 204 response is just what you get at the end of each file.


(Daniel Elliott) #12

Reviving an old thread to ask if there has been support added for 2.2 or 2.3? Or does the 2.1 version work on those versions as well? I’m on v2.3 and looking to make the jump to Influx.

Thanks,

Danny


(Rich Koshak) #13

The REST API hasn’t changed as far as I know so it should still work.