InfluxDB+Grafana persistence and graphing

My knowledge is the following:

  • All items can be defined by the * operator with a strategy
  • A Single item can be defined with a strategy
  • A Item-list can be defined with a strategy
  • A Group* with all items in that group can be defined with a strategy

I would have to read the docs again to know if things* can be defined too. (Maybe you should read them as well :wink: )

I will dive back into the docs.
The influx docs donā€™t address OH .persist directly.
I am willing to put in the time and will keep plodding along.

Again Thank you very much and I hope God The Father shows favor to you and yours

Please be aware of the following things:

  1. The configuration file for InfluxDB Database is influxdb.cfg, not Influxdb.cfg or Influxdb.persist
  2. You have to install the influxdb persistence service. This will result in the correct configuration file, mentioned in point 1.
  3. The persistence configuration is independent from the persistence itself, therefor there is no need for explanation in https://www.openhab.org/addons/persistence/influxdb/. Please take a look at https://www.openhab.org/docs/configuration/persistence.html for this part of configuration.
  4. The name of this file must be named influxdb.persist, as there is no other link to the persistence service, other than the file name.
  5. Obviously, you have to install InfluxDB1.x Database Server to use it. :wink:

just got all up and working it was some strech but now i have Influx for all my items

what is you opinion for a home with 300+ items that is saving all to influx once an hour and on every change, can this be an issue to system preformance?

its not that i am lazy to tag my itemsā€¦ some of them or created with paper UI and cannot be tagged with group item ā€¦(not sure why its like this )
but if found it useful to have all items show up in Influx\Grafna

just a litle worried with preformance
but to tell the turth looking at system moniter with Proxmox tools(Web GUI)
i dont see any spikes when influx is doing the once an hour update for all items
so i think i am ok

Not likely. It could be a lot of needless data though so I would only save those Items for which you actually care about itā€™s past states.

Not true.

not sure if you can see but i am not able to change anythiong there
it is grayed out

maybe its the link modeā€¦
but at this point i am not going to change it

You have Simple Mode turned on? If so thatā€™s why. Turn off simple Mode. Itā€™s a terrible option and should never be enabled.

1 Like

i know nowā€¦ but i already did my enitre setup with paper UI this way with no clue to what i was doing at that timeā€¦
and it works ā€¦ so lets keep it that way :slight_smile:

as i said i did * for all items and i dont see my VM going crazy or even streching for that so i am ok with that for now , but thanks for the heads up, i saw you say it in some other topics and i will not configr my home that way for next time

For anyone who comes across this, Iā€™ve posted the new way to render png image charts from Grafana as of version 6.4.x.

2 Likes

did you found a solution for your problem? facing the exact same issue. Only read here that there are problems with " * The timestamps cannot be calculated correctly on macOS".

Not sure whether it is working or not when working with the terminal at a macā€¦

@christoph_wempe any idea how I can fix the issue with the timestamp problem using mac terminal to execute the script?

Sorry.

I havenā€™t done anything related to this topic in the last two years and I donā€™t know if the new openhab version behaving any different than before.
And I donā€™t know anything about MacOS.

@Woogi: I donĀ“t use the script but IĀ“ve a mac - please write me the problem including the needed output an I will have a look at it. I canĀ“t promise anything but I will try.

Hi @controlc,

thank you.

Problem:
I run openhabian on a raspberry and access the system by a terminal session from my mac. I wanted to transfer my data from rrd4j to influxDB with the script. e.g. for item Klima_hum_wz

### timestamps
item: Klima_hum_wz
10y:  2009-11-01T14:36:44+01:00
1y:   2018-10-04T15:36:44+02:00
1m:   2019-10-03T15:36:44+02:00
1w:   2019-10-25T15:37:44+02:00
1d:   2019-10-31T14:37:44+01:00
8h:   2019-11-01T06:37:44+01:00
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   338  100   338    0     0   2624      0 --:--:-- --:--:-- --:--:--  2640
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   338  100   338    0     0  14382      0 --:--:-- --:--:-- --:--:-- 14695
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   338  100   338    0     0  16197      0 --:--:-- --:--:-- --:--:-- 16900
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   338  100   338    0     0  15557      0 --:--:-- --:--:-- --:--:-- 16095
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   338  100   338    0     0  14368      0 --:--:-- --:--:-- --:--:-- 14695
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   338  100   338    0     0  17425      0 --:--:-- --:--:-- --:--:-- 18777

### found values: 0
Warning: Couldn't read data from file "Klima_hum_wz-*", this makes an empty 
Warning: POST.
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: a57b457c-fcac-11e9-b4c9-b827eb33e913
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.7.8
X-Request-Id: a57b457c-fcac-11e9-b4c9-b827eb33e913
Date: Fri, 01 Nov 2019 13:36:45 GMT

Sleep for 5 seconds to let InfluxDB process the data...

### delete temporary files

So no values that have been transferred.

when browsing to the file http://IP/rest/persistence/items/Klima_hum_wz?serviceId=rrd4j I get a page with all the datapoint, see the extract here:

{"name":"Klima_hum_wz","datapoints":"360","data":[{"time":1570993680000,"state":"59.325"},
{"time":1570993920000,"state":"60.0"},{"time":1570994160000,"state":"59.95"},
{"time":1570994400000,"state":"59.8"},{"time":1570994640000,"state":"59.8"},
{"time":1570994880000,"state":"59.125"},{"time":1570995120000,"state":"58.9"},
{"time":1570995360000,"state":"58.925"},{"time":1570995600000,"state":"59.0"},
{"time":1570995840000,"state":"59.0"},{"time":1570996080000,"state":"58.925000000000004"},
{"time":1570996320000,"state":"58.9"},{"time":1570996560000,"state":"58.824999999999996"},
{"time":1570996800000,"state":"58.6"},{"time":1570997040000,"state":"58.6"},
{"time":1570997280000,"state":"58.75"},{"time":1570997520000,"state":"58.8"},

Why does the script does not take and transfer them? the script I use:

#!/bin/bash
# This script reads the values of an item from openhab via REST and imports the data to influxdb
# useage: get_item_states.sh <itemname>


itemname="$1"

if [ -z $itemname ]
then
  echo "Please define Item!"
  exit 0
fi

source /srv/openhab2-conf/services/rest2influx.cfg

# convert historical times to unix timestamps,
tenyearsago=`date +"%Y-%m-%dT%H:%M:%S%:z" --date="10 years ago"`
oneyearago=`date +"%Y-%m-%dT%H:%M:%S%:z" --date="-12 months 28 days ago"`
onemonthago=`date +"%Y-%m-%dT%H:%M:%S%:z" --date="29 days ago"`
oneweekago=`date +"%Y-%m-%dT%H:%M:%S%:z" --date="-6 days -23 hours 59 minutes ago"`
onedayago=`date +"%Y-%m-%dT%H:%M:%S%:z" --date="-23 hours 59 minutes ago"`
eighthoursago=`date +"%Y-%m-%dT%H:%M:%S%:z" --date="-7 hours 59 minutes ago"`


# print timestamps
echo ""
echo "### timestamps"
echo "item: $itemname"
echo "10y:  $tenyearsago"
echo "1y:   $oneyearago"
echo "1m:   $onemonthago"
echo "1w:   $oneweekago"
echo "1d:   $onedayago"
echo "8h:   $eighthoursago"

resturl="http://$openhabserver:$openhabport/rest/persistence/items/$itemname?serviceId=$serviceid"

# get values and write to different files
# curl -X GET --header "Accept: application/json" "$resturl&starttime=${tenyearsago}&endtime=${oneyearago}"  > ${itemname}_10y.xml
curl -X GET --header "Accept: application/json" "$resturl&starttime=${tenyearsago}&endtime=${oneyearago}"  > ${itemname}_10y.xml
curl -X GET --header "Accept: application/json" "$resturl&starttime=${oneyearago}&endtime=${onemonthago}"  > ${itemname}_1y.xml
curl -X GET --header "Accept: application/json" "$resturl&starttime=${onemonthago}&endtime=${oneweekago}"  > ${itemname}_1m.xml
curl -X GET --header "Accept: application/json" "$resturl&starttime=${oneweekago}&endtime=${onedayago}"    > ${itemname}_1w.xml
curl -X GET --header "Accept: application/json" "$resturl&starttime=${onedayago}&endtime=${eighthoursago}" > ${itemname}_1d.xml
curl -X GET --header "Accept: application/json" "$resturl&starttime=${eighthoursago}"                      > ${itemname}_8h.xml

# combine files
cat ${itemname}_10y.xml ${itemname}_1y.xml ${itemname}_1m.xml ${itemname}_1w.xml ${itemname}_1d.xml ${itemname}_8h.xml > ${itemname}.xml

# convert data to line protocol file
cat ${itemname}.xml \
     | sed 's/}/\n/g' \
     | sed 's/data/\n/g' \
     | grep -e "time.*state"\
     | tr -d ',:[{"' \
     | sed 's/time/ /g;s/state/ /g' \
     | awk -v item="$itemname" '{print item " value=" $2 " " $1 "000000"}' \
     | sed 's/value=ON/value=1/g;s/value=OFF/value=0/g' \
> ${itemname}.txt

values=`wc -l ${itemname}.txt | cut -d " " -f 1`
echo ""
echo "### found values: $values"


# split file in smaller parts to make it easier for influxdb
split -l $importsize ${itemname}.txt "${itemname}-"

for i in ${itemname}-*
do
  curl -i -XPOST -u $influxuser:$influxpw "http://$influxserver:$influxport/write?db=$influxdatbase" --data-binary @$i
  echo "Sleep for $sleeptime seconds to let InfluxDB process the data..."
  sleep $sleeptime
done

echo ""
echo "### delete temporary files"
rm ${itemname}*

exit 0

Hi @Woogi,

Well, now itĀ“s clear: OSX is using BSD date and not GNU date and the syntax of both commands are different. YouĀ“ve two options: Install gnu coreutils via brew (and replace ā€œdateā€ in the script with ā€œgdateā€) on your mac or adopt the script to BSD date syntax. As an example us this website. Hope this will help.

Update: You have to change only the six variables like this:

tenyearsago=`date -v -10y -R +ā€œ%Y-%m-%dT%H:%M:%S:%zā€`
oneyearago=`date -v -12m -v -28d -R +ā€œ%Y-%m-%dT%H:%M:%S:%zā€`
onemonthago=`date -v -29d -R +ā€œ%Y-%m-%dT%H:%M:%S:%zā€`
oneweekago=`date -v -6d -v -23H -v -59M -R +ā€œ%Y-%m-%dT%H:%M:%S:%zā€`
onedayago=`date -v -23H -v -59M -R +ā€œ%Y-%m-%dT%H:%M:%S:%zā€`
eighthoursago=`date -v -7H -v -59M -R +ā€œ%Y-%m-%dT%H:%M:%S:%zā€`

This should work.

Hi @controlc

I tried your hint, but I assume, this wasnā€™t the issue. further ideas?

date: invalid option -- 'v'
Try 'date --help' for more information.
date: invalid option -- 'v'
Try 'date --help' for more information.
date: invalid option -- 'v'
Try 'date --help' for more information.
date: invalid option -- 'v'
Try 'date --help' for more information.
date: invalid option -- 'v'
Try 'date --help' for more information.
date: invalid option -- 'v'
Try 'date --help' for more information.

### timestamps
item: Klima_hum_wz
10y:  
1y:   
1m:   
1w:   
1d:   
8h:   
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   229  100   229    0     0  14429      0 --:--:-- --:--:-- --:--:-- 15266
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   229  100   229    0     0  10986      0 --:--:-- --:--:-- --:--:-- 11450
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   229  100   229    0     0  15074      0 --:--:-- --:--:-- --:--:-- 16357
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   229  100   229    0     0  15272      0 --:--:-- --:--:-- --:--:-- 16357
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   229  100   229    0     0  17833      0 --:--:-- --:--:-- --:--:-- 19083
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   229  100   229    0     0  14906      0 --:--:-- --:--:-- --:--:-- 16357

### found values: 0
Warning: Couldn't read data from file "Klima_hum_wz-*", this makes an empty 
Warning: POST.
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: 078dc04b-fce8-11e9-bede-b827eb33e913
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.7.8
X-Request-Id: 078dc04b-fce8-11e9-bede-b827eb33e913
Date: Fri, 01 Nov 2019 20:41:50 GMT

Sleep for 5 seconds to let InfluxDB process the data...

### delete temporary files

Hi @Woogi,

please try the following date line in your local terminal window:

date -v -10y -R +"%Y-%m-%dT%H:%M:%S:%z"

and lets have a look at man date to study the syntax. If all of it doesnĀ“t work lets have a look at brew or why you donĀ“t execute the script on your Raspberry Pi?

hi @controlc

this is the result after typing the line into local terminal window:

2009-11-01T21:57:17:+0100

what is man date where can I find that?

I am connected to the raspberry by ssh so I thought I would execute it on the raspberry?!

Type man date on the command line and it will bring up the documentation for the date command.

Hi @Woogi,

That means that my conversion of the date command works for you (and your mac).

IĀ“m confused: Do you execute the script at a shell on the raspberry or at a shell on your mac?
Normally the script should run on your local workstation aka mac - you have to edit the config.cfg to setup up the connection details of your raspberry. By executing the script it connects to your raspberry and read/write the informations from rrd4j to influxdb.