Charts - long loading times

Setup, Configuration and Use Runtime
,
1

Hi!

I’m experiencing very long loading times for charts when the number of points grow.

I log temperature every 10 min, and viewing 1 month takes about 10 seconds. 6 months appears to never finish (perhaps a timeout somewhere?).

Is this to be expected on a raspberry pi 4?

I’m considering moving to graphana to see if it’s faster, but I’d prefer to stay with the native graphing since it’s quite convenient.

Any input is appreciated!

  • Platform information:
    • Hardware: Raspberry pi 4, 4GB
    • OS: openhabian
    • openHAB version: 3.3.0.M3
    • Persistence: Influxdb
2

You might want to disclose the persistence service used, where that lives (presumably on the same pi), and what other work it is doing in parallel.

3

Ah, good point. I’m using Influxdb running locally on the pi. I have 9 temperatures and 6 humidities that I log every 10 min. I have a few rules, but nothing I would consider heavy, and not relating to the persistence.

I have not installed anything else on the pi that should slow it down. When I look at processor usage during the opening of a chart, one processor works at 100 percent, but the others seem free.

4

Another thing I could mention is that each update of the temperature and humidity is updated tons of times with the same value. I counted around 75 updates within less than a second:

image
image1011×497 136 KB

I’m using the tellstick binding for this (over the cloud, Telldus Live), and my assumption has been that the devices simply send the same value over and over to ensure at least one reaches the bridge.

I don’t think it’s related, since these bursts happen with minutes in between, and the long loading times of the charts are very consistent.

5

Going to need to be very careful about what that is causing. First guess, you have default rrd4j alongside your influxdb, with default “persist everything”, and its trying to deal with.

But it’s going to be hammering all parts of openHAB, needs containing.

6

Can you also please show us your influxdb-configuration file?

7

Ok, so maybe this symptom is relevant after all… I don’t have rrd4j installed, so that should not be it. Anything I can do to dig further into this? Maybe enable some more logging?

8

Here are the contents of any relevant influxdb config file I could find:

influxdb.cfg:

url=http://localhost:8086
user=openhab
password=******
db=openhab
retentionPolicy=autogen

influxdb.persist (all persisted items are part of the gPersist group):

Strategies {
    everyMinute : "0 * * * * ?"
    everyHour   : "0 0 * * * ?"
    everyDay    : "0 0 0 * * ?"

    default = everyChange
}

Items {
    //gHeatAct*, gHeatSet*, gHeatValve*   : strategy = everyChange, everyHour
    //Presence_Phone                      : strategy = everyChange
    //gClimate* : strategy = everyUpdate
    gPersist* : strategy = everyUpdate
    // krypgrund_temp : strategy = everyUpdate
    // heat_pwr : strategy = everyUpdate
    // Livingroom_Temperature : strategy = everyUpdate
    // Hallway_Temperature : strategy = everyUpdate
    // Outdoor_Temperature : strategy = everyUpdate
}

influxdb.conf (at /etc/influxdb, all commented lines removed):

[meta]
  dir = "/var/lib/influxdb/meta"
[data]
  dir = "/var/lib/influxdb/data"
  wal-dir = "/var/lib/influxdb/wal"
  series-id-set-cache-size = 100
[coordinator]
[retention]
[shard-precreation]
[monitor]
  store-enabled = false
[http]
  enabled = true
  bind-address = "localhost:8086"
  auth-enabled = true
[logging]
[subscriber]
[[graphite]]
[[collectd]]
[[opentsdb]]
[[udp]]
[continuous_queries]
[tls]
9

The problem is here. You are persisting way too may measurements.
I don’t know what the tellstick binding does…
For a fast fix you could change the strategy to “everyChange” or log it in a fixed interval like every 5 minutes.
From that point your charts will load a lot faster.

10

Of course! :man_facepalming:All 75 updates are saved every 10 min. I didn’t see it on the chart because they are so close to each other. Thanks a lot for pointing it out!

I prefer to keep the everyUpdate setting if I can though, since I want information in the graph if something was actually sent from the device. If I go with everyChange, I cannot differentiate between it being the same temperature for a long time, and the device silently stopping to send updates.

It seems I need to solve the problem with the Telldus binding… Any ideas? Or perhaps I should create a new post?

Thanks in any case!

11

No it is not every 10 min. It is every second. I don’t know where you have the 10 min from.
You should really change the strategy. You can also put seconds if you like.

This i don’t know. Yes open a new topic where you describe your problem separate.

12

I thought the everyUpdate setting saves the state of the item every time it updates, rather than at fixed time intervalls? This would explain my issue since the temp and humidity values are sent every 10 min from the sensors, but for some reason the update is duplicated 75 times. Where does the every second update come from?

13

I created a new topic:

14

Ah ok, I misunderstood your analysis from the log. I was thinking this is permanently.
So you are right with your explanation for the strategy.

1 Like
15

There’s a better solution than changing the sampling rate and changing the data resolution: you can use continuous queries in influx.

They allow you to periodically pre-aggregate data for faster loading times.

For example, I am calculating and persisting energy consumption every minute and charting energy consumption per month and per year. For monthly consumption this means 60x24x30, i.e. 43,200 data points for one month, and 518,400 data points for annual charts.

Chart loading time on a Pi 5 with influx on a SSD is thus between 10 an 15 seconds.

When you use a continuous query instead, where you calculate daily aggregates once a day, you end up with 30 data points per month and 365 data points per year, thus no delay in loading times.