Influx db error: Re-queuing x elements, failed to write batch, cache-max-memory-size exceeded

stefan.hoehn · September 15, 2023, 7:41pm

I am on openHAB 4.1.0.M1

Since a few days I am getting the following error:

[DEBUG] [rnal.influx1.InfluxDB1RepositoryImpl] - Writing to database failed
org.influxdb.InfluxDBException$CacheMaxMemorySizeExceededException: engine: cache-max-memory-size exceeded: (1074461556/1073741824)
	at org.influxdb.InfluxDBException.buildExceptionFromErrorMessage(InfluxDBException.java:153) ~[?:?]

....

[WARN ] [.influxdb.InfluxDBPersistenceService] - Re-queuing 2118 elements, failed to write batch.

The first time I encountered this. I stopped influx (stopped, started zram) and restarted openHAB and didn’t have an error … for some time.

No it has reappeared. I read in a German forum a post of August that there was a type issue but that wasn’t available in a snapshot by then but I cannot find anything.

Now I stopped influxdb and then tried to restart and it doesn’t even start again:

Sep 15 21:43:53 openhabian systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
░░ Subject: A stop job for unit influxdb.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A stop job for unit influxdb.service has finished.
░░
░░ The job identifier is 25587 and the job result is done.
Sep 15 21:43:53 openhabian systemd[1]: influxdb.service: Start request repeated too quickly.
Sep 15 21:43:53 openhabian systemd[1]: influxdb.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit influxdb.service has entered the 'failed' state with result 'exit-code'.
Sep 15 21:43:53 openhabian systemd[1]: Failed to start InfluxDB is an open-source, distributed, time series database.
░░ Subject: A start job for unit influxdb.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit influxdb.service has finished with a failure.
░░
░░ The job identifier is 25587 and the job result is failed.

I have stopped OH and rebooted and it seems it has recovered but I doubt it will last for long.

Does anyone know what the root cause might be for ?

cache-max-memory-size exceeded: (1074461556/1073741824)

J-N-K · September 16, 2023, 4:57pm

It might be that there is a limit to the size of a write (i.e. some sort of “maximum batch size”). The question is why there are 2118 elements that need to be written to the database, that sound awfully much. How many items do you persist?

stefan.hoehn · September 16, 2023, 5:13pm

I have 603 items and to be honest, I haven’t yet filtered it which isn’t probably a good idea

I can at least filter the weather data but other than that I have to work harder on that to only persist the really needed ones.

I also think I should lower the amount of savings:

Strategies {
	everyMinute : "0 * * * * ?"
        default = everyChange, restoreOnStartup
}

Items {
	SysteminfoRP4_Used, SysteminfoRP4_UsedMemory, SysteminfoRP4_CPUTemperature, SysteminfoRP4_Available : strategy = everyHour
	* : strategy = everyUpdate, everyMinute
}

to

Strategies {
	everyMinute : "0 0/1 * * * ? *"
	everyHour : "0 0 * * * ?"
    default = everyChange, restoreOnStartup
}

Items {
	SysteminfoRP4_Used, SysteminfoRP4_UsedMemory, SysteminfoRP4_CPUTemperature, SysteminfoRP4_Available : strategy = everyHour
	OneCallAPIWeather* = everyHour
	* : strategy = everyUpdate, everyMinute
}

J-N-K · September 16, 2023, 5:24pm

No it should work. But even with 600 items, 2100 datapoints to persist is 3 points per item, which sounds much, because the add-on tries to commit with an interval of 3s.

So the question is: why are there so many datapoints? Is there any error in the log before? Or in the influxdb log at that time?

stefan.hoehn · September 16, 2023, 5:30pm

Now the problem has changed

org.influxdb.InfluxDBException: engine: error rolling WAL segment: error opening new segment file for wal (2): close /var/lib/influxdb/wal/openhab/autogen/2/_00025.wal: file already closed

which points to what was discussed here:

which though looks good to me:

[19:32:22] root@openhabian:/var/lib/influxdb/wal/openhab/autogen/2# ls -l
total 10492
-rw-r--r-- 1 influxdb influxdb 10741791 Sep 16 19:22 _00025.wal