InfluxDB Persistence stops permanently when Influx-DB-Server has outage

kose163 · September 9, 2023, 8:17am

Dear community,

I have the following setup:

openhab on a local Raspi (OpenHAB 4.0.2)
InfluxDB on a remote VM (InfluxDB 2.7.1)

The setup used to work stable for over one year with no interruptions. Now, since a few weeks, the persistence service stops working every 1-4 days, and values are not written to influx any longer.OpenHAB itself seems to work normally.

A reboot of the Raspi solves the issue, but I don’t want to create a cronjob to reboot every night. This is not really a clean solution.

The issue is not located in InfluxDB, since Influx still persists values from other sources.

Any idea how to debug and fix the issue? The Raspi itself seems to be ok (enough memory, enough disk space,…)

Thanks!

kose

Wolfgang_S · September 9, 2023, 8:41am

Enable logging on influxdb to check if access from openhab stops at some time or if the access is denied with a specific error message.
Enable debug log for influxdb on openhab to see if that gives a clue for the reason.

sihui · September 9, 2023, 9:08am

Probably related:

github.com/openhab/openhab-addons

[influxdb] Unhandled InfluxDBException (timeout)

opened 10:04AM - 17 Aug 23 UTC

closed 01:09PM - 09 Sep 23 UTC

Marco-Kawon

bug

## Expected Behavior InfluxDB Persistance Service should not crash and recover …from a InfluxDBException (timeout) exception. ## Current Behavior Hey there, for whatever reason I am getting a timeout every few days from my influxdb when openhab wants to write data to it. The issue is that once this error occurs, openHab stops writing to influxdb forever until I restart openhab, even if InfluxDB is working again. It does not recover from that failed state until openhab gets restarted. A very similar issue was already discussed and handled here in this commit: https://github.com/openhab/openhab-addons/commit/d2e10ab2820a987f273aa7ebbc64d575a39644a5 My Problem is that only "InfluxException" and "InfluxDBIOException" are handled there. But I am getting a "InfluxDBException", which is not catched here and then it crashes the persistance service. ``` 2023-08-11 21:04:45.131 [WARN ] [mmon.WrappedScheduledExecutorService] - Scheduled runnable ended with an exception: org.influxdb.InfluxDBException: timeout at org.influxdb.InfluxDBException.buildExceptionFromErrorMessage(InfluxDBException.java:161) ~[?:?] at org.influxdb.InfluxDBException.buildExceptionForErrorState(InfluxDBException.java:173) ~[?:?] at org.influxdb.impl.InfluxDBImpl.execute(InfluxDBImpl.java:837) ~[?:?] at org.influxdb.impl.InfluxDBImpl.write(InfluxDBImpl.java:470) ~[?:?] at org.openhab.persistence.influxdb.internal.influx1.InfluxDB1RepositoryImpl.write(InfluxDB1RepositoryImpl.java:133) ~[?:?] at org.openhab.persistence.influxdb.InfluxDBPersistenceService.commit(InfluxDBPersistenceService.java:285) ~[?:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] ``` ## Possible Solution The "InfluxDBIOException" extends "InfluxDBException" as you can see [here](https://github.com/influxdata/influxdb-java/blob/master/src/main/java/org/influxdb/InfluxDBIOException.java) so I think it should be safe to replace the recently added catch of "InfluxDBIOException" by "InfluxDBException" to catch both of them. InfluxDB's write() method always throws InfluxDBExceptions. Then I would just get a error thrown on console (DEBUG Level) and it would try again later until InfluxDB responds again ... ``` 2023-08-18 15:46:08.829 [DEBUG] [rnal.influx1.InfluxDB1RepositoryImpl] - Writing to database failed org.influxdb.InfluxDBIOException: java.net.SocketTimeoutException: timeout at org.influxdb.impl.InfluxDBImpl.execute(InfluxDBImpl.java:841) ~[?:?] ``` ## Context Has been discussed [here](https://github.com/openhab/openhab-addons/pull/15062) ## Your Environment * openHAB 4.0.2 * Openhabian on rpi4. * InfluxDB 1.8.x

kose163 · September 10, 2023, 10:41am

Hi,

I digged a bit into the logfiles and found only this hint in the openhab.log:

2023-09-08 14:49:01.704 [WARN ] [rite.events.WriteRetriableErrorEvent] - The retriable error occurred during writing of data. Reason: ‘timeout’. Retry in: 5s.
2023-09-08 14:49:16.720 [WARN ] [rite.events.WriteRetriableErrorEvent] - The retriable error occurred during writing of data. Reason: ‘xxx-Influx-Server-xxx.net: Temporary failure in name resolution’. Retry in: 25s.
2023-09-08 14:49:51.736 [WARN ] [rite.events.WriteRetriableErrorEvent] - The retriable error occurred during writing of data. Reason: ‘xxx-Influx-Server-xxx.net: Temporary failure in name resolution’. Retry in: 125s.

I will keep my eyes open when the next interruption occurrs. Is there a possibility to increase number of retries and timeout to avoid a complete stop of the persistence layer?

Wolfgang_S · September 10, 2023, 2:33pm

Based on the error about temporary failure in name resolution this looks like it would be a network problem.
Where is this influxdb server located ? In your local network or via internet remote connection ?
As long as it is a name resolution problem and not really related to network you can try to add the server name and ip address into the local server’s /etc/hosts file.

kose163 · September 10, 2023, 3:43pm

Hi Wolfgang,

the InfluxDB-Server is connected via an internet remote connection. Normally, writing to this server is working very well. The issue is that the persistence service stops completely. It seems that it is not too robust when the InfluxDB Server is not available. Or there is a timeout that thepersistence decides not to persist to this server any longer.
Anyway, I’d like to control this timeout, and the persistence should continue to persist once the server is back again.

kose163 · September 19, 2023, 5:44am

Some additional information, since the persistence stopped working tonight again:

The Influx-Server was down for ca. 15 Minutes.
once the server was back, Telegraf continued working
the OpenHab persistence layer did not recover from the outage

See attached screenshots.

Any idea how to increase the timeout of the persistence layer?

schup011 · September 20, 2023, 10:44am

I can confirm I had something similar after having upgraded to openHAB 4. In my system, the influxDB service is even on the same machine, but for reasons of CPU or memory overload, openHAB could not reach the service for a certain period of time, and it also could only be healed by restarting openHAB, not by restarting the influxDB service. I remember some timeout messages in the openhab log that said it was not able to reach the influxDB service.

Piffer · September 22, 2023, 6:27am

I just experienced the same after doing a Container Update for the InfluxDB on my NAS box.

2023-09-21 08:48:36.687 [WARN ] [rite.events.WriteRetriableErrorEvent] - The retriable error occurred during writing of data. Reason: 'Failed to connect to localhost/127.0.0.1:8086'. Retry in: 5s.
2023-09-21 08:48:41.692 [WARN ] [rite.events.WriteRetriableErrorEvent] - The retriable error occurred during writing of data. Reason: 'Failed to connect to localhost/127.0.0.1:8086'. Retry in: 25s.
2023-09-21 08:49:06.698 [WARN ] [rite.events.WriteRetriableErrorEvent] - The retriable error occurred during writing of data. Reason: 'Failed to connect to localhost/127.0.0.1:8086'. Retry in: 125s.

Nothing more regarding Influx in the logs after that, but no data written until I restarted Openhab4.
I use only InfluxDB as persistence

nagisa · October 15, 2023, 8:46pm

I think I have been seeing similar recovery failures with InfluxDB, except I also have some exceptions to share:

Click that fancy arrow for exception log

2023-10-13 23:19:06.268 [ERROR] [.client.write.events.WriteErrorEvent] - The error occurred during writing of data
com.influxdb.exceptions.InfluxException: interrupted
	at com.influxdb.client.internal.AbstractWriteClient.toInfluxException(AbstractWriteClient.java:546) ~[?:?]
	at com.influxdb.client.internal.AbstractWriteClient.lambda$new$12(AbstractWriteClient.java:178) ~[?:?]
	at io.reactivex.internal.subscribers.LambdaSubscriber.onNext(LambdaSubscriber.java:65) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableDoFinally$DoFinallySubscriber.onNext(FlowableDoFinally.java:84) ~[?:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapMaybe$ConcatMapMaybeSubscriber.drain(FlowableConcatMapMaybe.java:284) ~[?:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapMaybe$ConcatMapMaybeSubscriber.innerSuccess(FlowableConcatMapMaybe.java:179) ~[?:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapMaybe$ConcatMapMaybeSubscriber$ConcatMapMaybeObserver.onSuccess(FlowableConcatMapMaybe.java:322) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeOnErrorNext$OnErrorNextMaybeObserver.onSuccess(MaybeOnErrorNext.java:88) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybePeek$MaybePeekObserver.onSuccess(MaybePeek.java:122) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap$MapMaybeObserver.onSuccess(MaybeMap.java:89) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableSingleMaybe$SingleElementSubscriber.onComplete(FlowableSingleMaybe.java:104) ~[?:?]
	at io.reactivex.subscribers.SerializedSubscriber.onComplete(SerializedSubscriber.java:168) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableRetryWhen$RetryWhenSubscriber.onComplete(FlowableRetryWhen.java:82) ~[?:?]
	at io.reactivex.internal.subscriptions.DeferredScalarSubscription.complete(DeferredScalarSubscription.java:134) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeToFlowable$MaybeToFlowableSubscriber.onSuccess(MaybeToFlowable.java:70) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap$MapMaybeObserver.onSuccess(MaybeMap.java:89) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap$MapMaybeObserver.onSuccess(MaybeMap.java:89) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeFromCallable.subscribeActual(MaybeFromCallable.java:61) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14935) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14882) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenReceiver.onNext(FlowableRepeatWhen.java:100) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.tryEmit(FlowableFlatMap.java:282) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableFlatMap$InnerSubscriber.onNext(FlowableFlatMap.java:668) ~[?:?]
	at io.reactivex.subscribers.SerializedSubscriber.onNext(SerializedSubscriber.java:100) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableDelay$DelaySubscriber$OnNext.run(FlowableDelay.java:114) ~[?:?]
	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66) ~[?:?]
	at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.io.InterruptedIOException: interrupted
	at okio.Timeout.throwIfReached(Timeout.java:146) ~[?:?]
	at okio.Okio$1.write(Okio.java:76) ~[?:?]
	at okio.AsyncTimeout$1.write(AsyncTimeout.java:180) ~[?:?]
	at okio.RealBufferedSink.flush(RealBufferedSink.java:224) ~[?:?]
	at okhttp3.internal.http1.Http1ExchangeCodec.finishRequest(Http1ExchangeCodec.java:190) ~[?:?]
	at okhttp3.internal.connection.Exchange.finishRequest(Exchange.java:101) ~[?:?]
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:86) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:43) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at com.influxdb.client.internal.GzipInterceptor.intercept(GzipInterceptor.java:91) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at com.influxdb.client.internal.AuthenticateInterceptor.intercept(AuthenticateInterceptor.java:97) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:152) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at com.influxdb.internal.UserAgentInterceptor.intercept(UserAgentInterceptor.java:60) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229) ~[?:?]
	at okhttp3.RealCall.execute(RealCall.java:81) ~[?:?]
	at retrofit2.OkHttpCall.execute(OkHttpCall.java:188) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap$MapMaybeObserver.onSuccess(MaybeMap.java:82) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeFromCallable.subscribeActual(MaybeFromCallable.java:61) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14935) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14882) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenReceiver.onNext(FlowableRepeatWhen.java:100) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableRetryWhen.subscribeActual(FlowableRetryWhen.java:62) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14935) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableSingleMaybe.subscribeActual(FlowableSingleMaybe.java:34) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybePeek.subscribeActual(MaybePeek.java:56) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeOnErrorNext.subscribeActual(MaybeOnErrorNext.java:46) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapMaybe$ConcatMapMaybeSubscriber.drain(FlowableConcatMapMaybe.java:276) ~[?:?]
	... 33 more
2023-10-13 23:19:06.275 [ERROR] [.client.write.events.WriteErrorEvent] - The error occurred during writing of data
com.influxdb.exceptions.InfluxException: interrupted
	at com.influxdb.client.internal.AbstractWriteClient.toInfluxException(AbstractWriteClient.java:546) ~[?:?]
	at com.influxdb.client.internal.AbstractWriteClient.lambda$new$12(AbstractWriteClient.java:178) ~[?:?]
	at io.reactivex.internal.subscribers.LambdaSubscriber.onNext(LambdaSubscriber.java:65) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableDoFinally$DoFinallySubscriber.onNext(FlowableDoFinally.java:84) ~[?:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapMaybe$ConcatMapMaybeSubscriber.drain(FlowableConcatMapMaybe.java:284) ~[?:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapMaybe$ConcatMapMaybeSubscriber.innerSuccess(FlowableConcatMapMaybe.java:179) ~[?:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapMaybe$ConcatMapMaybeSubscriber$ConcatMapMaybeObserver.onSuccess(FlowableConcatMapMaybe.java:322) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeOnErrorNext$OnErrorNextMaybeObserver.onSuccess(MaybeOnErrorNext.java:88) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybePeek$MaybePeekObserver.onSuccess(MaybePeek.java:122) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap$MapMaybeObserver.onSuccess(MaybeMap.java:89) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableSingleMaybe$SingleElementSubscriber.onComplete(FlowableSingleMaybe.java:104) ~[?:?]
	at io.reactivex.subscribers.SerializedSubscriber.onComplete(SerializedSubscriber.java:168) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableRetryWhen$RetryWhenSubscriber.onComplete(FlowableRetryWhen.java:82) ~[?:?]
	at io.reactivex.internal.subscriptions.DeferredScalarSubscription.complete(DeferredScalarSubscription.java:134) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeToFlowable$MaybeToFlowableSubscriber.onSuccess(MaybeToFlowable.java:70) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap$MapMaybeObserver.onSuccess(MaybeMap.java:89) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap$MapMaybeObserver.onSuccess(MaybeMap.java:89) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeFromCallable.subscribeActual(MaybeFromCallable.java:61) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14935) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14882) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenReceiver.onNext(FlowableRepeatWhen.java:100) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.tryEmit(FlowableFlatMap.java:282) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableFlatMap$InnerSubscriber.onNext(FlowableFlatMap.java:668) ~[?:?]
	at io.reactivex.subscribers.SerializedSubscriber.onNext(SerializedSubscriber.java:100) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableDelay$DelaySubscriber$OnNext.run(FlowableDelay.java:114) ~[?:?]
	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66) ~[?:?]
	at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.io.InterruptedIOException: interrupted
	at okio.Timeout.throwIfReached(Timeout.java:146) ~[?:?]
	at okio.Okio$1.write(Okio.java:76) ~[?:?]
	at okio.AsyncTimeout$1.write(AsyncTimeout.java:180) ~[?:?]
	at okio.RealBufferedSink.flush(RealBufferedSink.java:224) ~[?:?]
	at okhttp3.internal.http1.Http1ExchangeCodec.finishRequest(Http1ExchangeCodec.java:190) ~[?:?]
	at okhttp3.internal.connection.Exchange.finishRequest(Exchange.java:101) ~[?:?]
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:86) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:43) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at com.influxdb.client.internal.GzipInterceptor.intercept(GzipInterceptor.java:91) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at com.influxdb.client.internal.AuthenticateInterceptor.intercept(AuthenticateInterceptor.java:97) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:152) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at com.influxdb.internal.UserAgentInterceptor.intercept(UserAgentInterceptor.java:60) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229) ~[?:?]
	at okhttp3.RealCall.execute(RealCall.java:81) ~[?:?]
	at retrofit2.OkHttpCall.execute(OkHttpCall.java:188) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap$MapMaybeObserver.onSuccess(MaybeMap.java:82) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeFromCallable.subscribeActual(MaybeFromCallable.java:61) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14935) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14882) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenReceiver.onNext(FlowableRepeatWhen.java:100) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableRetryWhen.subscribeActual(FlowableRetryWhen.java:62) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14935) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableSingleMaybe.subscribeActual(FlowableSingleMaybe.java:34) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybePeek.subscribeActual(MaybePeek.java:56) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeOnErrorNext.subscribeActual(MaybeOnErrorNext.java:46) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapMaybe$ConcatMapMaybeSubscriber.drain(FlowableConcatMapMaybe.java:276) ~[?:?]
	... 33 more
2023-10-13 23:19:06.280 [ERROR] [.client.write.events.WriteErrorEvent] - The error occurred during writing of data
com.influxdb.exceptions.InfluxException: interrupted
	at com.influxdb.client.internal.AbstractWriteClient.toInfluxException(AbstractWriteClient.java:546) ~[?:?]
	at com.influxdb.client.internal.AbstractWriteClient.lambda$new$12(AbstractWriteClient.java:178) ~[?:?]
	at io.reactivex.internal.subscribers.LambdaSubscriber.onNext(LambdaSubscriber.java:65) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableDoFinally$DoFinallySubscriber.onNext(FlowableDoFinally.java:84) ~[?:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapMaybe$ConcatMapMaybeSubscriber.drain(FlowableConcatMapMaybe.java:284) ~[?:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapMaybe$ConcatMapMaybeSubscriber.innerSuccess(FlowableConcatMapMaybe.java:179) ~[?:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapMaybe$ConcatMapMaybeSubscriber$ConcatMapMaybeObserver.onSuccess(FlowableConcatMapMaybe.java:322) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeOnErrorNext$OnErrorNextMaybeObserver.onSuccess(MaybeOnErrorNext.java:88) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybePeek$MaybePeekObserver.onSuccess(MaybePeek.java:122) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap$MapMaybeObserver.onSuccess(MaybeMap.java:89) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableSingleMaybe$SingleElementSubscriber.onComplete(FlowableSingleMaybe.java:104) ~[?:?]
	at io.reactivex.subscribers.SerializedSubscriber.onComplete(SerializedSubscriber.java:168) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableRetryWhen$RetryWhenSubscriber.onComplete(FlowableRetryWhen.java:82) ~[?:?]
	at io.reactivex.internal.subscriptions.DeferredScalarSubscription.complete(DeferredScalarSubscription.java:134) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeToFlowable$MaybeToFlowableSubscriber.onSuccess(MaybeToFlowable.java:70) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap$MapMaybeObserver.onSuccess(MaybeMap.java:89) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap$MapMaybeObserver.onSuccess(MaybeMap.java:89) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeFromCallable.subscribeActual(MaybeFromCallable.java:61) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14935) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14882) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenReceiver.onNext(FlowableRepeatWhen.java:100) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.tryEmit(FlowableFlatMap.java:282) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableFlatMap$InnerSubscriber.onNext(FlowableFlatMap.java:668) ~[?:?]
	at io.reactivex.subscribers.SerializedSubscriber.onNext(SerializedSubscriber.java:100) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableDelay$DelaySubscriber$OnNext.run(FlowableDelay.java:114) ~[?:?]
	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66) ~[?:?]
	at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.io.InterruptedIOException: interrupted
	at okio.Timeout.throwIfReached(Timeout.java:146) ~[?:?]
	at okio.Okio$1.write(Okio.java:76) ~[?:?]
	at okio.AsyncTimeout$1.write(AsyncTimeout.java:180) ~[?:?]
	at okio.RealBufferedSink.flush(RealBufferedSink.java:224) ~[?:?]
	at okhttp3.internal.http1.Http1ExchangeCodec.finishRequest(Http1ExchangeCodec.java:190) ~[?:?]
	at okhttp3.internal.connection.Exchange.finishRequest(Exchange.java:101) ~[?:?]
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:86) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:43) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at com.influxdb.client.internal.GzipInterceptor.intercept(GzipInterceptor.java:91) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at com.influxdb.client.internal.AuthenticateInterceptor.intercept(AuthenticateInterceptor.java:97) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:152) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at com.influxdb.internal.UserAgentInterceptor.intercept(UserAgentInterceptor.java:60) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[?:?]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229) ~[?:?]
	at okhttp3.RealCall.execute(RealCall.java:81) ~[?:?]
	at retrofit2.OkHttpCall.execute(OkHttpCall.java:188) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap$MapMaybeObserver.onSuccess(MaybeMap.java:82) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeFromCallable.subscribeActual(MaybeFromCallable.java:61) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14935) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14882) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenReceiver.onNext(FlowableRepeatWhen.java:100) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableRetryWhen.subscribeActual(FlowableRetryWhen.java:62) ~[?:?]
	at io.reactivex.Flowable.subscribe(Flowable.java:14935) ~[?:?]
	at io.reactivex.internal.operators.flowable.FlowableSingleMaybe.subscribeActual(FlowableSingleMaybe.java:34) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybePeek.subscribeActual(MaybePeek.java:56) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.maybe.MaybeOnErrorNext.subscribeActual(MaybeOnErrorNext.java:46) ~[?:?]
	at io.reactivex.Maybe.subscribe(Maybe.java:4290) ~[?:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapMaybe$ConcatMapMaybeSubscriber.drain(FlowableConcatMapMaybe.java:276) ~[?:?]
	... 33 more

I’ve also have seen some other components that talk to stuff over network (in my case rfc2217 serial ports) stop working in a similar manner occasionally, so this might be a broader issue in OpenHAB as far as I can tell. In my experience running something like bundle:restart org.openhab.persistence.influxdb (or the binding that has failed) makes things work again fine without restarting the openhab whole.

cc @rlkoshak I hate to ping people, but this seems like it could be pointing at a larger issue that could be affecting a non-trivial share of users and you’ve been really helpful before. If nothing else, perhaps you could point us to a correct place to report the issue (openhab-core? openhab-addons?) or connect us to the right people who work on the relevant code? I’m ready to help investigating this (and am technically able to do so with minimal guidance :))

rlkoshak · October 16, 2023, 2:24pm

If the problem is just happening for Influxdb report to the openhab-addons repo. If it’s affecting more than one persistence report to openhab-core.

vselitsky · December 3, 2023, 5:11pm

I have been trying to migrate from OH2.5 to OH4.0.3 with InfluxDB2.7 on Win11 and faced the same problem that it stopped often. At the beginning a restart of OH4 has been helping, but then it stopped working totally. Upgrade to OH4.0.4 as well as downgrade to Influx1.8.10 did not help. At the moment I even cannot make OH4 to connect with Influx with no reason. Does anybody still have the same problem?