OH3 InfluxDB2

I have two OX3 servers, one for the Windows 10 hypervisor, the second on a separate nettop. InfluxDB is the default on both databases. Both have the same problem that I wrote about here.
And also, only on the nettop there is another problem:
When loading OH3, data is written to the database for several hours, after which the following errors begin to appear in the log:

2020-11-26 20:18:51.777 [ERROR] [.client.write.events.WriteErrorEvent] - The error occurred during writing of data
com.influxdb.exceptions.UnauthorizedException: unauthorized access
	at com.influxdb.internal.AbstractRestClient.responseToError(AbstractRestClient.java:98) ~[bundleFile:?]
	at com.influxdb.client.internal.AbstractWriteClient.toInfluxException(AbstractWriteClient.java:574) [bundleFile:?]
	at com.influxdb.client.internal.AbstractWriteClient.lambda$new$12(AbstractWriteClient.java:181) [bundleFile:?]
	at io.reactivex.internal.subscribers.LambdaSubscriber.onNext(LambdaSubscriber.java:65) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableDoFinally$DoFinallySubscriber.onNext(FlowableDoFinally.java:84) [bundleFile:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapMaybe$ConcatMapMaybeSubscriber.drain(FlowableConcatMapMaybe.java:284) [bundleFile:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapMaybe$ConcatMapMaybeSubscriber.onNext(FlowableConcatMapMaybe.java:137) [bundleFile:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapSingle$ConcatMapSingleSubscriber.drain(FlowableConcatMapSingle.java:279) [bundleFile:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapSingle$ConcatMapSingleSubscriber.innerSuccess(FlowableConcatMapSingle.java:179) [bundleFile:?]
	at io.reactivex.internal.operators.mixed.FlowableConcatMapSingle$ConcatMapSingleSubscriber$ConcatMapSingleObserver.onSuccess(FlowableConcatMapSingle.java:317) [bundleFile:?]
	at io.reactivex.internal.operators.single.SingleMap$MapSingleObserver.onSuccess(SingleMap.java:64) [bundleFile:?]
	at io.reactivex.internal.operators.single.SingleMap$MapSingleObserver.onSuccess(SingleMap.java:64) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableCollectSingle$CollectSubscriber.onComplete(FlowableCollectSingle.java:119) [bundleFile:?]
	at io.reactivex.internal.subscribers.BasicFuseableSubscriber.onComplete(BasicFuseableSubscriber.java:120) [bundleFile:?]
	at io.reactivex.internal.subscribers.BasicFuseableConditionalSubscriber.onComplete(BasicFuseableConditionalSubscriber.java:119) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableGroupBy$State.checkTerminated(FlowableGroupBy.java:686) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableGroupBy$State.drainNormal(FlowableGroupBy.java:626) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableGroupBy$State.drain(FlowableGroupBy.java:558) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableGroupBy$State.onComplete(FlowableGroupBy.java:548) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableGroupBy$GroupedUnicast.onComplete(FlowableGroupBy.java:474) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableGroupBy$GroupBySubscriber.onComplete(FlowableGroupBy.java:213) [bundleFile:?]
	at io.reactivex.processors.UnicastProcessor.checkTerminated(UnicastProcessor.java:430) [bundleFile:?]
	at io.reactivex.processors.UnicastProcessor.drainRegular(UnicastProcessor.java:314) [bundleFile:?]
	at io.reactivex.processors.UnicastProcessor.drain(UnicastProcessor.java:397) [bundleFile:?]
	at io.reactivex.processors.UnicastProcessor.onComplete(UnicastProcessor.java:487) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableWindowBoundarySupplier$WindowBoundaryMainSubscriber.drain(FlowableWindowBoundarySupplier.java:254) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableWindowBoundarySupplier$WindowBoundaryMainSubscriber.innerNext(FlowableWindowBoundarySupplier.java:166) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableWindowBoundarySupplier$WindowBoundaryInnerSubscriber.onNext(FlowableWindowBoundarySupplier.java:316) [bundleFile:?]
	at io.reactivex.processors.PublishProcessor$PublishSubscription.onNext(PublishProcessor.java:360) [bundleFile:?]
	at io.reactivex.processors.PublishProcessor.onNext(PublishProcessor.java:243) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.tryEmit(FlowableFlatMap.java:282) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableFlatMap$InnerSubscriber.onNext(FlowableFlatMap.java:668) [bundleFile:?]
	at io.reactivex.subscribers.SerializedSubscriber.onNext(SerializedSubscriber.java:100) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableWindowTimed$WindowExactBoundedSubscriber.drainLoop(FlowableWindowTimed.java:498) [bundleFile:?]
	at io.reactivex.internal.operators.flowable.FlowableWindowTimed$WindowExactBoundedSubscriber$ConsumerIndexHolder.run(FlowableWindowTimed.java:579) [bundleFile:?]
	at io.reactivex.Scheduler$Worker$PeriodicTask.run(Scheduler.java:479) [bundleFile:?]
	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66) [bundleFile:?]
	at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57) [bundleFile:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

After a reboot, the work is restored again, for an indefinite time, after which these errors appear again and there is no access to the database until the next reboot of OH3.
Both systems ubuntu 20.04.1
OH3 Build # 2030 installed via apt.

service influxdb status:

â—Ź influxdb2.service - InfluxDB 2.0 service file.
     Loaded: loaded (/lib/systemd/system/influxdb2.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2020-11-28 18:47:35 +08; 16h ago
       Docs: https://v2.docs.influxdata.com/v2.0/get-started/
   Main PID: 598 (influxd)
      Tasks: 11 (limit: 2270)
     Memory: 109.3M
     CGroup: /system.slice/influxdb2.service
             └─598 /usr/local/bin/influxd

Nov 29 10:45:04 home-oh influxd[598]: ts=2020-11-29T02:45:04.132024Z lvl=info msg=Unauthorized log_id=0QkcLIKl000 error="session not found"
Nov 29 10:48:32 home-oh influxd[598]: ts=2020-11-29T02:48:32.268483Z lvl=info msg="Retention policy deletion check (start)" log_id=0QkcLIKl000 >
Nov 29 10:48:32 home-oh influxd[598]: ts=2020-11-29T02:48:32.268669Z lvl=info msg="Retention policy deletion check (end)" log_id=0QkcLIKl000 se>
Nov 29 10:49:04 home-oh influxd[598]: ts=2020-11-29T02:49:04.132333Z lvl=info msg=Unauthorized log_id=0QkcLIKl000 error="session not found"
Nov 29 10:52:04 home-oh influxd[598]: ts=2020-11-29T02:52:04.132097Z lvl=info msg=Unauthorized log_id=0QkcLIKl000 error="session not found"
Nov 29 10:54:04 home-oh influxd[598]: ts=2020-11-29T02:54:04.131576Z lvl=info msg=Unauthorized log_id=0QkcLIKl000 error="session not found"
Nov 29 10:56:04 home-oh influxd[598]: ts=2020-11-29T02:56:04.132089Z lvl=info msg=Unauthorized log_id=0QkcLIKl000 error="session not found"
Nov 29 11:04:04 home-oh influxd[598]: ts=2020-11-29T03:04:04.132747Z lvl=info msg=Unauthorized log_id=0QkcLIKl000 error="session not found"
Nov 29 11:05:04 home-oh influxd[598]: ts=2020-11-29T03:05:04.131296Z lvl=info msg=Unauthorized log_id=0QkcLIKl000 error="session not found"
Nov 29 11:06:04 home-oh influxd[598]: ts=2020-11-29T03:06:04.131965Z lvl=info msg=Unauthorized log_id=0QkcLIKl000 error="session not found"

What could be the problem? Has anyone else faced this problem?

I would like to bump this issue up, as I am facing the same problem. Different deployment as both OpenHab 3.0.2 and InfluxDB 2 are “dockerized” separately and running on Synology, however same behaviour. After several days/weeks the persistence stops working with the same errors as @Olymp described.
I don’t have to restart the OH3, as just clicking “Save” in OH3 “Settings->InfluxDB Persistence Service” restores the service.
It looks like there is an issue with some error handling and reconnect to InfluxDB that the connection/session is being lost for some reason.

I would apprecite some insights or comments on this issue.

@Olymp - Were you able to solve this issue eventually?

Regards

No, I could not solve the problem, I had other problems that I also could not solve, for example, influx + habpanel. I created several topics here on the forum, on github, but without success. I had to give up Influx altogether.

@Olymp - thank you for quick reply. I will wait for somebody else to answer - perhaps there is a solution to this issue as I think many OH3 users are using InfluxDB anyway.

BTW, so what other persistence did you go with eventually?

RRD4J

I’m running OH3 (M5) and influxdb2 on a debian machine (no containers) without any issues. Do you use a fresh influxdb install or have you migrated from v1? How have you configured authentication (username/password or token)?

I’m not an expert on this, but if we can figure out the differences between our setups we might find where the issue lies.

@pacive - I have all newly installed OH3 and InfluxDB 2 - so I have not been migrating/upgrading and all is running latest versions. Everything is configured properly - the reason I am saying this is that persistence works in general however from time to time (after several days or even weeks) there are errors like @Olymp reported - looks like the session/connectivity from OH to InfluxDB expired/got lost for some reason.
Going to OH3 influx DB setup tab and just clicking “Save” (without any changes) restores connectivity immediately … till next situation.

It clearly looks like some reconnect issue on OH side

I’m not sure how the influxdb rest API works but I find it strange that it should be session based. Generally in a rest API authentication is made on a request-by-request basis. That’s why I asked if you’re using username/password or token for authentication. Since it’s a simple request-response loop there shouldn’t be any problems due to lost connections either (except for perhaps losing the a batch of updates), each update to the database makes a new connection. Of course provided that influxdb doesn’t completely misuse the terminology.

@pacive - I am using username/pwd to authenticate. I understand your point, and perhaps the error log could be misleading and/or meaning different thing. Below is the example from my logs what I am seeing.

ts=2021-06-10T01:17:00.758908Z lvl=info msg=Unauthorized log_id=0UPC9ndG000 error="session not found"

After reading up a bit it seems that when using username/password it needs to exchange this for a (session-)token to make the actual write requests. It seems that OH fails to renew this token sometimes. Raising a github issue would be in order here I believe. But using a token to authenticate should work in the meantime. These can be easily created through the influxdb web interface (http://:8086)

@pacive - thank you for insights/help, I have changed this to token based authentication and will keep watching this for any issues there - just fyi as far as I remember token can be obtained only from influx CLI.

EDIT: It actually looks there is already open issue regarding the same problem and token way solved it,
and I should have also scanned git reported issues before too

No, the web interface is actually really convenient!


Just remember to grant openhab both read and write access.

@pacive - I missed that my bad :blush: … of course you are right. Anyway I have done it via CLI and so far so good everything works.

1 Like

Just wanted to confirm here after over a month by now, all works without any disruptions and problems. So switching from user/pwd to token made the trick. Thank you @pacive one more time for help!