Openhab cloud connection stops working, does not try to reconnect

Hey guys, same as above as recently as last night (12th Nov 2022) ping and pongs are fine. Cloud connector offline.disconnect and reconnects in logs, reboot of pi4 fixes the issue.

The problem was a lot more common over the last month and got a lot better with the Debug Binding (3.3) only failed once since its install (last night)

Potential Debug notes:
I did install the the cloud connector through text files (not GUI)

I live in South Africa (pongs are over 200ms) (Stable 1GB Fibre)

There are multiple disconnects and reconnects in log when issue occurs

App works fine (Local)

I dont have any notifications setup

It happened again Nov 18 and Nov 19, same thing as before.

2 Likes

Hi All,

Following this thread for a while as I am experiencing this issue as well. I wanted to inform that I upgraded from OH3.3 to 3.4 last week and encountered the problem again today. A simple “save” on the UI openHAB cloud page got me back online. There were no messages in the openhab.log after the last successful connect to openhabcloud. I have log level set to TRACE now and will send info as soon as I have the issue again. If anything else is helpfull for troubleshooting, please let me know.

Thanks

Hi @joseph @ssalonen

I have also been having this issue intermittantly, it’s been worse since October 2022. I was running 2.5.10 so upgraded to 3.4, but the issue remained. I changed the connection to the Pi from wifi to ethernet in case it was dropping, but that didn’t help.

Today it failed at 09:30 UK time (12-01-2023), apparently due to ping timeout. I’m not sure that it worked correctly after that, it was 14:30 that I noticed it failed as Alex said ‘the hub is not responding’. There is nothing else in the log apart from ping/pong even when I ask Alexa to turn on a light, I would have expected to see something.

Probably I’ll implement one of these scrpts that restarts the connector if it fails, clearly this is a hard bug to fix as has been around for a while.

Did yours also fail at 09:30 today? I would guess it did for everyone, unless it is my internet provider.

Maybe someone could increase the ping timeout threshold to make it less sensitive?

2023-01-12 09:28:12.765 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO ping
2023-01-12 09:28:12.789 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO pong: 24 ms
2023-01-12 09:28:37.790 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO ping
2023-01-12 09:28:37.816 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO pong: 26 ms
2023-01-12 09:29:02.819 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO ping
2023-01-12 09:29:03.342 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO pong: 522 ms
2023-01-12 09:29:28.344 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO ping
2023-01-12 09:29:29.872 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO pong: 1529 ms
2023-01-12 09:29:54.874 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO ping
2023-01-12 09:29:56.085 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO pong: 1212 ms
2023-01-12 09:30:21.091 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO ping
2023-01-12 09:30:26.095 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO CLOSE: ping timeout
2023-01-12 09:30:26.097 [WARN ] [io.openhabcloud.internal.CloudClient] - Socket.IO disconnected: ping timeout
2023-01-12 09:30:26.098 [INFO ] [io.openhabcloud.internal.CloudClient] - Disconnected from the openHAB Cloud service (UUID = 36…5d, base URL = http://localhost:8080)
2023-01-12 09:30:27.161 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connecting (attempt 1)
2023-01-12 09:30:27.164 [TRACE] [io.openhabcloud.internal.CloudClient] - Manager.EVENT_TRANSPORT
2023-01-12 09:30:27.166 [TRACE] [io.openhabcloud.internal.CloudClient] - Transport.EVENT_REQUEST_HEADERS
2023-01-12 09:30:37.337 [DEBUG] [io.openhabcloud.internal.CloudClient] - Error connecting to the openHAB Cloud instance: EngineIOException xhr poll error. Should reconnect automatically.
2023-01-12 09:30:37.341 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connect attempt error: SocketIOException Connection error
2023-01-12 09:30:38.517 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connecting (attempt 2)
.
.
.
2023-01-12 09:32:07.660 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connecting (attempt 8)
2023-01-12 09:32:07.662 [TRACE] [io.openhabcloud.internal.CloudClient] - Manager.EVENT_TRANSPORT
2023-01-12 09:32:07.664 [TRACE] [io.openhabcloud.internal.CloudClient] - Transport.EVENT_REQUEST_HEADERS
2023-01-12 09:32:07.841 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO OPEN
2023-01-12 09:32:07.845 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connected successfully (attempt 8)
2023-01-12 09:32:07.847 [TRACE] [io.openhabcloud.internal.CloudClient] - Manager.EVENT_TRANSPORT
2023-01-12 09:32:07.848 [TRACE] [io.openhabcloud.internal.CloudClient] - Transport.EVENT_REQUEST_HEADERS
2023-01-12 09:32:07.852 [TRACE] [io.openhabcloud.internal.CloudClient] - Transport.EVENT_REQUEST_HEADERS
2023-01-12 09:32:07.931 [TRACE] [io.openhabcloud.internal.CloudClient] - Socket.IO Packet: CONNECT (0)
2023-01-12 09:32:07.934 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO connected
2023-01-12 09:32:07.937 [INFO ] [io.openhabcloud.internal.CloudClient] - Connected to the openHAB Cloud service (UUID = 36…5d, base URL = http://localhost:8080)
2023-01-12 09:32:07.939 [TRACE] [io.openhabcloud.internal.CloudClient] - Transport.EVENT_REQUEST_HEADERS
2023-01-12 09:32:32.853 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO ping
2023-01-12 09:32:32.875 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO pong: 22 ms
2023-01-12 09:32:57.877 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO ping
2023-01-12 09:32:57.899 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO pong: 22 ms
Then nothing but ping/pongs.

Hi Miles,

For me it did not fail yet after I have set the log level to TRACE.’
The only messages I see are the “ping pong” messages.

Thanks
Jos

I have been on vacation. It seems, that my cloud connection also crashed on 11.01 or 12.01

Uh. This ist getting worse.
Two crashes within the last 4 days

Hey everyone,
I’m also having the same issue. Just like momo90 my connection stopped twice in a very short period. I have two instances running on different locations and both stopped communicating with the Cloud at the same time.

Read here for a rule to restart your cloud connector and some discussion about the ongoing problem. It’s a stopgap solution until the root cause can be determined.

1 Like

FYI, there’s maintenance tomorrow (January 29), so it is expected that there will be disconnections.

2 Likes

Yes, see my post in How to automatically restart cloud connector after an unexpected disconnection - #26 by digitaldan

Hopefully this helps with reconnects that get into a zombie state.

3 Likes

@digitaldan First, thank you for all your great work around openHAB.

I don’t want to play the party pooper, but I fear changes due to myopenHAB maintenance broke my openHAB installation that now it does not reconnect any more since January 30th.
By the way - no changes of my openHAB installation by my own since weeks.

I checked all my logs and this are the results of my research:

  • Every night shortly past 3am openHAB Cloud service is disconnected because my router performs an internet reconnect to prevent a provider forced disconnect.
  • After internet reconnect a few seconds later openHAB Cloud service connection was performed automatically – until January 29th.
  • Since January 30th no openHAB Cloud service reconnection was established automatically, so every morning I must restart openHAB manually to get openHAB Cloud service connection working.

See also my logs for details:

2023-01-29 03:24:20.174 [INFO ] [io.openhabcloud.internal.CloudClient] - Disconnected from the openHAB Cloud service (UUID = 2a3780fd-69ad-4e96-b5b8-69ed849xxxxx, base URL = http://localhost:8080)
2023-01-29 03:24:22.075 [INFO ] [io.openhabcloud.internal.CloudClient] - Connected to the openHAB Cloud service (UUID = 2a3780fd-69ad-4e96-b5b8-69ed849xxxxx, base URL = http://localhost:8080)
…
2023-01-30 03:23:09.889 [INFO ] [io.openhabcloud.internal.CloudClient] - Disconnected from the openHAB Cloud service (UUID = 2a3780fd-69ad-4e96-b5b8-69ed849xxxxx, base URL = http://localhost:8080)

=> No automated openHAB Cloud service reconnect since January 30th!

I saw the workaround described in the blog How to automatically restart cloud connector after an unexpected disconnection, but please be aware I’m still using openHAB 2.5 version.

Is there a chance to have back myopenHAB functionality in a way that the openHAB Cloud service reconnect is performed automatically – without workarounds?

Since you know it’s going to happen every day at that specific time, you don’t need to test the connection. You just need:

  • A single Exec thing/item with the console command to restart the binding.
    • openhab-cli console -p habopen bundle:restart org.openhab.io.openhabcloud
    • Don’t forget to add the command to your whitelist (per the tutorial)
  • A rule that turns the Exec item ON once a day.
    • I suggest 4:00am to ensure that the router has restarted and your Internet access has been fully restored.

It’s still a workaround, but it should keep you going until there’s a proper fix.

1 Like

So, there is an issue in the cloud connector bundle that would not reconnect to the service when the server specifically disconnects the client (which is now more common) . I suspect this is the issue. This is fixed in the next 3.4 release and also in the upcoming 4.0 release. As you are on 2.5, we don’t have a way of back porting this fix. I agree with @rpwong that manually restarting the binding shortly after your router restart is the best bet until you can upgrade to a supported OH version.

2 Likes

@andyzle: Off topic, but if you need more reason to upgrade, note that OH2.5 is subject to the Log4Shell vulnerability that was discovered in 2021. It was patched in OH3.2.

2 Likes

I will implement the scheduled binding restart, thank you.

@digitaldan Thank you for having a look on my issue.

Off topic too - In fact I started playing around with OH 3.2 in January 2022 to set up my openHAB installation from scratch as recommended by some experienced OH users. But after 4 month I lost all my developed content when my USB disk crashed totally due to my fault, only to backup openHAB content with its backup script storing my configuration in a zip file locally, not on a separated device. After that incident I was searching for a new backup environment (my old one is an outdated NAS). Due to other private activities I did not complete this search until now, but after closing that subject I will definitely start with OH 3.4.

Is there still no fix?

Only a workaround - see above comment from @rpwong.

Well there was this recently on server side: Disconnected from the openHAB Cloud service - #115 by rpwong

Further improvements in upcoming OH release as well to reconnection behaviour How to automatically restart cloud connector after an unexpected disconnection - #30 by digitaldan ([openhabcloud] Reconnection Fixes by digitaldan · Pull Request #14251 · openhab/openhab-addons · GitHub)

Is the server side fix making a difference?