Disconnected from the openHAB Cloud service

I can now definitely confirm that restarting the bundle from rules re-establishes the connection. Why the connection doesn’t re-establish itself is a different question.

Me and probably others are still having the following issue. Anyone know of any news on this issue?
Going to the “dashboard” on myopenhab returns a blank white page but e.g Google Assistant is connected fine.

OpenHAB retured empty sitemap list - Apps & Services / openHAB Cloud - openHAB Community

Hi LordLiverpool, I have the same problem of the cloud connector, and I am designing a solution just to check the connector status (up or down) in order to start it again.
What i do not understand of your solution is the reason why you need a timestamp, in my solution I just check if an item is reachable from myopenhab, using the rest API (a GET in particular), if not I restart the connector. It seems easyer than setting a timestamp, but perhaps there is something that I do not consider, can you help me to understand in which cases my solution does not work? Thanks

Hi. I suppose the benefit is being able to distinguish between brief outages and longer ones. If the cloud is briefly unreachable it won’t do anything. This could be due to any number of reasons. If it’s been a while since the connection was lost, then I try restarting the bundle.

:+1:

Thanks for you assistance, its taken a while as I’ve been away, but I’ve now setup something similar, I’m just changing a LED strip if I’m disconnected.

I was getting lots of disconnects but discovered I had a clone of my openhab docker running on my old host as well as the new and that was causing it to swap between them and confuse things.

1 Like

Is this issue widespread again?

It still happens time to time, no change there I’m afraid

2 Likes

Hi LordLiverpool,

I’m new with that and I have the same problem.
I would like to restart the binding too everytime it goes offline.

Can you please help my to do it and where put the files? I’ve created the item you posted but I don’t know how to menage with the script and in which folder put it to make it work and run everytime it goes offline.

Could you please help me?

Thanks a lot,
Matt

Same here. The connection was rock stable until I upgraded to 3.4.

Since then the connection is unstable. Here is the section in the log on DEBUG level, where the disconnection happened.

2023-01-08 02:28:52.411 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO pong: 25 ms
2023-01-08 02:29:17.411 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO ping
2023-01-08 02:29:20.911 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO pong: 3500 ms
2023-01-08 02:29:45.912 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO ping
2023-01-08 02:29:50.955 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO CLOSE: ping timeout
2023-01-08 02:29:50.956 [WARN ] [io.openhabcloud.internal.CloudClient] - Socket.IO disconnected: ping timeout
2023-01-08 02:29:50.956 [INFO ] [io.openhabcloud.internal.CloudClient] - Disconnected from the openHAB Cloud service (UUID = c3...c7, base URL = http://localhost:8080)
2023-01-08 02:29:52.269 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connecting (attempt 1)
2023-01-08 02:30:02.447 [DEBUG] [io.openhabcloud.internal.CloudClient] - Error connecting to the openHAB Cloud instance: EngineIOException xhr poll error. Should reconnect automatically.
2023-01-08 02:30:02.448 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connect attempt error: SocketIOException Connection error
2023-01-08 02:30:04.972 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connecting (attempt 2)
2023-01-08 02:30:15.070 [DEBUG] [io.openhabcloud.internal.CloudClient] - Error connecting to the openHAB Cloud instance: EngineIOException xhr poll error. Should reconnect automatically.
2023-01-08 02:30:15.071 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connect attempt error: SocketIOException Connection error
2023-01-08 02:30:20.071 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connecting (attempt 3)
2023-01-08 02:30:30.168 [DEBUG] [io.openhabcloud.internal.CloudClient] - Error connecting to the openHAB Cloud instance: EngineIOException xhr poll error. Should reconnect automatically.
2023-01-08 02:30:30.169 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connect attempt error: SocketIOException Connection error
2023-01-08 02:30:35.170 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connecting (attempt 4)
2023-01-08 02:30:45.270 [DEBUG] [io.openhabcloud.internal.CloudClient] - Error connecting to the openHAB Cloud instance: EngineIOException xhr poll error. Should reconnect automatically.
2023-01-08 02:30:45.307 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connect attempt error: SocketIOException Connection error
2023-01-08 02:30:50.308 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connecting (attempt 5)
2023-01-08 02:31:00.425 [DEBUG] [io.openhabcloud.internal.CloudClient] - Error connecting to the openHAB Cloud instance: EngineIOException xhr poll error. Should reconnect automatically.
2023-01-08 02:31:00.426 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connect attempt error: SocketIOException Connection error
2023-01-08 02:31:05.426 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connecting (attempt 6)
2023-01-08 02:31:15.546 [DEBUG] [io.openhabcloud.internal.CloudClient] - Error connecting to the openHAB Cloud instance: EngineIOException xhr poll error. Should reconnect automatically.
2023-01-08 02:31:15.550 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connect attempt error: SocketIOException Connection error
2023-01-08 02:31:20.550 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connecting (attempt 7)
2023-01-08 02:31:20.673 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO OPEN
2023-01-08 02:31:20.674 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connected successfully (attempt 7)
2023-01-08 02:31:20.774 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO connected
2023-01-08 02:31:20.775 [INFO ] [io.openhabcloud.internal.CloudClient] - Connected to the openHAB Cloud service (UUID = c3...c7, base URL = http://localhost:8080)
2023-01-08 02:31:30.777 [WARN ] [io.openhabcloud.internal.CloudClient] - Error during communication: EngineIOException xhr poll error
2023-01-08 02:31:30.869 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO CLOSE: transport error
2023-01-08 02:31:30.869 [WARN ] [io.openhabcloud.internal.CloudClient] - Socket.IO disconnected: transport error
2023-01-08 02:31:30.869 [INFO ] [io.openhabcloud.internal.CloudClient] - Disconnected from the openHAB Cloud service (UUID = c3...c7, base URL = http://localhost:8080)
2023-01-08 02:31:31.836 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connecting (attempt 1)
2023-01-08 02:31:41.940 [DEBUG] [io.openhabcloud.internal.CloudClient] - Error connecting to the openHAB Cloud instance: EngineIOException xhr poll error. Should reconnect automatically.
2023-01-08 02:31:41.940 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connect attempt error: SocketIOException Connection error
2023-01-08 02:31:44.275 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connecting (attempt 2)
2023-01-08 02:31:44.418 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO OPEN
2023-01-08 02:31:44.419 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO re-connected successfully (attempt 2)
2023-01-08 02:31:44.517 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO connected
2023-01-08 02:31:44.518 [INFO ] [io.openhabcloud.internal.CloudClient] - Connected to the openHAB Cloud service (UUID = c3...c7, base URL = http://localhost:8080)
2023-01-08 02:32:09.419 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO ping
2023-01-08 02:32:09.443 [DEBUG] [io.openhabcloud.internal.CloudClient] - Socket.IO pong: 24 ms

It’s strange that the third line from the bottom indicates a successful connection to the cloud service, but the service shows “Offline” in the myopenHAB Portal.

When I restart the bundle the connection to the cloud is restored properly and the portal shows “Online”.

Before that the cloud connection was rock stable for months. I never had to reestablish it.

Any idea what the cause could be?

1 Like

I am facing similar issues and I have the feeling that it is worse since I upgraded to 3.4
This morning there was no connection to the cloud. The debug log shows that the connection failed at 02:29:20 this morning. After 11 retries the connection was reestablished, at least according to the log. Effectively the cloud still did not work. The cloud worked again after pressing “save” in the cloud configuration settings. (the trace log is attached)

openhab.log (14.6 KB)

openHAB 3.4 running on Windows 11
Zulu Java

Hi
What rules/language are you using? As you may know, there are upteen different rule types and languages. I use Javascript/JSR223.

Hi and thanks for you reply.

I use javascript.

I already created the item and manually from the terminal in ssh using the obenhab>bundle:list I can restart the service using the command bundle:restart org.openhab.io.openhabcloud and in this way it works and getting online.

What now I would like to do is like what you did to send the command automatically every time it gets offline. I even tried to find a log about the openhabcloud but I didn’t find it to detect a string and send a timestamp from node-red to send the command.

If you can help me out to where to put the file you created in which folder etc I appreciate so I can do it automatically instead than by myself.

The rule files for Javascript go in:

/etc/openhab/automation/jsr223/javascript/personal

You need to import some stuff from the files in:

/etc/openhab/automation/lib/javascript/core

where you need to have the helper libraries installed (there should be 9 files I think).

var sOPENHAB_CONF_DIR = (sOPENHAB_CONF_DIR === undefined) ? Java.type("java.lang.System").getenv("OPENHAB_CONF") : sOPENHAB_CONF_DIR;

load(sOPENHAB_CONF_DIR+'/automation/lib/javascript/core/rules.js');
load(sOPENHAB_CONF_DIR+'/automation/lib/javascript/core/actions.js');

Then, to make the binding restart work requires extra steps. You have to pass a key file in the command line, which needs to be set up.

You need to generate public and private RSA keys:

cd /home/openhabian/karaf_keys
sudo -u openhab ssh-keygen -t rsa -f openhab.id_rsa

Then you need to copy the public key from:

/home/openhabian/karaf_keys/openhab.id_rsa.pub

(the long character string with no spaces) and put it in this file:

sudo nano /var/lib/openhab/etc/keys.properties

So that you end up with:

openhab=<LONG KEY STRING>,_g_:admingroup

Then you have to copy the private key to the openhab ssh directory (or at least I did), a directory which may need to be created. I couldn’t work out whether it needed to go in openhab2 or openhab, so I did both:

mkdir /var/lib/openhab2/.ssh
sudo cp /home/openhabian/karaf_keys/openhab.id_rsa /var/lib/openhab2/.ssh
mkdir /var/lib/openhab/.ssh
sudo cp /home/openhabian/karaf_keys/openhab.id_rsa /var/lib/openhab/.ssh

Then you have to give openhab ownership of .ssh. Again, I did both folders.

sudo chown openhab:openhab -R /var/lib/openhab/.ssh
sudo chown openhab:openhab -R /var/lib/openhab2/.ssh

Not sure if that’s every step. Then you should be able to pass the public key in the RestartBundle function from above. If you add the cloud heartbeat update and the call to RestartBundle from above when it’s out of date, you have almost everything. Perhaps the last missing piece is the timer function to update the item periodically and do the check:

function OnTimer_RunMasterScheduler_EveryTenMinutes(event) 
{
  try 
  {   
    logInfo("Running master scheduler (every 10 minutes)");

    // Do 10-minute tasks
    SendCloudHeartbeat() ;
    CheckIfAutomateAndCloudOffline();
  }
  catch(error)
  {
    logInfo("ERROR running master scheduler (every 10 minutes)! Message: " + error);
  }    
}
// @ts-ignore
when("Time cron 0 0/10 * * * ?")(OnTimer_RunMasterScheduler_EveryTenMinutes);
// @ts-ignore
rule
(
    "On timer: run master scheduler (every 10 minutes)",
    "Runs stuff every 10 minutes"
)(OnTimer_RunMasterScheduler_EveryTenMinutes);

I have most of my functions organised in classes, but to simplify I’ll remove the class stuff and also the export directive, which you need to stop Visual Code from complaining, but has to be stripped off before copying to openHab:

function GetOpenHabCloudCallAuthorisationHeaders()
{
  var headers = [];

  headers['Content-Type'] = '*/*';
  headers['accept'] = '*/*';
  headers['Authorization'] = 'Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxx'; // Cloud user and pw encoded
  headers['WWW-Authenticate'] = 'Basic';

  return headers;
}

function SendCloudHeartbeat() 
{
  // logInfo("SendCloudHeartbeat");

  try
  {
    var headers = GetOpenHabCloudCallAuthorisationHeaders();
    // logInfo("Cloud heartbeat headers: " + headers);

    var dateNow = new Date();

    DoHttpPutRequest("https://myopenhab.org:443/rest/items/Cloud_Heartbeat_TimeStamp/state", "Set cloud heartbeat", "*/*", dateNow.toISOString(), headers);     
  }
  catch(exception)
  {
    logInfo("Exception caught in SendCloudHeartbeat. Message: " + exception);   
  }
}

function CheckIfAutomateAndCloudOffline() 
{
   // Try restarting cloud client if down for a short amount of time
  if (nCloudHeartbeatAge > this.CLOUD_MAX_HEARTBEAT_AGE && (nCloudHeartbeatAge < this.CLOUD_MIN_AGE_ALERT_THRESHOLD)
  {
    RestartBundle("org.openhab.io.openhabcloud");
  }
}

function RestartBundle(sBundleID)
{
  var Exec = Java.type("org.openhab.core.model.script.actions.Exec");
  var Duration = Java.type("java.time.Duration");
  // @ts-ignore
  Exec.executeCommandLine(Duration.ofSeconds(6), "sudo", "/usr/bin/ssh", "-p", "8101", "-i", "/var/lib/openhab/.ssh/openhab.id_rsa", "openhab@localhost", "bundle:restart", sBindingID);
}

I also took out the minutes since reboot and push notification to simplify further.

1 Like

Opened a cloud connector issue #14200 for review by the developers.

In my point of view the workaround is very nicely crafted, but the root cause should be addressed.

1 Like

Developers are aware and have been trying to solve this for awhile now, but with no luck.

In reference to the GitHub issue you opened, I don’t think it has anything to do with OH 3.4. I’m still on OH 3.3 and have also noticed an increase in disconnects recently. Anecdotally, it seems that some users have experienced it more than others (well before 3.4 was released), while others may not be experiencing it at all. In my case, the disconnects stopped happening for awhile, and then started up again more frequently.

Yes. My personal opinion is similar, no significant change to the better or worse

The main issue is here, under openhab-cloud repo: Connection error recovery failed · Issue #134 · openhab/openhab-cloud · GitHub

We know that connection is re-established by the client seemingly successfully. Even notification sending works. Client sends a ping message pver the comms protocol and gets response back from the server.

However, the server has marked the openHAB as offline, making proxy access via myopenhab.org disfunctioning.

Unfortunately the symptomps match what has been observed already more than 2 years, see this comment Connection error recovery failed · Issue #134 · openhab/openhab-cloud · GitHub

1 Like

I’ve been working on a relatively simple approach to restarting the cloud connector when a disconnect happens. Seems like a good time to post it.

3 Likes

Same here. Either it was caused by the 3.4 update or coincidentally in the last days:

What should this picture tell us? This is only looking good to me - several succesful reconnections.