[SOLVED] Cloud connector refusing connections from outside using myopenhab.org

Hi,
I commented in another subject for the same problem with tag Home Automation but after many hours spent yesterday I’m pretty sure it is more general problem and should go to Cloud connector experts. Can anyone shed more light what typically causes this problem or may cause it, this is with myopenhab.org cloud:

[ERROR] [io.openhabcloud.internal.CloudClient] - Connection refused
[WARN ] [io.openhabcloud.internal.CloudClient] - Jetty request 17029647 failed: Connection refused
[WARN ] [io.openhabcloud.internal.CloudClient] - Connection refused
[WARN ] [io.openhabcloud.internal.CloudClient] - Connection refused

I’ve seen many subjects around it where people concluded - I have this problem and since now haven’t found the solution or rather: “I’ve done number or random actions and it is not there anymore but not sure what exactly helped as performed many reinstalls, re-creation of cloud account etc”. I guess there must be some people in this forum who have a clue what causes it and how to start debugging. The thing is annoying as it simply prevents any outside connections when using:

  • myopenhab.org cloud and the link suggested for OH2.x
  • OH app on the phone (Android in my case)
  • when trying to integrate Google Home
    At the same time notifications are working fine, the events and item changes can be seen on website. I’m having an impression it has something to do with http vs https access but no idea how to debug it. I tried to enable DEBUG on the cloud connector and on jetty but honestly I didn’t see and hint there. Thanks in advance for your assistance.

It is important to realize that the cloud server doesn’t connect to your OH instance. Your OH instance is connecting to the cloud server. Therefore the Connection Refused is because the Cloud Server is refusing the connection, not the other way around.

So indeed, I do think this is the right place for this topic.

But if the Cloud Server is refusing the connections, then the cause is either:

  • there is something wrong with the Cloud Server. You can always check http://status.openhab.org/ to see if the service appears healthy

  • there is something wrong with your network that is preventing the connection

  • there is something wrong with your ISP (e.g. recently it was discovered in the US that AT&T removed/black listed myopenhab.org from their DNS servers for some reason)

Rich,
first of all thanks for the response as not many really take part in this cloud discussions and seems this is quite common problem.

It is important to realize that the cloud server doesn’t connect to your OH instance. Your OH instance is connecting to the cloud server. Therefore the Connection Refused is because the Cloud Server is refusing the connection, not the other way around.

Right, this is what I’m getting, maybe I was not precise enough when forming the topic. What I meant however, is ability to “connect” from external to my instance, in other words to access it, and again - I understand the flow is actually the opposite direction (my instance -> cloud -> consumer (Android app/Google/whatever)).
I understood however (correct me if wrong) that communication happens within secure tunnel, therefore as long as https is allowed from my instance to the cloud, it should be OK.
It shouldn’t be network issue as I have this problem: 1) at home, 2) in my country in Europe using mobile connection (different than my home ISP), currently I’m in the US for couple of weeks and accessing it from 3) the hotel as well as 4) the office and experience same issue.
Being a technical person, I’d expect however a log file or any other message being present to help me to narrow down the problem to a specific place/area (maybe some config on my side, maybe on the link, maybe the OS).
I checked the status of myopenhab.org on the link you suggested and it reads ‘Operational’.
is it possible that connection within tunnel which is currently mentioned as http://localhost:8080 needs to be changed to something else for ‘Remote Access’ type of connection to the cloud (I mean there is access type of Notifications only or Notifications and Remote Access)

[io.openhabcloud.internal.CloudClient] - Connected to the openHAB Cloud service (UUID = myUUID, base URL = http://localhost:8080)

another idea is - maybe I need some certificates or so (eventhough I haven’t seen any place where myopenhab.org would allow to configure those) as from what I see in startup is:

2019-02-19 05:51:41.746 [WARN ] [.jetty.internal.ServerControllerImpl] - SSL connector will not be started
2019-02-19 06:01:49.390 [WARN ] [.jetty.internal.ServerControllerImpl] - SSL password and SSL keystore password must be set in order to enable SSL.

but maybe I’m saying rubbish (I am becoming desperate on this subject therefore might be mixing too many different things). This specific issue seems to not have a good logging support to identify what is rejected by whom and why.

One more important comment which was obvious to me but I realized maybe not for others, leading hints into wrong directions:

This “connection refused” message IS NOT related to when my instance tries to connect to myopenhab.org on startup or so. Connection is done properly.

This message appears EACH time I try to connect from either phone app, or link withing myopenhab.org (to use the 2.x link) or from Google Home cloud. In other words I read it as ‘incoming connection’ within the established tunnel from my instance to the cloud. The message is immediate which again suggests this is all happening within established tunnel/connection that is already present (hence I don’t blame network access as such but rather some configuration of ports within the tunnel)

Yes, the communication between your OH and myopenhab.org is encrypted and the communication from your browser to myopenhab.org is also encrypted. The primary distinction is that your OH is initiating the connection to myopenhab.org, which means you don’t have to open any ports on your firewall.

If there is a useful log it will be on the myopenhab.org servers. All your OH instance knows is that myopenhab.org refused the connection. It doesn’t know anything else so has nothing else to log.

Mentioned where?

Nothing needs to be changed. That is just telling the binding how to access your OH. To draw a more complete view of how this works:

The Cloud Connector Binding proxies between the Cloud Server and the OH instance it runs on. It therefore connects to both your local OH over the network (even though it is running in the same process as your OH) and myopenhab.org. Unless things have changed, it does not support the encrypted connection to your OH instance (doesn’t really matter as the network traffic never leaves the host) so it needs to connect to localhost:8080.

You would only need to change this setting if you changed the default port your local OH instance listens to, or disable the unencrypted port somehow.

But that is not a measure of how reliable the connection is between your OH instance and the cloud server. If that connection is spotty, then when you try to bring up your sitemap remotely it wouldn’t necessarily be able to but notifications get queued up and sent when the network connection does work.

One of the purposes of myopenhab.org is to not require you to mess with certificates. I think those warnings apply to http://localhost:8443, the connection to your local OH instance that uses https.

Rich,
picture worth thousand words! Eventually I got how the full chain works.
This however doesn’t explain to me how it happens that on each request from ‘remote clients’ I’m getting Jetty connection refused in other words there must be some messages flying from OH Cloud Server (myopenhab.org in this case) to the binding or even to Jetty to trigger Jetty to connect (and therefore complain it is refused). Do you know how it works?

Until now I thought it is connector trying to connect to my jetty on each ‘remote client’ requests and that my jetty is refusing the connection therefore I expected ports to get opened. From what you are saying this is not the case, right?

Mentioned where?

in the openhab.log on Cloud Connect. But from your further explanation this is all fine as long as I don’t change any ports

I would also add on this spotty connection - I don’t think it is about it as throughout the time I am having these problems (months; but never hurt me as I didn’t intend to integrate Google) it never worked. Even statistically it would be up for some time. Moreover what I observe:

  • when I’m restarting my OH instance and trying to connect the connection is not refused immediately, the cloud apparently waits for ports to be opened or process to be up, whatever
  • when however my OH is started I get immediate connection refused which suggests this could be something in the configuration not allowing me to connect
    It is so frustrating…

myopenhab.org and the Cloud Connector binding basically just proxy your OH’s Jetty and REST API. But I believe that the connections are always initiated by the Cloud Connector. Your remote device connects to myopenhab.org but myopenhab.org is already connected to your Cloud Connector binding so there is no need for a new connection. I could be wrong about that.

If I am wrong, then assuming the connection is being caused by trying to connect a remote client then the connection that is being refused would be between the Cloud Connector binding and your OH’s Jetty.

What are your Network settings in PaperUI -> Configuration -> System? Maybe you’ve accidentally excluded listening on localhost?

Do you have a host based firewall installed and running (IPTables, UFW, etc)?

How did you install OH?

Rich,

But I believe that the connections are always initiated by the Cloud Connector.

This is what I thought too. But this wouldn’t explain why I see refuse immediately after request from outside.
My understanding now is the following (can be very wrong though):

  • my client initiates connection to the cloud for creating the tunnel
  • within this tunnel, external element initiates connections back to my OH
    This would be logical explanation how to trigger from e.g. Google home switch off of my lights (Google Home -> Cloud [myopenhab.org] ----- over the tunnel ----> my own OH instance -> item switch off request)
    This in turn suggests there is some setting somewhere allowing/disallowing these internal requests (within the tunnel which is always initiated one way by my OH instance)

In my network setting I just specified my OH primary address with the netmask - 192.168.1.23/24 (the hint in line below in the PaperUI suggests however the whole subnet, I don’t know whether I should change it…)
then all the rest is default i.e.

  • use IPv6 if available (ON)
  • no broadcast address
  • single IP address per interface (OFF)

I installed OH using apt (about 1.5-2y ago so that’s why I was thinking that maybe I missed something important during installation and forgot about it).
I don’t use any iptables for traffic filtering

sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

more /var/lib/openhab2/config/org/eclipse/smarthome/network.config
primaryAddress=“192.168.1.23/24”
service.pid=“org.eclipse.smarthome.network”

some threads however suggest that the primary address should be the network (e.g. this one: Parameter setting org.eclipse.smarthome.network) but this is a bit counterintuitive. What do you have set there?

Let me say this with more detail. Connections from the Cloud Connector binding to openHAB are always initiated by the binding. Connections from the binding to OH I can see being initiated by the binding as well.

So perhaps the connection refused is in response to that.

There is no way for myopenhab.org it initiate a connection to your binding. The binding is inaccessible from the Internet.

That’s why I asked about host based firewalls and your network settings in PaperUI. Perhaps your OH is configured to only accept connections from certain IPs and that would exclude 127.0.0.1 from connecting.

That is defining the whole subnet. That is what the /24 is doing. All 192.168.1.* addresses would be allowed.

That maps to what I have, though I also defined the broadcast address but that shouldn’t impact this problem at all.

All I can recommend at this point is put OH into trace level logging and see if something shows up that might tell us a little more.

Assuming it’s not a fluke that you get the connection refused when some client tries to connect remotely, the problem is that your your OH Jetty is refusing connections on localhost:8080 from the Cloud Connector Binding.

Assuming it’s not a fluke that you get the connection refused when some client tries to connect remotely, the problem is that your your OH Jetty is refusing connections on localhost:8080 from the Cloud Connector Binding.

this starts forming a potential idea for resolution. Where would be such settings stored to change them, any idea?

I quickly had a look at all files with name jetty and there are too many to guess the relevant one, I took however the most probable suspect /usr/share/openhab2/runtime/etc/jetty.xml
and what I’ve found there:

                        <Property name="org.osgi.service.http.port.secure" default="8443" />
                                        <SystemProperty name="jetty.host" default="0.0.0.0" />:<SystemProperty name="org.osgi.service.http.port.secure" default="8443" />
                                <Set name="port">
                                        <SystemProperty name="org.osgi.service.http.port.secure" default="8443" />

nothing about 8080 but I’m a bit afraid to judge immediately this is my issue.

Edit. I’ve found actually something which would look even more relevant (still no 8080 there)

more /var/lib/openhab2/etc/jetty.xml
<?xml version="1.0"?>
<!--
 Licensed to the Apache Software Foundation (ASF) under one
 or more contributor license agreements.  See the NOTICE file
 distributed with this work for additional information
 regarding copyright ownership.  The ASF licenses this file
 to you under the Apache License, Version 2.0 (the
 "License"); you may not use this file except in compliance
 with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing,
 software distributed under the License is distributed on an
 "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 KIND, either express or implied.  See the License for the
 specific language governing permissions and limitations
 under the License.
-->
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_0.dtd">

<Configure id="Server" class="org.eclipse.jetty.server.Server">

    <!-- =========================================================== -->
    <!-- Set connectors -->
    <!-- =========================================================== -->
    <!-- One of each type! -->
    <!-- =========================================================== -->

    <!-- Use this connector for many frequently idle connections and for
        threadless continuations. -->
        <New id="httpConfig" class="org.eclipse.jetty.server.HttpConfiguration">
                <Set name="secureScheme">https</Set>
                <Set name="securePort">
                        <Property name="jetty.secure.port" default="8443" />
                </Set>
                <Set name="outputBufferSize">32768</Set>
                <Set name="requestHeaderSize">8192</Set>
                <Set name="responseHeaderSize">8192</Set>
                <Set name="sendServerVersion">true</Set>
                <Set name="sendDateHeader">false</Set>
                <Set name="headerCacheSize">512</Set>
        </New>

        <!-- =========================================================== -->
    <!-- Special server connectors -->
    <!-- =========================================================== -->
    <!-- This is a sample for alternative connectors, enable if needed -->
    <!-- =========================================================== -->
    <!--
        <Call name="addConnector">
                <Arg>
                        <New class="org.eclipse.jetty.server.ServerConnector">
                                <Arg name="server">
                                        <Ref refid="Server" />
                                </Arg>
                                <Arg name="factories">
                                        <Array type="org.eclipse.jetty.server.ConnectionFactory">
                                                <Item>
                                                        <New class="org.eclipse.jetty.server.HttpConnectionFactory">
                                                                <Arg name="config">
                                                                        <Ref refid="httpConfig" />
                                                                </Arg>
                                                        </New>
                                                </Item>
                                        </Array>
                                </Arg>
                                <Set name="host">
                                        <Property name="jetty.host" default="localhost" />
                                </Set>
                                <Set name="port">
                                        <Property name="jetty.port" default="8282" />
                                </Set>
                                <Set name="idleTimeout">
                                        <Property name="http.timeout" default="30000" />
                                </Set>
                                <Set name="name">jettyConn1</Set>
                        </New>
                </Arg>
        </Call>
        -->

    <!-- =========================================================== -->
    <!-- Configure Authentication Realms -->
    <!-- Realms may be configured for the entire server here, or -->
    <!-- they can be configured for a specific web app in a context -->
    <!-- configuration (see $(jetty.home)/contexts/test.xml for an -->
    <!-- example). -->
    <!-- =========================================================== -->
    <Call name="addBean">
        <Arg>
            <New class="org.eclipse.jetty.jaas.JAASLoginService">
                <Set name="name">karaf</Set>
                <Set name="loginModuleName">karaf</Set>
                <Set name="roleClassNames">
                    <Array type="java.lang.String">
                        <Item>org.apache.karaf.jaas.boot.principal.RolePrincipal
                        </Item>
                    </Array>
                </Set>
            </New>
        </Arg>
    </Call>
    <Call name="addBean">
        <Arg>
            <New class="org.eclipse.jetty.jaas.JAASLoginService">
                <Set name="name">default</Set>
                <Set name="loginModuleName">karaf</Set>
                <Set name="roleClassNames">
                    <Array type="java.lang.String">
                        <Item>org.apache.karaf.jaas.boot.principal.RolePrincipal
                        </Item>
                    </Array>
                </Set>
            </New>
        </Arg>
    </Call>

</Configure>


You wouldn’t find anything about port 8080 there. This is the settings for the encrypted connection, port 8443. 8080 is unencrypted so doesn’t need those settings.

Unless you messed around with the settings the problem is unlikely to be a settings problem. Thousands of users are running through myopenhab.org without any problem whatsoever. If it were installed wrong none of their systems would work either.

Also, this isn’t likely to be a setting that directly refers to port 8080, though it could be. It will be a network interface setting that defines what IP addresses it will accept connections from. Again, this is unlikely to be caused by the settings files because everyone else’s would be broken too.

<SystemProperty name="jetty.host" default="0.0.0.0" />

The 0.0.0.0 on this SystemProperty for port 8443 indicates that it will accept connections from any IP address.

Makes sense what you are saying. There are many however who have similar problem like mine and these threads (even in this forum) stay unanswered.
I accept your way of thinking though and therefore I need a step back and understanding where to find logs to understand what is happening (which part needs to be put in debug). As problem is 100% replicable and ‘on-demand’ I could collect them immediately.

At a minium the Cloud Connector Binding. Jetty may need to be put in trace log level too, though I wouldn’t expect much useful from that.

Like yours or exactly yours. All of the similar threads I recall reading had no connectivity between OH and myopenhab.org, or had problems exposing Items to IFTTT through myopenhab.org. This is the first thread I’ve seen that indicates remote connections being refused.

And if the problem were in the configs, and you didn’t change the configs, then we would all be seeing this problem, not just a few.

Rich,
I don’t guarantee I haven’t changed the config over these 2y I’m running OH (or rather I’d expect doing less than required somewhere).
I’m also thinking - some people with similar problem reported that after java upgrade these disappeared (my java is latest version). But maybe there are some java settings I should have (or shouldn’t have) to enable this? Something around security or so? Would you think of any?
regarding similar cases, I think this is exactly mine: Google Home Error linking openHAB account and even if people don’t have same conclusions I think it is because there was no deep dive debugging done on it.

I know what is the issue (finally!!!). My OH simply doesn’t listen on localhost:8080 hence refuses any connection to this address. It listens on the actual IP address.

One day I had to change it to some specific address and I followed the instructions which suggested to change it in here:
/var/lib/openhab2/etc/org.ops4j.pax.web.cfg
by commenting out this line:
org.ops4j.pax.web.listening.addresses = 0.0.0.0
and adding own address instead. As it never worked to me to provide a list instead of single address (it should accept it as per doc, comma separated but OH throws errors when trying this - I tried just now and I’m on OH2.4).
It seems I introduced the issue myself. I hope this thread will help others with similar problems.
Many thanks to @rlkoshak who assisted me above! We can close this thread I think

That has been my conclusion for awhile. The why is what it be been trying to solve.

That would do it. I assumed there wouldn’t be any changes in the config as most users don’t do this. I’m glad you found it. I’m going to have to remember that config file.

Do you still need the multiple addresses? If it’s supposed to accept a list, definitely file an issue on openhab-core.

I’m really glad we figured it out. Thanks for sticking to it until it was solved.

Rich,
today I don’t need it that much as in the past so I can survive with the current configuration. Lack of localhost was introduced indeed by the fact I couldn’t run it on multiple, selected addresses (not all possible as per current setting) therefore I had to decide for one and therefore cut out the others)