Thousands of OH installations accessible from everywhere

We can look at this as a good thing. It means that openHAB has enough users overall that the tiny percentage of those who are ignorant or foolish still runs into the thousands. :smiley:

One reason why I brought up the openHAB blog is it doesn’t require people to sign up, has a lot less going on so posts cannot be missed, and new posts to the blogs also get announced to Twitter. That provides a few more avenues for communication.

However, this is a forever issue. If we write a blog post or pinned announcement now it will work for a few months perhaps. Then it will fade back into history and become lost and ignored again. It’s a hard problem to solve.

Well, there are those rare few who run IP6 on their LAN (not me but we have enough users that there is bound to be a few). But those users will probably be knowledgable enough to know what’s not working and why.

It’s an intriguing idea but I think it leads to a chicken-and-egg problem. How can you tell OH what the site local IP4 subnet is without connecting to OH in the first place (I’m assuming this sort of setting would be wanted in the UI like all other settings are or are getting support for too). You can’t necessarily base it on the local machine’s IP address because:

  • might be running in Docker containers where they have their own 172 subnet
  • might have more than one IP address, which one to choose?

I’m sure these could potentially be overcome, I just want to explore the implications and how it would work from the user’s perspective.

I tried really hard to make that work. I eventually gave up. I know it’s possible but I couldn’t manage it (also with pfSense and HAProxy, I’ve recently moved to OPNsense for a number of reasons but support for Tailscale was one of them). Compared to OpenVPN or even messing with “raw” Wireguard Tailscale is super easy and super fast. It was really shocking how much easier and seamless the experience was and it doesn’t require any open ports.

I see this advice all the time but frankly it is completely out of the range of possibilities for the vast majority of our users. When your “server” is an RPi 3 and your ISP provided gateway doesn’t even let you assign static DHCP IP leases this is a complete non-starter. Even I, who has the equipment and the skills to do so don’t really do that except for my vault warden machine. I just can’t justify the effort/complexity costs for the risk. But that’s my personal calculation, others will have different concerns.

Instead I try to avoid wifi type IOT devices unless they are from “trusted” vendors or DIY. The degree of mitigation for the cost ends up being about the same for me. YMMV.

Yep, they are lovingly called bogons. :smiley: But not all users will have a bogon address on their machine. Some ISPs put their users right on the Internet without a gateway. A lot of those users might be the same ones that show up in shodan. It’s rare and I don’t know why but some users run OH in a VSP and they won’t have a bogon address either.

All of that could be overcome I’m sure but we have to consider all the potential use cases and make sure they are all covered. And even then there will be a few dozen users who show up and say “hey! why do you break my openHAB!”

I believe that is already implemented.

@mstormi, would Fail2Ban be something that could be easily added to openHABian? It’s been on my list to experiment with that on my own but haven’t got to it yet. It seems like it would be a good thing to run for both Mosquitto and openHAB, maybe other services openHABian supports as well.

But the overall problem is by default, there are parts of OH that can be accessed without any authentication what-so-ever. And those are parts that you wouldn’t want to expose to the Internet but might indeed want to have accessible without authentication on the LAN. For example, I want remote access but also have a QR code that house guests can use to bring up a sitemap to control the house when on my wifi.

I can’t (today) have my cake and eat it too. I need to have something between OH and the Internet. I choose and recommend myopenhab.org though have explored and experimented with HAProxy, and nginx in the past.

That one already only accepts connections from localhost by default. I don’t know if openHABian changes that default or not. It seems reasonable that it might.

Part of the amazing progress of OH in the last few years has been the ease of setup and usability improvements. It’s now, as we all see, quite possible to get a basic OH system up and functional without ever doing more than skimming one or two of the doc pages. We can’t have it both ways here, unfortunately. We can’t have a broad, open user base and have a 100% up-to-date educated user base.

Because entry to OH has gotten so much easier, I think that sign-up as opt-in is still a hurdle too high for many new users (a number of whom find themselves on the shodan list, I would guess). But I agree that something more proactive is probably called for here.

From a user psychology stand-point I think something more along the lines of a built-in security audit would be far more successful. Several other home-server suites do something like this (NextCloud’s being one of the better examples IMO - informative, well documented, and stratified with warnings vs suggestions). In general, a little bright red icon that constantly appears somewhere on the Main Administration page will eventually result in a user clicking on it (whether it’s concern, curiosity, nag fatigue, or the simple desire to turn that red icon green) and give the user a feeling of more control over that process. The user isn’t signing up for anything; nothing unsolicited is sent to the user; it happens at the user’s determined interval.

The audit itself wouldn’t have to do a whole world of things. A simple port scan such as shodan’s and then a few things that even a basic user would be familiar and comfortable with: a check of user password strengths, age, and maybe even haveibeenpwned (not that an OH user list is likely to be hacked and released but this might highlight password reuse), maybe also include other settings a new user might have changed without full understanding such as status of the security access to the API.

If it turned into something that was a little more general it might even see broader use (and sneak the net security in without some users even realizing it). It could easily check for crufty stuff: are there installed device bindings that have no associated things, or configured things that have no channels linked to items or triggers in rules - little things that would help keep an installation well-maintained for less advanced users.

2 Likes

That is an intriguing idea but how would openHAB determine that it’s exposed to the internet? That’s the specific problem we are facing in this thread. There are so many variables and so much between the OH instance and the way it’s accessed from the Internet I don’t see how it could ever know that. I could be wrong though and would happy to hear how that could be done.

Continuing to spitball ideas, OctoPrint has built-in notifications that pop up when I open the dashboard, and I can ignore them or click the links to go to webpages for more information. Perhaps something like that would be possible, maybe as a binding?

BTW, I’m learning a lot from this discussion, so thanks to everyone for participating. I feel that this is an important issue facing smart homes in general, and would love for openHAB to be ahead of the curve on handling it.

I believe a stock OH 3 when it first starts up goes through a wizard to first set it up. Maybe something can be mentioned there. It also links to the getting started tutorial and one can add a brief paragraph to the Introduction.

That’s what I was thinking about earlier when I suggested the opt-in mailing list. It’s the only place where I think you could guarantee a brand-new user would see it.

That a good point; the vast array of different implementations is a potential issue. This isn’t really my area of expertise (and possibly not even my area of basic competence) and certainly my proposal was more off-the-cuff than fleshed-out, but one imagines that a call to any of the standard external ip checking sites could serve as a basic check of outward connectivity in addition to returning the wan entry point that could then be scanned for open OH ports. Such a scan would be no more exact than the results from the shodan list, but, after all, those with OH properly obfuscated are not the ones we’re worried about here.

I think it is because our cloud is free and easy to use that the numbers are low, if you do the search for home assistant you get 80,000 results and the fact they charge would have a bearing on more users wanting to escape the fee. My understanding is they at least for the past year have had a user/pass and fail2ban may be implemented(?). It was a weekly occurrence that someone was ‘hacked’ on their forum for a while, now that has improved. The password is only asked for if you try and connect from a non site-local IP.

That is correct, you have to let a HTTP connection be made to determine if it is local or not, in the case of someone doing it ‘correctly’ via a VPN for example this would never occur, so it only happens when someone has already exposed the machine/port directly to the web. It does not need a user to select anything, it would be automatic and probably 10-20 lines of code to write and easy to reverse if another approach is developed. The server could refuse to make any reply back and hence they have no idea it is openHAB, which I am leaning more towards this approach or a generic error, because if you used a custom reply, then you can tell it is openHAB and then a targeted attack can be made.

My idea can automatically determine that, but it has to wait until someone trys to connect from a public IP. As for other ways to implement security audits I know of ways that can be done because other HA projects have already implemented them. But if we take the stance that it is not worth taking action because 3 people may have a whine they are inconvenienced and have to improve their security, then we can see openHAB slip further behind.

Sorry, we are too late to be leaders. We can only settle for playing catch up and the openHAB devs have done some great work into the area with V3.

If your interested in checking your system, another good site is GRC | ShieldsUP! — Internet Vulnerability Profiling where you can get them to probe your ports and the info on the site explains what it all means. Click on the PROCEED button and on the next page select which type of scan you wish to do, your also able to specify a custom port range.

Example of what you get back if you have a decent firewall…

2 Likes

Probably so. Haven’t used so dunno how much effort it is to integrate with new tools
If I just install it with some default config that (probably, haven’t checked) locks up SSH etc I fear we might create have lots of issues. Imagine telling some non-IT guy to unlock his fail2ban locked system… ouch.
But either way it’s not about adding but about tuning the rules, isn’t it ?
We would need a set to work on OH logs.
And what would be a proper reaction ? how to implement a “ban” in OH or mosquitto ?

How does it never occur? I host OH in a Docker container. As far as it’s concerned its IP address is in a 172.17.0.0/16 subnet. My lan is on a 10.10.1.0/24 subnet. My VPN was on a 10.10.8.0/24 subnet but is now on a 100.0.0.0/8 subnet. From OH’s perspective, any connection I make to it is from some other subnet and not local to the host’s LAN.

One could use a blanket “treat all bogons as local” I suppose but that doesn’t feel very effective. It may be the best that can be done on this one point.

I don’t think anyone will necessarily whine about it but making breaking changes like this without even considering and discussing how it impacts existing users is pretty user hostile IMO. One should at least consider the impacts a change like this will have to even those users who are outside the norm.

Fail2Ban watches the auth.log (or equivalent) for what ever services you want it to. When it finds a line that matches a regex that indicates a failed login attempt it starts to keep a count from that IP. If a configured number of failed attempts comes from that IP, it sets a temporary iptables rule to block that IP for a given amount of time. IP addresses can be allowed listed which could be set to all bogons so local access would never be blocked, even temporarily.

It also can be configured to just monitor OH’s logs so SSH would never be locked up. But it seems like anything that might be exposed should be monitored.

Note that this really just helps mitigate brute force login attempts so it’s not a magic cure all for all types of attacks.

As an example, my configs for code-server (VSCode in the browser) is as follows:

/etc/fail2ban/jail.conf: tells fail2ban where to look

[code-server]
port = 8080
enabled = true
backend = auto
logpath = /var/log/syslog

/etc/fail2ban/filter.d/code-server.conf: tells fail2ban what to look for and what to do

#Fail2Ban filter for code-server
[Definition[
failregex = ^.*: Failed login attemp\s{\"xForwardedFor\":\"<HOST>\"
ignoreregex =
datepattern = ^%%b %%d %%H:%%M:%%S

For OH it would look at /var/logs/openhab/auth.log and I’ve not looked at that log yet to see what the REGEX would need to be. Separate entries would need to be made for each service one wants to protect.

The behaviors and allowed lists are defined in other config files.

Fail2ban can be made pretty hands off while still providing mitigation against brute force login attempts.

I can second ShieldsUp! as a great resource for checking your system. And if you are interesting in following the latest in the computer security news Steve Gibson hosts the Security Now podcast which is somewhat entertaining and informative.

I also like this over all approach and have seen it taken in other products too like Nextcloud. Some security tests are made locally but that can only take you so far. But having a page we can host that will probe your exposed OH URL to find other problems would be cool too.

But we already provide a service that allows remote access without these sorts of security risks. Would a service like that encourage users not to use it? And any service like this or the self checks will not be a one time thing. It’ll need to be maintained for the long term. Do we have developers willing to do that?

will it work when the server is behind NAT? I guess that’s how most OH instances are deployed today, with the internet router providing some (fractional) security.

It depends on whether the originating IP address is the actual original IP address or it gets replaced with the gateway’s address. I can say for code-server at least that it worked behind a reverse proxy but back when I ran this the proxy was directly running on the gateway so NAT wasn’t involved.

I run fail2ban on my vaultwarden LXC which is behind pfsense which is behind my ISP modem.
If i remember well enough, you needed to add some setting in the reverse proxy (in my case HAProxy) to get the “real” IP through to fail2ban. Otherwise it will indeed just see a internal IP.

Yes and yes.
most of the openHAB installations (and other things as well) exposed on the internet are set up as port forwarding in thru the NATing.
The server inside the NAT will see the public IP of the connecting machine.
A quick google search is all that is needed to set up the forwarding.

This is what I have been talking about doing a check on @rlkoshak not the ip of the network card NIC address.

Then that is a valid use case that makes the idea of only accepting site local address’ (with no reply) not worth doing unless it was only done for a security audit idea. Perhaps it could still be used to require a valid openhab user login that does not appear when your internal as V3 has this ability now, but without a ipbanning and throttling service it would only slow a hacker down. By putting a user/pass there it could also make people use it more thinking it is a good idea to just directly expose.

A “ban” is best handled with nftables on Debian and its offsprings.
(Nftables is the standard from Buster, iptables is now translated into nftables)
Nftables can take care of automatic timeouts of banned IPs.
Fail2ban just deliver a “ban” to nftables.
Another related feature to look into is ratelimits.

The Debian nftables apt package do exist in Rasberry OS from Buster and onwards. It provides a empty /etc/nftables.conf to build on.

Of course, so my question was merely if that’s always the case.
If for some reason it does not and the router IP gets banned that’ll likely have severe impacts (think DNS,DHCP).

Just an idea:

  • how about OH3 would check if it’s exposed directly to the web itself, once a day automatically?
  • maybe this could be checked via pinging/asking a script at myopenhab.org.
  • if it does, put a big fat red “WARNING Your OH is exposed…” message or box overlayed on all the screens.

With 25 years of experience writing programs for users I’ve learned one thing:

  • users do not care anything, unless it’s annoying enough.

Most routers are automatically forwarding ports via UPnP option turned on.

I recommend: always TURN OFF at each router at first setup. It’s a huge security risk.
kép

1 Like

Reliably doing so is not trivial at all.
And false positive warnings have a huge impact particularly on trust in OH as a reliable system (think vaccinations…) so I would not want to shoot early.

And ‘overlayed on all the screens’ isn’t trivial either.
We have many “screens” / UIs, some java generated, some HTML/CSS based, external programs, text only …

While probably correct this is an entirely different thing you cannot check or change automatically, it’s closely related to user education, different already across routers, and almost every installation is unique.

But unless you treat all bogon addresses as local, how do you tell whether the address is or is not a local connection?

If you can look that up or remember I bet a lot of readers of this thread would appreciate seeing what needs to be set.

True, but you can allow list the local subnet(s) so those addresses never get banned. The gateway and subnet can be discovered and the added to the config through scripts so it very much could be added as an openHABian config.

Also, by default the ban is only for a time, not permanent.

I think this could be implemented in a low risk way while still providing some reasonable protections. As with anything though, it’s one tool in an overall toolbox, not a single tool to fix all security problems.