My personal openHAB server infrastructure hacked

Setup, Configuration and Use Installation
1

Dear all

I would like to inform you about an attack against my openHAB infrastructure. The hacker fund a way into my system through somekind of open i.e. exposed ports and placed some kind of malware into my server.

As a result my system further tried to hack other systems on ssh outgoing connections. I’ve got informed by my internet provider about this issue as they received reports from security providers.

The details from provider came as follows (IP addresses and useranonymized):
description: This host has most likely been performing brute-force attacks against third parties.
feeder: shadowserver
destination ip: 216.X.Y.Z
destination port: 22
source ip: 80.A.B.C
source port: 60540
protocol: ssh
source: GCA-IoT
start time: 2019-06-29T02:20:29.036033UTC+0
end time: 2019-06-29T02:20:29.323124UTC+0

description: This host has most likely been performing brute-force attacks against third parties.
feeder: shadowserver
destination ip: 195.X.Y.Z
destination port: 22
source ip: 80.A.B.C
source port: 46892
protocol: ssh
source: GCA-IoT
username: ****
password: ****
start time: 2019-06-29T03:26:44.557520UTC+0
end time: 2019-06-29T03:26:46.919524UTC+0

At first I closed all exposed ports, but the bad guy is still sitting in my system and continued its attacks against other systems. Therefore I updated my firewall rules to block all outgoing connections completely. Resulting in malfunctioning of some OH processes such as homeconnect.
Meanwhile I have not received any further reports from my provider, but I still haven’t found the root cause of it.

Yesterday I’ve tried to track down what’s going on, I can see outgoing connection requests. The same IP target is not only used on port 22 but also 80 (using a JSON request), the occurrence is randomly no regular time.

I hope will find the damn nasty bit, but it doesn’t look good. I assume the only way is a fresh install :frowning: .

Platform information:

  • Hardware: MacMini /intel x64/16MB/130GB
  • OS: Debian 9
  • Java Runtime Environment: oracle java 1.8
  • openHAB version: 2.5 M1

No cloud services involved.

Formerly open ports:

  • SSH
  • OH webUI

through a port forwarding (i.e. public port different)

This post is not to blame openHAB. It should be seen more in the way as a reminder how important security is and how careful one needs to be.

Cheers
Stefan

1 Like
2

Mind sharing some details of your setup (like OS of your server, exposed ports, cloud services used…)?

3

Were you using a reverse proxy?

4

You should never ever trust a compromised system. Always perform a reinstall and be very carefull when placing back any config backups.

Just on a side note: The title of this post is somehow disturbing. There is no proof that openHAB was the entrypoint for the hack. Your system could be hacked by so many exploits are even a weak password. So without having any proof that there is a vulnarability in openHAB, i would suggest you to change the title.

Edit: :slight_smile: After title change, a suggestion: ‘My local openHAB infrastructure is hacked’ just to make it clear it is not the infrastructure hosted by openHAB foundation and used by other users.

2 Likes
5

no, that would probably have helped to prevent the current mess

6

Was port 22 open to the outside?

7

It was (see my edit) but not directly, the port used to access SSH was different and in non common range.

8

There are a lot of mad guys out just want to destroy something …
But there are also guys professionally trying to place malware to use your PC as spam-mail proxy or get email addresses or whatever.
These guys using automated portscanner, user and password libraries etc. and having deep knowledge about IP protocol.
I have just one port open, this is (i think) UDP 1194 for OpenVPN …

9

I absolutely agree with you, and that is one of the reason why I prefer to keep such sensitive environments as separate as possible. The more external connections involved the higher the risk.

10

Yes, this is the matter why I have chosen Openhab :slightly_smiling_face:
Following is for sure off topic but whant to share.
We have customers wolrdwide, when I am corresponding via email with a customer e.g. in Asia it takes mostly just a few days until I get spam in Asian letters.
So there are compromised mail servers harvesting email addresses …

11

this is also reminder why one should never expose any ports to public internet, ever.
Always use key-key VPN to your home and then use it as normal local network.

as others stated, start from scratch, do not use that system anymore as you never ever will know for sure if something is hidden somewhere.

12

I don’t want to sound too negative, but exposing the openHAB web UI to the Internet is just a bad idea. I doubt a professional penetration test was ever performed against the various UI’s that might be installed (e.g. dashboard, Paper UI, REST API, Basic UI, HABpanel, etc.). This attack surface is quite substantial, and it would not surprise me if there were multiple entry points where malicious code could be injected.

13

Dear Mark

I fully agree with you :slight_smile: , that it was not a good idea and another reason for me to speak that open and show to others what happened to me (even as an IT guy, being a bit too less restrict in securing my env).

Cheers
Stefan

1 Like
14

You do not know how networking works then. If no ports are exposed, you cannot communicate over the Internet.
Restricting port access through firewalls makes more sense.

15

To add some two cents to the topic, while IT security can deal with firewalls and port controls; passwords are also a over looked category. Sometimes the password is to make it easy, sometimes we set it to get the OS up quickly due to a lack of time with that infamous sticky note to remind us to change it yet never get around to doing so…

Wanted to point future readers to this as an option to just randomly generate your passwords via something like KeePass (I prefer to keep my passwords out of cloud services, and like the password + key file to unlock my passwords). https://keepass.info/

@smhaller like others have stated I would wipe that machine, but also keep this in mind a year or two ago I read some articles that malware was confirmed loaded on various firmwares of Macs; I don’t know if there is a way to verify if the infections have found places other than just the hard drive to hide making a wipe a waste of time if you just get re-infected.

1 Like
16

some people could understand the meaning of the sentence even without explicitly specifiing every single detail in it.
… so yes, exposing 22 or openhab port to public is or any other port which is not maintained by proper router configuration is wrong

17

And some, who are more paranoid, would take what you said at face value.
Things could have been communicated more clearly. The main function of language is clear communication.

18

Relax, guys. I agree that it’s good to be clear for the benefit of less-knowledgeable readers, but we can come at this more collaboratively. No need to attack each others’ comprehension or communication skills.

@smhaller, thanks for sharing this story. We all know that there are bad actors out there, but it’s a little more meaningful when it feels like something that could happen to us personally. Good luck getting your system back in working order!

4 Likes
19

Some questions and advice.

  • Can you tell which machine(s) are infected?

  • Search the IP addresses for both incoming and outgoing connections that the mysterious malware is making. At least one of those will be to a control server and if this is a known malware there might be something published about identifying and cleaning the malware from your machine associated with one of those IP addresses. Also check your DNS requests from this machine. You may need to set up WireShark on another machine to capture all the packets coming out of the compromised machine.

  • What kind of authentication did you have set up for your SSH server?

  • What kind of authentication did you have set up for your OH server (reverse proxy)?

:scream:

The two most common uses these days are adding a machine to a Denial of Service bot farm or crypto mining.

  • Someone can add a Rule through the REST API.
  • A Rule can call executeCommandLine.

You can guess the rest. It doesn’t really even matter if there are vulnerabilities in the UIs (there almost certainly are). OH is unsuitable to be put on the Internet without added encryption and authentication by design. One of it’s major use cases is itself a more than sufficient vulnerability to compromise a machine.

And this is why the openhab user needs to be a limited right user. And this is why you shouldn’t give the openhab user access to ssh to other machines or sudo permissions.

KeePass also has a plug-in that will let you put your SSH certificates into it and it acts like Pagent on a Windows machine.

Depending on how you set up your ssh access, that might actually be the more likely route of entry. I doubt that OH is popular enough yet to be worth building an automated tool to look for and compromise machines. Though that is always a possibility, and there might be some vulnerability with Jetty that could have been attacked without the attacker ever knowing or caring that they were attacking openHAB.

  • Did you have certificate only logins or did you allow paswords? Bonus if you have a password on the certificate.
  • You already posted that you are using a non-standard port which is good.
  • Fail2Ban?

For those looking to expose OH to the internet, just don’t do it. If you can’t or won’t use myopenhab.org, rent a virtual server somewhere and host your own instance of the Cloud Server. At a minimum, if you know how to monitor it for compromise, use a reverse proxy.

For those looking to use SSH:

  • Configure it to only allow logins using certificates, preferably put a password on your certificate. No password logins allowed.
  • Use a non-standard port which will avoid a lot of the automated dumb attacks
  • Set up Fail2Ban
  • Consider setting up a remote SSH proxy. How to access a Linux server behind NAT via reverse SSH tunnel

Also consider other options like VPN and remote management proxies like https://www.pitunnel.com/. Anything you can do to avoid opening a port in your firewall the better off you will be.

4 Likes
20

Dear @rlkoshak

Thank you for your very detailed post as usual.

Firewall was constantly active since begin, but by default all outgoing connections are open.
Also Fail2ban was installed from the very begin on.

Meanwhile I found something suspect and while using google I found the following:

I don’t know yet if my case is the same ground stuff but anyway there is some significant parallel.

My root crontab is looking like that

0 0 */3 * * /tmp/.rsync/a/upd>/dev/null 2>&1
5 8 * * 0 /tmp/.rsync/b/sync>/dev/null 2>&1 
@reboot /tmp/.rsync/b/sync>/dev/null 2>&1  
#5 1 * * * /tmp/.rsync/c/aptitude>/dev/null 2>&1

but rsync is also used in relation to Samba processes…

Cheers