You do not know how networking works then. If no ports are exposed, you cannot communicate over the Internet.
Restricting port access through firewalls makes more sense.
To add some two cents to the topic, while IT security can deal with firewalls and port controls; passwords are also a over looked category. Sometimes the password is to make it easy, sometimes we set it to get the OS up quickly due to a lack of time with that infamous sticky note to remind us to change it yet never get around to doing so…
Wanted to point future readers to this as an option to just randomly generate your passwords via something like KeePass (I prefer to keep my passwords out of cloud services, and like the password + key file to unlock my passwords). https://keepass.info/
@smhaller like others have stated I would wipe that machine, but also keep this in mind a year or two ago I read some articles that malware was confirmed loaded on various firmwares of Macs; I don’t know if there is a way to verify if the infections have found places other than just the hard drive to hide making a wipe a waste of time if you just get re-infected.
1 Like
some people could understand the meaning of the sentence even without explicitly specifiing every single detail in it.
… so yes, exposing 22 or openhab port to public is or any other port which is not maintained by proper router configuration is wrong
And some, who are more paranoid, would take what you said at face value.
Things could have been communicated more clearly. The main function of language is clear communication.
Relax, guys. I agree that it’s good to be clear for the benefit of less-knowledgeable readers, but we can come at this more collaboratively. No need to attack each others’ comprehension or communication skills.
@smhaller, thanks for sharing this story. We all know that there are bad actors out there, but it’s a little more meaningful when it feels like something that could happen to us personally. Good luck getting your system back in working order!
4 Likes
Some questions and advice.
-
Can you tell which machine(s) are infected?
-
Search the IP addresses for both incoming and outgoing connections that the mysterious malware is making. At least one of those will be to a control server and if this is a known malware there might be something published about identifying and cleaning the malware from your machine associated with one of those IP addresses. Also check your DNS requests from this machine. You may need to set up WireShark on another machine to capture all the packets coming out of the compromised machine.
-
What kind of authentication did you have set up for your SSH server?
-
What kind of authentication did you have set up for your OH server (reverse proxy)?
The two most common uses these days are adding a machine to a Denial of Service bot farm or crypto mining.
- Someone can add a Rule through the REST API.
- A Rule can call executeCommandLine.
You can guess the rest. It doesn’t really even matter if there are vulnerabilities in the UIs (there almost certainly are). OH is unsuitable to be put on the Internet without added encryption and authentication by design. One of it’s major use cases is itself a more than sufficient vulnerability to compromise a machine.
And this is why the openhab user needs to be a limited right user. And this is why you shouldn’t give the openhab user access to ssh to other machines or sudo permissions.
KeePass also has a plug-in that will let you put your SSH certificates into it and it acts like Pagent on a Windows machine.
Depending on how you set up your ssh access, that might actually be the more likely route of entry. I doubt that OH is popular enough yet to be worth building an automated tool to look for and compromise machines. Though that is always a possibility, and there might be some vulnerability with Jetty that could have been attacked without the attacker ever knowing or caring that they were attacking openHAB.
- Did you have certificate only logins or did you allow paswords? Bonus if you have a password on the certificate.
- You already posted that you are using a non-standard port which is good.
- Fail2Ban?
For those looking to expose OH to the internet, just don’t do it. If you can’t or won’t use myopenhab.org, rent a virtual server somewhere and host your own instance of the Cloud Server. At a minimum, if you know how to monitor it for compromise, use a reverse proxy.
For those looking to use SSH:
- Configure it to only allow logins using certificates, preferably put a password on your certificate. No password logins allowed.
- Use a non-standard port which will avoid a lot of the automated dumb attacks
- Set up Fail2Ban
- Consider setting up a remote SSH proxy. How to access a Linux server behind NAT via reverse SSH tunnel
Also consider other options like VPN and remote management proxies like https://www.pitunnel.com/. Anything you can do to avoid opening a port in your firewall the better off you will be.
4 Likes
Dear @rlkoshak
Thank you for your very detailed post as usual.
Firewall was constantly active since begin, but by default all outgoing connections are open.
Also Fail2ban was installed from the very begin on.
Meanwhile I found something suspect and while using google I found the following:
I don’t know yet if my case is the same ground stuff but anyway there is some significant parallel.
My root crontab is looking like that
0 0 */3 * * /tmp/.rsync/a/upd>/dev/null 2>&1
5 8 * * 0 /tmp/.rsync/b/sync>/dev/null 2>&1
@reboot /tmp/.rsync/b/sync>/dev/null 2>&1
#5 1 * * * /tmp/.rsync/c/aptitude>/dev/null 2>&1
but rsync is also used in relation to Samba processes…
Cheers
Yes but that’s not the rsync command. That is some hidden folder in /tmp named .rsync. They just named it .rsync to fool you, or they are running some rsync process for it’s own nefarious purposes. I run samba on a couple of machines and there is no /tmp/.rsync folder and there are no entries in the root crontab.
This definitely points to either a virus or signs of compromise. I searched for the command in the crontavb and found a number of posts talking about it. This Reddit thread seemed to have the best advice.
It could be fun to research this and figure out what it’s doing and how it works, but ultimately you can never ever trust this machine again. You should trash the drive and start afresh. If you are very paranoid, switch SD cards/hard drive. But there is some benefit in researching this so you can figure out how it got on your machine. If openHAB users are being targeted it could help us figure out how to protect ourselves.
Dear all
I found a further one. mentioned crontab entries in this Thread, were found in crontab of openhab user.
I will further dig into this. I wonder what is next to be found.
Worth to mention is also that, I found my system using unusual high CPU load od 100% more ore less constantly. The same is reported in other threads regarding this subject.
Update:
while still analysing I found that fail2ban was continue to report:
2019-07-31 09:20:49,286 fail2ban.filter [1105]: INFO [sshd] Ignore 127.0.0.1 by ip
2019-07-31 09:20:49,293 fail2ban.filter [1105]: INFO [sshd] Ignore 127.0.0.1 by ip
2019-07-31 09:20:51,447 fail2ban.filter [1105]: INFO [sshd] Ignore 127.0.0.1 by ip
Also I found following suspect auth.log:
Jul 29 19:47:56 sshd[19184]: pam_unix(sshd:auth): check pass; user unknown
Jul 29 19:47:56 sshd[19184]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=127.0.0.1
Jul 29 19:47:59 sshd[19184]: Failed password for invalid user administrator from 127.0.0.1 port 42286 ssh2
Jul 29 19:49:02 su[19068]: pam_unix(su:session): session closed for user openhab
Jul 29 20:03:14 sshd[20404]: Invalid user test from 127.0.0.1 port 42940
Jul 29 20:03:14 sshd[20404]: input_userauth_request: invalid user test [preauth]
Jul 29 20:03:14 sshd[20404]: pam_unix(sshd:auth): check pass; user unknown
Jul 29 20:03:14 sshd[20404]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=127.0.0.1
Jul 29 20:03:15 sshd[20404]: Failed password for invalid user test from 127.0.0.1 port 42940 ssh2
mail.log: Exim ?? I never used it.
Jul 28 00:07:05 exim[32411]: 2019-07-28 00:07:05 1hrUqH-0008Ql-AE <= root@orlando.homeland U=root P=local S=679
Jul 28 00:07:05 exim[32411]: 2019-07-28 00:07:05 1hrUqH-0008Ql-AE Cannot open main log file "/var/log/exim4/mainlog": Permission denied: euid=106 egid=114
Jul 28 00:07:05 exim[32411]: exim: could not open panic log - aborting: see message(s) above
Jul 31 09:06:09 exim[1497]: 2019-07-31 09:06:09 exim 4.89 daemon started: pid=1497, -q30m, listening for SMTP on [127.0.0.1]:25 [::1]:25
Jul 31 09:06:09 exim[1497]: 2019-07-31 09:06:09 Cannot open main log file "/var/log/exim4/mainlog": Permission denied: euid=106 egid=114
Jul 31 09:06:09 exim[1497]: exim: could not open panic log - aborting: see message(s) above
Jul 31 09:27:15 exim[1385]: 2019-07-31 09:27:15 exim 4.89 daemon started: pid=1385, -q30m, listening for SMTP on [127.0.0.1]:25 [::1]:25
Jul 31 09:27:15 exim[1385]: 2019-07-31 09:27:15 Cannot open main log file "/var/log/exim4/mainlog": Permission denied: euid=106 egid=114
Jul 31 09:27:15 exim[1385]: exim: could not open panic log - aborting: see message(s) above
What crap is that…
But after I removed all suspect crontab entries there were no more reports about such susbect entries found and in fact the last try of outbound ssh connection was reported at 09:20
Jul 31 09:20:32 kernel: [ 944.973810] [UFW BLOCK] IN= OUT=eth0 SRC=192.168.223.200 DST=5.255.86.129 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=30613 DF PROTO=TCP SPT=49442 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0
tcpdump is still running, but nothing reported so far.
CPU usage is back to normal (1% to 5%)
After some more research I found the entry on trendmicro that is pretty much what I’ve got here.
Cheers
Stefan
In addition, since myopenhab.org code is opensource too. You could get your own cloud vps, not expose anything to it other than the myopenhab code and abstract again.
Overall thought here (being a server admin as my day job)
The box has been compromised. Take your openhab configs and do a fresh install. If it’s a VM, hopefully you have a backup, remove the external access and restore from backup. If you don’t think the backups are clean, restore with no nics active. I would not advise trying to clean it up.
- my less than 2 cents lol
1 Like
Dear Jason
I will definitely do so (as soon as i have time for).
Cheers
I’d strongly recommend turning off the machine until it can be refreshed.
1 Like
Also, use geo IP filtering to block all connections from countries in which you do not need to connect from. I use free pfsense firewall and drastically reduced attacks by only allowing USA or North American with pfsense Geo-IP blocking.
1 Like
Opening up your openhab instance to internet is never a good idea.
You have not followed general guide lines for setting up something somewhat secured.
Use VPN (openvpn, wireguard etc) and only allow connections to your openhab instance from the local network.
If you expose ssh to the outside world I would:
- Use a non standard port (I know you did this)
- Use a non standard username, and only allow that username to connect
- Use a key-file rather then a password to connect (configure ssh to not accept passwords)
- Use geo-ip blocking, be restrictive.
Regards S
+1! 
I been using KeePass for years, and sync it across devices via syncthing. Version 2 somehow allows you to simultaneously edit your database from different locations and then automatically reconcile the differences(!). It’s a really nice piece of software, and native clients available for every platform you can think of.
Another really nice piece of kit, that I also use!
(sorry for double post)
for this type of exposure i would not recommend to allow any passwords at all. Personal keys are way more secure than this.
Which brings us back to proper key-key VPN connection to your home and then work as you are at home.
harder to set but way safer
1 Like
Gents and ladies
to those who use a VPN to connect, what would you actually suggest? Or what are you using?
I actually had a look at ExpressVPN, NordVPN and also OpenVPN, but I’m really not sure what to go for.
Cheers
Stefan
depends on your infrastructure.
if you are using router which is flashed with openwrt or something like that, openvpn is way to go
if you have something else, depends on how much time and effort you wanna put into it
I’m using openvpn and key-key on each device i want to have to be able to connect. phone, notebook, tablet each one has got own key which i can invalidate anytime and as well I do have forbidden password login for obvious reasons. 