Does this look like I have been hacked?

I recently upgraded to the 2.5.11.

Yesterday, I went to the greenhouse and the temps on the equipment had been reset to 100 degrees and the lights had been turned on. Not something we had done.

I spent some time last night working on some alerting. And looking through logs. Everything looked good when I signed off.

This morning, everything we reset to 100 and the lights were on again.

I found these entries in the logs:

This was the first odd entry. I had setup a way to record info at another farm of ours but we dont farm there now and I just haven’t turned off the code. It really doesn’t impact us. So this flipping to ON was very weird. And at midnight.

2021-01-20 23:42:22.370 [ome.event.ItemCommandEvent] - Item 'Midway_Pyganic_Trigger' received command ON

Right before that I had this entry in the openhab log:

2021-01-20 23:42:27.946 [ERROR] [ntime.internal.engine.RuleEngineImpl] - Rule 'Midway Nutrients Water Plants Counter': An error occurred during the script execution: Couldn't invoke 'assignValueTo' for feature JvmVoid:  (eProxyURI: midwaywater.rules#|::0.2.2.2.0.0::0::/1)

Again, not a big deal as we dont use this anymore and didn’t think it was something that would harm anything…

Then the weird things start happening…
All of the settings in the greenhouse started changing or being changed. Here is just one example:

2021-01-20 23:42:42.049 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 82
2021-01-20 23:42:42.086 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 81.0 to 82
2021-01-20 23:42:42.609 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 83
2021-01-20 23:42:42.634 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 82 to 83
2021-01-20 23:42:42.991 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 84
2021-01-20 23:42:43.014 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 83 to 84
2021-01-20 23:42:43.311 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 85
2021-01-20 23:42:43.332 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 84 to 85
2021-01-20 23:42:43.568 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 86
2021-01-20 23:42:43.620 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 85 to 86
2021-01-20 23:42:43.898 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 87
2021-01-20 23:42:43.920 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 86 to 87
2021-01-20 23:42:44.078 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 88
2021-01-20 23:42:44.113 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 87 to 88
2021-01-20 23:42:44.367 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 89
2021-01-20 23:42:44.389 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 88 to 89
2021-01-20 23:42:44.577 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 90
2021-01-20 23:42:44.601 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 89 to 90
2021-01-20 23:42:44.775 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 91
2021-01-20 23:42:44.792 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 90 to 91
2021-01-20 23:42:45.030 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 92
2021-01-20 23:42:45.055 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 91 to 92
2021-01-20 23:42:45.223 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 93
2021-01-20 23:42:45.250 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 92 to 93
2021-01-20 23:42:45.424 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 94
2021-01-20 23:42:45.443 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 93 to 94
2021-01-20 23:42:45.599 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 95
2021-01-20 23:42:45.626 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 94 to 95
2021-01-20 23:42:45.851 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 96
2021-01-20 23:42:45.880 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 95 to 96
2021-01-20 23:42:46.080 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 97
2021-01-20 23:42:46.108 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 96 to 97
2021-01-20 23:42:46.268 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 98
2021-01-20 23:42:46.301 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 97 to 98
2021-01-20 23:42:46.493 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 99
2021-01-20 23:42:46.515 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 98 to 99
2021-01-20 23:42:46.876 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 100
2021-01-20 23:42:46.895 [vent.ItemStateChangedEvent] - Shutter1_Thermostat_SP_F changed from 99 to 100
2021-01-20 23:42:47.900 [ome.event.ItemCommandEvent] - Item 'Shutter1_Thermostat_SP_F' received command 100
2021-01-20 23:42:49.718 [ome.event.ItemCommandEvent] - Item 'SingleSpeedFan_Thermostat_SP_F' received command 98
2021-01-20 23:42:49.752 [vent.ItemStateChangedEvent] - SingleSpeedFan_Thermostat_SP_F changed from 97.0 to 98
2021-01-20 23:42:50.255 [ome.event.ItemCommandEvent] - Item 'SingleSpeedFan_Thermostat_SP_F' received command 99
2021-01-20 23:42:50.271 [vent.ItemStateChangedEvent] - SingleSpeedFan_Thermostat_SP_F changed from 98 to 99
2021-01-20 23:42:57.563 [ome.event.ItemCommandEvent] - Item 'SingleSpeedFan_Thermostat_SP_F' received command 100
2021-01-20 23:42:57.594 [vent.ItemStateChangedEvent] - SingleSpeedFan_Thermostat_SP_F changed from 99 to 100
2021-01-20 23:42:58.996 [ome.event.ItemCommandEvent] - Item 'SingleSpeedFan_Thermostat_SP_F' received command 100

It gets reset until it gets to 100.

Just trying to determine if this is something that could be triggered in openhab or if this is a hack or a card issue?

I will take my lashes as I dont run much security on the system. This looks like it would be done through the app in some way or a bot? Wow, that was some work by someone for whatever reason. I need to have them code some good stuff for me…

I tried to download the auth.log file but its very large and would not come down. Yesterday when I looked through there you can see where there are various login attempts.

I plan to pull the card and rebuild today. Maybe on 3. I will work to put in better security but thought I would share for some feedback.

Thanks!

Is the machine openHAB is running on exposed to the internet (e.g. OpenVPN server, SSH, etc.)?

Is openHAB itself exposed to the Internet either directly or via a reverse proxy?

This is interesting. If those log in attempts are not you than your machine is under attack. That by itself may not be a big deal if they haven’t been successful but the weird behavior on your openHAB may be a sign that they have been successful.

That behavior could be triggered in openHAB if you have a rule or something like that that might cause it to run amok like that. But given the other evidence I don’t know.

  1. Shut down any port you have exposed to the Internet now.
  2. Review all of your systems for weird behaviors, failed login attempts that are not you, etc. Pay particular attention to machines that are not exposed to the Internet. If you see failed logins there or other weirdness that means they got into your network and all of your machines are suspect. There might be a back door there. The two most popular attacks against individuals are extortion (e.g. encrypt all your data and demand bitcoin to unencrypt) and crypto mining (watch the temp and CPU loads on your machines). This looks more like pranksters though.
  3. Rebuild and don’t expose stuff to the internet. Use a VPN. If you do expose stuff, monitor it like a hawk and set up stuff like Fail2Ban and maybe even snort. A DNS blocker like pihole can be useful too as it blocks the known bad destinations.
1 Like

Hey @rlkoshak thanks! That’s the direction I am working.

I agree, just weird that someone would hack and do that. They would have to know openhab I would think.

I use a Fing box to monitor things. I like it. Lets me know if the pi goes offline or the router goes down. I even monitor my “things” like my mqtt devices. I did a scan with the fing and it only finds three open ports; 53, 80, and 443. Would think those would be normal. I know i opened 3000 for grafana. That should be it.

Thanks for the insight.

It’s not that hard to figure out that you are running openHAB. You can search Shodan and find all sorts of people who have openHAB exposed to the internet without protection. From there it’s a quick google search to figure out what it is and how to interact with it.

Why would you expose a DNS server to the Internet? And what machines are those ports mapped to? Port 80 is http and port 443 is https and port 53 is usually DNS. If you opened Fing should show port 3000 too. If it doesn’t perhaps fing isn’t showing you what you think it is. Maybe it’s showing what your reouter/firewall has open and listening on your LAN (which would be normal really) and not what ports are open on the internet.

I can’t think of a single reason why you as an individual would want to expose a DNS server to the internet. And you definitely shouldn’t be exposing anything as HTTP and all your HTTPS should be password protected.

So right now I don’t trust that those fing scan results mean what you think they mean.

You can go to shodan and search for your public IP address and it will show you what you have exposed to the Internet.

These are a bit horrid to track down, but probably associated with some missing or uninitialized Item. Feels like that fits with your redundant site stuff.
It does tell you the rule - what does that look at, what state is that in?

I would say your errors and your unexpected commands happening in a close timeframe are way too much of coincidence, and there is some rule(s) problem in play.
If the rules are looking backwards using persistence, e.g. last months figures. it might be that your remaining “old data” is now out of range.

You could check logons to the device. If you SSH in you could try the below

cat /var/log/auth.log | grep 'Accepted'

Should show you logon times and from what IP they came from. Basic but might help

Hey @rlkoshak yeah I dont know on the DNS either. I am sure I was trying to run something on there that required it at one point. This server is not in my house so I am not as sensitive to what goes on there. But yes, should be a little more careful.

I did use shodan and it shows a different set of ports, all standard stuff, but not the 53. Go figure…

@rossko57 yes, I agree. Quite a coincidence. But still weird that it would just assign a value. Then start assigning values to other equipment. And, it just happened for the first time a day ago. And then a second time. Sure feels like a hack.

thanks @Gerry_Maguire. was wondering how to do that. I am going to look that up and see if i can find where it might have compromised.

I spent yesterday just rebuilding on a new machine and new card. Not going to load the things that were not really operational. Probably a bad idea to have them floating around. Also working through the security. I did have some questions on ssl that I didn’t see a good answer for on the forum so far so may have to post on that. I have the old machine/card still available so will still do some forensics and see what I can find out. I will post anything I can locate.

One thing that I keep working through is alerting. I know @rlkoshak you have posted some good information. I use the fing to tell me when the net drops or a device goes offline, well, off the network. Different than just not communicating through say mqtt. I need to setup lwt alerts. I did add the watchdog to the pi. Almost feel like I want to setup two units. One that runs everything and another that just alerts if something goes out of whack. Tied to a couple of separate sensors that check for out of line activity. Try to lock that one down really tight.

I will say this, that pi is rock solid. I know they can flake but you would not believe the conditions that thing has operated in for over three years. I have had one card failure. It just has a simple case that actually allows for connections for the gpio so its not really closed up. The temps where it sits runs over 100 degrees in the summer and down in the 30’s in the winter (we have heat but where it is it can drop below the heat settings). Its dusty, damp, vibrations from the fan next to it on the wall. Brutal for a computer. But it just keeps on going. Better to be lucky than good but right now its been both.

Same thing happening twice at similar times is what leads me to think it’s likely in-house rules/system. No matter, starting over clears it.

FYI, I recall that I added a Steinel Z-wave floodlight to my setup a while back and after a few days I started seeing “weird” behaviour with OH. Stuff turning on by itself, rules not working, etc.

At first glance I thought I’d been hacked too, but it turned out to be the Steinel device that was causing it all. When I removed it, everything returned to normal again. So, I’d take a look at any changes you may have made recently and see if they could be your root cause. You may not have been hacked at all here!

Keep in mind that a lot of that sort of stuff will largely be outside of the scope of openHAB so you won’t find a whole lot of information or help here on the forum. If it’s directly related to SSL and the openHAB server itself yes; if it’s SSL in general, probably not.

That’s a common approach. You often need something separate to monitor. But if your main concern is being hacked and detecting that (this should not be your only concern of course but it is what started this thread) than detecting when a system goes offline is probably not sufficient. The hackers won’t want your machine off the network any more than you do because then they lose access too.

There are services and tools that are designed for this sort of monitoring and alerting. It would be much better to deploy and use one of those instead of rolling your own. Nagios and Zabbix are two that I’ve seen mentioned and spent a little time looking into. These will tell you if a machine is offline or having troubles (disk full, CPU temp out of range, stuff like that). Snort and Bro are a couple of options for monitoring the network for malicious activity (kind of like antivirus for network traffic, only with way more false positives).

I personally run pfSense as my firewall and I’ve put Snort on that. The few services that I have exposed (OpenVPN, an SSH port, and Guacamole through HAProxy running on pfSense all the time, sometimes I’ll enable others temporarily) are monitored and I check the logs once a day to look for suspicious behavior and such. I’ve also configured the firewall to block devices that shouldn’t be reaching out to the Internet from being able to and limit where others can reach. Stuff I look for are weird login attempts, a device that has started using an unusual amount of network traffic, etc.