All lights and switches flipping state - very strange

Hammar · January 31, 2021, 3:17am

Configuration

Platform information:
- OS: Ubuntu 18.04.5 LTS
- Java Runtime Environment: build 11.0.9.1+1-Ubuntu-0ubuntu1.18.0
- openHAB version: 2.5.11 (Docker)

Issue of the topic
Occasionally, I have a situation where ALL my lights and switches flipped state (mostly ON, but the pond pump turned off for example) during the evening. This has happened approximately 4x in the last 8 weeks and it is difficult to diagnose. It has only happened since upgrading to OpenHab 2.5.11 but I think it is a coincidence.

This happened to all items, including Zwave, Zigbee and non-linked items.

What I have checked and can confirm:

nginx access and error log
I thought this might be an attack on the rest api as it is open via nginx to Alexa etc. However, my access log doesn’t show anything that concerns me and nothing at the start of the issue. There was a lot of network traffic during the 2 hours that his happened last night with Openhab and Fail2Ban sending and receiving a lot of network traffic.

Question: Is there a way to see what triggered the state change (rule, rest etc?)

System Metrics

The light blue container at the bottom is OpenHAB and the dark blue on top is Fail2Ban.

Fail2Ban didn’t pick anything unique up and I am at a loss to explain the increase in network traffic beyond OpenHAB issuing state changes.

Rules
I believe I have ruled (sic) this out as some items, for example the pond pump, aren’t in any of my rules

Logs
I am not sure there is a lot of value in posting the logs that I have as they clearly show the items turning on - but here is an example at the exact moment it happened:

Event Log

2021-01-31 03:57:54.870 [vent.ItemStateChangedEvent] - MyAir_AC2_Zone1_RoomTemp changed from 23.4 to 23.3
2021-01-31 03:58:02.865 [vent.ItemStateChangedEvent] - MyAir_AC2_Zone6_RoomTemp changed from 22.7 to 22.6
2021-01-31 03:58:23.666 [vent.ItemStateChangedEvent] - Date changed from 2021-01-31T03:57:23.661+1100 to 2021-01-31T03:58:23.664+1100
2021-01-31 03:58:23.667 [vent.ItemStateChangedEvent] - DateString changed from 2021-01-31 03:57:23 AEDT to 2021-01-31 03:58:23 AEDT
2021-01-31 03:58:43.469 [ome.event.ItemCommandEvent] - Item 'Light_FG_Porch' received command ON
2021-01-31 03:58:43.477 [nt.ItemStatePredictedEvent] - Light_FG_Porch predicted to become ON
2021-01-31 03:58:43.482 [vent.ItemStateChangedEvent] - Light_FG_Porch changed from OFF to ON
2021-01-31 03:58:43.998 [ome.event.ItemCommandEvent] - Item 'Light_FG_Stairs' received command ON
2021-01-31 03:58:44.001 [nt.ItemStatePredictedEvent] - Light_FG_Stairs predicted to become ON
2021-01-31 03:58:44.010 [vent.ItemStateChangedEvent] - Light_FG_Stairs changed from OFF to ON
2021-01-31 03:58:44.745 [ome.event.ItemCommandEvent] - Item 'Light_BF_Driveway' received command ON
2021-01-31 03:58:44.748 [nt.ItemStatePredictedEvent] - Light_BF_Driveway predicted to become ON
2021-01-31 03:58:44.759 [vent.ItemStateChangedEvent] - Light_BF_Driveway changed from OFF to ON
2021-01-31 03:58:46.494 [ome.event.ItemCommandEvent] - Item 'Pond_Pump' received command OFF
2021-01-31 03:58:46.497 [nt.ItemStatePredictedEvent] - Pond_Pump predicted to become OFF
2021-01-31 03:58:46.504 [vent.ItemStateChangedEvent] - Pond_Pump changed from ON to OFF
2021-01-31 03:58:47.561 [ome.event.ItemCommandEvent] - Item 'Light_FG_BinAlley' received command OFF
2021-01-31 03:58:47.566 [nt.ItemStatePredictedEvent] - Light_FG_BinAlley predicted to become OFF
2021-01-31 03:58:47.580 [vent.ItemStateChangedEvent] - Light_FG_BinAlley changed from ON to OFF
2021-01-31 03:58:48.572 [ome.event.ItemCommandEvent] - Item 'Light_FG_Side' received command OFF
2021-01-31 03:58:48.574 [nt.ItemStatePredictedEvent] - Light_FG_Side predicted to become OFF
2021-01-31 03:58:48.586 [vent.ItemStateChangedEvent] - Light_FG_Side changed from ON to OFF
2021-01-31 03:58:51.690 [ome.event.ItemCommandEvent] - Item 'Light_FG_Gate' received command OFF
2021-01-31 03:58:51.692 [nt.ItemStatePredictedEvent] - Light_FG_Gate predicted to become OFF
2021-01-31 03:58:51.699 [vent.ItemStateChangedEvent] - Light_FG_Gate changed from ON to OFF
2021-01-31 03:58:55.558 [ome.event.ItemCommandEvent] - Item 'Doorbell_OpenDoor2' received command ON
2021-01-31 03:58:55.560 [nt.ItemStatePredictedEvent] - Doorbell_OpenDoor2 predicted to become ON
2021-01-31 03:58:55.566 [vent.ItemStateChangedEvent] - Doorbell_OpenDoor2 changed from OFF to ON
2021-01-31 03:59:01.347 [ome.event.ItemCommandEvent] - Item 'Doorbell_OpenDoor2' received command OFF
2021-01-31 03:59:01.352 [nt.ItemStatePredictedEvent] - Doorbell_OpenDoor2 predicted to become OFF
2021-01-31 03:59:01.362 [vent.ItemStateChangedEvent] - Doorbell_OpenDoor2 changed from ON to OFF
2021-01-31 03:59:05.328 [ome.event.ItemCommandEvent] - Item 'Light_FG_Porch' received command OFF
2021-01-31 03:59:05.331 [nt.ItemStatePredictedEvent] - Light_FG_Porch predicted to become OFF
2021-01-31 03:59:05.340 [vent.ItemStateChangedEvent] - Light_FG_Porch changed from ON to OFF
2021-01-31 03:59:05.828 [ome.event.ItemCommandEvent] - Item 'Light_FG_Stairs' received command OFF
2021-01-31 03:59:05.830 [nt.ItemStatePredictedEvent] - Light_FG_Stairs predicted to become OFF
2021-01-31 03:59:05.839 [vent.ItemStateChangedEvent] - Light_FG_Stairs changed from ON to OFF
2021-01-31 03:59:07.306 [ome.event.ItemCommandEvent] - Item 'Light_BF_Driveway' received command OFF
2021-01-31 03:59:07.310 [nt.ItemStatePredictedEvent] - Light_BF_Driveway predicted to become OFF
2021-01-31 03:59:07.323 [vent.ItemStateChangedEvent] - Light_BF_Driveway changed from ON to OFF

What I plan to do
(1) Change all passwords but I can’t find evidence of an attack.
(2) Upgrade to 2.5.12 due to security incidents resolved in it

Any help or direction appreciated.

Thanks

BobMiles · January 31, 2021, 7:00am

Hi,

hmm sounds a little familiar to: HELP! Someone Hacked my system? Lights and other turning on and off randomly. OH3 Snapshot - #26 by openhab2

Do you have any ports open to the outer world?

Other than that, could you check your thing activity before the incident? I had things reinitialize during the night and somehow the items flipped to NULL and back, which triggered some of my rules…

I think this is a very interesting hint:

This happened to all items, including Zwave, Zigbee and non-linked items.

Hammar · January 31, 2021, 7:36am

I have a couple of ports open (80 and 443) all routed through nginx. I have checked my ip routing tables and nginx can’t be bypassed.

rlkoshak · February 1, 2021, 6:02pm

And have you checked the auth log for nginx?

It may not be the result of an attack but if you don’t audit the logs from nginx it might be compromised and you’d never know. It’s not enough to just set up a service like nginx and add a username and password. You have to constantly monitor it too.

Beyond that you’ll have to look at your logs and rules to try and find anything that looks a little off right before and during the time that the weird behaviors occurred.

Hammar · February 5, 2021, 6:45am

Thanks. I can’t find anything unexpected in the nginx logs.

There are items changing state that aren’t even attached to rules. I have grep’ed all my rules and those items aren’t any there.

Is there any way to increase logging to see what process/action/etc triggered that item state change?

rlkoshak · February 5, 2021, 3:12pm

Not really. You can increase the logging on various parts of OH and then compare time stamps. But the origin of an event is not part of the event.

Hammar · February 11, 2021, 9:40pm

I found the issue and it is a little embarrassing.

I had switched ISPs and there was an open http port on their side that was blocked with my previous ISP. I had previously configured this port for test purposes. It allowed bots to hit my sitemaps without authentication.

Thanks for the help everyone

rossko57 · February 11, 2021, 9:47pm

That’s a good find though, that there are bots out there targeting OH by design or accident.