Hacked after opening up port 8080 - any other way to expose Ipcamera binding externally?

jamesfawcett · November 14, 2022, 11:30am

Platform information:
- Hardware: CPUArchitecture/RAM/storage 32GB SD Card Openhabian
  Z-wave devices with z-wave bindings
  IPCamera bindings
  All default passwords changed
- Java Runtime Environment: which java platform is used and what version
- openHAB version: Latest 0H3
Issue of the topic:

I have a standard OpenHabian install with z-wave devices. For years, since OH1, I have used Linux Motion to track camera motion and alerts and to generate the compressed camera feeds. I displayed the Motion camera feeds on a basic html page served by nginx on a nonstandard port.

Last week I opened port 8080 on my router after setting up IPCamera bindings for Hikvision cameras. Today I checked my logs and it looks like I was compromised. Every couple of seconds there’s a message like this:

2022-11-14 09:12:22.931 [WARN ] [org.eclipse.jetty.server.HttpChannel] - qx.zpuszee.cn:443
java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 17
	at java.lang.String.checkBoundsBeginEnd(String.java:3319) ~[?:?]
	at java.lang.String.substring(String.java:1874) ~[?:?]
	at org.ops4j.pax.web.service.spi.model.ServerModel.matchPathToContext(ServerModel.java:370) ~[?:?]
	at org.ops4j.pax.web.service.spi.model.ServerModel.matchPathToContext(ServerModel.java:310) ~[?:?]
	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:73) ~[?:?]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.46.v20220331]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2022-11-14 09:12:23.413 [WARN ] [org.eclipse.jetty.server.HttpChannel] - qx.zpuszee.cn:443
java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 17
	at java.lang.String.checkBoundsBeginEnd(String.java:3319) ~[?:?]
	at java.lang.String.substring(String.java:1874) ~[?:?]
	at org.ops4j.pax.web.service.spi.model.ServerModel.matchPathToContext(ServerModel.java:370) ~[?:?]
	at org.ops4j.pax.web.service.spi.model.ServerModel.matchPathToContext(ServerModel.java:310) ~[?:?]
	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:73) ~[?:?]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.46.v20220331]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2022-11-14 09:12:23.654 [WARN ] [org.eclipse.jetty.server.HttpChannel] - www.habbocity.me:443
java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 20
	at java.lang.String.checkBoundsBeginEnd(String.java:3319) ~[?:?]
	at java.lang.String.substring(String.java:1874) ~[?:?]
	at org.ops4j.pax.web.service.spi.model.ServerModel.matchPathToContext(ServerModel.java:370) ~[?:?]
	at org.ops4j.pax.web.service.spi.model.ServerModel.matchPathToContext(ServerModel.java:310) ~[?:?]
	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:73) ~[?:?]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.46.v20220331]
	at java.lang.Thread.run(Thread.java:829) [?:?]

My only other recent change was trying another HikVision method to get the cameras working, which involved installing:

npm install git+lshttps://github.com/ragingcomputer/node-hikvision-api.git

This wasn’t successful, but I didn’t uninstall this either.

To allow viewing my cameras externally, I had to open up port 8080 and point it to my Raspberry Pi. The camera url’s look like this:

http://{URL}:8080/ipcamera/71f23c503d/ipcamera.mjpeg

The problem I now realise, is that I’ve also displayed my full OpenHab dashboard at :8080 too, even though it does have the admin password set.

If anyone has any suggestions on how this could have happened I would be grateful to know, or any other suggestions how to change the external port for the camera feeds.

Thank you

puterfixer · November 14, 2022, 2:12pm

Do not ever, EVER expose your internal network to the internet through port forwarding. That may have been a solution in the internet’s naive times, when trust was high, security breaches were not so frequent and malicious traffic didn’t make up a significant portion of the internet traffic overall. Those times are gone. Plus, we have better tools at consumer level disposal.

Case in point: network-enabled Chinese products are notoriously bad at security. This wasn’t such a big issue when cameras were analogue and only the DVR was network-connected, but nowadays even cameras are IP-based and their security S U C K S. Just a couple of years ago, a worm was discovered to exploit such video surveillance devices, which apparently had secret admin passwords hardcoded into the firmware. The worm was able to obtain full admin privileges to any such device, and even burn a modified firmware into them which included the worm and added the device to a remotely controlled botnet. Since pretty much nobody ever checks on these devices (install and forget), this attack vector was low on the radar. But here we are, knowing that any Hikvision DVR/NVR exposed to the Internet can be entered with a default admin password from the factory, different than what you set in the admin interface for the admin user. Same for hundreds of other manufacturers, but I’m naming Hikvision as they’re among the top 10 video surveillance manufacturers. Dahua also has its own admin bypass mechanism, although that’s not a fixed password but a number generated with a formula based on current date. (That’s also the official admin password recovery method, since the Dahua DVRs don’t have a factory reset button or jumper or any other means to reset a forgotten admin password.)

Someone security conscious wouldn’t even put these security devices on the same internal subnet as the rest of your home computers/phones, but on a separate VLAN with highly restricted access permissions. But I digress.

Nowadays, to securely access your home network’s systems from the internet, you should establish a communication channel that firstly checks your identity and permissions (authentication and authorization), before you can reach any of those internal systems’ own login page. That’s done through a VPN tunnel to your home router. Most of them offer this capability, and you should use it. VPN into your home router, then access your DVR cameras as if you’re at home. It is an extra step, however as the internet matured, we have also learned that convenience is opposing security.

jamesfawcett · November 14, 2022, 5:43pm

Thanks for your reply, I use OpenVPN to connect to my home network, but the remote machines I use I can’t use an OpenVPN client.

I managed to solve my issue by adding another nginx location for each of the camera feeds which are generated by the IPCamera binding on port 8080.

location /camera1 {
                proxy_pass  http://127.0.0.1:8080/ipcamera/503d71f23c/ipcamera.mjpeg;
        }

        location /camera2 {
                proxy_pass  http://127.0.0.1:8080/ipcamera/0db87395c5/ipcamera.mjpeg;
        }

        location /camera3 {
                proxy_pass  http://127.0.0.1:8080/ipcamera/1d4ea39fe3/ipcamera.mjpeg;
        }

        location /camera4 {
                proxy_pass  http://127.0.0.1:8080/ipcamera/e554f30a28/ipcamera.mjpeg;
        }

I do still have my non-standard http port forwarded to my Raspberry PI though for the nginx server. I will look for a safer solution.

If anyone has any further tips on how the compromise could have happened earlier please let me know.

Wolfgang_S · November 14, 2022, 6:33pm

e.g. CVE - CVE-2021-36260 of course depending on your product and firmware version.

jamesfawcett · November 14, 2022, 6:50pm

Thanks for the link - I didn’t realise this affected HikVision cameras without an NVR too! That’s looking very likely to be the cause then! A couple of the cameras are 7+ years old, but all are on the latest firmware.

Thanks for your help

puterfixer · November 15, 2022, 3:35pm

An HTTP(S) proxy can also protect vulnerable backend systems by stripping down the intrusion attempts sent through HTTP(S) GET/POST requests. If you can set up nginx to 1) require authentication and 2) completely ignore all parameters in the URL and always send back the stream, then the security holes in DVR/NVR/IP camera firmware cannot be exploited.

maDDin1338 · November 15, 2022, 3:41pm

you could also think about:

jamesfawcett · December 22, 2022, 9:54pm

I may be being paranoid… but… I am not sure if I’ve been hacked again.

I’ve been super careful this time, set up from OpenHabian, only trusted z-wave devices, Hikvision cameras on latest firmware and one TP-Link camera.

Something has happened and its being extremely slow, and in the logs I get errors like this repeated. The part that concerns me is the “$Proxy333”. Has anyone seen anything like this before, or know where I can locate and trace the $Proxy333" message?

2022-12-22 21:46:34.554 [WARN ] [ore.internal.scheduler.SchedulerImpl] - Scheduled job '<unknown>' failed and stopped
java.lang.reflect.UndeclaredThrowableException: null
	at com.sun.proxy.$Proxy333.apply(Unknown Source) ~[?:?]
	at org.openhab.core.model.script.actions.ScriptExecution.lambda$0(ScriptExecution.java:97) ~[?:?]
	at org.openhab.core.internal.scheduler.SchedulerImpl.lambda$12(SchedulerImpl.java:191) ~[?:?]
	at org.openhab.core.internal.scheduler.SchedulerImpl.lambda$1(SchedulerImpl.java:88) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.openhab.core.model.script.engine.ScriptExecutionException: The name '<unkown>' cannot be resolved to an item or type; line 0, column 0

Wolfgang_S · December 23, 2022, 8:55am

see thread Dimmer rule between scene light [SOLVED] there $Proxy333 appears as well in one of the log messages.

jamesfawcett · December 23, 2022, 10:18am

Thank you Wolfgang_S!

I guess I am just being super paranoid now since the compromise, and I really don’t want to lose the heating rules over Christmas!

Thanks for finding that which has put my mind at ease!