I just want to share this with you. Maybe you have a closer look at the Cloud-Service. First to be clear I am not a linux security expert!
Since one year I use openHab with its cloud service. Yesterday my wife called me: “all lights are on and its not possible to control them” (She did not turn on the lights).
Short facts:
The openHab tasks are not running
Restart does not work (also systemrestart)
Manual start does not work
apt-get upgrade does not work anymore (mysterious)
“top” does not work anymore !! (mysterious)
not one single log entry in openHab since the problem rises
the raspberry pi has restared before this hole problem appears (seen into the linux logs)
automatic updates are not enabled (hole system)
only two things are installed: Oracle Java and openHab nothing more!
the system is behind a FritzBox without any special network rules its not possible to reach the system from the internet, only via openHab cloud
the pi runs for sure not with its standard password! (maybe the reason that nothing more happend)
In a conclusion since one year the system runs like a swiss clockwork. And now many weird things happend. In my opinion a hardware defect is unlikely. There was no power failure. Maybe there is a possiblility that the openHab cloud service was a gateway to get into the pi to compromise it!? I reinstalled everything. And everything is working fine now. Sorry that I dont have the time to work out in detail what happend. I only want to share this with you and I have to say thats mysterious. Maybe there is a securityproblem into the cloud.
Indeed given every experience I’ve ever had with SD and microSD cards and even USB sticks for storage and as boot drives, I can’t for the life of me understand why anyone would use one as a HDD
But then I should read something about “disk failure”, “IO exception”, “write error”, “read error” or? And your experience is after one year with a operating system on a SD card (expensive sandisk) thats normal? If that is your very likely experience, then I have to drop all my SD card systems!?
benhelps: Cause thats the way from the raspberry guys…
Have you checked your SD card in another computer there are solutions for windows and linux (google). The symptoms you describe are consistent with a typical Raspi SD card failure. No matter how expensive the card, it WILL fail. It only needs to be a critical block in the card and your whole system can go down.
No sorry, I told you that its reinstalled and in use again. But Windows was reading the boot partition fine before I formated it. Also not the hole system was down… I logged on via SSH I was watching one hour what happend before I decided to reinstall. Funny that top was’nt working but htop fine… nano also fine… if I would read only one single row about problems with the file system I would totally agree with your “very likely” guess that the hardware is the problem. But at the end there is a bad feeling. Thats what I want to share with you. If the hole experiance from the community says: Thats a SD card problem, than I am fine.
Again after the SD card has problems (your assessment) openHab decide to activate only all tradfri lamps (no ZWave systems) and then linux decide to drop top, rise apt-get problems and would not run openHab…!? And not one system has a log that give a clear statement?
No you wouldn’t and that is another really big reason (IMHO the biggest reason) why running off of an SD card is a bad idea. When an SD card wears out the system continues on thinking that its writes are working but when you look at the files:
deleted files come back
changes saved to file disappear on a reboot
files won’t delete
changes made to files revert mysteriously
unexplained weird behavior
Depends on the size of the SD card and the amount of writes that are occuring over time. But the answer is yes. For an 8 Gig SD card running OH with default logging and persistence on the SD card, a year sounds about right. It isn’t an exact science but based on the many many reports of failures on this forum.
when an SD card wears out the problem is that the operating system continues running along happily writing to the SD card and for a time everything is hunky dory because the writes are cached in memory while waiting to write it to the SD card. Then it tries to write it to the SD card and the SD card says “Got it, it’s written.” But the SD card is a lying liar that lies. It tried to write to the SD card, maybe it was successful for part of the writes. But ultimately the files written on the SD card itself are corrupted. Later on, when some service or program tries to read those files they read corrupted files. Sometimes the corruption is subtle, sometimes catastrophic. The result is weird inexplicable behavior.
From a personal experience, I had an RPi with a camera running motion and a little service I wrote to control and report the status of my garage doors. The SD card on it failed but the first time I discovered the problem was when I discovered that my syslog wasn’t rotating. It just kept growing until it filled the card. I could delete the file but every time I rebooted the file came back. Motion kept sending fantom motion events. Updates to the config of my service the changes would revert.
I’ve seen even weirder stuff reported on this forum (search for “SD Failure” and you will find dozens of threads).
Now to address your security concerns. Let’s assume that myopenhab.org were compromised. The only thing that gives the hacker access to is your OH REST API. So to do anything on your RPi they would have to:
compromise myopenhab.org to gain access to the REST API of your openHAB
compromise openHAB through it’s REST API in such a way that the hacker gains access to your machine
compromise your RPi to elevate their privileges from the openhab user to root (the only way they can do anything that would cause problems with top), unless you are running openHAB as root
Is this in the realm of possible, sure. But it would be very very challenging to do so quietly and it would take a whole lot of work and require at least three separate exploits against at least three separate vulnerabilities. This is what is often referred to as a “state level attack.” So unless you are personally an enemy of Russia or the like it is unlikely that you were hacked.
And even if you were hacked, given the challenging nature of that hack, all the hacker does is some petty vandalism? Again, highly unlikely. Any attack that requires this much work will be done by a motivated group or hackers who will most likely be motivated by profit. There is no profit in petty vandalism. They would be more likely to set up surveilance so they can perform identity theft or join your machines up to a botnet for SPAM or DDoS attacks.
Given the behavior you describe is explainable by a worn out SD card (keep on eye on your rebuilt system, reformatting does not fix the worn out portions of the card) Occam’s Razer points to that being the simpler explanation.
No one can say for sure though without a forensic analysis of your SD card which you rendered impossible by reformatting and reinstalling. But IMHO the likelihood that you were hacked through myopenhab.org is so low as to not be worth the analysis without further evidence to indicate it is a likelihood.
And while I’m not offering professional advice here or anything like that, I am a computer security professional so I’m not just talking. I do have some knowledge and experience in this area.
Than I will change to another system to run my openHab cause if I would do my projects how I would prioritize them, a garagedoor would be unrecognized opened - and the normal murphy law says: This happend when you are on holidays. And then the OH community should scream out: Don’t DON’T use a PI (or something else) with an OS on a SD card!! Cause if I have to guess >80% did!
Just one more small thought:
If you think you can build a big community like OH with a cloud service and hundred thousands of RPi’s (and other systems) don’t feel save that the work what you are talking about above to hack this cloud won’t be done! That’s money for the right people - much money! And another thought: Think about how many OH-Users has the standard RPi user/password and the cloud service active!? I am really glad that they dissabled the ssh!
That is one option but not the only one. You can run the RPi off of an external HDD/SDD. You can move the expensive writes to a network shared drive. You can live with the unreliability of SD cards and have a good backup and restore process (you need this anyway no matter your solution). Just use a name brand SD card with lots of spare room (16 GB or more) in this case.
Again, search the forum and you will find dozens of threads with lots of approaches to mitigate this problem. The biggest thing you can do is to move the openHAB logs and persistence off of the SD card. The remaining writes are small enough that an SD card will last a very long time.
The number of users of myopenhab.org, in the scheme of things, is really small. In the tens to hundreds of thousands of users. I’m not going to be one to say that myopenhab.org wouldn’t be attacked some day or that it already hasn’t been attacked. And the maintainers use (as far as I can tell) industry best practices to secure the service. The security of myopenhab is taken seriously.
But looking at it from the attacker’s perspective, am I going to spend the effort to find and exploit three different vulnerabilities to get on a system that is likely just a RPi with no financial information on it or am I going to spend my effort compromising the millions of poorly secured IoT devices (IP cameras and the like) that are directly exposed to the internet with a default password?
Again, the risk isn’t zero, but it is far far less than exposing your RPi to the Internet directly.
Irrelevant. And if they are using openHABian, they are not using the standard user/password.
Anyway, the only connection between myopenhab.org and your openHAB is through the Cloud Connector. The Cloud Connector only allows the REST API traffic to pass through. So the only way the attacker can get on your system is by causing openHAB to crash or causing openHAB to run something on the attacker’s behalf (this is the more likely scenario).
But if you are running openHAB in the recommended configuration that means, in either case, the attacker can only run things as the openhab user. The openhab user has no password so unless the user has gone out of their way to give the openhab user sudoer permission without password the attacker can’t become root. And if they did give the openhab user sudoer permission without password, then it doesn’t matter what your login user and password is.
Again, ssh is irrelevant. If an attacker is going to be successful attacking you through myopenhab.org then the login username and password doesn’t matter and whether or not sshd is running doesn’t matter. They got on your system through openHAB. If you don’t have your RPi exposed to the Internet directly then whether or not sshd is running or you are using the default username and password doesn’t matter unless and until some other machine on your LAN is compromised.
I’m not saying that worrying about the security of your systems is not important and that we should not be concerned about the risks of using myopenhab.org. But we do need to be reasonable and knowledgeable about what the risks actually are.
Risk = Likelihood an attacker will exploit the vulnerability * Impact should the vulnerability be exploited
So in this case we have:
Vulnerability: Attacker gains access to your RPi running openHAB through the myopenhab.org service
Likelihood: 5%
Impact: Depends on your LAN, information on your LAN, and other mitigating features
Thanks everyone for your insightful responses. While it is always possible we have a security issue in the cloud service, I think its highly unlikely in this scenario. That being said, we do take these matters seriously.
As with anything connected to the internet, it is also important to keep you account safe by using strong passwords and be careful about what you expose through openHAB. For example, a shell script that runs any command posted to an item might not be a good idea.
As others have mentioned, the most likely compromise would give an attacker the ability to access the HTTP based api’s on the OH system, which would allow control over the items in your system, but again it does not look like this is the case here.
If you think your account has been compromised, stop openHAB, or at least disable the cloud connector and send an email to support@myopenhab.org so we can investigate throughly.
This will download a script from some evil URL and run it. Even if you don’t give openhab sudo permissions, this could be bad as it could easily download and run arbitrary code on your machine, perhaps setting up a backdoor or other code to join up with a botnet or something. An attacker doesn’t need root to cause damage or do bad stuff with your machine.
But this is not something inherent with openHAB. This is a vulnerability created by an incautious configuration of the openhab user and/or the Exec binding.
NOTE: I don’t know for sure that these attacks would work. I don’t think the Exec binding gets a full shell so I don’t know if symbols like ; and | are processed and I don’t have a good way to test it without borking my own security vulnerability mitigations, but it does illustrate the point. Someone with more time might make the test and if it works file an issue on the binding to sanitize out those from the input.
So how to mitigate this (assuming it works at all)?
Don’t give openhab unrestricted sudo permissions. I wouldn’t give any sudo permissions at all but if you must, give it for only those commands you need.
Don’t use the input argument eliminating the ability for an attacker to pass an injection attack to your system in the first place.
Create a rule to sanitize the input stripping out the ; and | symbols
Add code to the binding (if it doesn’t already exist) to sanitize out the ; and | symbols
Do not allow access to your OH from outside your controlled network. For example, use a VPN instead of myopenhab.org.
Personally, for this and other reasons (i.e. I run in Docker so don’t have access to a lot of command line programs), I don’t use the Exec binding at all. Instead, I send an MQTT message to a Python script I have running on another host and that script sanitizes the input (actually that is a work in progress) before calling the command.
SD cards are wear leveled as well. In the cheaper SD cards that leveling is often weak leading to one of the reasons the cheaper ones wear out sooner than they should. But the problem is fundamentally the same. eMMC will also eventually wear out and will do so in the same ways as an SD card or USB flash drive.
The amount of time it takes to ware out varies widely and when it starts to wear out you may not know it.
The code is already posted here. See the code in execActuator.py in the onMessage function.
There are two for arg loops. The first one strips out the arguments that include ;, |, or \\ from the command that is loaded from the .ini file. The second one strips out the same types of arguments that are passed as part of the MQTT message.
The way it works is sensorReporter loads a .ini file and dynamically creates senors and/or actuators based on the contents of the .ini file. For the execActuator that includes a command to execute. This actuator is configured to listen for MQTT messages on a certain topic. The contents of the MQTT message is either “NA” (no arguments) or it includes the command line arguments to pass to the command in the .ini file.
If you try to use ;, |, or \\ in either the .ini file or the MQTT message you will likely end up with a command that doesn’t run.
I’ve not spent a lot of time analyzing this to make sure there are no other characters or strings that should be excluded and that is the part that is a work in progress. I’m pondering whether sudo and su should be excluded too, but that is already mitigated by not adding the sensorReporter user to sudoers. I probably should include & as well.
An example .ini section for an execActuator is as follows:
Just a thought on the rest API via myopenhab.org: does it provide access to the whole API or just the /items- endpoint? Otherwise someone could remotely create the exec-thing @rlkoshak posted and then run arbitrary scripts with it. Would this be possible?
It will proxy the entire http api, so yes that is a possible scenario if an account was compromised, and a good example why openHAB should not be run as root unless in some type of container (like docker). it brings up an interesting thought about limiting the scope of what we will proxy, but I am not sure how many people depend on remote access to their system through something like the paperUI which uses most of the API. It might be a nice option in the cloud binding config to limit what parts of the http API can be called.
People concentrate on wear levelling and writes wearing out sd far too much whilst possible it is the unlikely cause when a cause that is far more likely is probably to blame. From experience it is usually data corruption from dirty shutdowns when a write is occurring that causes issues. Wipe the sd card clean setup openhab again and use these tweaks to test with and use a UPS to stop power losses from causing reboots. The card will be fine. All flash based storage writes in blocks and the block does not store only the file you are wanting to write to, so this is how a log file write can corrupt an important file in a five second explanation.
It would be great to be able to configure what parts of the REST API are proxied through myopenhab. Personally I use openVPN whenever I need remote access to my system, and the only thing I use myopenhab for is google assistant which only needs acces to the items.
I do have a strong password for my account, but the more I could limit the attack surface on my system the better. An alternative would be to uninstall the cloud connector and manage without voice control (which i rarely use anyway).
Should I raise an issue so you developers can discuss if/how this could be done? Which repo should it posted on?
first of all I once more have to thank you again for your insightfull posts.
I learned so much from those during my initial OpenHAB learning experience and still do (I love your design patterns!).
When it comes to transferring those writes to an shared drive: How would I actually do this? (I have a network drive setup and the RPi is able to access it for backup purposes).
Do I understand correctly, that I could actually transfer writes of my persistence services to my network share instead of my SD card?
I didn’t find a tutorial on the forum on how to enable network writes.
Well, once you’ve mounted your network share on a local dir, you just need to change your persistence service configuration to store its data there. As there’s various persistence services, refer to the documentation of yours.
Make sure to use NFS for your mount, don’t use CIFS !
While you’re at it, also move your logs.
But this is getting off-topic so please open a new thread if needed.
