Yeah I can do, I’ll try to come up with an actual exploit aswell so I can test if this is really fixing this.
1 Like
I don’t think that’s correct. My understanding is that when a certain string is logged that string is being parsed as an instruction to connect to a server and load a class from there and execute it. It doesn’t matter of you use jndi for logging in my opinion. I’m sure with OH there’s a way to request some URL that then show’s up in the log file, and that could potentially be used to exploit this.
Karaf 4.3.4 is currently on vote (see mailing list) and the staged artifacts were created a couple of days ago so it still depends on Pax Logging 2.0.10 (without the fix).
Besides that, Karaf 4.3.4 does seem to work well with OH 3.2 after doing some testing with it this afternoon. 
https://www.lunasec.io/docs/blog/log4j-zero-day/ states:
JDK versions greater than
6u211,7u201,8u191, and11.0.1are not affected
As we are on Java 11 and version 11.0.1 is from 2018 (the very first patch after the initial release), I would actually say that we should be safe for openHAB 3.x.
Yes and no: While code might not be executed it still tries to connect to that server and if you let it timeout by making it connect to something that doesn’t respond you probably would be able to cause a denial of service as it would wait for a connect. If it’s multithreaded then you will have to spawn a few threads but at some point you will have too many which are all trying to connect to that server and waiting for a timeout, if it’s singlethreaded you will have to wait for that timeout anyways (which can be long…).
Also from the site you quoted:
However, there are other attack vectors targeting this vulnerability which can result in RCE.
I would say that even though no exploit was published that works on those versions yet we shouldn’t assume that we are safe.
If I didn’t make a mistake then there should be a PR now at Mitigate potential Remote-Code-Execution caused by CVE-2021-44228 by Flole998 · Pull Request #1343 · openhab/openhab-distro · GitHub which solved/mitigates this issue. Please look at it and please also double-check that this is at the right place.
As mentioned above, I believe that using karaf it’s possible to check if it’s applied properly:
system:property log4j2.formatMsgNoLookups
If it says “true” then all is good, “false” or “null” is not good.
1 Like
Since it is usually not usual for an OpenHAB server to be accessible from the Internet and there should also be no port forwarding on it, we can actually go back to DefCon 4, right?
For those who want to “try” the exploit: You can using karaf:
log:log -l ERROR '${jndi:ldap://127.0.0.1/a}'
and use Wireshark or tcpdump to see if a connection to 127.0.0.1:389 is opened or attempted. This doesn’t do anything harmful as there is no server running on port 389 to do the next steps necessary, but it proves that we are indeed affected by it. This should/could be used after a test to verify if we have mitigated it properly aswell.
1 Like
There is demo.openhab.org… And I could probably come up with a way to exploit it by visiting a website aswell, but I really don’t want to spent much time to actually exploit this as there is no point in doing that for me. For me it was important to have a way to check if we are affected or not by triggering that exploit, and I have done that and provided a fix aswell.
No, problem…everything is fine.
i have tested
log:log -l ERROR '${jndi:ldap://127.0.0.1/a}'
in my installation (with activated option ‘formatMsgNoLookups’ and with WireShark i can not see any connection attemps with LDAP-Port 389 to the server itself…
so i think, the workaround should work until upcoming versions use the fixed log4j-library…
OK, after writing a big post you’ve found a way to do what appeared to not be possible based on the docs.
Since it is possible to provide a log string that does cause OH to attempt to reach out to a non-configured JNDI service OH users do have a little more exposure. Of course to exploit it an attacker must have access to something in OH that forces it to log out a carefully selected String as well as having a compromised LDAB server somewhere out on the Internet.
However, What if you connect to some other arbitrary IP address? It seems not only possible but reasonable that Karaf itself is hosting it’s own internal LDAP service to centralize logging configs (it’s kind of the whole point of JNDI Context and JNDI Lookups). By connecting to 127.0.0.1 you might be triggering OH to use it’s already configured JDNI service and it might still be impossible to connect out to an arbitrary service. There are some allow listing capabilities built into this and localhost is usually alllowed.
But one thing to be careful of, if Karaf or Pax uses Lookups behind the scenes, the environment variable mitigation could break all of OH logging.
I’ve just found a way to have demo.openhab.org connect to me… So just because I don’t post a ready-to-use exploit that doesn’t mean it doesn’t exist… In general exploits are held back until patches are available, and using karaf you can do worse things so there is no point in running that exploit from karaf, that’s why I posted that one publicly.
Just to be clear: That was just done for testing purposes, there was no server listening here and there was no attempt to actually transfer and execute a payload.
Reading your post again I don’t think you understand how this actually works: There is no JNDI/LDAP normally involved in logging. But with a certain message you can cause OH to connect to an LDAP server which then makes it connect to a HTTP Server which then serves the exploit. If it is just blindly executed or not by default depends on the Java version. I don’t know why that feature even exists in log4j or what it’s supposed to be used for, but it’s not used in any normal logging scenario. Even if this would break things, we would notice it soon enough when someone reports issues with it. If we don’t want to break things accidentally we have to stop development right now. I think the risk/reward ratio here is well on the “reward” side. A karaf-update is probably more risky.
In OSGi containers and JavaEE application servers JNDI absolutely is involved in logging. That’s how you can set logging properties in one place and have all your various applications use the same values. Instead of pointing all the modules to the same file to load, they can pull the properties they want from a JNDI service. The config therefore lives in the JNDI service.
I haven’t got too far into the Pax code yet to see if this is how it works in OH (it is how it worked in Weblogic the last time I coded something on it, which was prior to Oracle acquiring it). The logging config gets ingested and stored in a central service (I can’t remember if it was LDAP or not) and all the modules would pull the config from that central service.
But consider the following scenario for a more powerful use for this feature. Let’s say I have a dozen instances of a web service hosted across several cloud servers with a load balancer in front of it. Let’s make it even more interesting and have the number of instances of the service change based on the real time load.
One way (not the only way for sure) would be to host a central service, let’s use LDAP, and load up the config there. Then the logging config files or, apparently even the logging statement themselves, can pull properties from the LDAP service. To change the logging config across the whole cluster one need only update the LDAP service. That’s way better, for example, to handle something like needing to route the logging to a different logging collector because of a failure than needing to deploy a new config file to each of the currently running services.
The only thing I didn’t understand here was that it was possible to cause log4j2 to connect to an arbitrary JNDI service through a specially crafted JNDI Lookup in a log statement. That isn’t in and of itself an exploit, that’s it working as designed. And the fix doesn’t appear to remove that capability either. But based on the docs I managed to read and understand in this short period of time, that did not seem possible and that you had to already configure the connection to the JNDI service . If it were in fact not possible then the risk to OH users is exceptionally low. Since it is possible the risk to OH users is much more significant.
What you demonstrated is a feature in log4j2 called a lookup. Lookups let you set log4j parameters in a central location (e.g. an LDAP server) so you can change the logging behavior of a cluster of apps or even the whole enterprise from one central location.
I don’t disagree but then what if it did break? No mitigation is possible until an upgrade in that case.
However, if it does break there might be alternative mitigations possible (maybe my time spent with the docs wasn’t a waist of time). The log4j2.allowedJndiProtocols property could be configured to just “java” (assuming that Pax is only using java and not LDAP) and/or the log4j2.allowedLdapHosts property can be used to limit connections to only localhost. Though that last property should be limited to localhost by default already so not sure why your tests worked unless that property is being set somewhere in Pax or Karaf to allow arbitrary connections out.
For such cases bundle:update is your friend. You can update pax logging. After restart new version should be loaded.
For openHAB, the distribution build has an option to blacklist old logging libs as well as override version to a hotfix. We had it in past for one of jetty errors as far I remember.
1 Like
It is an exploit if you have a logging statement like this:
logger.log("Test: {}", userSuppliedString);
and as userSuppliedString then get’s evaluated aswell. Normally you would think that you are safe in this case, but you are not. It’s similar to printf which works in a somehow similar way, but a string that is passed as an argument isn’t evaluated again (like it should be). And it gets worse, as the LDAP server can make it pull and run code then, unless the java version prevents that.
I am still hoping that the new karaf gets ready, then we can get both fixes in. If this breaks something for someone then yes, there is no mitigation possible or other attempts need to be used. For the majority of users I don’t expect anything to break though. It’s documented now in this thread how it can be tested, so everyone should do that and then we will soon see if it breaks something, ideally before the release. That’s why I would like to get this posted as an annoucement, then many people would read it, do it and we could react if it breaks by delaying a release until the new karaf version is ready. But that’s not my decision to make if and how long a release is delayed, I can only stress that everyone should use this mitigation until a release with a fix is out, just to be on the safe side.
By the way:
openhab> system:property log4j2.allowedLdapHosts
null
and still it works.
1 Like
Then either the docs are wrong on the log4j2 site or there is something somewhere overriding that property. Log4j has a crazy complex hierarchy in what variables where take precedence. Something must be setting that property at a higher priority than the console causing it to overwrite it.
For OH I see no reason why anything other than localhost host should be allowed ever, if LDAP is even needed at all, so that’s probably worth further exploration.
Edit: Found it.
The following is a list of available global configuration properties. Note that these can only be set once per JVM process unlike configuration settings available in configuration files
Since it’s already set before you get to the console you can’t override it. We would need to set the property before OH starts up.
I’m not sure it would actually work. Based on
https://news.ycombinator.com/item?id=29507263
This mitigation likely will not work anyways.
@Kai wrote above that we are currently on 2.0.10
“The Internet”, namely this site, explains why this still works:
Those config measures were put in place as part of the fix for this issue. I.e they didn’t exist before the fix was released.
For OH I don’t see any reason why on earth JNDI Lookups from logged strings should be possible at all. Just turn that stuff off with the option I described above. Even if you fix LDAP then someone might come up with the next JNDI-compatible protocol and we will have the same mess all over again. I assume there is a fix for the weird string parsing aswell in the new version, but still: Just turn that feature off, it is not needed at all for OH and is just insecure.
Because JNDI isn’t the only Lookup and that mitigation turns all Lookups off, not just the JNDI ones.
There are 12 different types of lookups, only two of which have anything to do with being able to connect to remote services. Some of them like date and base64 don’t even refer to anything external to the log statement.
That’s why it might break all the logging. There’s lookups for environment variables, system properties, inserting a DateTime, command line args to the JVM. I’m sure that’s why they added those new properties (I wish there were a way to see their docs by version more easily) in the first place.
But hey, everyone here clearly knows more than me. I’m sorry I’ve wasted everyone’s time.
I have just checked and for me the fix works. Maybe @Kai got confused with log4j2 and PAX Logging versions there, or he meant to say 2.10.0, no clue.