Log4j vulnerability

I‘m using this one and it’s working well:

I‘m still having some issues with some file encodings but I will sort this out once I finished scanning all our log files :grimacing:

At work this link is just making the rounds:

Money quote:

You may still be vulnerable to Log4Shell (RCE) if you only enabled the noMsgFormatLookups flag or set %m{nolookups} when you also set data in the ThreadContext with attacker controlled data. In this case, you must upgrade to 2.16.0 or else you will still be vulnerable to RCE.

I have no idea if this is applicable here, but at the moment at work, we’re patching a few things where we were only using the option before

2 Likes

I’m selfhosting OH cloud and a Nginx as reverse proxy. I found approximately 10 requests with jndi:// and most of them got 503 as reponse. But four of them got replied a http 400 and those are the ones that worry me because I couldn’t reproduce this behavior so far.
I think that it is pretty unlikely that those four requests got forwarded but I can’t tell for sure…

OH cloud isn’t using java as far as I know, so unless someone has your credentials they can’t do much.

The page at 400 Bad Request Error: What It Is and How to Fix It describes reasons for error code 400. At least a subset of reasons is most probably also the reason why you cannot reproduce the behavior.

It looks like Pax logging 2.0.12 includes log4j 2.16.0 Upgrade to log4j 2.16.0 · Issue #416 · ops4j/org.ops4j.pax.logging · GitHub

and Karaf 4.3.4 will include Pax 2.0.12 https://issues.apache.org/jira/projects/KARAF/issues/KARAF-7293?filter=allissues , and its ETA is Dec 21 Apache Karaf - Download

Kai mentioned earlier in this thread that openHAB is using Karaf 4.3.3 and upgrading to 4.3.4 would be a logical step. I know openHAB 3.2.0 final was planned for Dec 20, but maybe it could be delayed to include the new version of Karaf? Or a 3.2.1 release could happen shortly after? I don’t know how much work it is to upgrade Karaf in openHAB.

Most probably they got rejected by the Nginx because of invalid characters or so, but I couldn’t find any logging for that.
And the fact that if it got somehow redirected to OH cloud it then (hopefully) only gets passed to the OH instance if someone has the correct credentials makes me feel a bit better :grin:
Thanks for your replies @J-N-K, @Flole and @Wolfgang_S

Apparently according to Log4j – enabling log4j2.formatMsgNoLookups is not deemed sufficient enough…

The reason these measures are insufficient is that, in addition to the Thread Context attack vector mentioned above, there are still code paths in Log4j where message lookups could occur: known examples are applications that use Logger.printf(“%s”, userInput), or applications that use a custom message factory, where the resulting messages do not implement StringBuilderFormattable. There may be other attack vectors.

Wondering is openHAB is affected?

The new advice (apart from updating to log4j 2.16) is to remove org/apache/logging/log4j/core/lookup/JndiLookup.class from the JAR.

zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

For OH 2.5.12?
Is Pax logging 1.11.11 the version which is equivalent to 2.16.0? Which means the fixed version.

It is a little bit confusing that the numbers are not the same
Pax logging 2.0.12 = 2.16.0
Pax logging 1.1.11 = 2.16.0 ?

This will not solve your problem because you modify JAR in system/ directory (this is what I find from zip command), which is a “reference” location. At runtime pax logging is loaded from cached directory under data/cache. You still need to do an update call to force re-loading of module and refreshing of cache directory contents.

Your finding is correct. Pax logging does wrap log4j2 which causes us to track two versions at once. Usually it doesn’t matter, cause you get that from distro, yet this time it is important to know how to apply security. Have a look on my earlier answer Log4j vulnerability - #49 by splatch. This applies to any version of runtime and will work with 2.6.x too, despite of no patch release for it.

There is already a PR open on GitHub and the Karaf mailing list is watched closely. At Karaf there’s currently a vote for the release going on which will stay open for a little longer, the hope is that it will be closed early so Karaf will be released early. Then it could very much happen that it could be ready just in time.

3 Likes

If you ask me, the proper fix for this vulnerability does not require openhab to wait for Karaf release. Solution is here: Override pax logging version to address #1349. by splatch · Pull Request #1350 · openhab/openhab-distro · GitHub. It forces use of newest release of pax logging which contains fixed version of log4j.

Note old version of pax is still installed in filesystem, it can be removed later on as its early banishment causes build to fail.

1 Like

The 3.2.0 release includes 2.17.0, so it looks like this is fixed! Thanks everyone for your hard work addressing this so quickly

Hi.

How do i fix my OH 2 to use the patched pax files?

I manually put the new ones 1.11.13 in the openhab/runtime/system/org/ops4j/pax/ dir and removed the old ones…

I changed startup.properties to be …
mvn:org.apache.karaf.features/org.apache.karaf.features.extension/4.2.7 = 1
mvn:org.apache.felix/org.apache.felix.metatype/1.2.2 = 5
mvn:org.apache.karaf.services/org.apache.karaf.services.eventadmin/4.2.7 = 5
mvn:org.ops4j.pax.url/pax-url-aether/2.6.1 = 5
mvn:org.fusesource.jansi/jansi/1.18 = 8
mvn:org.ops4j.pax.logging/pax-logging-api/1.11.13 = 8
mvn:org.ops4j.pax.logging/pax-logging-log4j2/1.11.13 = 8
mvn:org.apache.felix/org.apache.felix.coordinator/1.0.2 = 9
mvn:org.apache.felix/org.apache.felix.configadmin/1.9.16 = 10
mvn:org.apache.felix/org.apache.felix.fileinstall/3.6.4 = 11
mvn:org.apache.karaf.features/org.apache.karaf.features.core/4.2.7 = 15

I removed the cache and restarted but seeing this is logs…

    Caused by: java.io.IOException: Error resolving artifact org.ops4j.pax.logging:pax-logging-api:jar:1.11.2: [Could not find artifact org.ops4j.pax.logging:pax-logging-api:jar:1.11.2 in openhab (https://openhab.jfrog.io/openhab/libs-release/)]
            at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.configureIOException(AetherBasedResolver.java:803) ~[?:?]
            at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:774) ~[?:?]
            at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:657) ~[?:?]
            at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:598) ~[?:?]
            at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:565) ~[?:?]
            at org.apache.karaf.features.internal.download.impl.MavenDownloadTask.download(MavenDownloadTask.java:52) ~[?:?]
            at org.apache.karaf.features.internal.download.impl.AbstractRetryableDownloadTask.run(AbstractRetryableDownloadTask.java:60) ~[?:?]
            ... 6 more
            Suppressed: shaded.org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact org.ops4j.pax.logging:pax-logging-api:jar:1.11.2 in openhab (https://openhab.jfrog.io/openhab/libs-release/)
                    at shaded.org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:48) ~[?:?]
                    at shaded.org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:368) ~[?:?]
                    at shaded.org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75) ~[?:?]
                    at shaded.org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:642) ~[?:?]
                    at shaded.org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:262) ~[?:?]
                    at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:489) ~[?:?]
                    at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:390) ~[?:?]
                    at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:215) ~[?:?]
                    at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:192) ~[?:?]
                    at shaded.org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:247) ~[?:?]
                    at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:767) ~[?:?]
                    at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:657) ~[?:?]
                    at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:598) ~[?:?]
                    at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:565) ~[?:?]
                    at org.apache.karaf.features.internal.download.impl.MavenDownloadTask.download(MavenDownloadTask.java:52) ~[?:?]
                    at org.apache.karaf.features.internal.download.impl.AbstractRetryableDownloadTask.run(AbstractRetryableDownloadTask.java:60) ~[?:?]
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
                    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
                    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
                    at java.lang.Thread.run(Thread.java:829) [?:?]

I couldn’t find a proper guide to fix a 2.x version for the log4j issue… Looks like I’m messingthis up.

Anyone got a clear guide to sort my 2.x for pax/log4j issue?

A lot depends on actual path you put in filesystem and startup properties. Practically filesystem location is not that important cause all metadata is read from actual file. You dont need to create new files, it is sufficient to override existing jar file.

Thanks for the reply and Happy New Year!! :slight_smile:

I’ve messed it up a bit right now. So are you saying it i revert to what it was, then I change just replace the files with 1.11.2 jars with the 1.11.13 jars, but rename them to 1.11.2?

Also, I see that Karaf 4.2.14 has the fix in. Is it possible to overlay the newer version of that over OH 2.x ? If so how?

I’ve downloaded the latest clean 2.5.12 for OH, which still has the bad log4f. So what’s the best way of downloading a clean linux binary install and patching it to be safe?

Sorry for jumping in here again. I’m confused with version numbering. Is there a change log in the marven repository to know which marven version belongs to which apache version?

Only appache log4j version 2.17.1 is bug free at the time of writing this.

What does this mean for OH 2.5.12?
Is Pax logging 1.11.13 the version which is equivalent to 2.17.0 or 2.17.1?

It is a little bit confusing that the numbers are not the same.

I think you can lookup the relation/dependency here:
https://mvnrepository.com/artifact/org.ops4j.pax.logging/pax-logging-log4j2

From what I can see 1.11.13 has 2.17.1 in it.

However, I’m still lost how I change a OH 2.5.12 (which has 1.11.2 in) to upgrade to 1.11.13.

Can someone please explain?

I would have thought a 2.5.13 should be there too, with 1.11.13 in, since this is such a bad issue. 2.5.12, really shouldn’t be used.

In theory third part of version number is irrelevant. In practice it might cause troubles. Most of stuff you see in openHAB or Karaf specify minimum version and as long as you move within x.y range of xy.z version you are on safe side.
Regardless of which exact version of pax logging you use you shall be able to update .z part of version with no major repercussions. My earlier post outlined tips for OH 3.0 but same way shall work for 2.5.x which is on older Apache Karaf version.
My advice (which you already did) - look for pax logging versions your OH 2.5.x uses, take maximum patch (.z) available and override filesystem files. Then do clean start without cache dirs.
Obviously you still need to double check if you get log4j2 >= 2.17.1for pax logging but with little effort you can fix issue without awaiting next 2.5.x release.