SSL protocol error on OH 5.2.0

Setup, Configuration and Use Beginners
1

Hi all. I had reported this problem before in the milestone discussion for 5.2.0 but apparently this problem has not been fixed before the release. The problem is that when I connect to OH locally via SSL, i.e., port 8443, I get an SSL protocol error.

Chrome on my MacBook shows me only this:

image
image946×256 10.7 KB

The OH Android App is more detailed, it shows me this:

image
image460×1000 96.9 KB

Now one might argue that when using a local connection one can just go without SSL, which is my workaround (connecting to port 8080 instead), but I nonetheless believe that people prefer to have this fixed. Hence I am bringing this up again.

I am running OH 5.2.0 in Docker on a Synology NAS. This seems to have been still working fine on OH 5.1.4.

2

Hi,

I can’t confirm this bug. Maybe your docker Image has a wrong port mapping?
Looks like you do a https requst to the 8080 http port.
What network setting did you set in container manager?

3

It’s hard to understand how this can be a OH problem since OH doesn’t provide HTTPS at all. It is the reverse proxy that you install in front of it that does that, if you won’t use one, you will have to use HTTP.

I really cannot see the point in using HTTPS locally, but I know that some people do it.

4

OH provides HTTPS on port 8443 by default using a self signed certificate. The reverse proxy approach is to enable additional authentication before reaching OH itself (since OH’s authentication and authorization is kind-of lame) and make it possible to integrate with a trusted CA more easily (e.g. letsencrypt). But OH has always supported HTTPS on port 8443 by default. You just have to deal with the "untrusted certificate warning from the browser. The Android app should ignore the trustworthiness of the certificate for the local connection though.

If it were a certificate issue, the error would be different.

I think @cd-tronic is on to something. The error shown in the Android app is telling you that openhab:8443 doesn’t support HTTPS at all. What is your port mappings on the container? The error is telling us port 8443 is not supporting HTTPS which implies you’ve either changed the HTTPS port on openHAB (e.g. set the OPENHAB_HTTPS_PORT environment variable or changed the config in userdata) or you’ve mapped the wrong port from openHAB to port 8443 in the container.

5

Ok, I wasn’t aware, I see no point in using HTTPS locally so I’ve never tried it (yes, I know that basic auth sends the user/pass in clear text, but the traffic isn’t that easy to intercept unless you control either end). I thought HTTPS was something you stuck in front of OH, so my bad.

6

There is some issue with HTTP and the number of websockets it can have open at the same time. If you experience timeouts in MainUI or a very slow UI, it might be worth moving to HTTPS even locally.

I don’t know the details but apparently we can only increase the number of sockets allowed under HTTPS. I didn’t follow the PRs that closely to know specifically why.

7

I think those are browser limitations imposed by Chrome. I use Firefox, so I have bumped it from 6 to 20 connections or something like that. It’s the browsers that limits this, not OH.

By the way: The limit applies to SSE connections, not websocket. The number of websocket connections are unlimited as far as I know, but as of today, very few things in MainUI actually use websockets - I think the exception is the log viewer.

8

The SIP widget in MainUI requires HTTPS, because the SIP card uses the Camera/Microphone API and WebRTC, which most/all browsers only allow over SSL: SIP Client cannot be used over insecure HTTP · Issue #1690 · openhab/openhab-webui · GitHub

9

Luckily I don’t use it - and I generally stay away from things that “require” HTTPS. It tends to mean that they have something to hide in their communication (not saying that’s the case here, but in general).

10

Hi all. Please find here my container configuration attached. Do you see anything that might explain this?

openhab (2).json (4.2 KB)

11

I’ve never done bridged networking so I could be on the wrong track, but this looks suspicious.

   "network" : [
      {
         "driver" : "bridge",
         "name" : "openhab_bridge"
      },
      {
         "driver" : "macvlan",
         "name" : "openhab_network"
      }
   ],
   "network_mode" : "openhab_bridge",
   "port_bindings" : [
      {
         "container_port" : 5007,
         "host_port" : 0,
         "type" : "tcp"
      },
      {
         "container_port" : 8080,
         "host_port" : 0,
         "type" : "tcp"
      },
      {
         "container_port" : 8101,
         "host_port" : 0,
         "type" : "tcp"
      },
      {
         "container_port" : 8443,
         "host_port" : 0,
         "type" : "tcp"
      }
   ],

If I understand this you are mapping all the ports to port 0? Often choosing port 0 means “wildcard” telling the system to “scan the OS and choose th enext available port”. I don’t know for sure that’s the case with docker network bridging but it’s the first thing I’d check. In short, your OH ports are being exposed randomly.

12

Thanks Rich! I am not really a network expert. Hence I debugged that further with the help of Claude - with some success! Some of the output that follows was generated with Claude…

The port bindings turned out to be a red herring, though a reasonable thing to suspect.

You’re right that host_port: 0 means “pick a random ephemeral port”, and docker port openhab confirmed it:

8443/tcp -> 0.0.0.0:32780
8080/tcp -> 0.0.0.0:32782

But those mappings are never used here. The container sits on two networks: the bridge (openhab_bridge) and a macvlan (openhab_network). The macvlan gives it its own LAN IP (192.168.178.210), which is what openhab resolves to — so I connect straight to the container and the published bridge ports play no part. Vestigial clutter, but harmless.

The real cause: two separate TLS problems

The confusing part was that curl -k https://openhab:8443/ returned a clean HTTP/2 200 while browsers and the Android app both failed. It turned out there were two independent issues, and curl happened to be immune to both.

1. The certificate has no subjectAltName (breaks Chrome)

openHAB’s auto-generated self-signed cert carries only a common name:

subject: CN=openhab.org; OU=None; O=None; L=None; C=None

No subjectAltName exists. Chrome has required a SAN since v58 and will not fall back to the CN, so there’s no name to match — it fails at the protocol level with ERR_SSL_PROTOCOL_ERROR and no click-through interstitial. curl -k skips validation entirely, which is why it never complained.

Deleting the keystore and letting openHAB regenerate does not help — it produces the same SAN-less cert.

Probably not a 5.2 regression. My original cert dated from June 2023 and had worked for years. openHAB has likely been generating SAN-less certs the whole time; what changed is browser tolerance. The trigger was most likely a Chrome update, and the timing with the 5.2 upgrade was coincidental.

Fix (example for my setup here) — generate a cert with a proper SAN:

docker exec openhab cp /openhab/userdata/etc/keystore /openhab/userdata/etc/keystore.bak

docker exec openhab keytool -delete -alias mykey \
  -keystore /openhab/userdata/etc/keystore -storepass openhab

docker exec openhab keytool -genkeypair \
  -alias mykey -keyalg EC -groupname secp256r1 \
  -keystore /openhab/userdata/etc/keystore \
  -storepass openhab -keypass openhab \
  -dname "CN=openhab" -validity 3650 \
  -ext "SAN=dns:openhab,dns:openhab.fritz.box,ip:192.168.178.210"

docker restart openhab

Adjust the SAN list to your own hostnames and IP. Verify:

openssl s_client -connect <ip>:8443 -showcerts </dev/null 2>/dev/null \
  | openssl x509 -noout -text | grep -A2 "Alternative"

After having performed this, I can access OH through Chrome and port 8443 just fine now! But not yet through the OH Android app…

2. CRYPTO_POLICY=limited breaks TLS 1.3 (breaks the Android app)

With the cert fixed, Chrome was happy — but the openHAB Android app still failed:

javax.net.ssl.SSLProtocolException: Read error: Failure in SSL library, usually a protocol error
error:10000089:SSL routines:OPENSSL_internal:DECODE_ERROR
(external/boringssl/src/ssl/tls13_client.cc:1181)

That’s BoringSSL failing to decode the TLS 1.3 handshake — a protocol fault, not a trust fault. The same symptom is visible in openssl s_client, right at the end of an otherwise successful TLS 1.3 handshake:

error:0A00009F:SSL routines:tls_process_new_session_ticket:length mismatch

The server completes the handshake but then emits a NewSessionTicket that clients cannot decode. OpenSSL shrugs it off (so curl and s_client appear to “work”); BoringSSL — i.e. Android — rejects the connection outright. Forcing TLS 1.2 gives a completely clean handshake with no such error:

openssl s_client -connect openhab:8443 -tls1_2   # clean
openssl s_client -connect openhab:8443           # length mismatch on the ticket

The trigger is CRYPTO_POLICY. The openHAB Docker image defaults to limited — the Dockerfile deliberately rewrites crypto.policy=unlimited to limited in java.security, to comply with local laws that may restrict unlimited-strength cryptography.

Setting it to unlimited fixes the TLS 1.3 problem completely:

CRYPTO_POLICY=unlimited

Restart afterwards. TLS 1.3 then negotiates cleanly and the Android app connects.

Why this isn’t widely reported: every official example — the Docker Hub README, the openhab-docker README, the openhab.org Docker install docs, the Ansible playbook — sets CRYPTO_POLICY: "unlimited" explicitly. So most people never run with the default. My container had been created from the image defaults (Synology’s Container Manager captures the image env into its config), so I was on limited without ever having chosen it. If you’re in the same boat and use HTTPS, you will hit this.

I have not traced why a restricted crypto policy produces a malformed ticket rather than a clean failure or a graceful fallback — that’s for someone who knows the pax-web / Jetty / JDK internals. But the correlation is solid and reproducible: flip the variable, the behaviour flips. Arguably a restricted crypto policy should never cause the server to emit a structurally invalid handshake message, so I suspect there is a real bug somewhere in the stack.

Environment, for the record:

  • openhab/openhab:5.2.0 (Debian)
  • Jetty 9.4.58.v20250814
  • OpenJDK 21.0.11+10-1-deb13u2

tl;dr: not a Docker networking issue. Two TLS problems stacked on top of each other:

  1. openHAB’s self-signed cert has no subjectAltName → modern Chrome rejects it outright. Regenerate the cert with an explicit SAN.
  2. The image default CRYPTO_POLICY=limited causes the server to emit an undecodable TLS 1.3 NewSessionTicket → the Android app (BoringSSL) refuses to connect. Set CRYPTO_POLICY=unlimited.

curl -k was immune to both, which made the server look perfectly healthy when it wasn’t.

13

Please open an issue on openhab-core (?). This might be an upstream issue. I don’t really know where the cert is generated, in openHAB or karaf.

The TLS issues are certainly in upstream libraries. Which one will take some research. But I doubt OH is the only one having problems.

14

openHAB-Android, at least in the past, asks the user to confirm explicitly, when the certificate does not match - User must validate twice the certificate of the server · Issue #3638 · openhab/openhab-android · GitHub .

15

Yes that is right. But in my case, with CRYPTO_POLICY=limited this dialog is never shown because the app crashes earlier.

16

Just to close the loop and cross-link this: I hit the exact same thing on 5.2.0 (official Docker image, JDK 21.0.11+10-1-deb13u2, Jetty 9.4.58), and CRYPTO_POLICY=unlimited fixes it, as you found.

On the root cause, in case it helps someone landing here: the limited crypto policy caps AES at 128-bit, but the TLS 1.3 session ticket (NewSessionTicket) is encrypted with AES-256. So under limited the server ends up sending a malformed/empty ticket. Strict TLS 1.3 clients (Chrome, and a plain JSSE client) reject it right after the handshake; TLS 1.2 doesn’t use that path, which is why forcing 1.2 works. A plain SSLServerSocket with TLSv1.3 reproduces it with no Jetty involved, and it happens with both RSA and EC keystores, so it is the JDK crypto policy, not Jetty or the certificate. With unlimited the negotiated cipher also goes back up to AES-256.

I reported it upstream so the image can default to unlimited:

17

Awesome thanks a lot @edo89b !