Image URL with HTTPS seems not to work

Hi,

little by little I switch all my internal servers/services to my own PKI - yesterday my IP cams became new certificates. But since then, I do not get the image snapshots running in my OH sitemap (in OH habpanel it works without any problems).

I use the following image item config in my sitemap:
Image url="https://192.168.xx.yy/cgi-bin/api.cgi?cmd=Snap&channel=0&rs=test&user=xxx&password=xxx" refresh=5000

As mentioned above, the URL is correct. I can open it in a browser. My habpanel also works without any problems and displays the image. It is only a problem in the sitemap.

My OH server also uses a SSL certificate from same private CA.

Does anyone has an idea what is going wrong?

Thx + Regards!!

Any error message in the browsers developer consile ?

Hi,

good idea!

I’ve checked the page load sequence via the developer console in FF:

For sitemap:

For habpanel:

  • The URL is not translated, the genuine URL is used here.

Sitemap UIs are using OH server proxy for image URL

Hi,

yes, that is exactly what I can observe on the developer console resp. in the generated HTML code. But the question is, what is going wrong here? The only idea I have is, that it could be a problem with the certificate chain (e.g. if Jetty does not trust my custom CA; my browser has imported the custom CA into his CA store, but what about Jetty?)

The only thing which have been changed was, that the URL now uses HTTPS (=updated sitemap due to SSL certificates for IP cams) instead of HTTP (=former sitemap version).

Hi,

just an update, maybe to raise this one up again. In the meantime, I’ve tried different things to solve it without a workaround.

I’m wondering why Jetty answers the request with a 502 message. Therefore I had the idea, that this could be a problem with my custom root CA and imported it into the (raspian/Linux) trust store where OH (incl. Jetty) is running on to complete the certificate chain. But no effect - the error message still occure.

Then I’ve tested it with external images, e.g. by including external images like https://www.openhab.org/openhab-logo.png into the sitemap image URL. With such external resources it works, i.e. the image is correctly displayed in browser and OH app.

Therefore I assume, that there is a problem with the custom CA and its handling in the chain nginx, jetty and/or OH. IP cams and my OH instance are using custom certificates based on the same custom root CA. IP cams and OH instance have different IPs (but in the same subnet).

I know, there is a workaround by using the static html folder. But I don’t like that. Are there any ideas how this could be solved without that workaround?

Thx + regards!!

Earlier you wrote

80 and 443 - but you use different/appropriate methods to do so ? It is not happening, that http requests are send to https reps. https requests to http ports …?

I use the nginx config as mentioned here:

More in detail, my config looks here like that:

server {
    listen                                    80;
    server_name                               192.168.xx.yy;
    return 301                                https://$server_name$request_uri;
}

server {
    listen                                    443 ssl;
    server_name                               192.168.xx.yy;
    ...
    location / {
        proxy_pass                            http://localhost:8080/;
        proxy_set_header Host                 $http_host;
        proxy_set_header X-Real-IP            $remote_addr;
        proxy_set_header X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto    $scheme;
        proxy_read_timeout                    3600;
    }

Nginx is listening on port 80 and 443 and redirects the requests to 8080 (all on the same machine).

The image URL entry in the sitemap to capture the snapshots from the IP cams looks like that:

Image url="https://192.168.xx.zz/cgi-bin/api.cgi?cmd=Snap&channel=0&rs=test&user=xxx&password=xxx" refresh=5000

Such image URL in the sitemap are then translated into this:

https://192.168.xx.yy/proxy?sitemap=mysitemap&widgetId=1234&t=123456789

This setup runs into that trouble.

If I change the image URL the sitemap to an external resource like

Image url="https://www.openhab.org/openhab-logo.png"

which is translated into something like this

https://192.168.xx.yy/proxy?sitemap=mysitemap&widgetId=1235t=123456789

works without any problems.

Anything that is logged in the nginx log files ( access / error ) ?

where did you import your root CA into ? Into /etc/ssl/certs or into the java keystore ?

Anything that is logged in the nginx log files ( access / error ) ?

The access log only contains entries saying that the GET request results into 502 / 505. The error log is empty. Is there a possibility to set the log level for Jetty events (I did not find a matching package.subpacke to set the log level)?

where did you import your root CA into ? Into /etc/ssl/certs or into the java keystore ?

Into /etc/ssl/certs via update-ca-certificates. The keystore is afaik only relevant if I install the SSL server certificates directly for use by Jetty (if Jetty performs the SSL stuff and reacts on https requests). One of my first ideas was, to NOT use ningx and better Jetty directly. But nginx looks like the default solution in context of OH. Maybe I should switch from nginx to Jetty?

Does the entry look ok with regard to the requested object/file/server ?

how does the proxy part work ? Isn’t Jetty the client in case it acts as a proxy and then needs to validate the servers certificate ?

Does the entry look ok with regard to the requested object/file/server ?

The access log contains the GET requests. The respective entry entry looks OK, it only documents the 502 status. Example:
192.168.aa.bb - - [15/Apr/2023:15:23:03 +0200] "GET /proxy?sitemap=mysitemap&widgetId=1234&t=123456789 HTTP/1.1" 502 505 "https://192.168.xx.yy/basicui/app?w=1234&sitemap=mysitemap" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0"

how does the proxy part work ? Isn’t Jetty the client in case it acts as a proxy and then needs to validate the servers certificate ?

Good question. I’ve tried to analyze the Java code for that feature, but until now, I did not find the respective piece of code to understand what happens in detail. For me the process chain looks like that:

Step 1: Client (e.g. Browser) installs the custom root CA. By this, the custom SSL server certificate won’t results into any warnings.

Step 2: Client (Browser, App) opens Sitemap with https://192.168.30.2/basicui/app?sitemap=mysitemap. Nginx redirects that request to Jetty. Nginx deals with the SSL stuff, i.e. delivers the SSL server certificate. Jetty answers the request, Nginx passes the Jetty answer.

Step 3: Image (proxy) item in the Sitemap send a request to https://192.168.xx.yy/proxy?… This request goes first to Nginx which transmits it to Jetty. Now Jetty (resp. the OH piece of code) has to request the IP cam on https://192.168.xx.zz/… This one results into that error. One reason for the error could be, that Jetty does not trust the IP cam server certificate.

But not comprehensible:

  • The custom root CA for both, OH and IP cam is the same. This one is installed in the Linux trust store.
  • If Jetty does not work with the trust store, then I would assume that it would also answer the “test” with the external image available under https://www.openhab.org/openhab-logo.png with such an error.
  • Why? Because Jetty then could also NOT know the “official” root CA Baltimore CyberTrust Root (which is the root CA of the OH website certificate chain). This root CA is installed by default in OS (like Linux trust store under /etc/ssl/certs) and Browsers.

Java provides a keystore for root certificates as well.
Search for cacerts in the java installation treee ( …/lib/security/cacerts ).
This keystore contains root certificates.

keytool -list -v -keystore $(find $JAVA_HOME -name cacerts)  | grep Owner | grep Baltimore

when being asked for its password enter: changeit
Result of above command then is:

Owner: CN=Baltimore CyberTrust Root, OU=CyberTrust, O=Baltimore, C=IE

This keystore contains root certificates.

You’re right. Initially it looked like the missing link in the certificate chain. But after checking that store: It already contains my custom Root CA certificate. Probably by performing update-ca-certificates, that Java keystore has been updated, too.

I’ve re-added incl. option -trustcacerts and re-started the system, but without any change in the mentioned behaviour. Jetty stills answers with 502 Bad Gateway

Is it possible to enable deeper logging with focus on Jetty to get a bit more insight, what is happening there? I’ve tried to add logger for package org.eclipse.jetty, but it is no effect?

I tried

log:set DEBUG org.eclipse.jetty

in the karaf console. Followed by

log:tail

This seems to show logging output. Checking the openhab.log file the same output is appended there as well.

Yes, that is exactly what I’ve done - without any new entries in openhab.log. But with log:tail it works, i.e. a bunch of hundreds of debug log entries from Jetty:

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
	at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
	at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:340) ~[?:?]
	at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:186) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) ~[?:?]
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637) ~[?:?]
	at org.eclipse.jetty.io.ssl.SslConnection.unwrap(SslConnection.java:429) ~[bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:718) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:168) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:80) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:131) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:172) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.46.v20220331]
	at java.lang.Thread.run(Thread.java:829) [?:?]

The entries document a SSL Handshake Problem. OK, now it is a confirmed problem with the SSL certificates.

I feel like a milimeter away from the solution, but not able to see it …

is that the complete block that belongs to the initial error message or are there more following lines that provide more information about the root cause / problem ?
Before the error you also should see information about the connection request itself.
In case DEBUG does not help there also should be TRACE level.
Keep attention in case all this is being logged to a file that you may run out of disk space.

is that the complete block that belongs to the initial error message or are there more following lines that provide more information about the root cause / problem ?

The log snipped in my former post only contains the exception stack trace. The debug log entries before are not very helpful; they are only saying that is suddenly going wrong (“what” happens, but no indicates for “why” it happens.

In case DEBUG does not help there also should be TRACE level.
Keep attention in case all this is being logged to a file that you may run out of disk space.

Oh yes, even with “only” debug, the performance of the web server is going down. I’ll try it later on again with TRACE level.

Additionally I’ve check with curl and openssl s_client, that there are no general problems in connecting between OH server and IP cams. Calling the same URL with openssl s_client from OH to IP cam everything works fine - even the SSL handshake. Therefore it must be something in context of Jetty.