TimeOutError when using "HttpUtil.executeUrl"

Development Misc
, ,
1

A user of the Tankerkoenig binding reports errors (Error Report) when using the binding.
All reported errors show to the use of httpUtil.executeUrl, either showing a TimeoutExeception or a ChannelClosedException.

10:56:36.395 [DEBUG] [lipse.smarthome.io.net.http.HttpUtil] - About to execute https://creativecommons.tankerkoenig.de/json/detail.php?id=51d4b529-a095-1aa0-e100-80009459e03a&apikey=[Removed]
10:56:37.335 [DEBUG] [ig.internal.data.TankerkoenigService] - getTankerkoenigDetailResult IOException:
java.io.IOException: java.util.concurrent.TimeoutException
at org.eclipse.smarthome.io.net.http.HttpUtil.executeUrlAndGetReponse(HttpUtil.java:255)[113:org.eclipse.smarthome.io.net:0.9.0.b5]
at org.eclipse.smarthome.io.net.http.HttpUtil.executeUrl(HttpUtil.java:149)[113:org.eclipse.smarthome.io.net:0.9.0.b5]
at org.eclipse.smarthome.io.net.http.HttpUtil.executeUrl(HttpUtil.java:122)[113:org.eclipse.smarthome.io.net:0.9.0.b5]

or

11:11:42.667 [DEBUG] [lipse.smarthome.io.net.http.HttpUtil] - About to execute https://creativecommons.tankerkoenig.de/json/detail.php?id=51d4b529-a095-1aa0-e100-80009459e03a&apikey=[Removed]
11:11:43.467 [DEBUG] [ig.internal.data.TankerkoenigService] - getTankerkoenigDetailResult IOException:
java.io.IOException: java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException
at org.eclipse.smarthome.io.net.http.HttpUtil.executeUrlAndGetReponse(HttpUtil.java:255)[113:org.eclipse.smarthome.io.net:0.9.0.b5]
at org.eclipse.smarthome.io.net.http.HttpUtil.executeUrl(HttpUtil.java:149)[113:org.eclipse.smarthome.io.net:0.9.0.b5]
at org.eclipse.smarthome.io.net.http.HttpUtil.executeUrl(HttpUtil.java:122)[113:org.eclipse.smarthome.io.net:0.9.0.b5]

I compiled a binding version for the user with a timeout of 20 seconds instead of 5 seconds, with no effect. I would have assumed that the observed error would be reported somewhat later by having a timeout of 20 seconds. The second posted log-listing is from the binding with 20 seconds timeout, Notice that the time between the start of the request and the reporting of the error is still less then a second!

What could cause such an error?
The network-connection to the internet is reported to be stable, a missing connection would have rissen another error anyway!

2

I let the user test the WeatherUnderground binding, which is also using the HttpUtil.
I would have assumed that the error would come up in that binding as well, however this binding worked!
As far as I can see the main difference in the use of “executeURL” (beside the use of a different overload) is the TimeOut of 30 seconds in the WeatherUnderground binding. I don’t think that a shorter TimeOut is the cause, since in the test with the TimeOut extended to 20 seconds in the Tanklerkoenig binding, the error was observed after 1 second.
Shouldn’t the time between the GET-request and the error be something like the TimeOut setting?

3

The user observing the initial problem did find the reason as well as a code solution to overcome it.

The reason was the handling of the https request (more precise the keystore wasn’t read). Even the posted workaround (FAQ Tankerkoenig) didn’t help, although he can get a valid response using the browser on the system running OH.

He found a solution to load the keystore, which I think isn’t part of OH yet (I could be wrong in this), however it relies on the usage of the packages:

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;

import java.net.URL;
import java.util.Properties;

import java.security.KeyStore;
import java.security.cert.X509Certificate;

import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;

Are those packages, especially the java.security- and javaxx-ones, in the scope of being used for OH?

I do think that such a change would affect core bundles, with my limited depth of knowledge I’m hesitating to do anything in there at all.
@martinvw @Kai

4

imho, they are part of the JDK so that should not be a problem

5

In that case I’d give it a try.
Please correct me if I’m wrong, such a change should/would be done in eclipse.smarthome io.net.httpUtil or even a httpsUtil.

6

By default the JDK should load a certain default keystore, maybe we should describe properly how to add keys there.

7

Could you share to code which makes it work, it could give a feeling how viable a solution could be.

8

That is what I was thinking in the first place.

I haven’t tested the code it myself, but that is what has been sent to me (hopefully I’ve posted all needed);

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
//import java.net.HttpUrlConnection;
import java.net.URL;
import java.security.KeyStore;
import java.security.cert.X509Certificate;
import java.util.Properties;

import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;

import org.eclipse.smarthome.io.net.http.HttpUtil;
import org.openhab.binding.tankerkoenig.internal.config.TankerkoenigDetailResult;
import org.openhab.binding.tankerkoenig.internal.config.TankerkoenigListResult;
import org.openhab.binding.tankerkoenig.internal.serializer.CustomTankerkoenigDetailResultDeserializer;
import org.openhab.binding.tankerkoenig.internal.serializer.CustomTankerkoenigListResultDeserializer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;

/***
 * Serivce class requesting data from tankerkoenig api and providing result objects
 *
 * @author Dennis Dollinger
 * @author Juergen Baginski
 */
public class TankerkoenigService {
    private final Logger logger = LoggerFactory.getLogger(this.getClass());
    private final GsonBuilder gson_Builder = new GsonBuilder().registerTypeAdapter(TankerkoenigListResult.class,
            new CustomTankerkoenigListResultDeserializer());
    private final Gson gson = gson_Builder.create();
    private final GsonBuilder gson_Builder_Detail = new GsonBuilder()
            .registerTypeAdapter(TankerkoenigDetailResult.class, new CustomTankerkoenigDetailResultDeserializer());;
    private final Gson gson_Detail = gson_Builder_Detail.create();
    private static final int REQUEST_TIMEOUT = 5000;

    public TankerkoenigListResult getStationListData(String apikey, String locationIDs, String userAgent,
            boolean bUseStdSSL, String strKeystorePath, String strKeystorePass, String strCertAlias) {
        return this.getTankerkoenigListResult(apikey, locationIDs, userAgent, bUseStdSSL, strKeystorePath,
                strKeystorePass, strCertAlias);
    }

    public TankerkoenigDetailResult getStationDetailData(String apikey, String locationID, String userAgent,
            boolean bUseStdSSL, String strKeystorePath, String strKeystorePass, String strCertAlias) {
        return this.getTankerkoenigDetailResult(apikey, locationID, userAgent, bUseStdSSL, strKeystorePath,
                strKeystorePass, strCertAlias);
    }

    private String executeGetRequest(String apiKey, String locationIDs, String userAgent, boolean detail,
            String strKeystorePath, String strKeystorePass, String strCertAlias) {
        javax.net.ssl.HttpsURLConnection conn = null;

        // Create a trust manager that does not validate certificate chains
        TrustManager[] trustAllCerts = new TrustManager[] { new X509TrustManager() {
            @Override
            public java.security.cert.X509Certificate[] getAcceptedIssuers() {
                return new java.security.cert.X509Certificate[0];
            }

            @Override
            public void checkClientTrusted(java.security.cert.X509Certificate[] certs, String authType) {
            }

            @Override
            public void checkServerTrusted(java.security.cert.X509Certificate[] certs, String authType) {
            }
        } };

        try {
            // Create URI
            StringBuilder sb = new StringBuilder();
            sb.append("https://creativecommons.tankerkoenig.de/json/");
            if (detail) {
                sb.append("detail.php?id=");
            } else {
                sb.append("prices.php?ids=");
            }
            sb.append(locationIDs);
            sb.append("&apikey=");
            sb.append(apiKey);

            logger.debug("Building URL : " + sb.toString());
            URL url = new URL(sb.toString());

            // Loading Keystore
            KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
            File ksFile = new File(strKeystorePath);
            FileInputStream inFS = new FileInputStream(ksFile);
            ks.load(inFS, strKeystorePass.toCharArray());
            X509Certificate cert = (X509Certificate) ks.getCertificate(strCertAlias);

            // Setting Keystore to SSLContext
            SSLContext sc = SSLContext.getInstance("TLS");
            sc.init(null, trustAllCerts, new java.security.SecureRandom());
            javax.net.ssl.HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());

            // Create Connection
            conn = (javax.net.ssl.HttpsURLConnection) url.openConnection();
            conn.setRequestMethod("GET");

            conn.setRequestProperty("USER-AGENT", userAgent);
            logger.debug("Request sent");

            // Get Response
            InputStream is = conn.getInputStream();
            BufferedReader rd = new BufferedReader(new InputStreamReader(is));
            StringBuilder response = new StringBuilder();
            logger.debug("Response received");
            String line;
            while ((line = rd.readLine()) != null) {
                response.append(line);
                response.append('\r');
            }
            rd.close();
            logger.debug("Response data: " + response.toString());
            return response.toString();
        } catch (Exception e) {
            logger.debug("Error in executeGetRequest: ", e);
            return null;
        } finally {
            if (conn != null) {
                conn.disconnect();
            }
        }
    }
9

I’m sorry that I’m bursting in here.
I’m the author of that code. Feel free to ask me, if I should explain, what the problem was and why I have resolve it in this way.

1 Like
10

Then you are very welcome here :slight_smile:

Would it also work just importing the certificate or even keychain in the jdk’s truststore?

11

What do you mean with jdk’s truststore? The cacerts-File assigned to the jvm?
This is the solution described in the Tankerkönig-FAQ and it not works for me and the main problem was, to understand, that this solution not works. There was some different exceptions at runtime, sometimes a SSL failure, sometimes a Timeout, often a ClosedChannel exception.
After change the code to my new executeGetRequest-Method (without manipulate some SSL crap), the exception messages was more in detail and I’m sure, that the certificate was not loaded by the runtime or there is a problem by the SSL handshake.
My theory was substantiated by a TCPDump and here I could see, that the DNS resolv was success, but no connection attempt after this. So my idea was, to set the certifcate chain direct to the SSL connection and, to be sure, I disable the server certificate validation.

Maybe my “special” runtime environment is a reason for the problem. It’s a simple RaspPi2 with a running OH1 instance under Java7 and I’m in the migration phase to OH2 (running under Java8). I added the certificate, that was missing, to the cacerts-File in both Java environments with the procedure described in the Tankerkönig FAQ.

Please note, that I’m a totally beginner in OH development and my Java knowledge is from the middle of the 90’s. So the code would be very ugly and the solution probably a dirty hack.

I want to thank JĂĽrgen (Opus) here for his support and assistance and the time/patience he has spent with me.

12

What certificates did you import in the custom defined truststore?

I would love to see the minimal required changes, so does it work because of trustAllCerts or because of loading the truststore or only in the combination.

13

I did this:
sudo wget http://www.startssl.com/certs/ca.crt
keytool -import -keystore cacerts -alias startssl -file ca.crt

14

can you check which keytool

15

pi@pi:/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/security $ keytool -list -keystore cacerts -v -alias startssl
Keystore-Kennwort eingeben:
Aliasname: startssl
Erstellungsdatum: 11.11.2017
Eintragstyp: trustedCertEntry

EigentĂĽmer: CN=StartCom Certification Authority, OU=Secure Digital Certificate Signing, O=StartCom Ltd., C=IL
Aussteller: CN=StartCom Certification Authority, OU=Secure Digital Certificate Signing, O=StartCom Ltd., C=IL
Seriennummer: 1
GĂĽltig von: Sun Sep 17 21:46:36 CEST 2006 bis: Wed Sep 17 21:46:36 CEST 2036
Zertifikat-Fingerprints:
MD5: 22:4D:8F:8A:FC:F7:35:C2:BB:57:34:90:7B:8B:22:16
SHA1: 3E:2B:F7:F2:03:1B:96:F3:8C:E6:C4:D8:A8:5D:3E:2D:58:47:6A:0F
SHA256: C7:66:A9:BE:F2:D4:07:1C:86:3A:31:AA:49:20:E8:13:B2:D1:98:60:8C:B7:B7:CF:E2:11:43:B8:36:DF:09:EA
Signaturalgorithmusname: SHA1withRSA
Version: 3

Erweiterungen:

#1: ObjectId: 2.16.840.1.113730.1.13 Criticality=false
0000: 16 29 53 74 61 72 74 43 6F 6D 20 46 72 65 65 20 .)StartCom Free
0010: 53 53 4C 20 43 65 72 74 69 66 69 63 61 74 69 6F SSL Certificatio
0020: 6E 20 41 75 74 68 6F 72 69 74 79 n Authority

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:true
PathLen:2147483647
]

#3: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
[DistributionPoint:
[URIName: http://cert.startcom.org/sfsca-crl.crl]
, DistributionPoint:
[URIName: http://crl.startcom.org/sfsca-crl.crl]
]]

#4: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
[CertificatePolicyId: [1.3.6.1.4.1.23223.1.1.1]
[PolicyQualifierInfo: [
qualifierID: 1.3.6.1.5.5.7.2.1
qualifier: 0000: 16 23 68 74 74 70 3A 2F 2F 63 65 72 74 2E 73 74 .#http://cert.st
0010: 61 72 74 63 6F 6D 2E 6F 72 67 2F 70 6F 6C 69 63 artcom.org/polic
0020: 79 2E 70 64 66 y.pdf

], PolicyQualifierInfo: [
qualifierID: 1.3.6.1.5.5.7.2.1
qualifier: 0000: 16 29 68 74 74 70 3A 2F 2F 63 65 72 74 2E 73 74 .)http://cert.st
0010: 61 72 74 63 6F 6D 2E 6F 72 67 2F 69 6E 74 65 72 artcom.org/inter
0020: 6D 65 64 69 61 74 65 2E 70 64 66 mediate.pdf

], PolicyQualifierInfo: [
qualifierID: 1.3.6.1.5.5.7.2.2
qualifier: 0000: 30 81 C3 30 27 16 20 53 74 61 72 74 20 43 6F 6D 0…0’. Start Com
0010: 6D 65 72 63 69 61 6C 20 28 53 74 61 72 74 43 6F mercial (StartCo
0020: 6D 29 20 4C 74 64 2E 30 03 02 01 01 1A 81 97 4C m) Ltd.0…L
0030: 69 6D 69 74 65 64 20 4C 69 61 62 69 6C 69 74 79 imited Liability
0040: 2C 20 72 65 61 64 20 74 68 65 20 73 65 63 74 69 , read the secti
0050: 6F 6E 20 2A 4C 65 67 61 6C 20 4C 69 6D 69 74 61 on Legal Limita
0060: 74 69 6F 6E 73 2A 20 6F 66 20 74 68 65 20 53 74 tions of the St
0070: 61 72 74 43 6F 6D 20 43 65 72 74 69 66 69 63 61 artCom Certifica
0080: 74 69 6F 6E 20 41 75 74 68 6F 72 69 74 79 20 50 tion Authority P
0090: 6F 6C 69 63 79 20 61 76 61 69 6C 61 62 6C 65 20 olicy available
00A0: 61 74 20 68 74 74 70 3A 2F 2F 63 65 72 74 2E 73 at http://cert.s
00B0: 74 61 72 74 63 6F 6D 2E 6F 72 67 2F 70 6F 6C 69 tartcom.org/poli
00C0: 63 79 2E 70 64 66 cy.pdf

]] ]
]

#5: ObjectId: 2.5.29.15 Criticality=false
KeyUsage [
DigitalSignature
Key_Encipherment
Key_Agreement
Key_CertSign
Crl_Sign
]

#6: ObjectId: 2.16.840.1.113730.1.1 Criticality=false
NetscapeCertType [
SSL CA
S/MIME CA
Object Signing CA]

#7: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 4E 0B EF 1A A4 40 5B A5 17 69 87 30 CA 34 68 43 N…@[…i.0.4hC
0010: D0 41 AE F2 .A…
]
]

16

I literally mean the command which keytool

17

Ups. Sorry

pi@pi:~ $ which keytool
/usr/bin/keytool

18

A that is not usefull:

ls -la `which keytool`
19

pi@pi:~ $ ls -la which keytool
lrwxrwxrwx 1 root root 25 Jan 5 2016 /usr/bin/keytool → /etc/alternatives/keytool

20

That was again the correct command, but it does not help me any further :smiley:

I don’t know whether you acted on that but the keytool is part of the Java runtime, so if you have multiple java versions running you most likely also have multiple versions of keytool running. So for predictability it would be great to use the Java 8 keytool to update the java 8 cacerts file and the other way around.

Did you also see my other question above:

I would love to see the minimal required changes, so does it work because of trustAllCerts or because of loading the truststore or only in the combination.