sendHttpGetRequest does not respect User-Agent in header

Setup, Configuration and Use Scripts & Rules
1
  • Platform information:
    • Hardware: RaspberryPi 4, 8GB
    • OS: Raspbian GNU/Linux 10 (buster)
    • Java Runtime Environment: Zulu11.50+19-CA (build 11.0.12+7-LTS)
    • openHAB version: 3.4.1 - Release Build

Problem: as a user, I would expect sendHttpGetRequest to let the header information I am giving it take precedence over sendHttpGetRequest’s default values.

Description: Using javascript;version=ECMAScript-2021, I am trying to use send a sendHttpGetRequest with certain headers defined in a hashmap. All of the values in the hashmap is used by sendHttpGetRequest except for User-Agent. In the case of User-Agent, two values for User-Agent is sent to the server; the value I gave it in the hashmap and User-Agent: Jetty/9.4.46.v20220331.
Some servers accept the last User-Agent in the header while others (e.g. Microsoft-IIS/10.0 at AWS) will return an error. Technically, it is correct to return an error if multiple User-Agents are given in the header.
Here is the relevant part of the code:

var serverIP = "www.somehost.se";
var serviceURL = "some_service";
var headerHashMap = {
  'Host' : 'www.somehost.se',
  'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.59',
  'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
  'Accept-Language' : 'sv-SE,sv;q=0.8,en-US;q=0.5,en;q=0.3',
  'Accept-Encoding' : 'gzip, deflate',
  'DNT' : '1',
  'If-Modified-Since' : 'Mon, 08 Nov 2021 11:14:19 GMT',
};
var hostURL = "http://" + serverIP + "/" + serviceURL ;
console.info("url=" + hostURL);
var jsonData = actions.HTTP.sendHttpGetRequest(hostURL,headerHashMap,3000);
console.info(jsonData);

Using tcpdump and Wireshark, I have verified that two User-Agents are in fact sent from openHAB :

GET /some_service HTTP/1.1
Accept-Encoding: gzip
**User-Agent: Jetty/9.4.46.v20220331**
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
**User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.59**
If-Modified-Since: Mon, 08 Nov 2021 11:14:19 GMT
Host: www.somehost.se
Accept-Encoding: gzip, deflate
DNT: 1
Accept-Language: sv-SE,sv;q=0.8,en-US;q=0.5,en;q=0.3

If I remove the User-Agent string from the hashmap, everything works fine except that the server of course returns a response for a Jetty client rather than the Mozilla/5.0 I was trying to get.
I have tried to find information in the documentation for sendHttpGetRequest that indicates that the content of the header is not handled but found nothing.

(I have tried http as well as https - no difference. The example is edited to hide the specific server/service)

Any clue as to what is going on here?

2

See

and the mentioned github feature request

3

@brusper Can you try renaming this file from .txt to .jar, then drop it into your addons folder. Make sure the original bundle org.openhab.core.io.net is not active.

org.openhab.core.io.net-3.4.3-SNAPSHOT.txt (43.8 KB)

For it to work, I had to completely uninstall the original org.openhab.core.io.net. Simply stopping it doesn’t work.

openhab> list -s | grep org.openhab.core.io.net
174 │ Active   │  80 │ 3.4.2                  │ org.openhab.core.io.net
294 │ Active   │  80 │ 3.4.3.202302281335     │ org.openhab.core.io.net

openhab> bundle:stop 174
openhab> bundle:uninstall 174

Note 174 is the bundle id on my system. It will most likely be different on yours, so adjust accordingly.
then restart openhab

If you want to restore things to the original bundle:

  • stop openhab
  • delete the jar from the addons folder
  • Delete the files inside the userdata/tmp/ directory, but don’t delete the tmp dir itself.
  • Delete the userdata/cache/org.eclipse.osgi/ directory
  • start openhab again

Disclaimer: The above steps worked on my system (docker). I’m not 100% of the repercussions of deleting the files inside tmp/ although I think it should be pretty safe to do so.

4

A workaround is this:

var headerHashMap = {
  'Host' : 'www.somehost.se',
  'User-Agent' : null, // delete previous User-Agent
  'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.59',
  'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
  'Accept-Language' : 'sv-SE,sv;q=0.8,en-US;q=0.5,en;q=0.3',
  'Accept-Encoding' : 'gzip, deflate',
  'DNT' : '1',
  'If-Modified-Since' : 'Mon, 08 Nov 2021 11:14:19 GMT',
};
5

Jim, thank you for your input and ideas. I will test them ASAP! The “null”-version seems promising given the discussions I have seen over at the Eclips forum.

6

@JimT Unfortunately, the User-Agent : null idea did not work for me. With the header hashmap as in your example above, Jetty still sends the same:

Hypertext Transfer Protocol
    GET /some_service HTTP/1.1\r\n
    Accept-Encoding: gzip\r\n
    User-Agent: Jetty/9.4.46.v20220331\r\n
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\r\n
    User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0\r\n
    If-Modified-Since: Mon, 08 Nov 2021 11:14:19 GMT\r\n
    Host: some.server.se\r\n
    Accept-Encoding: gzip, deflate\r\n
    DNT: 1\r\n
    Accept-Language: sv-SE,sv;q=0.8,en-US;q=0.5,en;q=0.3\r\n
    Cookie: ASP.NET_SessionId=mebkmgp2bjzbltm5id0xgrib\r\n
    \r\n
    [Full request URI: http://some.server.se/some_service]
    [HTTP request 1/1]

I will now try to drop in the modified org.openhab.core.io.net-3.4.3-SNAPSHOT.jar instead.

7

Finally! The modified org.openhab.core.io.net-3.4.3-SNAPSHOT.jar did the trick :slight_smile:

Thank you so much @JimT for your support. Now I just hope I’ll remember to take care of this modification when upgrading OpenHAB next time.

1 Like
8

This fix will be included in 3.4.3 as well as 4.0.

2 Likes