Issues with httpPostRequest in openHAB 3.4.1's JavaScript Automation Rule

GeorgeX · September 17, 2023, 8:50am

Hello openHAB Community!

I’m currently running openHAB 3.4.1 inside a Docker container on my Synology. I’m facing an issue with my automation rule written in JavaScript, located at .../conf/automation/js/some_rule.js.

When I try to update my Controme Gateway using the httpPostRequest method from this rule, it doesn’t behave as expected.

curl "http://192.168.178.81/m_setup/1/hardware/gwedit/3/" \
     -X POST \
     -H "Cookie: csrftoken=OBFUSCATED; sessionid=OBFUSCATED" \
     --data-raw "csrfmiddlewaretoken=OBFUSCATED&...rest_of_the_data..."

To replicate this in my openHAB rule, I’m using the following:

let url = "http://192.168.178.81/m_setup/1/hardware/gwedit/3/";
let contentType = "application/x-www-form-urlencoded";
let content = /*structured_content_data_here*/;
let headers = {
    "Cookie": "csrftoken=OBFUSCATED; sessionid=OBFUSCATED"
};
let timeout = 5000;  // Just as an example

actions.HTTP.sendHttpPostRequest(url, contentType, content, headers, timeout);

Even after ensuring the content is correctly formatted and URL-encoded, and headers are set up as in the curl command, the httpPostRequest fails to mirror the effect of the curl command.

Has anyone encountered a similar issue, especially in the context of JavaScript rules in openHAB? Any pointers or guidance would be greatly appreciated!

Thank you for your time and assistance!

Warm regards,
Frank

Wolfgang_S · September 17, 2023, 10:36am

And what does it mean ?
Do both fail ? Does one of them fail ? Is there any error message ?

As it is an http connection you can use a network sniffer like wireshark to analyze the traffic. Then you can compare the differences and take actions to make correct the request - as long as you have influence on all the header fields.

GeorgeX · September 17, 2023, 3:56pm

Hello Wolfgang,

Thank you so much for your swift response. I truly value your insight.

To provide a clearer picture:

The curl command operates seamlessly when I run it directly from the command line or within a script.
However, the httpPostRequest within my openHAB rule seems problematic. There isn’t a direct error message, but the outcome suggests the POST request isn’t executing as anticipated.

For better clarity, I’ve captured the entire curl command from the Firefox inspector, presented in Windows style:

curl "http://192.168.178.81/m_setup/1/hardware/gwedit/1/" -X POST 
-H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/117.0" 
-H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8" 
-H "Accept-Language: de,en-US;q=0.7,en;q=0.3" 
-H "Accept-Encoding: gzip, deflate" 
-H "Content-Type: application/x-www-form-urlencoded" 
-H "Origin: http://192.168.178.81" -H "Connection: keep-alive" 
-H "Referer: http://192.168.178.81/m_setup/1/hardware/gwedit/1/" 
-H "Cookie: csrftoken=SomeTokenData; sessionid=SomeSessionId" 
-H "Upgrade-Insecure-Requests: 1" 
--data-raw "csrfmiddlewaretoken=SomeTokenData&name=90-b2-e5-05-19-37&description=HKV+EG&out1=21&out2=10&out3=28&out4=21&out5=12&out6=24&out7=24&out8=28&out9=100&out10=100&out11=100&out12=100&out13=100&out14=100&out15=100&regelbereich_min=0&regelbereich_max=100"

From my analysis, the essential components for modifying the valve settings are the raw data and the cookie header.

To give a visual of the website I’m interfacing with: upon pressing the “speichern” (Save) button, there’s a redirect to the identical page. Below is an illustrative screenshot:

However, when initiating this POST request from openHAB, I encounter a divergent response:

Your idea of leveraging a network sniffer like Wireshark intrigues me. Could you guide me on employing Wireshark to capture the data exchange between the Controme mini server and my openHAB instance? I’m treading unfamiliar waters here and would appreciate any direction or reference materials.

Lastly, my overarching aim is to automate the adjustment of my floor heating circuits’ maximum opening to achieve hydraulic balancing. This effort is to enhance the efficiency of heating and cooling systems.

Thank you again for your invaluable guidance, and I’m eager to hear your further insights.

Warm regards,
Frank

Wolfgang_S · September 17, 2023, 7:01pm

Based on the fact that you get a login screen when you use the httpPostRequest it looks like the authentication would be missing.
Try to add the referer and the accepted languages in the header.

You need to install wireshark on one of the involved computers. Or you have a router that allows you to redirect a copy of the complete network to one of it’s interfaces.
I think on some routers it is possible to capture network traffic through the router.

GeorgeX · September 17, 2023, 8:00pm

Hi Wolfgang,

I in fact got my managed switch to mirror the ports on which my Synology and my Controme Miniserver are running to my wireshark PC and got it all up and running. I can see the difference in both calls - curl vs. rule.

It seems that my openhab JavaScripting call of actions.HTTP.sendHttpPostRequest adds a unwanted cookie into the headers hash map.

This is the wireshark recording of the POST triggered by the RULE:
Here you can see the additional cookie entry.
Yet another unwanted and uncalled header entry here is: Accept-Encoding:

Hypertext Transfer Protocol
    POST /m_setup/1/hardware/gwedit/3/ HTTP/1.1\r\n
    Accept-Encoding: gzip\r\n
    User-Agent: Jetty/9.4.46.v20220331\r\n
    Cookie: csrftoken=0xyL....cC3I; sessionid=5fw8....iurp\r\n
        Cookie pair: csrftoken=0xyL....cC3I
        Cookie pair: sessionid=5fw8....iurp
    Content-Type: application/x-www-form-urlencoded\r\n
    Host: 192.168.178.81\r\n
    Cookie: csrftoken=qwOk....V4Hf\r\n
        Cookie pair: csrftoken=qwOk....V4Hf
    Content-Length: 302\r\n
        [Content length: 302]
    \r\n
    [Full request URI: http://192.168.178.81/m_setup/1/hardware/gwedit/3/]
    [HTTP request 1/2]
    [Response in frame: 97546]
    [Next request in frame: 97550]
    File Data: 302 bytes

This is the successfully working CURL call in contrast:

Hypertext Transfer Protocol
    POST /m_setup/1/hardware/gwedit/3/ HTTP/1.1\r\n
    Host: 192.168.178.81\r\n
    User-Agent: curl/7.55.1\r\n
    Accept: */*\r\n
    Cookie: csrftoken=0xyL....cC3I; sessionid=5fw8....iurp\r\n
        Cookie pair: csrftoken=0xyL....cC3I
        Cookie pair: sessionid=5fw8....iurp
    Content-Length: 302\r\n
        [Content length: 302]
    Content-Type: application/x-www-form-urlencoded\r\n
    \r\n
    [Full request URI: http://192.168.178.81/m_setup/1/hardware/gwedit/3/]
    [HTTP request 1/1]
    [Response in frame: 6472]
    File Data: 302 bytes

In my preparation for the JavaScripting call of sendHttpPostRequest I tried to set the cookie value to ‘null’ prior to setting it to my desired value.

    // HTTP.sendHttpPostRequest(String url, String contentType, String content, Map<String, String> headers, int timeout)
    var url = "http://192.168.178.81/m_setup/1/hardware/gwedit/3/"
    var contentType = "application/x-www-form-urlencoded";
    var headers = {
        "Cookie": null,
        "Cookie": "csrftoken=0xyL....cC3I; sessionid=5fw8...iurp"
    };
    var payload = "csrfmiddlewaretoken=O6vY....9wvN&name=90-b2-e5-05-19-48&description=HKV+DG&out1=21&out2=17&out3=24&out4=7&out5=100&out6=100&out7=100&out8=100&out9=100&out10=100&out11=100&out12=100&out13=100&out14=100&out15=100&regelbereich_min=0&regelbereich_max=100"
    var response = actions.HTTP.sendHttpPostRequest(url, contentType, payload, headers, 40000);
    console.info("responseBody: " + response);

I found a Similar Community Topic that refers to the same behaviour of header values being pre set to something unwanted.

It seems I need to upgrade to something past 3.4.3 now. :-/

Any suggestions before I do that?

Cheers
Frank

Wolfgang_S · September 18, 2023, 1:56am

In case you do not want to upgrade or try something else you could try to create a shell script that uses curl. The shell script then is started by executing executeComandLine.

jimtng · September 18, 2023, 9:48am

In case you decided to upgrade, please let us know whether the upgrade solved the problem.

jimtng · September 18, 2023, 12:53pm

I tested this on openhab 3.4.5, using jruby script (but it should work the same on javascript)

require 'openhab/dsl'

url = "http://192.168.1.10:8088/"
contentType = "application/x-www-form-urlencoded"
content = "Data here"
headers = {
  Test1: "Test1",
  Cookie: "csrftoken=OBFUSCATED; sessionid=OBFUSCATED",
}.transform_keys(&:to_s)

logger.warn HTTP.send_http_post_request(url, contentType, content, headers: headers)

I listened on that ip on port 8088 using nc -l 8088 and this is what I see from nc:

# nc -l 8088
POST / HTTP/1.1
Accept-Encoding: gzip
User-Agent: Jetty/9.4.46.v20220331
Test1: Test1
Cookie: csrftoken=OBFUSCATED; sessionid=OBFUSCATED
Content-Type: application/x-www-form-urlencoded
Host: 192.168.1.10:8088
Content-Length: 9

Data here

The cookie header isn’t duplicated or doing anything funny.

GeorgeX · September 24, 2023, 4:03pm

Hi All,

After some deep dive into the issue I had with the actions.HTTP.sendHttpPostRequest, I wanted to share my observations and a workaround.

Issue Recap: Using actions.HTTP.sendHttpPostRequest in openHAB JavaScripting resulted in multiple cookies being sent in the header. My hunch is that upon the initial request, my controme system might be sending a new cookie, which somehow becomes a permanent part of subsequent requests. Even if I provided a custom header with the intended cookie, the function seemed to add the initially received cookie to the request.

This behavior persisted even after I updated openHAB from version 3.4.1 to 3.4.5.

Workaround: Following @wolfgang’s advice, I decided to execute the curl command directly from my rule. Although it was a bit challenging to get actions.Exec.executeCommandLine to work properly, I succeeded in the end. Each parameter snipped apparently needs to be its own string in the parameter list.

For those facing similar issues, here’s my implementation (note: sensitive tokens, session IDs, and cookies have been redacted for privacy):

javascript

var response = actions.Exec.executeCommandLine(
    "/usr/bin/curl", 
    'http://192.168.178.81/m_setup/1/hardware/gwedit/3/',
    '-X', 'POST',
    '-H', 'Cookie: [REDACTED_COOKIE_CONTENT]',
    '--data-raw', '[REDACTED_POST_DATA]'
);

Though not the most straightforward solution, it works reliably, bypassing the quirks of the sendHttpPostRequest function.

Hope this helps someone out there who might be facing similar challenges. If there’s any update on a fix or if the behavior of sendHttpPostRequest changes in future releases, do share.

Best regards,
Frank

jimtng · September 25, 2023, 4:17am

If sendHttpPostRequest is inserting extra headers / cookies, it needs to get fixed. I’d like to be able to first reproduce the issue, but I haven’t been able to.

So is this the exact code you used that caused the duplicate Cookie headers?

// HTTP.sendHttpPostRequest(String url, String contentType, String content, Map<String, String> headers, int timeout)
    var url = "http://192.168.178.81/m_setup/1/hardware/gwedit/3/"
    var contentType = "application/x-www-form-urlencoded";
    var headers = {
        "Cookie": null,
        "Cookie": "csrftoken=0xyL....cC3I; sessionid=5fw8...iurp"
    };
    var payload = "csrfmiddlewaretoken=O6vY....9wvN&name=90-b2-e5-05-19-48&description=HKV+DG&out1=21&out2=17&out3=24&out4=7&out5=100&out6=100&out7=100&out8=100&out9=100&out10=100&out11=100&out12=100&out13=100&out14=100&out15=100&regelbereich_min=0&regelbereich_max=100"
    var response = actions.HTTP.sendHttpPostRequest(url, contentType, payload, headers, 40000);
    console.info("responseBody: " + response);

Can you provide a code that is not redacted, i.e. feel free to replace the secrets with something else, as long as they still cause the double headers, but I need a verbatim code to copy paste / reproduce without any modifications on my part.

GeorgeX · September 25, 2023, 8:36am

Hello JimT,

Thank you for taking the time to look into the issue of duplicate Cookie headers.

Here’s the exact code that I used, which I believe resulted in the double headers. This is a non-redacted version, although I’ve obfuscated the sensitive tokens. If you can run this code exactly as it is, you might be able to reproduce the issue.

javascript sitting here:

docker\openhab\conf\automation\js\controme-hydrolic-control.js

function testHttpRequest() {
    var url = "http://192.168.178.81/m_setup/1/hardware/gwedit/3/";
    var contentType = "application/x-www-form-urlencoded";
    var headers = {
        "Accept": "*/*",
        "Accept-Encoding": "*/*",
        "Cookie": "csrftoken=nw4DX...O4gA; sessionid=jgxv...pkk4"
    };
    var payload = "csrfmiddlewaretoken=06Rr...xRlB&name=90-b2-e5-05-19-48&description=HKV+DG&out1=21&out2=16&out3=23&out4=7&out5=100&out6=100&out7=100&out8=100&out9=100&out10=100&out11=100&out12=100&out13=100&out14=100&out15=100&regelbereich_min=0&regelbereich_max=100";
    var response = actions.HTTP.sendHttpPostRequest(url, contentType, payload, headers, 40000);
    console.info("responseBody: " + response);
}

rules.JSRule({
    name: "Hydrolic Equilibrium Control",
    description: "Ensures that the hydrolic equilibrium is maintained",
    triggers: [
        triggers.ItemStateChangeTrigger('DEBUG_VIRTUAL_SWITCH', 'OFF', 'ON')
    ],
    execute: (event) => {
        var debug_itm = items.getItem("DEBUG_VIRTUAL_SWITCH");
        if(debug_itm.state == "ON") { debug_itm.sendCommand("OFF"); }        
        else { return; }
        console.info("Hydrolic Equilibrium Control: triggered by DEBUG_VIRTUAL_SWITCH");
        testHttpRequest();
        return;
    }
});

Note: To possibly reproduce this issue, the server at http://192.168.178.81/m_setup/1/hardware/gwedit/3/ would need to suggest a cookie to the requester system, as you mentioned.

Please let me know if you are able to reproduce the issue with this code. Thanks again for your help!

Best regards,
Frank

system · November 6, 2023, 12:37am

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.