[JSR223] [JYTHON] Jython with requests successful?

JimT · December 13, 2019, 1:10pm

Definitely. However, I ran into problems with the requests module. One error was as reported in the OP. Another error was the lack of support for SNI as I explained several posts above. So requests is a no go for me unfortunately.

Spaceman_Spiff · December 13, 2019, 1:16pm

You could use HABApp and the normal request module since it’s python >= 3.6.

JimT · December 13, 2019, 1:16pm

Do you have a helper class to deal with curl / executeCommandLine? If none exists, I might write one. It would be super simple, but it would avoid code duplications. I’m thinking something like

curl = Curl("https://www.something.com")
curl.add_header(....)
curl.set_params(....)
curl.add_data(...) to be called multiple times - for POST), or curl.set_data(pass the data in a list)
curl.get() or curl.post()

In the background it will deal with executeCommandLine’s weird “@@” thing

Dibbler42 · December 13, 2019, 1:19pm

Is the requests error only within jython? On my system i get an error using requests within jython. If i use pur python it works like a charm.

To day i opend an issue at jythons github page

JimT · December 13, 2019, 1:19pm

Wow, I’ve never heard of HABApp. Took a quick look at it. It seems like a viable solution. Must investigate more.

JimT · December 13, 2019, 1:22pm

Correct. See Post #9

I doubt if they can do much, but I’d be curious to know. It would take a while to trickle down to OpenHAB too, so it’s a very very long shot. For the time being curl works perfectly for me. Just need to write a helper class for it to make it nicer to use.

5iver · December 13, 2019, 1:33pm

No… I’m familiar enough with curl and OH’s requirements that I haven’t needed one.

If there were to be a fix in Jython, it would be immediately available in OH. You can manually install whatever version of Jython you’d like. I’ve experienced errors causing rules not to run in 2.7.1 and 2.7.2b2. Oddly, the 2.7.2a1+ build that I made never had an issue.

rlkoshak · December 13, 2019, 3:54pm

I can confirm, the Java HTTP libraries support them. You can see examples (Rules DSL but the Java will still be the same) at:

You could also use executeCommandLine or subprocess (subprocess works a bit better in my experience) to call curl as a work around. Here is an example using subprocess and wget (I’m just grabbing an icon).

from core.rules import rule
from core.triggers import when
from configuration import weather_icon_path
import subprocess
from javax.imageio import ImageIO
from java.io import File

@rule("Weather Icon",
      description="Copy the current weather conditions icon",
      tags=["weather"])
@when("Item vWeather_Conditions_Icon changed")
@when("System started")
def cond_icon(event):
    """
    Download the weather conditions icon from DarkSky and convert it from gif to
    png.
    """
    # OpenWeatherMap
#    cond_icon.log.info("Fetching weather conditions icon to {} from {}"
#                  .format(weather_icon_path, items["vWeather_Conditions_Icon"]))
#
#    results = subprocess.check_output(['/usr/bin/wget', '-q', '-O',
#                                       weather_icon_path,
#                                       str(items["vWeather_Conditions_Icon"])])
#
#    input_file  = File(weather_icon_path)
#    if not input_file.exists():
#        cond_icon.log.warn("Failed to fetch the weather icon!")
#        return
#
#    output_file = File(weather_icon_path.replace('gif', 'png'))
#    ImageIO.write( ImageIO.read(input_file), 'png', output_file)
#
#    results = subprocess.check_output(['rm', weather_icon_path])

    # DarkSky
    cond_icon.log.info("Fetching the weather conditions icon... {}".format(ir.getItem("vWeather_Conditions_Icon").state))
    dl = subprocess.Popen(['/usr/bin/wget', '-qO-',
                  'http://argus:8080/rest/items/vWeather_Conditions_Icon/state'],
                  stdout=subprocess.PIPE)
    dd = subprocess.Popen(['/bin/dd', 'bs=22', 'skip=1'], stdin=dl.stdout, stdout=subprocess.PIPE)
    dl.wait()
    f = open(weather_icon_path, "w")
    subprocess.call(['/usr/bin/base64', '-d'], stdout=f, stdin=dd.stdout)
    dd.wait()

Note, the ImageIO class I use to convert the gif to png is a Java class.

If you make a helper I strongly recommend using subprocess instead of executeCommandLine. It can be quite challenging if not impossible to get executeCommandLine to work sometimes. The way the arguments are pass as an array or tuple in subprocess helps avoid the whole @@ nonsense. In fact, if you look at the simple command I try to run in the commented out OpenWeatherMap example above, I never was able to get that simple wget to run using executeCommandLine. I used subprocess out of desperation and was very happy with how easy it was to use.

5iver · December 13, 2019, 4:18pm

The trouble with subprocess is that it does not have a built-in timeout. Early on when I was still getting an understanding of the HLs, I submitted a PR that used subprocess, but cancelled the PR since executeCommandLine worked just fine.

Rather than wrap executeCommandLine or subprocess, we should just make executeCommandLine easier to use or add the functionality to the HTTTP Action. It is really easy to test things out in scripted automation, but we’ll have a mess if we don’t put the functionality back into OH.

rlkoshak · December 13, 2019, 4:27pm

But that’s the problem. executeCommandLine doesn’t just work fine. Sometimes you need @@, sometimes you don’t. Sometimes even with the @@ it doesn’t work. Take that simple wget I have above. I tried about 30 different combinations over about two hours trying to get it to work with executeCommandLine. None of them worked. The only way I could get it to work was to put the wget into a script or call subprocess.

Perhaps bringing in subprocess32 would be a solution since it does support a timeout.

I agree, ultimately fixing executeCommandLine would be better, but right now I assert that executeCommandLine is fundamentally broken. What ever was done in it to handle the arguments before the command get’s passed to the Java libraries that execute the command only works sometimes. I would not try to base anything on it unless and until it get’s fixed.

And it’s worth noting that I have some preliminary suspicions that the timeout argument passed to executeCommandLine is not always followed either. I don’t have enough yet to file an issue though.

As for fixing the HTTP Actions, I assumed, perhaps incorrectly, that the 2.x version of the binding would implement replacements for those Actions, similar to how MQTT replaced publish.

5iver · December 13, 2019, 8:35pm

Hmmm… I didn’t have any problems with this, unless I left out the timeout…

from core.actions import Exec

test = Exec.executeCommandLine("/usr/bin/wget -qO- https://ci.openhab.org/job/openHAB2-Bundles/lastSuccessfulBuild/console", 10000)

rlkoshak · December 13, 2019, 8:46pm

That was one of the variations that I tried. I don’t remember the exact behavior I saw with that specific variation but the end result was it didn’t work. For all attempts I would verify that it runs from the command prompt first as the openHAB user exactly as typed (minus any @@ of course) and verify it worked. I’d try to run it from executeCommandLine and I’d get no output or an error message. Maybe it’s because I’m running in Docker (note, I executed the command lines from inside the container). But I’m not alone. Search the forum and you will see many threads where the solution was “¯\_(ツ)_/¯, put it in a shell script and call the script from executeCommandLine.”

JimT · December 14, 2019, 4:25am

I actually tried subprocess.check_output before resorting to executeCommandLine. For me, subprocess.check_output didn’t work. It caused a python error about popen argument being wrong or missing argument.

executeCommandLine is not bad if you break it down. Basically it wants the “arguments” (not space - the document is unclear and misleading) separated by @@. I should probably create a PR to edit the doc

Here’s an excerpt:

    def get_command(self, url, params):
        command = ["/usr/bin/curl", "-s", "-H", self.header, self.get_url(path, params)]
        return "@@".join(command)
...
   def something(self, blahblah)
        output = Exec.executeCommandLine(self.get_command(path, params), 10000)

Note that there can be spaces in self.header, path / params etc. Once you’ve figured this out, executeCommandLine isn’t so bad.

See, in the unix shell, you would call curl like this:

curl -s -H 'Header: blah blah' 'https://xxx.yy.zz/path?blahblah'

You would enclose the argument containing spaces with single or double quotes to tell the shell that it’s a part of a single argument. The shell will strip them off before passing them to the program (curl). With executeCommandLine, such syntax is not needed because it uses @@ as an argument separator, so spaces do not cause ambiguity as to whether it’s a part of the argument or an argument separator.

Sure, subprocess is a lot more versatile with input/output. I just wish I could make it work, and I didn’t spend too much time bothering with it. Maybe I will… and I’ll ping @rlkoshak

Hmmmm… I never got that far with subprocess. Did it fail because of the lack of timeout?

That would probably be the ideal. Anyone working on it?

JimT · December 14, 2019, 6:13am

I’ve just spent some time adding this to HTTP Action. It was very easy.

Now my code looks like:

        headers = {"Authorization": "FPDAPI SubscriberToken={}".format(self.token)}
        output = HTTP.sendHttpGetRequest(url, headers, 10000)

I will submit a PR to core!

JimT · December 14, 2019, 10:55am

5iver · December 14, 2019, 11:28am

And that is how it is done! Nice! The helper libraries could get huge if we don’t spend the time to get the functionality back into OH.

jswim788 · March 24, 2020, 5:03pm

Did this make it into the 2.5.x builds? I get this error using 2.5.3:

TypeError: sendHttpGetRequest(): expected 1-2 args; got 3

where the call is:

TokenResponse = HTTP.sendHttpGetRequest('https://data.tankutility.com/api/getToken', headers, 10000)

JimT · March 28, 2020, 2:15pm

I believe it made it into 2.5.2, so 2.5.3 should have it too. Have you tried to empty your cache and tmp directories?

jswim788 · March 28, 2020, 5:02pm

Emptied the cache and tmp but it made no difference. Isn’t this in the core, not an add-on? I thought the updates are now only for add-ons, not for the core. Are you running the standard 2.5.2 or 2.5.3? Or do you have your own customized core for this fix? Thanks.

JimT · April 1, 2020, 2:47am

I tried this code on my installation and it worked fine, no errors about the number of arguments. I am on 2.5.2. The only things in my addons folder are a custom daikin binding jar and the jython engine from scott (5iver).