Adding python modules to jython

I redownloaded the two jar files from your google drive today.

By having it there, I don’t need to do it in each and every script. In fact, if the “syspath” rule gets reloaded, the path will go in there again and again. That means if I added the path in multiple scripts, there’ll be multiple entries. I’ll need to check before appending.

If you are using S1775 or newer, or 2.5 stable (once released), you will not need the org.openhab.core.automation.script jar, since this PR was merged. The ScriptEngineFactory jar is still needed.

I use containers to run OH and surrounding services (on Debian 10) and am struggling a bit with finding an optimal way to handle Python libraries needed by the scripts.

OH configuration files and python scripts are maintained in a Github repo (with git-crypt for the secret stuff) and I’d rather not add libraries into conf/automation/lib/python/... because they will either end up in the repo or must be excluded and reinstalled in some other way when redeploying a container.
Currently I install them with a requirements.txt in the Dockerfile and they are installed in /usr/local/lib/python2.7/dist-packages in the container image. But when I add this path with EXTRA_JAVA_OPTS="-Dpython.path=/usr/local/lib/python2.7/dist-packages", the library cannot be imported:

[ERROR] [ipt.internal.ScriptEngineManagerImpl] - Error during evaluation of script 'file:/openhab/conf/automation/jsr223/python/personal/astro_sun_tomorrow.py': ImportError: No module named astral in <script> at line number 20

In the container, the settings are:

# echo $EXTRA_JAVA_OPTS
-Duser.timezone=Europe/Amsterdam -Dpython.path=/usr/local/lib/python2.7/dist-packages

# ls /usr/local/lib/python2.7/dist-packages
astral-1.10.1.dist-info  astral.py  astral.pyc  pytz  pytz-2019.3.dist-info

# python
Python 2.7.16 (default, Oct 10 2019, 22:02:15) 
[GCC 8.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print sys.path
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages']
>>> 

What am i missing?

Let’s discuss in the beta bundle thread so that this is all in one place…

To my knowledge you must install your Python2 packages locally, i.e. in OPENHAB_CONF/automation/lib/python/ for openHAB+Jython to see & use it. On openHABian:

$ sudo -u openhab pip install --target=/etc/openhab2/automation/lib/python PYTHON_PACKAGE_NAME

Here’s what I did to make it work - credits go to the astral documentation:

$ sudo -u openhab pip install --target=/etc/openhab2/automation/lib/python astral

Here’s a test rule that works:

from core.rules import rule
from core.triggers import when

from core.actions import LogAction

import pprint

pp = pprint.PrettyPrinter(indent=4)

from org.joda.time import DateTime

import datetime
from datetime import date

from astral import Astral

rule_init_timestamp = DateTime.now()
logTitle = "astral.py@{ts}".format(ts=rule_init_timestamp.toString("HH:mm:ss"))
ruleTimeStamp = " -- (Rule set initialised {date} at {ts})".format(
    date=rule_init_timestamp.toString("E d MMM yyyy"),
    ts=rule_init_timestamp.toString("HH:mm:ss (z)"),
)
rulePrefix = "Astral | "


a = Astral()
a.solar_depression = "civil"

city_name = "Brussels"
city = a[city_name]
LogAction.logInfo(logTitle, "Information for %s/%s\n" % (city_name, city.region))


timezone = city.timezone
LogAction.logInfo(logTitle, "Timezone: %s" % timezone)

LogAction.logInfo(
    logTitle,
    u"Latitude: %.04f ° %s; Longitude: %.04f ° %s\n"
    % (
        city.latitude,
        "S" if city.latitude < 0.0 else "N",
        city.longitude,
        "W" if city.longitude < 0.0 else "E",
    ),
)

sun = city.sun(date=date.today(), local=True)

LogAction.logInfo(logTitle, pp.pformat(sun))

moon_phase = a.moon_phase(date=date.today())

LogAction.logInfo(logTitle, "Moon phase: %.2f (%.2f %%)" % (moon_phase, 100*moon_phase/28.0))

Here’s the output for today:

==> /var/log/openhab2/openhab.log <==
2020-01-22 12:14:55.124 [INFO ] [me.core.service.AbstractWatchService] - Loading script 'python/personal/astral.py'
2020-01-22 12:14:55.809 [INFO ] [home.model.script.astral.py@12:14:55] - Information for Brussels/Belgium

2020-01-22 12:14:55.812 [INFO ] [home.model.script.astral.py@12:14:55] - Timezone: Europe/Brussels
2020-01-22 12:14:55.815 [INFO ] [home.model.script.astral.py@12:14:55] - Latitude: 50.8500 ° N; Longitude: 4.3500 ° E

2020-01-22 12:14:56.186 [INFO ] [home.model.script.astral.py@12:14:55] - {   u'dawn': datetime.datetime(2020, 1, 22, 7, 55, 2, tzinfo=<DstTzInfo 'Europe/Brussels' CET+1:00:00 STD>),
    u'dusk': datetime.datetime(2020, 1, 22, 17, 52, 46, tzinfo=<DstTzInfo 'Europe/Brussels' CET+1:00:00 STD>),
    u'noon': datetime.datetime(2020, 1, 22, 12, 53, 54, tzinfo=<DstTzInfo 'Europe/Brussels' CET+1:00:00 STD>),
    u'sunrise': datetime.datetime(2020, 1, 22, 8, 32, tzinfo=<DstTzInfo 'Europe/Brussels' CET+1:00:00 STD>),
    u'sunset': datetime.datetime(2020, 1, 22, 17, 15, 48, tzinfo=<DstTzInfo 'Europe/Brussels' CET+1:00:00 STD>)}
2020-01-22 12:24:28.670 [INFO ] [home.model.script.astral.py@12:24:28] - Moon phase: 25.00 (89.29 %)

Please note that in this script I’m using JodaTime (DateTime) for generating a script timestamp and that the rule uses regular datetime for execution.

1 Like

Here is what I’ve used. The --ignore-installed ignores what you have already installed so that the dependencies all come down with the package in the external directory.

sudo -u openhab pip2 install --ignore-installed --install-option="--prefix=/opt/openhab2/conf/automation/lib/python" PYTHON_PACKAGE_NAME

There was a bug in the beta Jython bundle, but it now properly appends the python.path to any that already exists. This was the issue Ron was running into.

Installing in the automation directory works (indeed using pip with the -t option).
But that’s not my issue. What I am struggling a bit with is the location of the libraries, somewhat deeply buried inside the /conf/automation/... tree.

For easy automated (re)deployment of OH - in my case with Ansible and Docker - I must be careful to not overwrite stuff within the automation directory that’s already there. I use this automation to deploy both new installations (not only of OH) and redeploy or upgrade existing ones. The conf directory is managed as a Github repository. In general, I would like to avoid more static stuff that is maintained outside my scope out of the conf directory.
Having to juggle with stuff inside directories in the /conf/automation tree with some stuff ‘immutable’ and some stuff dependent on my specific configuration or scripts, is sub-optimal. :sunglasses:

Also, it is not yet clear to me why the added sys.path doesn’t work in the first place. I double-checked the setting inside the container for Python with the openhab user…

It can be made to work for sure, but it feels like there should be a cleaner way by splitting the personal configuration (fully maintained by me) from the more static stuff (maintained in the OH community) on a higher level (outside conf for instance as I already mentioned). Your thoughts?

You can put your libraries wherever you’d like… just add the path to python.path. The beta Jython bundle was not handling this properly, but this should now be corrected. This means you can run pip to install to the default location and add that path to your python.path in EXTRA_JAVA_OPTS.

As an alternative, you should be able to append to sys.path. What problem are you having?

None, after I corrected a typo in my Docker file… :innocent:

TL;DR
I removed the manually installed python libraries from the /conf/automation/ path and installed them from the Docker file using requirements.txt. They end up in /usr/local/lib/python2.7/dist-packages in the container image. The EXTRA_JAVA_OPTS paths are picked up now and the installed libraries are found by the scripting engine.

Thanks for this quick fix!

1 Like

Need some help getting the requests package installed. When I try the above like this:

sudo -u openhab pip2 install --ignore-installed --install-option="--prefix=/etc/openhab2/automation/lib/python" requests

Then I get an error:

/usr/lib/python2.7/dist-packages/pip/commands/install.py:212: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
  cmdoptions.check_install_build_global(options)
The directory '/home/john/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/john/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting requests
  Downloading https://files.pythonhosted.org/packages/f5/4f/280162d4bd4d8aad241a21aecff7a6e46891b905a4341e7ab549ebaf7915/requests-2.23.0.tar.gz (114kB)
    100% |████████████████████████████████| 122kB 1.5MB/s
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/41/bf/9d214a5af07debc6acf7f3f257265618f1db242a3f8e49a9b516f24523a6/certifi-2019.11.28.tar.gz (156kB)
    100% |████████████████████████████████| 163kB 2.9MB/s
Collecting chardet<4,>=3.0.2 (from requests)
  Downloading https://files.pythonhosted.org/packages/fc/bb/a5768c230f9ddb03acc9ef3f0d4a3cf93462473795d18e9535498c8f929d/chardet-3.0.4.tar.gz (1.9MB)
    100% |████████████████████████████████| 1.9MB 456kB/s
Collecting idna<3,>=2.5 (from requests)
  Downloading https://files.pythonhosted.org/packages/cb/19/57503b5de719ee45e83472f339f617b0c01ad75cba44aba1e4c97c2b0abd/idna-2.9.tar.gz (175kB)
    100% |████████████████████████████████| 184kB 4.6MB/s
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
  Downloading https://files.pythonhosted.org/packages/09/06/3bc5b100fe7e878d3dee8f807a4febff1a40c213d2783e3246edde1f3419/urllib3-1.25.8.tar.gz (261kB)
    100% |████████████████████████████████| 266kB 2.8MB/s
Installing collected packages: certifi, chardet, idna, urllib3, requests
  Running setup.py install for certifi ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-vOYhi8/certifi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-HbGyvr-record/install-record.txt --single-version-externally-managed --compile --prefix=/opt/openhab2/conf/automation/lib/python --user --prefix=:
    running install
    error: [Errno 13] Permission denied: '/home/john/.local'

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-vOYhi8/certifi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-HbGyvr-record/install-record.txt --single-version-externally-managed --compile --prefix=/opt/openhab2/conf/automation/lib/python --user --prefix=" failed with error code 1 in /tmp/pip-build-vOYhi8/certifi/

Okay, so try with the -H option and it works, but it installs in /var/lib/openhab2/.local. My script is not able to find “requests” when it is there:

2020-03-22 20:16:05.139 [ERROR] [ipt.internal.ScriptEngineManagerImpl] - Error during evaluation of script 'file:/etc/openhab2/automation/jsr223/python/personal/tankUtility.py': ImportError: No module named requests in <script> at line number 7

So then I pulled out the ‘-u openHAB’ and the -H and install again with pip. This time I do end up with the requests where they were requested to be but owned by root, so I then reset the permissions. Sample location:

/etc/openhab2/automation/lib/python/lib/python2.7/site-packages/requests

Still, though, I get the same import error after a restart.

The script starts out like this, line 7 is the import line that fails.

"""
TankUtility Jython Rule
"""

from core.rules import rule
from core.triggers import when
import requests  <-- line 7, dead here
from requests.auth import HTTPBasicAuth

What next? Thanks for any help.

First, why are you trying to use requests, since there are several other options?

The heart of the script is making GET requests with authentication. Is there a better way to do this?

    jsonTokenResponse = requests.get('https://data.tankutility.com/api/getToken', auth=HTTPBasicAuth(TankUser, TankPassword)).json()
    jsonDeviceResponse = requests.get('https://data.tankutility.com/api/devices?token=' + jsonTokenResponse["token"]).json()
    # below is querying the first tank in the account - adjust the [0] if you want other tanks
    jsonTankDataResponse = requests.get('https://data.tankutility.com/api/devices/' + jsonDeviceResponse["devices"][0] + '?token=' + jsonTokenResponse["token"]).json()

thanks

GET requests can be done through HTTP Actions. Work is progressing on a 2.x HTTP binding which will (hopefully) incorporate these Actions. There are also several Java libraries. If these are better is subjective… but they work. If you are having difficulty installing requests, you may find these easier to get going.

The Actions are okay, but the authentication is a bit painful with them as you have to do it yourself with encoding it into a header string (unless I missed something). I was looking forward to easily porting Python scripts as it makes this much easier. I’ve already got the script running externally as a Python script, so I can take some time to debug this.

If you have requests installed, you just need to make them accessible in the classpath (python.path) to be able to use it within Jython. Based on your post, you were installing to

/etc/openhab2/automation/lib/python

Is it in this directory? If not, you can add your Python2 module directory to the python.path or create a symlink to it.

I installed and using request, this my instruction to install:

sudo pip install --upgrade --target=/datavol/openhab-2.5.1/openhab_conf/automation/lib/python requests

/datavol/openhab-2.5.1/openhab_conf maps to the OH conf directory, which here is mapped to my docker container.

putting the module to automation/lib/python you do not need to set classpath

1 Like

That appears to almost work for me. The automation/lib/python directory does get populated with a requests directory that has the files in it. And the “import requests” line works. Progress.

But the next line fails:

from requests.auth import HTTPBasicAuth

Error during evaluation of script ‘file:/etc/openhab2/automation/jsr223/python/personal/tankUtil.py’: ImportError: No module named auth in at line number 8

There is an auth.py and auth.pyc in the requests directory. Any idea why that can’t be found?

thanks

EDIT: happened to reboot, and now it appeared to load. Not sure why that is needed, but it looks better. Let me test this a bit further. Thanks

EDIT #2: onward to the next problem. This does indeed load requests, but then it breaks exactly as seen here. That’s the new challenge. I was hoping that Jython would drop in for Python, but I guess not.

Try:

import requests
from requests.auth import HTTPBasicAuth

together

1 Like

That works - thank you. I originally didn’t read your post carefully enough, but it does indeed install. But I hit this problem now as seen in an another thread, so I’ll try to follow that. I don’t think it is related to importing - that’s fine with your pip install line.

I had the same problem. The problem is not that I cannot extend the system path or configure my PYTHONPATH in /etc/default/openhab. The problem is that the openhab user needs to access it and this is not the user I work with on my operating system. I am using Linux in a virtual machine. Maybe this even works with openhabian, I can’t guarantee that.

You can install the package as follows, for example:

sudo -u openhab pip install --user requests

As you can see:

root@OpenHABVM:/var/lib/openhab/.local/lib/python2.7/site-packages# ls -la
total 48
drwx------ 12 openhab openhab 4096 Jul  5 16:08 .
drwx------  3 openhab openhab 4096 Jul  5 16:08 ..
drwxrwxr-x  2 openhab openhab 4096 Jul  5 16:08 certifi
drwxrwxr-x  2 openhab openhab 4096 Jul  5 16:08 certifi-2021.10.8.dist-info
drwxrwxr-x  4 openhab openhab 4096 Jul  5 16:08 chardet
drwxrwxr-x  2 openhab openhab 4096 Jul  5 16:08 chardet-4.0.0.dist-info
drwxrwxr-x  2 openhab openhab 4096 Jul  5 16:08 idna
drwxrwxr-x  2 openhab openhab 4096 Jul  5 16:08 idna-2.10.dist-info
drwxrwxr-x  2 openhab openhab 4096 Jul  5 16:08 requests
drwxrwxr-x  2 openhab openhab 4096 Jul  5 16:08 requests-2.27.1.dist-info
drwxrwxr-x  5 openhab openhab 4096 Jul  5 16:08 urllib3
drwxrwxr-x  2 openhab openhab 4096 Jul  5 16:08 urllib3-1.26.16.dist-info

In /etc/default/openhab I have following line:

EXTRA_JAVA_OPTS="-Xms9830m -Xmx9830m -Dpython.path=/var/lib/openhab/.local/lib/python2.7/site-packages:/usr/local/lib/python2.7/site-packages:/usr/local/lib/python2.7/dist-packages:/usr/local/lib/python2.7/dist-packages"