I’d like to import the “requests” python module for use in my jython script. How do I add the “requests” module? Is there a way/utility to manage the python libraries in /conf/automation/jsr223/ ? It would be great if we could do automatically download, list, delete etc.
Libraries do not go into /automation/jsr223/. They go into /automation/lib/. The location of the libraries can be modified, but I’m still working out how this might work with the bundle.
That is what pip provides… Installing Packages - Python Packaging User Guide. Just be careful and backup everything up!
You’re beta testing the Jython bundle, so this may not be possible, since you will need to install it first using pip. If you search the forum, there have been some discussions, but with using manual or standalone installs of Jython. I’m working on some instructions for using pip with Jython and OH, but I haven’t tried yet while using the bundle.
However, you can probably just use the HTTP actions… Actions — openHAB Helper Libraries documentation. If this does not suit your purposes, then explain what you intend to use it for and maybe someone can help.
I need to send a custom http header on the request, and afaik, the HTTP Action cannot do this.
I managed to use pip to install the modules into conf/automation/lib/python and that works. However, to keep things clean, I’d like to put pip-installed modules inside conf/automation/lib/python/libs (got any better name ideas?). My problem is to make jython add this into the search path.
It works if I added this to the top of my script:
import sys
sys.path.append('/openhab/conf/automation/lib/python/libs')
However, I’d rather not have to do this in every script. Is there a central / global place to add to python’s sys.path? I tried putting those two lines into conf/automation/lib/python/personal/init.py and then restarted the automation bundles (with bundle:restart xx), but this init.py didn’t seem to be loaded/read.
EDIT: Maybe a good name is conf/automation/lib/python/system ?
I’m no Python expert, but I think you can put the import in __init__.py. I don’t know though whether that is loaded and used for scripts.
But the better solution for you would probably be to add that folder to the -Dpython.path option for your Java Opts. Separate the two with a colon on Linux or semicolon on Windows.
-Dpython.path=/openhab/conf/automation/lib/python:/openhab/conf/automation/lib/python/libsI see no point in doing this, but you will need to make /automation/lib/python/libs a package by adding an __init_.py to it. You can just leave it empty.
https://docs.python.org/2.7/tutorial/modules.html#the-module-search-path
He is using the beta Jython bundle. I made a change a few days ago to append the existing python.path, so that users could specify paths to third party libs. My IDE is loaded with some other changes that I was testing, so give me a bit to clear it out and do a build for you. The bundle adds OPENHAB_CONF/automation/lib/python to python.path, so just this would be needed to add this other path…
EXTRA_JAVA_OPTS="-Dpython.path=/openhab/conf/automation/lib/python/libs"
1 Like
Solution:
I’m using openhab in a docker, on a centos 7 host system, where the python libraries are stored in /usr/lib/python2.7/site-packages
The solution is to mount it into docker in my docker-compose.yaml:
volumes:
- /usr/lib/python2.7/site-packages:/openhab/conf/automation/lib/python/system
- /usr/lib64/python2.7/site-packages:/openhab/conf/automation/lib/python/system64
I had to create an empty directory for conf/automation/lib/python/system and system64 in order for the above mount to work.
Next to add them to python sys.path, I created
conf/automation/jsr223/python/personal/__init__.py (not sure if the name init matters or just that it starts with an underscore so it loaded before other scripts)
from core.rules import rule
from core.triggers import when
import sys
@rule("syspath")
@when("System started")
def syspath(event):
sys.path.append('/openhab/conf/automation/lib/python/system')
sys.path.append('/openhab/conf/automation/lib/python/system64')
Now my openhab docker jython will pick up the system’s python libraries which I can manage using pip.
I think it’s also possible to have pip manage a separate python library for openhab. To do this, instruct pip with --target=/path/to/conf/automation/lib/python/system or perhaps using virtualenv
2 Likes
It seems that this doesn’t seem to work for me, and I had to use a script to call sys.path.append()
Had you upgraded the Jython bundle jar with the version that had this change in it?
An __init__.py file is used to create a package. Anything under /automation/jar223/ are scripts and can be named whatever you’d like. What you have will work, but probably better to rename it to avoid confusion. If you have not already, you should have a /automation/lib/python/personal/__init__.py to create your personal package. And yes, the name matters.
There’s no issue I can think of, but the rule here is unnecessary… the sys.path.append lines can go straight into the script.
I redownloaded the two jar files from your google drive today.
By having it there, I don’t need to do it in each and every script. In fact, if the “syspath” rule gets reloaded, the path will go in there again and again. That means if I added the path in multiple scripts, there’ll be multiple entries. I’ll need to check before appending.
If you are using S1775 or newer, or 2.5 stable (once released), you will not need the org.openhab.core.automation.script jar, since this PR was merged. The ScriptEngineFactory jar is still needed.
I use containers to run OH and surrounding services (on Debian 10) and am struggling a bit with finding an optimal way to handle Python libraries needed by the scripts.
OH configuration files and python scripts are maintained in a Github repo (with git-crypt for the secret stuff) and I’d rather not add libraries into conf/automation/lib/python/... because they will either end up in the repo or must be excluded and reinstalled in some other way when redeploying a container.
Currently I install them with a requirements.txt in the Dockerfile and they are installed in /usr/local/lib/python2.7/dist-packages in the container image. But when I add this path with EXTRA_JAVA_OPTS="-Dpython.path=/usr/local/lib/python2.7/dist-packages", the library cannot be imported:
[ERROR] [ipt.internal.ScriptEngineManagerImpl] - Error during evaluation of script 'file:/openhab/conf/automation/jsr223/python/personal/astro_sun_tomorrow.py': ImportError: No module named astral in <script> at line number 20
In the container, the settings are:
# echo $EXTRA_JAVA_OPTS
-Duser.timezone=Europe/Amsterdam -Dpython.path=/usr/local/lib/python2.7/dist-packages
# ls /usr/local/lib/python2.7/dist-packages
astral-1.10.1.dist-info astral.py astral.pyc pytz pytz-2019.3.dist-info
# python
Python 2.7.16 (default, Oct 10 2019, 22:02:15)
[GCC 8.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print sys.path
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages']
>>>
What am i missing?
To my knowledge you must install your Python2 packages locally, i.e. in OPENHAB_CONF/automation/lib/python/ for openHAB+Jython to see & use it. On openHABian:
$ sudo -u openhab pip install --target=/etc/openhab2/automation/lib/python PYTHON_PACKAGE_NAME
Here’s what I did to make it work - credits go to the astral documentation:
$ sudo -u openhab pip install --target=/etc/openhab2/automation/lib/python astral
Here’s a test rule that works:
from core.rules import rule
from core.triggers import when
from core.actions import LogAction
import pprint
pp = pprint.PrettyPrinter(indent=4)
from org.joda.time import DateTime
import datetime
from datetime import date
from astral import Astral
rule_init_timestamp = DateTime.now()
logTitle = "astral.py@{ts}".format(ts=rule_init_timestamp.toString("HH:mm:ss"))
ruleTimeStamp = " -- (Rule set initialised {date} at {ts})".format(
date=rule_init_timestamp.toString("E d MMM yyyy"),
ts=rule_init_timestamp.toString("HH:mm:ss (z)"),
)
rulePrefix = "Astral | "
a = Astral()
a.solar_depression = "civil"
city_name = "Brussels"
city = a[city_name]
LogAction.logInfo(logTitle, "Information for %s/%s\n" % (city_name, city.region))
timezone = city.timezone
LogAction.logInfo(logTitle, "Timezone: %s" % timezone)
LogAction.logInfo(
logTitle,
u"Latitude: %.04f ° %s; Longitude: %.04f ° %s\n"
% (
city.latitude,
"S" if city.latitude < 0.0 else "N",
city.longitude,
"W" if city.longitude < 0.0 else "E",
),
)
sun = city.sun(date=date.today(), local=True)
LogAction.logInfo(logTitle, pp.pformat(sun))
moon_phase = a.moon_phase(date=date.today())
LogAction.logInfo(logTitle, "Moon phase: %.2f (%.2f %%)" % (moon_phase, 100*moon_phase/28.0))
Here’s the output for today:
==> /var/log/openhab2/openhab.log <==
2020-01-22 12:14:55.124 [INFO ] [me.core.service.AbstractWatchService] - Loading script 'python/personal/astral.py'
2020-01-22 12:14:55.809 [INFO ] [home.model.script.astral.py@12:14:55] - Information for Brussels/Belgium
2020-01-22 12:14:55.812 [INFO ] [home.model.script.astral.py@12:14:55] - Timezone: Europe/Brussels
2020-01-22 12:14:55.815 [INFO ] [home.model.script.astral.py@12:14:55] - Latitude: 50.8500 ° N; Longitude: 4.3500 ° E
2020-01-22 12:14:56.186 [INFO ] [home.model.script.astral.py@12:14:55] - { u'dawn': datetime.datetime(2020, 1, 22, 7, 55, 2, tzinfo=<DstTzInfo 'Europe/Brussels' CET+1:00:00 STD>),
u'dusk': datetime.datetime(2020, 1, 22, 17, 52, 46, tzinfo=<DstTzInfo 'Europe/Brussels' CET+1:00:00 STD>),
u'noon': datetime.datetime(2020, 1, 22, 12, 53, 54, tzinfo=<DstTzInfo 'Europe/Brussels' CET+1:00:00 STD>),
u'sunrise': datetime.datetime(2020, 1, 22, 8, 32, tzinfo=<DstTzInfo 'Europe/Brussels' CET+1:00:00 STD>),
u'sunset': datetime.datetime(2020, 1, 22, 17, 15, 48, tzinfo=<DstTzInfo 'Europe/Brussels' CET+1:00:00 STD>)}
2020-01-22 12:24:28.670 [INFO ] [home.model.script.astral.py@12:24:28] - Moon phase: 25.00 (89.29 %)
Please note that in this script I’m using JodaTime (DateTime) for generating a script timestamp and that the rule uses regular datetime for execution.
1 Like
Here is what I’ve used. The --ignore-installed ignores what you have already installed so that the dependencies all come down with the package in the external directory.
sudo -u openhab pip2 install --ignore-installed --install-option="--prefix=/opt/openhab2/conf/automation/lib/python" PYTHON_PACKAGE_NAME
There was a bug in the beta Jython bundle, but it now properly appends the python.path to any that already exists. This was the issue Ron was running into.
Installing in the automation directory works (indeed using pip with the -t option).
But that’s not my issue. What I am struggling a bit with is the location of the libraries, somewhat deeply buried inside the /conf/automation/... tree.
For easy automated (re)deployment of OH - in my case with Ansible and Docker - I must be careful to not overwrite stuff within the automation directory that’s already there. I use this automation to deploy both new installations (not only of OH) and redeploy or upgrade existing ones. The conf directory is managed as a Github repository. In general, I would like to avoid more static stuff that is maintained outside my scope out of the conf directory.
Having to juggle with stuff inside directories in the /conf/automation tree with some stuff ‘immutable’ and some stuff dependent on my specific configuration or scripts, is sub-optimal. 
Also, it is not yet clear to me why the added sys.path doesn’t work in the first place. I double-checked the setting inside the container for Python with the openhab user…
It can be made to work for sure, but it feels like there should be a cleaner way by splitting the personal configuration (fully maintained by me) from the more static stuff (maintained in the OH community) on a higher level (outside conf for instance as I already mentioned). Your thoughts?
You can put your libraries wherever you’d like… just add the path to python.path. The beta Jython bundle was not handling this properly, but this should now be corrected. This means you can run pip to install to the default location and add that path to your python.path in EXTRA_JAVA_OPTS.
As an alternative, you should be able to append to sys.path. What problem are you having?
None, after I corrected a typo in my Docker file… 
TL;DR
I removed the manually installed python libraries from the /conf/automation/ path and installed them from the Docker file using requirements.txt. They end up in /usr/local/lib/python2.7/dist-packages in the container image. The EXTRA_JAVA_OPTS paths are picked up now and the installed libraries are found by the scripting engine.
Thanks for this quick fix!
1 Like
Need some help getting the requests package installed. When I try the above like this:
sudo -u openhab pip2 install --ignore-installed --install-option="--prefix=/etc/openhab2/automation/lib/python" requests
Then I get an error:
/usr/lib/python2.7/dist-packages/pip/commands/install.py:212: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
cmdoptions.check_install_build_global(options)
The directory '/home/john/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/john/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting requests
Downloading https://files.pythonhosted.org/packages/f5/4f/280162d4bd4d8aad241a21aecff7a6e46891b905a4341e7ab549ebaf7915/requests-2.23.0.tar.gz (114kB)
100% |████████████████████████████████| 122kB 1.5MB/s
Collecting certifi>=2017.4.17 (from requests)
Downloading https://files.pythonhosted.org/packages/41/bf/9d214a5af07debc6acf7f3f257265618f1db242a3f8e49a9b516f24523a6/certifi-2019.11.28.tar.gz (156kB)
100% |████████████████████████████████| 163kB 2.9MB/s
Collecting chardet<4,>=3.0.2 (from requests)
Downloading https://files.pythonhosted.org/packages/fc/bb/a5768c230f9ddb03acc9ef3f0d4a3cf93462473795d18e9535498c8f929d/chardet-3.0.4.tar.gz (1.9MB)
100% |████████████████████████████████| 1.9MB 456kB/s
Collecting idna<3,>=2.5 (from requests)
Downloading https://files.pythonhosted.org/packages/cb/19/57503b5de719ee45e83472f339f617b0c01ad75cba44aba1e4c97c2b0abd/idna-2.9.tar.gz (175kB)
100% |████████████████████████████████| 184kB 4.6MB/s
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Downloading https://files.pythonhosted.org/packages/09/06/3bc5b100fe7e878d3dee8f807a4febff1a40c213d2783e3246edde1f3419/urllib3-1.25.8.tar.gz (261kB)
100% |████████████████████████████████| 266kB 2.8MB/s
Installing collected packages: certifi, chardet, idna, urllib3, requests
Running setup.py install for certifi ... error
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-vOYhi8/certifi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-HbGyvr-record/install-record.txt --single-version-externally-managed --compile --prefix=/opt/openhab2/conf/automation/lib/python --user --prefix=:
running install
error: [Errno 13] Permission denied: '/home/john/.local'
----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-vOYhi8/certifi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-HbGyvr-record/install-record.txt --single-version-externally-managed --compile --prefix=/opt/openhab2/conf/automation/lib/python --user --prefix=" failed with error code 1 in /tmp/pip-build-vOYhi8/certifi/
Okay, so try with the -H option and it works, but it installs in /var/lib/openhab2/.local. My script is not able to find “requests” when it is there:
2020-03-22 20:16:05.139 [ERROR] [ipt.internal.ScriptEngineManagerImpl] - Error during evaluation of script 'file:/etc/openhab2/automation/jsr223/python/personal/tankUtility.py': ImportError: No module named requests in <script> at line number 7
So then I pulled out the ‘-u openHAB’ and the -H and install again with pip. This time I do end up with the requests where they were requested to be but owned by root, so I then reset the permissions. Sample location:
/etc/openhab2/automation/lib/python/lib/python2.7/site-packages/requests
Still, though, I get the same import error after a restart.
The script starts out like this, line 7 is the import line that fails.
"""
TankUtility Jython Rule
"""
from core.rules import rule
from core.triggers import when
import requests <-- line 7, dead here
from requests.auth import HTTPBasicAuth
What next? Thanks for any help.
First, why are you trying to use requests, since there are several other options?