OH3: exec binding with python script in virtenv not executing

Max_G · March 20, 2023, 7:36am

openHABian 3.3.0 on rPi4 with 4GB

[Post updated]

I thought I give the exec binding a go, and used the full example in the documentation

The example seems to make sense.

I have whitelisted the command.

# For security reasons all commands that are used by the exec binding or
# transformation need to be whitelisted.
# Every command needs to be listed on a separate line below.
/home/openhabian/VirtualEnvironments/myenergi/set_zappi_mode.py eco
/home/openhabian/VirtualEnvironments/myenergi/set_zappi_mode.py %2$s

Running the command manually works.

(myenergi) # [2023-03-20 16:45] openhabian@openhabian ~/VirtualEnvironments/myenergi $ 
./set_zappi_mode.py eco
Selected mode is eco.
Zappi is in Eco mode.
(myenergi) # [2023-03-20 16:45] openhabian@openhabian ~/VirtualEnvironments/myenergi $

I am logged in as openhabian:

whoami
openhabian

pwd
/home/openhabian/VirtualEnvironments/myenergi

The Thing is configured as follows:

The error message I am getting is:

2023-03-20 18:04:12.090 [INFO ] [penhab.core.model.script.Zappi.01.01] - zap_test_switch........: OFF
2023-03-20 18:04:15.796 [INFO ] [penhab.core.model.script.Zappi.03.01] - Script has completed.
2023-03-20 18:04:24.177 [INFO ] [penhab.core.model.script.Zappi.01.01] - zap_CommandParameter...: eco
2023-03-20 18:04:24.379 [INFO ] [penhab.core.model.script.Zappi.03.01] - Script has completed.
2023-03-20 18:04:24.381 [INFO ] [penhab.core.model.script.Zappi.04.01] - Raw return result......: Traceback (most recent call last):
ModuleNotFoundError: No module named 'daemon'
Traceback (most recent call last):
ModuleNotFoundError: No module named 'daemon'

What might be worthwhile pointing out is that the python script runs in a virtual environment… ans suspect since the virtenv is not activated by the OH command line, the script does not find its related modules.

Maybe I need to create a bash script containing the virtenv activation?!

Any hints appreciated.

[edit]
I have created a bash script:

#!/usr/bin/env bash
set -e
cd /home/openhabian/VirtualEnvironments/myenergi
source "./bin/activate"
python set_zappi_mode.py $1

resulting in this error

2023-03-20 21:42:25.272 [INFO ] [penhab.core.model.script.Zappi.01.01] - zap_CommandParameter...: eco
2023-03-20 21:42:25.490 [INFO ] [penhab.core.model.script.Zappi.03.01] - Script has completed.
2023-03-20 21:42:25.491 [INFO ] [penhab.core.model.script.Zappi.04.01] - Raw return result......: Traceback (most recent call last):
  File "/home/openhabian/VirtualEnvironments/myenergi/set_zappi_mode.py", line 7, in <module>
    import run_zappi
  File "/home/openhabian/VirtualEnvironments/myenergi/run_zappi.py", line 13, in <module>
    import daemon
ModuleNotFoundError: No module named 'daemon'
Traceback (most recent call last):
  File "/home/openhabian/VirtualEnvironments/myenergi/set_zappi_mode.py", line 7, in <module>
    import run_zappi
  File "/home/openhabian/VirtualEnvironments/myenergi/run_zappi.py", line 13, in <module>
    import daemon
ModuleNotFoundError: No module named 'daemon'
==> /var/log/openhab/events.log <==
2023-03-20 21:42:25.262 [INFO ] [openhab.event.ItemCommandEvent      ] - Item 'zap_test_switch' received command ON
2023-03-20 21:42:25.268 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'zap_test_switch' changed from OFF to ON
2023-03-20 21:42:25.273 [INFO ] [openhab.event.ItemCommandEvent      ] - Item 'zap_CommandParameter' received command eco
2023-03-20 21:42:25.289 [INFO ] [penhab.event.ItemStatePredictedEvent] - Item 'zap_CommandParameter' predicted to become eco
2023-03-20 21:42:25.297 [INFO ] [openhab.event.ItemCommandEvent      ] - Item 'zap_CommandRunning' received command ON
2023-03-20 21:42:25.300 [INFO ] [penhab.event.ItemStatePredictedEvent] - Item 'zap_CommandRunning' predicted to become ON
2023-03-20 21:42:25.304 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'zap_CommandRunning' changed from OFF to ON
2023-03-20 21:42:25.488 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'zap_CommandRunning' changed from ON to OFF
2023-03-20 21:42:25.491 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'zap_LastExecution' changed from 2023-03-20T21:28:20.825387+1000 to 2023-03-20T21:42:25.487209+1000

Calling the bash file directly works.

This must have something to do with the virtual environment.
I can’t figure out where the problem is.
It seems OH opens a very limited shell??

I also tried executeCommandLine instead of using the binding; same problem.

executeCommandLine(Duration.ofSeconds(10), "/bin/sh", "/home/openhabian/VirtualEnvironments/myenergi/zap.sh", "eco")

Wolfgang_S · March 20, 2023, 1:31pm

Permissions are set that files and directories are readable for openhab user?

Lionello_Marrelli · March 20, 2023, 1:38pm

You should test your commands (or probably create the environment) with the user openhab and not openhabian. You can test your command preceeding it with sudo -u openhab.
I think you should be able to re-install the virtual environment for openhab with sudo -u openhab

Max_G · March 21, 2023, 8:08am

Yes.

drwxr-xr-x 7 openhab openhab  4096 Mar 21 08:25 .
drwxr-xr-x 3 openhab openhab  4096 Mar 21 08:19 ..
drwxr-xr-x 2 openhab openhab  4096 Mar 20 20:53 bin
-rwxr-xr-x 1 openhab openhab   441 Mar 20 07:58 get_eddi_boost.py
-rwxr-xr-x 1 openhab openhab   495 Mar 20 07:58 get_zappi_boost.py
-rwxr-xr-x 1 openhab openhab  5924 Mar 20 07:58 get_zappi_history.py
-rw-r--r-- 1 openhab openhab    40 Mar 20 07:58 .gitignore
drwxr-xr-x 3 openhab openhab  4096 Mar 20 07:58 lib
-rw-r--r-- 1 openhab openhab 35149 Mar 20 07:58 LICENSE
drwxr-xr-x 2 openhab openhab  4096 Mar 20 21:21 logs
drwx------ 3 openhab openhab  4096 Mar 20 07:58 mec
-rw-r--r-- 1 openhab openhab 46573 Mar 20 07:58 mec-master.zip
drwxr-xr-x 2 openhab openhab  4096 Mar 20 12:28 __pycache__
-rw-r--r-- 1 openhab openhab   206 Mar 20 07:58 pyvenv.cfg
-rw-r--r-- 1 openhab openhab  5179 Mar 20 07:58 README.md
-rwxr-xr-x 1 openhab openhab   421 Mar 20 07:58 report_on_sockets.py
-rw-r--r-- 1 openhab openhab    59 Mar 20 07:58 requirements.txt
-rwxr-xr-x 1 openhab openhab 21286 Mar 20 12:27 run_zappi.py
-rwxr-xr-x 1 openhab openhab  4758 Mar 20 07:58 set_boost_charge.py
-rwxr-xr-x 1 openhab openhab   530 Mar 20 07:58 set_zappi_mode_orig.py
-rwxr-xr-x 1 openhab openhab  3122 Mar 20 07:58 set_zappi_mode.py
-rw-r--r-- 1 openhab openhab   208 Mar 20 07:58 .zappirc
-rwxr-xr-x 1 openhab openhab   289 Mar 21 08:23 zap.sh

Yes. Have created a home directory for the user openhab.
created directory named VirtualEnvironments in it
created a directory called myenrgi under it.

# [2023-03-21 17:58] openhabian@openhabian /home/openhab/VirtualEnvironments $ 
sudo -u openhab pip3 install virtualenv
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting virtualenv
  Downloading https://www.piwheels.org/simple/virtualenv/virtualenv-20.21.0-py3-none-any.whl (8.7 MB)
     |████████████████████████████████| 8.7 MB 608 kB/s 
Collecting distlib<1,>=0.3.6
  Downloading https://www.piwheels.org/simple/distlib/distlib-0.3.6-py2.py3-none-any.whl (468 kB)
     |████████████████████████████████| 468 kB 107 kB/s 
Collecting filelock<4,>=3.4.1
  Downloading https://www.piwheels.org/simple/filelock/filelock-3.10.0-py3-none-any.whl (9.9 kB)
Collecting platformdirs<4,>=2.4
  Downloading https://www.piwheels.org/simple/platformdirs/platformdirs-3.1.1-py3-none-any.whl (14 kB)
Installing collected packages: platformdirs, filelock, distlib, virtualenv
  WARNING: The script virtualenv is installed in '/var/lib/openhab/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed distlib-0.3.6 filelock-3.10.0 platformdirs-3.1.1 virtualenv-20.21.0

# [2023-03-21 17:59] openhabian@openhabian /home/openhab/VirtualEnvironments $ 
sudo -u openhab export PATH=$PATH:/home/$(whoami)/.local/bin
sudo: export: command not found

# [2023-03-21 18:00] openhabian@openhabian /home/openhab/VirtualEnvironments $ 
export PATH=$PATH:/home/$(whoami)/.local/bin

# [2023-03-21 18:00] openhabian@openhabian /home/openhab/VirtualEnvironments $ 
echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games:/home/openhabian/.local/bin:/home/openhabian/.local/bin

# [2023-03-21 18:00] openhabian@openhabian /home/openhab/VirtualEnvironments $ 
sudo -u openhab virtualenv myenergi
sudo: virtualenv: command not found

I am at my ends wits.

I even added openhab' to the sudoers; no difference.. I can't su openhab` either, as it is asking for a password, which I do not have.

Any hints appreciated.

Lionello_Marrelli · March 21, 2023, 8:45am

It seems to me that it has been installed in
/var/lib/openhab/.local/bin
There should be no need of creating a home directory.

Anyway, try giving only the command
sudo -u openhab
alone and press enter
In this way you have a shell in which you can give all the commands to install the environment.
To exit the shell you have to type
exit
Before giving exit you should be able to test the command

There is an old thread in the forum, that now I can’t find as I am not at my pc, about the exec binding, treating a similar problem.

Max_G · March 21, 2023, 9:09am

This is the result:

# [2023-03-21 18:08] openhabian@openhabian /home/openhab/VirtualEnvironments $ 
sudo -u openhab
usage: sudo -h | -K | -k | -V
usage: sudo -v [-AknS] [-g group] [-h host] [-p prompt] [-u user]
usage: sudo -l [-AknS] [-g group] [-h host] [-p prompt] [-U user] [-u user] [command]
usage: sudo [-AbEHknPS] [-r role] [-t type] [-C num] [-D directory] [-g group] [-h host] [-p prompt] [-R directory] [-T timeout]
            [-u user] [VAR=value] [-i|-s] [<command>]
usage: sudo -e [-AknS] [-r role] [-t type] [-C num] [-D directory] [-g group] [-h host] [-p prompt] [-R directory] [-T timeout] [-u
            user] file ...

Maybe you meant this link: Exec binding with sudo

The following allowed me to be openhab w/o a password:

# [2023-03-21 20:13] openhabian@openhabian /home/openhab/VirtualEnvironments $ 
sudo -u openhab /bin/bash
[20:13:21] openhab@openhabian:/home/openhab/VirtualEnvironments$ la

Now trying to install virtualenv, which says it is already there.

[20:14:01] openhab@openhabian:/home/openhab/VirtualEnvironments$ pip3 install virtualenv
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: virtualenv in /var/lib/openhab/.local/lib/python3.9/site-packages (20.21.0)
Requirement already satisfied: distlib<1,>=0.3.6 in /var/lib/openhab/.local/lib/python3.9/site-packages (from virtualenv) (0.3.6)
Requirement already satisfied: filelock<4,>=3.4.1 in /var/lib/openhab/.local/lib/python3.9/site-packages (from virtualenv) (3.10.0)
Requirement already satisfied: platformdirs<4,>=2.4 in /var/lib/openhab/.local/lib/python3.9/site-packages (from virtualenv) (3.1.1)
[20:14:15] openhab@openhabian:/home/openhab/VirtualEnvironments$ virtualenv myenergi/
bash: virtualenv: command not found

However, it does not find the virtualenv command.

Is it fair to say that the exec binding cannot handle running commands form a python virtual environment?

rlkoshak · March 21, 2023, 1:42pm

No, it could do it. But everything has to be just right and getting everything just right is going to be a chore. You’ll likely need to use the full path to any commands like virtualenv and you may need to create a shell to cd to the right folder to execute the Python program.

Essentially you can assume that the openhab user has nothing. No path. No shell. Often no permissions.

Lionello_Marrelli · March 21, 2023, 5:59pm

As Rich writes, it is a matter of finding a way to write a script defining all the required variables into it. Once you are in the openhab environemnt provided by sudo -u openhab /bin/bash
you should be able to issue the command
/var/lib/openhab/.local/bin/virtualenv

If ‘export’ is not available, instead of bash you may try with sh. You can google ‘export not found with bash’: there are web pages stating that it may be due to .bashrc file being modified or missing. Probably .bashrc is not available to the openhab user but you can try to copy its content from the openhabian user.

rlkoshak · March 21, 2023, 6:20pm

Be careful here though. When OH runs a command using the Exec binding or executeCommandLine Action, it doesn’t have a shell. No Bash, no sh, nothing. It’s been a long time since I’ve messed with virtualenv but if it’s depending on something provided by a shell, you’re going to have to run it from a shell. So from OH’s perspective running bash -c <command> may be required which often means it’ll be easier to put it all inside a shell script anyway.

Max_G · March 21, 2023, 10:21pm

Thank you all; I appreciate the feedback; but this seems to go over my head/
While I might able to gobbledegook this together, it seems all a bit much for issuing a command via a python script requiring a virtual environment.

I gave it a shot just to be stuck as usual.


[07:03:15] openhab@openhabian:/home/openhab/VirtualEnvironments$ /var/lib/openhab/.local/bin/virtualenv myenergi
created virtual environment CPython3.9.2.final.0-32 in 1589ms
  creator CPython3Posix(dest=/home/openhab/VirtualEnvironments/myenergi, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/var/lib/openhab/.local/share/virtualenv)
    added seed packages: pip==23.0.1, setuptools==67.4.0, wheel==0.38.4
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
[07:03:30] openhab@openhabian:/home/openhab/VirtualEnvironments$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

[07:04:47] openhab@openhabian:/home/openhab/VirtualEnvironments/myenergi$ source bin/activate

(myenergi) [07:05:43] openhab@openhabian:/home/openhab/VirtualEnvironments/myenergi$ pip install -r requirements.txt
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting ascii-graph==1.5.1
  Downloading https://www.piwheels.org/simple/ascii-graph/ascii_graph-1.5.1-py3-none-any.whl (14 kB)
Collecting daemon==1.2
  Downloading https://www.piwheels.org/simple/daemon/daemon-1.2-py3-none-any.whl (3.4 kB)
Collecting PyYAML==6.0
  Downloading https://www.piwheels.org/simple/pyyaml/PyYAML-6.0-cp39-cp39-linux_armv7l.whl (45 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.3/45.3 kB 100.9 kB/s eta 0:00:00
Collecting tabulate==0.8.9
  Downloading https://www.piwheels.org/simple/tabulate/tabulate-0.8.9-py3-none-any.whl (25 kB)
Installing collected packages: tabulate, daemon, ascii-graph, PyYAML
Successfully installed PyYAML-6.0 ascii-graph-1.5.1 daemon-1.2 tabulate-0.8.9


(myenergi) [07:07:00] openhab@openhabian:/home/openhab/VirtualEnvironments/myenergi$ ./zap.sh eco
Selected mode is eco.
Traceback (most recent call last):
  File "/home/openhab/VirtualEnvironments/myenergi/set_zappi_mode.py", line 101, in <module>
    main()
  File "/home/openhab/VirtualEnvironments/myenergi/set_zappi_mode.py", line 97, in main
    set_mode(mode)
  File "/home/openhab/VirtualEnvironments/myenergi/set_zappi_mode.py", line 50, in set_mode
    config = run_zappi.load_config()
  File "/home/openhab/VirtualEnvironments/myenergi/run_zappi.py", line 67, in load_config
    with open(os.path.expanduser(RC_FILE), 'r') as ofh:
FileNotFoundError: [Errno 2] No such file or directory: '/home/openhabian/VirtualEnvironments/myenergi/.zappirc'


(myenergi) [07:08:20] openhab@openhabian:/home/openhab/VirtualEnvironments/myenergi$ python set_zappi_mode.py eco
Selected mode is eco.
Traceback (most recent call last):
  File "/home/openhab/VirtualEnvironments/myenergi/set_zappi_mode.py", line 101, in <module>
    main()
  File "/home/openhab/VirtualEnvironments/myenergi/set_zappi_mode.py", line 97, in main
    set_mode(mode)
  File "/home/openhab/VirtualEnvironments/myenergi/set_zappi_mode.py", line 50, in set_mode
    config = run_zappi.load_config()
  File "/home/openhab/VirtualEnvironments/myenergi/run_zappi.py", line 67, in load_config
    with open(os.path.expanduser(RC_FILE), 'r') as ofh:
FileNotFoundError: [Errno 2] No such file or directory: '/home/openhabian/VirtualEnvironments/myenergi/.zappirc'
(myenergi) [07:08:37] openhab@openhabian:/home/openhab/VirtualEnvironments/myenergi$

Anyway; I think this is it.

Spaceman_Spiff · March 22, 2023, 3:03am

You can always try HABApp.
It has the option to execute arbitrary python scripts in the context of the venv where HABApp was installed as a sub process.
You’ll have to run HABApp alongside openHAB but that’s never a mistake and if you’re on openhabian it’s really easy to set up (option 2B).

Wolfgang_S · March 22, 2023, 6:24pm

Looks like the file .zappirc is not available.
I am not sure if I found the right python module the description for that one tells:

A config file (~/.zappirc) is required and should contain at least the credentials used to authenticate against the MyEnergi severs,
username: <serial number of Hub>
password: <Password as used in App>

That sounds like the file needs to e created manually but it wasn’t …
So once that is fixed you may need to do at least one further trial

Max_G · March 22, 2023, 8:57pm

Thanks
Post 4 shows the file is present.
I have given up on this; too much of a hassle.
I started another post offering a bounty to fix the Zappi binding, which as it turns out is in progress.

Spaceman_Spiff · March 23, 2023, 5:06am

Would you be willing to try it with HABApp? It should be straight forward and easy and if you need help you can just ask. That way you could use it now and you’d learn about the best python rule engine for openHAB which is also a plus.