HABapp won't start: failing module dependencies (openhabian)

I am trying to instal HABapp, however, after installing with openhabian-config the output of sudo systemctl status habapp shows:

type or● habapp.service - HABApp
   Loaded: loaded (/etc/systemd/system/habapp.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2021-12-13 08:40:28 CET; 12s ago
     Docs: https://habapp.readthedocs.io
  Process: 14364 ExecStart=/opt/habapp/bin/habapp -c /etc/openhab/habapp (code=exited, status=1/FAILURE)
 Main PID: 14364 (code=exited, status=1/FAILURE)

Dec 13 08:40:28 Jedha habapp[14364]:     import HABApp.config
Dec 13 08:40:28 Jedha habapp[14364]:   File "/opt/habapp/lib/python3.7/site-packages/HABApp/config/__init__.py", line 1, in <module>
Dec 13 08:40:28 Jedha habapp[14364]:     from .config import Openhab, Mqtt
Dec 13 08:40:28 Jedha habapp[14364]:   File "/opt/habapp/lib/python3.7/site-packages/HABApp/config/config.py", line 7, in <module>
Dec 13 08:40:28 Jedha habapp[14364]:     from ._conf_location import Location
Dec 13 08:40:28 Jedha habapp[14364]:   File "/opt/habapp/lib/python3.7/site-packages/HABApp/config/_conf_location.py", line 3, in <module>
Dec 13 08:40:28 Jedha habapp[14364]:     import eascheduler
Dec 13 08:40:28 Jedha habapp[14364]: ModuleNotFoundError: No module named 'eascheduler'
Dec 13 08:40:28 Jedha systemd[1]: habapp.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 08:40:28 Jedha systemd[1]: habapp.service: Failed with result 'exit-code'. paste code here

My systems (al config changes done on the Pi2 which is not essential for ‘production use’):

Release = Raspbian GNU/Linux 10 (buster)
Kernel = Linux 5.10.63-v7+
Platform = Raspberry Pi 2 Model B Rev 1.1

ACME System CM3-Home:
Release = Raspbian GNU/Linux 10 (buster)
Kernel = Linux 5.10.63-v7+
Platform = Raspberry Pi Compute Module 3 Rev 1.0

OpenHAB 3.2.0.M5

I have no experience with venv. But use pyenv on mac and thus have some understanding of python version seggregation in virtual environments.
Suspecting it had something to do with the installed virtual environment I went to /opt/habapp and did a source ./bin/activate. From the activated venv I confirmed Python 3.7 with python -V and did a python -m pip install pydantic.

The above output is the third output after trying to install libraries.
Initially HABapp startup fails because of the said ModuleNotFoundError for ‘pydantic’.
Thereafter ‘EasyCo’ is missing and a python -m pip install easyco also downloads ‘voluptuous’. This however gives the above ModuleNotFoundError for ‘eascheduler’.

The HABapp documentation https://habapp.readthedocs.io/en/latest/configuration.html does not adress the need to install modules. That leeds me to think it should not be necessary.

This table shows the differences (both python 3.7.3 environment) between the Raspberry Pi Compute Module 3 (non modified after HABapp installation) and the Rasperry Pi 2 (updated libraries with python -m pip install ).

Package              Version       Versions after 
                            pip install    original install
-----------------        -------           ------
EasyCo                0.2.3           
HABApp               0.31.1        0.31.1
pip                        21.3.1        21.3.1
pkg_resources     0.0.0          0.0.0
pydantic               1.8.2
ruamel.yaml         0.17.17
ruamel.yaml.clib   0.2.6
setuptools             40.8.0       40.8.0
typing_extensions 4.0.1
voluptuous            0.12.2

Is this an installation issue? Am I doing something wrong?

Hi,

could you try this and then the manual installation of dependencies 2 posts after the linked one?
It’s the pendulum installation that is making issues.

That did not entirely work…

(habapp) <...>@<...>:/opt/habapp $ sudo python -V
Python 2.7.16
(habapp) <...>@<...>:/opt/habapp $ python -m pip install pendulum==2.1.1
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pendulum==2.1.1
  Downloading pendulum-2.1.1.tar.gz (84 kB)
     |████████████████████████████████| 84 kB 200 kB/s            
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting python-dateutil<3.0,>=2.6
  Downloading https://www.piwheels.org/simple/python-dateutil/python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     |████████████████████████████████| 247 kB 2.3 MB/s            
Collecting pytzdata>=2020.1
  Downloading https://www.piwheels.org/simple/pytzdata/pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
     |████████████████████████████████| 489 kB 2.5 MB/s            
Collecting six>=1.5
  Using cached https://www.piwheels.org/simple/six/six-1.16.0-py2.py3-none-any.whl (11 kB)
Building wheels for collected packages: pendulum
  Building wheel for pendulum (pyproject.toml) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/habapp/bin/python /opt/habapp/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpkgmyhlkz
       cwd: /tmp/pip-install-4_jvd_3h/pendulum_83fe98ff807449168986447fb5cc0578
  Complete output (25 lines):
  A setup.py file already exists. Using it.
  Traceback (most recent call last):
    File "/tmp/pip-install-4_jvd_3h/pendulum_83fe98ff807449168986447fb5cc0578/setup.py", line 2, in <module>
      from setuptools import setup
  ModuleNotFoundError: No module named 'setuptools'
  Traceback (most recent call last):
    File "/opt/habapp/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/opt/habapp/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/opt/habapp/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 262, in build_wheel
      metadata_directory)
    File "/tmp/pip-build-env-qy_6v0z5/overlay/lib/python3.7/site-packages/poetry/core/masonry/api.py", line 69, in build_wheel
      return WheelBuilder.make_in(poetry, Path(wheel_directory))
    File "/tmp/pip-build-env-qy_6v0z5/overlay/lib/python3.7/site-packages/poetry/core/masonry/builders/wheel.py", line 77, in make_in
      wb.build()
    File "/tmp/pip-build-env-qy_6v0z5/overlay/lib/python3.7/site-packages/poetry/core/masonry/builders/wheel.py", line 108, in build
      self._build(zip_file)
    File "/tmp/pip-build-env-qy_6v0z5/overlay/lib/python3.7/site-packages/poetry/core/masonry/builders/wheel.py", line 141, in _build
      self._run_build_command(setup)
    File "/tmp/pip-build-env-qy_6v0z5/overlay/lib/python3.7/site-packages/poetry/core/masonry/builders/wheel.py", line 185, in _run_build_command
      str(self._path / "build"),
    File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/opt/habapp/bin/python', '/tmp/pip-install-4_jvd_3h/pendulum_83fe98ff807449168986447fb5cc0578/setup.py', 'build', '-b', '/tmp/pip-install-4_jvd_3h/pendulum_83fe98ff807449168986447fb5cc0578/build']' returned non-zero exit status 1.
  ----------------------------------------
  ERROR: Failed building wheel for pendulum
Failed to build pendulum
ERROR: Could not build wheels for pendulum, which is required to install pyproject.toml-based projects

To be sure about procedure:

I am running commands as the openhabian user, thus:

oh-user@server:~ $
oh-user@server:~ $ cd /opt/habapp
oh-user@server:/opt/habapp $ source bin/activate
(habapp) oh-user@server:/opt/habapp $python -V
Python 3.7.3
(habapp) oh-user@server:/opt/habapp $python -m pip install pip --upgrade

because using sudo (as indicated by Spaceman_Spiff) gives the wrong venv.

(habapp) oh-user@server:/opt/habapp $sudo python -V
Python 2.7.1

Can you try

python - m pip install --upgrade setuptools

and then retry the installation of pendulum?


Under /opt/habapp/bin you’ll find the python binary which you then can call with sudo.

Yuck, failed again.

First the pip upgrade. I’m not sure if sudo python3 is the same binary as /opt/habapp/bin/python3.

(habapp) oh-user@server:/opt/habapp $ sudo python3 -m pip install pip --upgrade
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Collecting pip
  Using cached https://files.pythonhosted.org/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 18.1
    Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'pip'. No files were found to uninstall.
Successfully installed pip-21.3.1

So, to be sure I continued:

(habapp) oh-user@server:/opt/habapp $ sudo bin/python3 -m pip install pip --upgrade
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: pip in ./lib/python3.7/site-packages (21.3.1)
(habapp) oh-user@server:/opt/habapp $ sudo bin/python3 -m pip install setuptools
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: setuptools in ./lib/python3.7/site-packages (40.8.0)

pip up-to-date and setuptools already in place, so continued with pendulum install, but failed.

(habapp) oh-user@server:/opt/habapp $ sudo bin/python3 -m pip install pendulum==2.1.1
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pendulum==2.1.1
  Downloading pendulum-2.1.1.tar.gz (84 kB)
     |████████████████████████████████| 84 kB 497 kB/s            
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting python-dateutil<3.0,>=2.6
  Downloading https://www.piwheels.org/simple/python-dateutil/python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     |████████████████████████████████| 247 kB 2.2 MB/s            
Collecting pytzdata>=2020.1
  Downloading https://www.piwheels.org/simple/pytzdata/pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
     |████████████████████████████████| 489 kB 2.6 MB/s            
Collecting six>=1.5
  Using cached https://www.piwheels.org/simple/six/six-1.16.0-py2.py3-none-any.whl (11 kB)
Building wheels for collected packages: pendulum
  Building wheel for pendulum (pyproject.toml) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/habapp/bin/python3 /opt/habapp/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpmj8q22ag
       cwd: /tmp/pip-install-tosvwy3o/pendulum_ff0d22ffd2b44973b07caef0261fc4a9
  Complete output (25 lines):
  A setup.py file already exists. Using it.
  Traceback (most recent call last):
    File "/tmp/pip-install-tosvwy3o/pendulum_ff0d22ffd2b44973b07caef0261fc4a9/setup.py", line 2, in <module>
      from setuptools import setup
  ModuleNotFoundError: No module named 'setuptools'
  Traceback (most recent call last):
    File "/opt/habapp/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/opt/habapp/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/opt/habapp/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 262, in build_wheel
      metadata_directory)
    File "/tmp/pip-build-env-2j07l8_4/overlay/lib/python3.7/site-packages/poetry/core/masonry/api.py", line 69, in build_wheel
      return WheelBuilder.make_in(poetry, Path(wheel_directory))
    File "/tmp/pip-build-env-2j07l8_4/overlay/lib/python3.7/site-packages/poetry/core/masonry/builders/wheel.py", line 77, in make_in
      wb.build()
    File "/tmp/pip-build-env-2j07l8_4/overlay/lib/python3.7/site-packages/poetry/core/masonry/builders/wheel.py", line 108, in build
      self._build(zip_file)
    File "/tmp/pip-build-env-2j07l8_4/overlay/lib/python3.7/site-packages/poetry/core/masonry/builders/wheel.py", line 141, in _build
      self._run_build_command(setup)
    File "/tmp/pip-build-env-2j07l8_4/overlay/lib/python3.7/site-packages/poetry/core/masonry/builders/wheel.py", line 185, in _run_build_command
      str(self._path / "build"),
    File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/opt/habapp/bin/python3', '/tmp/pip-install-tosvwy3o/pendulum_ff0d22ffd2b44973b07caef0261fc4a9/setup.py', 'build', '-b', '/tmp/pip-install-tosvwy3o/pendulum_ff0d22ffd2b44973b07caef0261fc4a9/build']' returned non-zero exit status 1.
  ----------------------------------------
  ERROR: Failed building wheel for pendulum
Failed to build pendulum
ERROR: Could not build wheels for pendulum, which is required to install pyproject.toml-based projects

Oops: missed the upgrade setuptools portion.
Now upgraded successfully:

Successfully uninstalled setuptools-40.8.0

Successfully installed setuptools-59.6.0

And try again with pendulum.

Could you try this:

sudo bin/python3 -m pip install poetry
sudo bin/python3 -m pip install --no-build-isolation pendulum

Did not proceed with your above suggested steps: I first wanted to try upgrading setuptools. From there I continued as follows.

Installing pendulum 2.1.1 after upgrading setuptools failed too.
Pendulum 2.1.0. installed nicely though:

(habapp) oh-user@server:/opt/habapp $ sudo bin/python3 -m pip install pendulum==2.1.0
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pendulum==2.1.0
  Downloading https://www.piwheels.org/simple/pendulum/pendulum-2.1.0-cp37-cp37m-linux_armv7l.whl (107 kB)
     |████████████████████████████████| 107 kB 2.4 MB/s            
Collecting pytzdata>=2018.3
  Using cached https://www.piwheels.org/simple/pytzdata/pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
Collecting python-dateutil<3.0,>=2.6
  Using cached https://www.piwheels.org/simple/python-dateutil/python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting six>=1.5
  Using cached https://www.piwheels.org/simple/six/six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, pytzdata, python-dateutil, pendulum
Successfully installed pendulum-2.1.0 python-dateutil-2.8.2 pytzdata-2020.1 six-1.16.0

Upgraded pendulum to 2.1.2 via pip.

However, HABapp still fails to start ( ModuleNotFoundError: No module named 'eascheduler' )

I understand correctly that installing pendulum 2.1.0 worked without any further tries?

Since you have fixed the cause for a broken setup you either have to remove habapp and install it again oder install the required dependencies by hand.

I’am at a loss: won’t uninstalling and reinstalling (I used the openhabian-config ui) just repeat the failure? Or have you, like lightning, updated de source in the meanwhile :slightly_smiling_face:

You should have not used the openhabian-config ui but rather pip install habapp.
Uninstalling through openhabian-config removes the venv so the work you’ve done was deleted too.
Sorry - I should have been more clear.

You can also try a manual installation, just follow the steps in the docs (it’s the same as the openhabian installation).
But before installing HABApp (step 6) install pendulum so it’s already in the venv.
Only if this installation is successful you go on to the installation of HABApp.

Well, I’m running into a new problem:

I’ve deinstalled HABapp (openhabian-config) and removed .local python3.7 instances from the /home/oh-user directory.

I then made the /opt/habapp venv and followed the steps from de manual.
Before installing HABapp I installed pendulum (no pinning to previous version needed, installed latest without problems).
Then installed HABapp.
Then habapp -c /etc/openhab/habapp.
It then hangs. After a ctr-c I check /etc/openhab/habapp for logging.yml, config.yml and rules. All are there. So everything seems fine nevertheless.

Make systems startup file as per the manual.
Cannot startup.

Make sure all files and directories are owned by openhab.

Test with habapp -h and habapp -b:

Config file "config.yml" not found!
Checked folders:
 - /opt/habapp/bin/HABApp
 - /opt/habapp/HABApp
 - /opt/HABApp
 - /home/ohadmin/HABApp
 - /opt/habapp/HABApp

ugh…

That’s what I don’t understand. These installation problems are totally random! :confused:

Of course - because it is running. Take a look in the HABApp.log and you’ll see the startup.

You have to start it as a service in the background.
See docs on how to run it as a service.

Could it have anything tot do with file permissions and a changed login user for openhab? As stated in the first post: am I messing things up?

Okay, I, wrongly, assumed it would just create te needed config files and then exit.
But I have created the systemd service file to run as a service and that too does not work.

But backstepping a bit:

Habapp interfaces with openhab via the REST-API. It has its credentials (username and password) in the config.yml file.
For it to be able to read that file it needs directory and file acces to /etc/openhab/habapp and /opt/habapp. For interfacing with openhab said username and password suffice.
So I would think it could run as nonpriviliged user habapp. Or if done as per the manual it gets installed as the login user for openhab, which is not necessarily openhabian (it can be changed at openhab install time).

It looks like all of my problems are related to the versions of libraries that habapp depends on and the venv they are installed with.
As stated in my first post, I am only vaguely familiar with virtual environments from the sporadic use of pyenv in macOS. There you either globally set an environment, with its files centrally located in a folder or declare a local (per directory) environment that has its files equally centrally stored. When working from such a localised directory you automatically work in that environment (no activation needed). Or such is as I understand it …

The venv environments are PATH’s to the correct python version and locations per user login shell, am I right? So if I do source ./bin/activate as user ‘me’ and then do a sudo ./bin/python3 -m pip <something> as root / privileged-user, I am doing things in two entirely different environments.
So to run a command as root in the habapp venv I first need to do sudo source ./bin/activate and then do the python3 pip stuff.

So I’ve repeated all steps as the non-privileged login user for openhab over ssh and then changed all file and subdirectory ownership related to habapp to openhab:openhab. Ran habapp --config ‘/etc/openhab/habapp’ and shut it down.

But no go:

sudo systemctl status habapp
● habapp.service - HABApp
   Loaded: loaded (/etc/systemd/system/habapp.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2021-12-13 14:15:01 CET; 4s ago
     Docs: https://habapp.readthedocs.io
  Process: 23532 ExecStart=/opt/habapp/bin/habapp -c PATH_TO_CONFIGURATION_FOLDER (code=exited, status=1/FAILURE)
 Main PID: 23532 (code=exited, status=1/FAILURE)

Dec 13 14:14:57 Jedha systemd[1]: Started HABApp.
Dec 13 14:15:01 Jedha habapp[23532]: Config file "config.yml" not found!
Dec 13 14:15:01 Jedha habapp[23532]: Checked folders:
Dec 13 14:15:01 Jedha habapp[23532]:  - //PATH_TO_CONFIGURATION_FOLDER
Dec 13 14:15:01 Jedha habapp[23532]: Please create file or specify a folder with the "-c" arg switch.
Dec 13 14:15:01 Jedha systemd[1]: habapp.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 14:15:01 Jedha systemd[1]: habapp.service: Failed with result 'exit-code'.

The file exists though:

(habapp) oh-user@server:/etc/openhab/habapp $ ls -l
total 12
-rw-r--r-- 1 openhab openhab 1604 Dec 13 14:13 config.yml
-rw-r--r-- 1 openhab openhab 1465 Dec 13 12:45 logging.yml
drwxr-xr-x 2 openhab openhab 4096 Dec 13 12:45 rules

and since habapp runs as openhab (I did not alter the habapp.service file) it should be able to read the file.

The manual (https://habapp.readthedocs.io/en/latest/installation.html) does not stipulate to do the steps as root. So I haven’t. Is it necessary nevertheless?

Correct - the problem was that the dependencies have not been installed.
But now this is fixed and everything should run fine.

You have to replace PATH_TO_CONFIGURATION_FOLDER with the folder or direct path to your config, e.g. /etc/openhab/habapp/config.yml

I have tried both habapp --config '/etc/openhab/habapp' and habapp --config '/etc/openhab/habapp/config.yml'.
And /opt/habapp/bin/habapp -c '/etc/openhab/habapp'
And /opt/habapp/bin/habapp -c /etc/openhab/habapp
No joy.

You can try the calls in the shell until everything works correctly.
Imho in your case habapp -c /etc/openhab/habapp/config.yml should work properly.
Could you please post the error message if it doesn’t?

There is no error message with either of the four commands.

What do you mean by this? What is a “call” in this context?

I mean you can try it in the shell - you don’t have to modify the service every time.


So does the service start? What does the HABApp.log say?

2021-12-13 14:42:03.248 [INFO ] [HABApp                              ] - HABApp Version 0.31.1
2021-12-13 14:42:03.264 [INFO ] [HABApp.openhab.thing                ] - Config folder does not exist - textual thing config disabled!
2021-12-13 14:42:03.265 [INFO ] [HABApp.RuleParameters               ] - Parameter files disabled: Folder /etc/openhab/habapp/params does not exist!
2021-12-13 14:42:03.273 [INFO ] [HABApp.mqtt.connection              ] - MQTT disabled
2021-12-13 14:42:03.362 [WARN ] [HABApp.openhab.connection           ] - Status 401 for GET http://localhost:8080/rest/uuid/
2021-12-13 14:42:03.363 [INFO ] [HABApp.openhab.connection           ] - ... offline!
2021-12-13 14:42:05.189 [WARN ] [HABApp.Rule                         ] - /usr/lib/python3.7/logging/handlers.py:1326: ResourceWarning:unclosed file <_io.TextIOWrapper name='/var/log/openhab/HABApp_events.log' mode='a' encoding='utf-8'>
/var/log/openhab/HABApp.log (END)

No idea why it says that openhab is offline, I can log in just fine in the Main-UI.

Setting file path variable in the shell produces this:

(habapp) oh-user@server:/var/log/openhab $ ls /etc/openhab/habapp/
config.yml  logging.yml  rules

(habapp) ohuser@server:/var/log/openhab $ PATH_TO_CONFIGURATION_FOLDER='/etc/openhab/habapp'
(habapp) ohuser@server:/var/log/openhab $ echo $PATH_TO_CONFIGURATION_FOLDER
/etc/openhab/habapp
(habapp) ohuser@server:/var/log/openhab $ habapp -b
Config file "config.yml" not found!

It sais 401 forbidden - you propably have to activate basic auth or use a token.
So HABApp already has been running fine, you just have to modify the config.yml.

No - this is not a shell variable but a command line argument.
My hint was that you can just type in
habapp -c /etc/openhab/habapp/config.yml
and if no error comes back it is working.
Only if this step works without error you should modify your background service.