Zigbee2mqtt adapter discovery problem after openhabian update

[stonke (Kristoph)]

@hafniumzinc

What where the updates? OH 3.4.4 is the latest release already, so I presume it was other things that updated?

You may be right 3.4.4 is the current firmware version that I read just before writing the post. I’m not sure what the version was before the unlucky update but probably 3.2.3.

This updates were from Openhabian option 02 “Upgrade system”. There were updates for three packages. I don’t know which ones exactly (can we check it in some log?).

@Wolfgang_S

aren’t there any log entries in the system log files ? What is the output of the dmesg command ?
Does journalctl output give any hint about the problem ?

journalctl - shows tons of lines but starts on 24hours after problematic update been done, should we look for some specific?

dmesg - looks more compact. The only red messages it contains are:
[ 1.385846] bcm2708_fb soc:fb: Unable to determine number of FBs. Disabling driver.
[ 8.873865] vc_sm_cma_vchi_init: failed to open VCHI service (-1)
[ 8.873897] [vc_sm_connected_init]: failed to initialize shared memory service[[ 9.029407] bcm2835_mmal_vchiq: Failed to open VCHI service connection (status=-1)
[ 9.215513] bcm2835_mmal_vchiq: Failed to open VCHI service connection (status=-1)
[ 9.249819] bcm2835_mmal_vchiq: Failed to open VCHI service connection (status=-1)

the lines regarding usb seem to look fine:
[ 0.121937] usbcore: registered new interface driver usbfs
[ 0.122011] usbcore: registered new interface driver hub
[ 0.122093] usbcore: registered new device driver usb
[ 0.122437] usb_phy_generic phy: supply vcc not found, using dummy regulator
[ 0.122670] usb_phy_generic phy: dummy supplies not allowed for exclusive requests
[ [ 1.490310] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.01
[ 1.490346] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.490375] usb usb1: Product: xHCI Host Controller
[ 1.490398] usb usb1: Manufacturer: Linux 6.1.21-v8+ xhci-hcd
[ 1.490421] usb usb1: SerialNumber: 0000:01:00.0
[ 1.491166] hub 1-0:1.0: USB hub found
[ 1.491275] hub 1-0:1.0: 1 port detected
[ 1.492298] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.01
[ 1.492336] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.492363] usb usb2: Product: xHCI Host Controller
[ 1.492386] usb usb2: Manufacturer: Linux 6.1.21-v8+ xhci-hcd
[ 1.492409] usb usb2: SerialNumber: 0000:01:00.0
[ 1.493116] hub 2-0:1.0: USB hub found
[ 1.493215] hub 2-0:1.0: 4 ports detected
[ 1.895686] usb 1-1: New USB device found, idVendor=2109, idProduct=3431, bcdDevice= 4.21
[ 1.895745] usb 1-1: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[ 1.895773] usb 1-1: Product: USB2.0 Hub
[ 1.897805] hub 1-1:1.0: USB hub found
[ 1.898078] hub 1-1:1.0: 4 ports detected
[ 2.193011] usb 1-1.2: new full-speed USB device number 3 using xhci_hcd
[ 2.300565] usb 1-1.2: New USB device found, idVendor=10c4, idProduct=ea60, bcdDevice= 1.00
[ 2.300608] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 2.300637] usb 1-1.2: Product: Sonoff Zigbee 3.0 USB Dongle Plus
[ 2.300661] usb 1-1.2: Manufacturer: ITead
[ 2.300684] usb 1-1.2: SerialNumber: 4e57bf320813ec11b92923c7bd930c07
[ 2.381023] usb 1-1.3: new full-speed USB device number 4 using xhci_hcd
[ 2.486470] usb 1-1.3: New USB device found, idVendor=10c4, idProduct=ea60, bcdDevice= 1.00
[ 2.486516] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 2.486544] usb 1-1.3: Product: Sonoff Zigbee 3.0 USB Dongle Plus
[ 2.486568] usb 1-1.3: Manufacturer: Silicon Labs
[ 2.486591] usb 1-1.3: SerialNumber: 0001
[ 9.225505] usbcore: registered new interface driver usbserial_generic
[ 9.225685] usbserial: USB Serial support registered for generic
[ 9.287729] usbcore: registered new interface driver cp210x
[ 9.287915] usbserial: USB Serial support registered for cp210x
[ 9.288100] cp210x 1-1.2:1.0: cp210x converter detected
[ 9.339320] usb 1-1.2: cp210x converter now attached to ttyUSB0
[ 9.341482] cp210x 1-1.3:1.0: cp210x converter detected
[ 9.353369] usb 1-1.3: cp210x converter now attached to ttyUSB1
[ 9.757088] usbcore: registered new interface driver brcmfmac

it detected both adapters in both usb ports, and yet when I try to install Z2M from the 2A option in Openhabian, I see this screen:

[stonke (Kristoph)]

any tips? (except complete reinstallation from scratch)

[Larsen (Locke)]

what is the output of

ls -l /dev/serial/by-id ?
openhabian-config offers the devices in this folder.

[Benjy (Ben Clark)]

If you can’t find anything relevant in /dev/serial/by-id you may have a version of udev with a bug preventing these symlinks from working.

No Connection to Conbee with Docker

It’s not always a good idea to desynchronise system packages, but I would try upgrading udev and systemd as suggested in the above issue.

[stonke (Kristoph)]

@Larsen

ls -l /dev/serial/by-id

Hi Larsen, thank you for reply.

openhabian@openhabian:~ $ sudo ls -l /dev/serial/by-id
[sudo] password for openhabian:
ls: cannot access '/dev/serial/by-id': No such file or directory

within /dev/serial there is only “by-path” folder which contains:
platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.3:1.0-port0

@Benjy

It’s not always a good idea to desynchronise system packages, but I would try upgrading udev and systemd as suggested in the above issue.

Hi Benjy, thank you for reply.
Tried as per link (created bullseye-backports.list with inside configuration) but received some error about PUBKEY and not sure if should follow or step down and start living without Z2M

Hit:1 http://raspbian.raspberrypi.org/raspbian bullseye InRelease
Get:2 http://deb.debian.org/debian bullseye-backports InRelease [49.0 kB]
Get:3 http://davesteele.github.io/comitup/repo comitup InRelease [4,659 B]
Hit:4 https://deb.nodesource.com/node_16.x bullseye InRelease
Hit:5 http://archive.raspberrypi.org/debian bullseye InRelease
Hit:6 https://openhab.jfrog.io/artifactory/openhab-linuxpkg stable InRelease
Err:2 http://deb.debian.org/debian bullseye-backports InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY 6ED0E7B82643E131
Reading package lists... Done
W: GPG error: http://deb.debian.org/debian bullseye-backports InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY 6ED0E7B82643E131
E: The repository 'http://deb.debian.org/debian bullseye-backports InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

[Larsen (Locke)]

I guess that’s the debian bug that prevents the links from being created.
I described what to do here:

RFXcomm unstable since 3.4.4

I guess you were hit by a new debian bug that occurs when updating the system. It has nothing to do with openhab but occurs when updating e.g. with openhabian-config. The result is that the symbolic links in /dev/serial/by-id are missing. It’s reported here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035094#26 The file with the bug is /usr/lib/udev/rules.d/60-serial.rules. I changed it with udev: fix by-id symlinks by yuwata · Pull Request #25246 · systemd/systemd · GitHub and it work…

[Benjy (Ben Clark)]

You can change your /usr/lib/udev/rules.d/60-serial.rules in the way @Larsen suggests which would be the quickest way to change it and has the benefit of not needing an internet connection.

If you wanted to update udev/systemd (note: this method is riskier since the package comes from outside Raspberry Pi OS’s maintained packages - there’s no guarantee that updating udev will work for your raspberry pi configuration):

@stonke

NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY 6ED0E7B82643E131

You simply don’t have the signing keys, Debian usually comes with a package for it but since you’re using Raspberry Pi OS, this package does not exist.

(Adapted from: From Fix the apt-key deprecation error in Linux | Opensource.com)

gpg --no-default-keyring --keyring gnupg-ring:~/manual-debian-archive.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 0E98404D386FA1D9 6ED0E7B82643E131
sudo mv manual-debian-archive.gpg /etc/apt/trusted.gpg.d/
sudo chown root:root /etc/apt/trusted.gpg.d/manual-debian-archive.gpg
sudo chmod ugo+r /etc/apt/trusted.gpg.d/manual-debian-archive.gpg
sudo chmod go-w /etc/apt/trusted.gpg.d/manual-debian-archive.gpg

Note: The bug is fixed in Debian/Raspberry Pi OS bookworm, and you should remove the gpg file from trusted.gpg.d as soon as you’re able to. For a more secure way of adding the repository, follow similar guidance to our own key:

[stonke (Kristoph)]

Thank you Larsen, that helped kind of
Edited /usr/lib/udev/rules.d/60-serial.rules file and now

ls -l /dev/serial/by-id

finally return values

openhabian@openhabian:~ $ ls -l /dev/serial/by-id
total 0
lrwxrwxrwx 1 root root 15 Jun 28 12:41 -0xa0200859-if -> ../../mmcblk0p1
lrwxrwxrwx 1 root root 11 Jun 28 12:41 --if -> ../../zram2
lrwxrwxrwx 1 root root 21 Jun 28 12:41 usb-2109_USB2.0_Hub-if -> ../../bus/usb/001/002
lrwxrwxrwx 1 root root 21 Jun 28 12:41 usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_4e57bf320813ec11b92923c7bd930c07-if -> ../../bus/usb/001/003
lrwxrwxrwx 1 root root 15 Jun 28 12:41 usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_4e57bf320813ec11b92923c7bd930c07-if00 -> ../../gpiochip2
lrwxrwxrwx 1 root root 13 Jun 28 12:41 usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_4e57bf320813ec11b92923c7bd930c07-if00-port0 -> ../../ttyUSB0
lrwxrwxrwx 1 root root 21 Jun 28 12:41 usb-Linux_6.1.21-v8+_xhci-hcd_xHCI_Host_Controller_0000:01:00.0-if -> ../../bus/usb/002/001

but once when trying install Z2M as 2A Optional component at openhabian-config there is no more screen to choose zigbee adapter.

when hit <Continue it goes back to main screen of openhabian not to let me choose adapter…

On debugmode it looks like this:

+ whiptail --title 'Zigbee2MQTT installation' --yes-button Continue --no-button Cancel --yesno 'A MQTT-server is required for Zigbee2mqtt. If you                haven'\''t installed one yet, please select <cancel> and come back after installing one (e.g. Mosquitto).\n\nZigbee2MQTT will be installed from                the official repository.\n\nDuration is about 4 minutes... ' 14 80
++ whiptail --noitem --title 'Zigbee2MQTT installation' --radiolist 'Please select your zigbee adapter:' 14 100 4 -0xa0200859-if 1 --if 0 usb-210               9_USB2.0_Hub-if 0 usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_4e57bf320813ec11b92923c7bd930c07-if 0 usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_4               e57bf320813ec11b92923c7bd930c07-if00 0 usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_4e57bf320813ec11b92923c7bd930c07-if00-port0 0 usb-Linux_6.1.21               -v8+_xhci-hcd_xHCI_Host_Controller_0000:01:00.0-if 0
+ selectedAdapter='-0xa0200859-if: unknown option'
+ return 0
+ '[' 0 -ne 0 ']'
+ true
+ show_main_menu
+ local choice

Is there some easy way to downgrade OH to version 3.4.3 where everything worked perfectly? Installation from scratch would solve that problem but makes no sense as OH4 is coming within 1month <3

[Larsen (Locke)]

no need to downgrade. Just check what’s going wrong by installing manually as described here:

Linux | Zigbee2MQTT

It’s the same that is done within openhabian-config.

[jswim788 (John Schmitz)]

@stonke

lrwxrwxrwx 1 root root 15 Jun 28 12:41 -0xa0200859-if -> ../../mmcblk0p1
lrwxrwxrwx 1 root root 11 Jun 28 12:41 --if -> ../../zram2

Those links look very strange and you can see that the script fails when it encounters them:

@stonke

+ selectedAdapter='-0xa0200859-if: unknown option'

I don’t have the solution to this, but I would check those links to be sure they are correct. Starting with a ‘-’ is unusual. Maybe it is correct, but it looks odd.

[Larsen (Locke)]

you’re right. You only need the link pointing to ttyUSB0. Probably the patched serial.rules file doesn’t work with your version.

[stonke (Kristoph)]

We`re really close guys
Everything was great up to section
"Starting Zigbee2MQTT
Now that we have setup everything correctly we can start Zigbee2MQTT.

cd /opt/zigbee2mqtt
npm start

"
setup was correct and it works great - frontend easily to be open, devices could be paired - Z2M looks to be back alive.
But then you have to hit CTRL+C and then after npm stop there is no option to start Z2M as expected:

sudo systemctl start zigbee2mqtt
[sudo] password for openhabian:
Failed to start zigbee2mqtt.service: Unit zigbee2mqtt.service not found.

Tried to solve by implement next section ((Optional) Running as a daemon with systemctl)

Linux | Zigbee2MQTT

so created zigbee2mqtt.service file with recommend block of code, then

systemctl start zigbee2mqtt.service

systemctl status zigbee2mqtt.service

returns

● zigbee2mqtt.service - zigbee2mqtt
     Loaded: loaded (/etc/systemd/system/zigbee2mqtt.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Thu 2023-06-29 11:12:54 CEST; 9s ago
    Process: 14128 ExecStart=/usr/bin/npm start (code=exited, status=217/USER)
   Main PID: 14128 (code=exited, status=217/USER)
        CPU: 0

Service down
so to sum up, we managed to partially solve the problem - the adapter is visible during Z2M installation manually, but reinstatement Z2M service to the state before openhabian update still causes problems.

It seems that the old Chinese proverb holds true:
If something works, don’t update it

[Larsen (Locke)]

for raspberry pi you need
ExecStart=/usr/local/bin/npm start

[stonke (Kristoph)]

Tried your way but later found this is for pi1 only.

Finally I checked what is the meaning of

ExecStart=/usr/bin/npm start (code=exited, status=217/USER)

and found that on file /etc/systemd/system/zigbee2mqtt.service

User=pi

should be replaced with

User=openhabian

and now it works - thank you all

[stonke (Kristoph)]

Huh, looks like bug still present on OH4.0.1 and OH4.0.2 so I guess problem is related with openhabian 1.8 based on “Raspbian GNU/Linux 11 (bullseye)”. Openhabian is updated every half year so hopefully it would be fixed with 1.9

[leshqo (Leszek)]

Hey @Larsen, I’m on fresh installation of OH 4.0.2 on RPi4 where I’m moving all my stuff from OH 3 and RPi 2 and faced the same issue (I am not able to select my zigbee adapter connected via usb during Z2M installation. My 60-serial rules file however looks slightly different with some reference to 50-default rules so I’m not sure whether it is still save to apply your solution or how it should be modified

# do not edit this file, it will be overwritten on update

ACTION=="remove", GOTO="serial_end"
SUBSYSTEM!="tty", GOTO="serial_end"

SUBSYSTEMS=="pci", ENV{ID_BUS}="pci", ENV{ID_VENDOR_ID}="$attr{vendor}", ENV{ID_MODEL_ID}="$attr{device}"
# We already ran the hwdb builtin for devices with MODALIAS in 50-default.rules.
# Let's cover the remaining case here, where we walk up the tree to find a node with $MODALIAS.
ENV{MODALIAS}=="", SUBSYSTEMS=="pci", IMPORT{builtin}="hwdb --subsystem=pci"

# /dev/serial/by-path/, /dev/serial/by-id/ for USB devices
KERNEL!="ttyUSB[0-9]*|ttyACM[0-9]*", GOTO="serial_end"

SUBSYSTEMS=="usb-serial", ENV{.ID_PORT}="$attr{port_number}"

IMPORT{builtin}="path_id"
ENV{ID_PATH}=="?*", ENV{.ID_PORT}=="", SYMLINK+="serial/by-path/$env{ID_PATH}"
ENV{ID_PATH}=="?*", ENV{.ID_PORT}=="?*", SYMLINK+="serial/by-path/$env{ID_PATH}-port$env{.ID_PORT}"

IMPORT{builtin}="usb_id"
ENV{ID_SERIAL}=="", GOTO="serial_end"
SUBSYSTEMS=="usb", ENV{ID_USB_INTERFACE_NUM}="$attr{bInterfaceNumber}"
ENV{ID_USB_INTERFACE_NUM}=="", GOTO="serial_end"
ENV{.ID_PORT}=="", SYMLINK+="serial/by-id/$env{ID_BUS}-$env{ID_SERIAL}-if$env{ID_USB_INTERFACE_NUM}"
ENV{.ID_PORT}=="?*", SYMLINK+="serial/by-id/$env{ID_BUS}-$env{ID_SERIAL}-if$env{ID_USB_INTERFACE_NUM}-port$env{.ID_PORT}"

LABEL="serial_end"

[Larsen (Locke)]

thanks for your tests.

• did you do a fresh debian installation (etcher or raspberry imager)?
• did you do a “02 upgrade system” before installation of zigbee2mqtt?
• what is the content of /dev/serial/by-path and /dev/serial/by-id

In your case I would not edit 60-serial-rules but prefer to edit
/opt/openhabian/functions/nodejs-apps.bash
by replacing “by-id” with “by-path”. Then you will see your usb devices but with cryptic names.

general information: The problem that /dev/serial/by-id is empty has absolutely nothing to do with openhab or openhabian. It is/was a debian-bullseye bug which we have to deal with.

[leshqo (Leszek)]

1. I’ve made a fresh installation using etcher as described on OH page
2. Yes, system and all packages are in their latest versions
3. Can’t check right now, however in by-path there was one entry of my zigbee stick and by-id returned error ’ No such file or directory’

I’ll go with your suggestion of changing /opt/openhabian/functions/nodejs-apps.bash

Will keep you posted

[leshqo (Leszek)]

This one worked, thank you, however looks that in other place the same issue persists.

openhabian@openhabian:~ $ sudo service zigbee2mqtt restart && sudo journalctl -u zigbee2mqtt.service -f
-- Journal begins at Thu 2023-08-24 12:01:26 CEST. --
Aug 24 12:25:43 openhabian npm[3536]:                     ^
Aug 24 12:25:43 openhabian npm[3536]: TypeError: Cannot read properties of undefined (reading 'end')
Aug 24 12:25:43 openhabian npm[3536]:     at MQTT.disconnect (/opt/zigbee2mqtt/lib/mqtt.ts:129:21)
Aug 24 12:25:43 openhabian npm[3536]:     at Controller.stop (/opt/zigbee2mqtt/lib/controller.ts:198:9)
Aug 24 12:25:43 openhabian npm[3536]:     at stop (/opt/zigbee2mqtt/index.js:111:5)
Aug 24 12:25:43 openhabian npm[3536]:     at process.handleQuit (/opt/zigbee2mqtt/index.js:117:9)
Aug 24 12:25:43 openhabian systemd[1]: zigbee2mqtt.service: Succeeded.
Aug 24 12:25:43 openhabian systemd[1]: Stopped Zigbee2MQTT.
Aug 24 12:25:43 openhabian systemd[1]: zigbee2mqtt.service: Consumed 6.436s CPU time.
Aug 24 12:25:43 openhabian systemd[1]: Started Zigbee2MQTT.
Aug 24 12:25:44 openhabian npm[3560]: > zigbee2mqtt@1.32.2 start
Aug 24 12:25:44 openhabian npm[3560]: > node index.js
Aug 24 12:25:49 openhabian npm[3574]: Zigbee2MQTT:error 2023-08-24 12:25:49: Error while starting zigbee-herdsman
Aug 24 12:25:49 openhabian npm[3574]: Zigbee2MQTT:error 2023-08-24 12:25:49: Failed to start zigbee
Aug 24 12:25:49 openhabian npm[3574]: Zigbee2MQTT:error 2023-08-24 12:25:49: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start.html for possible solutions
Aug 24 12:25:49 openhabian npm[3574]: Zigbee2MQTT:error 2023-08-24 12:25:49: Exiting...
Aug 24 12:25:49 openhabian npm[3574]: Zigbee2MQTT:error 2023-08-24 12:25:49: Error: Error while opening serialport 'Error: Error: No such file or directory, cannot open /dev/serial/by-id/platform-fd50                                                                                  0000.pcie-pci-0000:01:00.0-usb-0:1.4:1.0-port0'
Aug 24 12:25:49 openhabian npm[3574]:     at SerialPort.<anonymous> (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/src/adapter/z-stack/znp/znp.ts:146:28)
Aug 24 12:25:49 openhabian npm[3574]:     at SerialPort._error (/opt/zigbee2mqtt/node_modules/@serialport/stream/dist/index.js:75:22)
Aug 24 12:25:49 openhabian npm[3574]:     at /opt/zigbee2mqtt/node_modules/@serialport/stream/dist/index.js:111:18
Aug 24 12:25:49 openhabian systemd[1]: zigbee2mqtt.service: Main process exited, code=exited, status=1/FAILURE
Aug 24 12:25:49 openhabian systemd[1]: zigbee2mqtt.service: Failed with result 'exit-code'.

This is result of mine by-id and by-path

openhabian@openhabian:~ $ ls -l /dev/serial/by-id
ls: cannot access '/dev/serial/by-id': No such file or directory
openhabian@openhabian:~ $ ls -l /dev/serial/by-path
total 0
lrwxrwxrwx 1 root root 13 Aug 24 12:20 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4:1.0-port0 -> ../../ttyUSB0

[stonke (Kristoph)]

@Larsen

/opt/openhabian/functions/nodejs-apps.bash

You should go to

nano /opt/zigbee2mqtt/data/configuration.yaml

and modify “port section” by doing replacement by-id with by-path