Openhabian unresponsive after first clean installation

  • Platform information:

    • Hardware: Raspberry Pi 4 Model B Rev 1.2, 4GB RAM,

    • OS: Linux 5.4.51-v7l+

    • Java Runtime Environment:

      • openjdk version “1.8.0_265”
      • OpenJDK Runtime Environment (Zulu 8.48.3.246-CA-linux_aarch32hf) (build 1.8.0_265-b11)
      • OpenJDK Client VM (Zulu 8.48.3.246-CA-linux_aarch32hf) (build 25.265-b11, mixed mode)
    • openHAB version: openHAB 2.5.8-1

  • Issue of the topic: after installing the openhabian solution on a new SD cards (tested 2 card: 32GB and 8GB), the system runs for less than 24 hours and after that it is unresponsive: not possible to login via SSH or the 8080 port. This issue happens on a daily based since almost 2 weeks. At the beginning I tought it was due to my wrong settings, but now it is the installlation only. In order to get openhabian responsive agian, the hard reset is necessary and after that I checked the /var/log/syslog
    file and I do not detect an issue (uploaded with the openhab.log).

  • Configurations : there are no items, no sitemap, no bindings, no rules, no services

openhab.log (6.3 KB) syslog.txt (720.1 KB)

Would anyone have advices on what and where to look for?

No that could be anything. Disk full, some memory leak to cause process size grow beyond bounds, or a corrupted SD card or other.
Monitor processes and RAM usage closely to find out more.

Is the entire PI unresponsive? Can you access it with a local terminal?

  • Disk full : not possible since it is an SD car of 32GB with only few GB used
  • memory leak : it is the pure and simple installation. No modification applied. And the same approach was working before 2 weeks ago.
  • corrupted SD card: I tried 2, and exactly same behaviour with both

Is there a way to register what is happening during my absence. I am not able to trigger the crash.

These are the messages before to end of the log before the stop:

  • it is possible that the “Network Time Synchronization.” or the “ICMPv6” are the root cause?

Sep 14 22:25:53 openhab systemd[1]: Stopping Network Time Synchronization…
Sep 14 22:25:53 openhab systemd[1]: systemd-timesyncd.service: Succeeded.
Sep 14 22:25:53 openhab systemd[1]: Stopped Network Time Synchronization.
Sep 14 22:25:53 openhab systemd[1]: Starting Network Time Synchronization…
Sep 14 22:25:53 openhab systemd[1]: Started Network Time Synchronization.
Sep 14 22:25:53 openhab systemd-timesyncd[5777]: Synchronized to time server for the first time 212.51.144.46:123 (0.debian.pool.ntp.org).
Sep 14 22:25:54 openhab kernel: [ 4742.458958] ICMPv6: NA: dc:a6:32:7c:6d:51 advertised our address ::54a3:dc51:c04e:883d on wlan0!
Sep 14 22:25:55 openhab kernel: [ 4743.459977] ICMPv6: NA: dc:a6:32:7c:6d:51 advertised our address ::54a3:dc51:c04e:883d on wlan0!
Sep 14 22:25:57 openhab systemd[1]: Stopped target Timers.
Sep 14 22:25:57 openhab systemd[1]: Stopped target Sound Card.
Sep 14 22:25:57 openhab systemd[1]: Stopped target Bluetooth.
Sep 14 22:25:57 openhab bluetoothd[584]: Terminating
Sep 14 22:25:57 openhab systemd[1]: Stopping Bluetooth service…
Sep 14 22:25:57 openhab systemd[1]: systemd-rfkill.socket: Succeeded.
Sep 14 22:25:57 openhab systemd[1]: Closed Load/Save RF Kill Switch Status /dev/rfkill Watch.
Sep 14 22:25:57 openhab systemd[1]: man-db.timer: Succeeded.
Sep 14 22:25:57 openhab systemd[1]: Stopped Daily man-db regeneration.
Sep 14 22:25:57 openhab systemd[1]: Stopped target Graphical Interface.
Sep 14 22:25:57 openhab systemd[1]: logrotate.timer: Succeeded.
Sep 14 22:25:57 openhab systemd[1]: Stopped Daily rotation of log files.
Sep 14 22:25:57 openhab systemd[1]: Stopped target Multi-User System.
Sep 14 22:25:57 openhab systemd[1]: rpi-eeprom-update.service: Succeeded.
Sep 14 22:25:57 openhab systemd[1]: Stopped Check for Raspberry Pi EEPROM updates.
Sep 14 22:25:57 openhab bluetoothd[584]: Stopping SDP server
Sep 14 22:25:57 openhab bluetoothd[584]: Exit
Sep 14 22:25:57 openhab systemd[1]: Stopping Frontail openHAB instance, reachable at http://openhab:9001
Sep 14 22:25:57 openhab systemd[1]: Stopping Samba SMB Daemon…
Sep 14 22:25:57 openhab avahi-daemon[338]: Got SIGTERM, quitting.
Sep 14 22:25:57 openhab systemd[1]: Stopping Avahi mDNS/DNS-SD Stack…
Sep 14 22:25:57 openhab avahi-daemon[338]: Leaving mDNS multicast group on interface wlan0.IPv6 with address ::54a3:dc51:c04e:883d.
Sep 14 22:25:57 openhab systemd[1]: Stopped target Login Prompts.
Sep 14 22:25:57 openhab avahi-daemon[338]: Leaving mDNS multicast group on interface wlan0.IPv4 with address 192.168.1.106.
Sep 14 22:25:57 openhab systemd[1]: Stopping Getty on tty1…
Sep 14 22:25:57 openhab avahi-daemon[338]: avahi-daemon 0.7 exiting.

I tried to connect a screen via the HDMI, but I didn’t see any signal. So I do not know how to access the PI in order to see what it does.

I would grab a keyboard and HDMI, Wack a key see if it wakes up. Also check the temperature of the chips. There were some Pi4’s that had thermal issues. Bad PSU can also cause this.

I’ll try at the next crash.

Is there a way to trigger an extra load in order to force a crash? if it is that one, I would add a fun.

I tried a second PSU and I got the same behaviour. In addition, if it is a PSU issue, the PI will restart, but it does not.

You need to start with raspian lite image. If it does the same thing then you have a hardware issue.

The other option would be to stop openHAB and see if the crash happens.

I did stop the openhab service (with sudo systemctl stop openhab2.service) and the system crashed again (unresponsive). I tried to plug keyboard and HDMI screen, hitting keys but nothing happened.
I start to doubt about my PI and for that reason I am going to ordering a new PI.

Based on this first experience with openhab on PI + the possible corruption on SD card, I am wondering if the PI is the best hardware for a 24/7 running solution. In parallel, I’ll look for a more stable alternative.

Forgetting about openHAB completely for a second, can this Pi run just Raspbian (or whatever the latest OS is for a Pi) without crashing?

Many, many people are running openHAB on a Pi, so it’s possible that you’ve been unlucky…

Since the shop may took the PI back, I cannot check your valuable advice.

Nevertheless up to 2 weeks ago, the PI was running perfectly with the original raspian and/or the openhabian based solution (tested both for several days). During the last 2 weeks, it didn’t last more than 24 hours. I repeated the installation via BalenaEtcher (with the exactly same process) several times with different SD, with different PSU, but it was always showing the same outcome: unresponsive.

Regarding the fact that many people make use of PI, it is true. But many of them shared concerns related to the SD corruption. When I discovered that (too late) I was very surprised. I am wondering if it is worth the spend hours on this harward+software in order to master it. Shouldn’t be better to spend the same amount of time on a more stable hardware? I have in mind a NAS like Synology or a server like NUC. In short, I would like to build a solid elementary brick that I can rely on for further development.

1 Like

Then stay away from hobby - as cheap as possible - world. One small step toward more reliable hardware would already be a nice step - I am using this HW for about 2 years now. I have two x86 servers and high-end NAS running in the same room - where I could also run OpenHab - but this one is doing just fine. It doesn’t sweat, it has plenty of memory, reliable eMMC storage, attached to UPS + it has all / more improvements that comes with Openhabian. Wife doesn’t complain - not a single problem.

Still your proposal (Nanopi K1 plus) is based on SD card. Based on the amount of contributors available on the net, I think that PI is the most appropriate solution for beginner. Moreover, in a second step, I’ll move to a more stable hardware solution (unless PI reliability surprises me).

Not exactly a proposal. Just a board I use and it has a removable eMMC which I use it.

Most of the contribution for Rpi is re-branded / copy pasted / resold community work that works on any Linux and Linux can run on anything … even on proprietary video players https://www.cnx-software.com/2020/09/14/banana-pi-bpi-m5-amlogic-s905x3-sbc/#comment-575479

I have 4 pi’s running. 3 are on sd cards the last my most important one openHAB is on an ssd. I ran on an sd card with no issues, the ssd was easier for me to backup and faster.

If you have backups the sd cards are a non issue. You need two things good power supply and good backup.

USB storage is not a reliable storage (also on high end x86 servers or on Mac if you like) and you can consider yourself lucky if you don’t have troubles. Look on this forum, on OMV and RPi forums how many problems are around Raspberry Pi and how many of them are associated with USB storage. USB drive is connected to a long chain of cheap components - SBC is having internally a PCI where USB hub is attached to. Then you have USB A connectors, which solo ads troubles at high speed transfers, where you attach USB2SATA converter or USB back to PCI. This means that even the best SSD on the planet can’t match reliability (and usually also the speed) of mentioned eMMC based setup. Better machines (x86 or Rockchip RK3399 SBC for example) has SSD connected directly to PCI or worse case directly to SATA which can also be problematic under certain conditions.

Problems can be mitigated up to some degree (zram log, zswap, USB storage is you gain something you loose something), they can’t be fixed. There are thousands of people seeing those problems and they keep trying to fix the impossible over and over again. What a waste!

How many years do you run this setup and how do you know your data is safely stored?

Over a year on ssd and not one blip. Good enough for me!

After having received a new PI, new SD, new PS, I installed openhabian. This solution ran for 2 days without issue. Thanks to this first encuraging result, I loaded all the bindings, things and few items, in particular:

  • openhabcloud (for remote connection)
  • deconz (for the raspbee interface) and phoscon
  • shelly (for the shelly interface)
  • mail (for sending email)
    After less than 12 hours, the PI turned out to be unreacheable again.
    So, I’ll step back and remove all those components and I’ll see again what happen.
    I looked for in the /var/log/syslog, but I do not see any strange message (to my best knowledge).
    The only strange this I remark is that the last information I see, before the crash, is related to zram:

openhab zram-config[479]: + systemctl restart rsyslog.service

why this service is restarting, who/what initiated this requet? Is there a way to detect?

Any advice on where to look for some sign of issue or any sign to search?

I did a further step in the understanding of the issue:

  • I was connecting to the PI via wireless only and probably this connection is compromised because of something.
  • I swithched on a wired connection and since more than 24 hours the system is reachable (still without bindings, items or whatever, I’ll add them later in a step by step process).

In order to understand what was happening, I still cheched the syslog and I dicovered the following interesting events:

Sep 27 00:00:04 openhab logrotate[3328]: error: error renaming /usr/local/share/zram-config/log/zram-config.log.4.gz to /usr/local/share/zram-config/log/zram-config.log.5.gz: Read-only file system
Sep 27 00:00:04 openhab systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Sep 27 00:00:04 openhab systemd[1]: logrotate.service: Failed with result ‘exit-code’.
Sep 27 00:00:04 openhab systemd[1]: Failed to start Rotate log files.
Sep 27 00:00:05 openhab systemd[1]: man-db.service: Succeeded.

After the previous logs, the following lines of logs (related to the time synchronization) are repeated every minutes forever:

Sep 27 00:00:08 openhab systemd[1]: Stopping Network Time Synchronization…
Sep 27 00:00:08 openhab systemd[1]: systemd-timesyncd.service: Succeeded.
Sep 27 00:00:08 openhab systemd[1]: Stopped Network Time Synchronization.
Sep 27 00:00:08 openhab systemd[1]: Starting Network Time Synchronization…
Sep 27 00:00:08 openhab systemd[1]: Started Network Time Synchronization.
Sep 27 00:00:08 openhab systemd-timesyncd[3428]: Synchronized to time server for the first time 195.186.1.101:123 (0.debian.pool.ntp.org).

Someone understand what is happening or can address me to relevant documentations?

same thing happening to me. Fresh installation of Openhabian 3.0.1, 5.10.11-v7l+ , Raspberry Pi 4 Model B Rev 1.2, 4GB RAM, after 24 hours completely unresponsive.

After deactivation of time synchronization via systemd-timesyncd, not happening anymore.

so I assume that for you it is not a problem to lose the time synchronization as explained here:
https://wiki.archlinux.org/index.php/systemd-timesyncd.