OH3.1 Fresh Installation Issues: Mosquitto, Grafana, Permissions... how to check if rest is OK?

[stm (Stefan)]

• Platform information:
  • Hardware: Raspberry 3B+ LAN connected with Openhabian on 32GB SD card
    Fresh Install, in openhabian.conf: hostname, password, debug maximum changed

Issue1:
Mosquitto not available after Install:
Search forum and applied
sudo mkdir -p /var/log/mosquitto /opt/zram/log.bind/mosquitto sudo chown mosquitto /var/log/mosquitto /opt/zram/log.bind/mosquitto sudo systemctl start mosquitto.service

Create /lib/systemd/system/mosquitto.service.d/override.conf

to contain

[Unit] After=network-online.target zram-config.service Wants=network-online.target zram-config.service
Mosquitto works after reboot

Issue 2:
Grafana not available after Install
Search forum and applied:

sudo mkdir -p /var/log/grafana /opt/zram/log.bind/grafana
sudo chown grafana /var/log/grafana /opt/zram/log.bind/grafana

Grafana Works after reboot

Issue 3:
Menue Item “Fix Permissions” returned with Errors.
From Log output it seems ‘/var/log/openhab’ missing

Search Web and applied:

‘sudo mkdir /var/log/openhab’
‘sudo chown openhab: /var/log/openhab’

after reboot fix permissions works, frontail gives more output

Questions:

1. What did I do wrong in installing?
2. How can I be sure, that system is correctly setup, especially zram setup, and additional functions like logging, backup, etc.?

Thank you for help

[mstormi (Markus Storm)]

@stm

What did I do wrong in installing?

What did you do at all in installing ?
which image, downloaded when ? which install instructions ? which branch ? which version is shown ?

Log ?

[stm (Stefan)]

Thank you for fast reply,
I used openhabian-pi-raspios32-v1.6.5.img.xz, Main branch, unattended installation, the Install via openhabian-config Mosquitto, Influx/Grafana

I will do a new clean install in the evening and provide log.

[mstormi (Markus Storm)]

@stm

I used openhabian-pi-raspios32-v1.6.5.img.xz, Main branch, unattended installation,

What do you mean by that? Name the link. Flashing the image always is followed by sort of an unattended install on booting while “unattended installation” would means to install Raspi OS, then clone openhabian repo, then execute openhabian-setup unattended.
Name the openHABian version you have now (title bar/menu 00 of oh-config).

[stm (Stefan)]

Short version:
fresh install mosquito and grafana worked, after reboot no longer…and Still permissiin issues

long version:

1. Create SD Card

Downloaded Image from github.com/openhab/openhabian/releases/tag/v1.6.5 today

Image Name: openhabian-pi-raspios32-v1.6.5.img.xz

Install via Balena Etcher to 32GB SD Card

2. Change openhabian.conf
   ‘hostname=…
   userpw="…"
   ipv6=disable
   debugmode=maximum
   clonebranch=main (unchanged)’

3. Insert in Raspi 3B+ with LAN Connect and start Pi with auto install

4. login via SSh and run openhabian.- config
   Headerline is ‘[main]{2021-08-04T15:31:50+02:00}(daa0885)’

5. Install Mosquitto log in install-mosquitto.log
   install-mosquitto.log (84.9 KB)

Connection to MQTT Broker from external machine possible

6. ‘sudo reboot’ …
   after Reboot still connection to MQTT possible

7. Install Influxdb and Grafana log in install-grafana.log
   install-grafana.log (248.1 KB)

connection to Grafana via port 3000 possible

8. ‘sudo reboot’
   no longer possible to connect to MQTT and Grafana

9. ‘openhabian-config’ apply improvenets/fix permission gives Error … log in fix-perm.log
   fix-perm.log (74.0 KB)

After reboot: Mosquitto and Grafana still gone,

[mstormi (Markus Storm)]

@stm

‘openhabian-config’ apply improvenets/fix permission gives Error … log in fix-perm.log

for some to me unknown reason it seems /var/log/openhab or /var/log are missing.
That in turn is probably why Mosquitto and Grafana do not start.
Get /var/log back and I’d guess they will start again.

Do those dirs exist ?
Did you mess with zram config ? what does zramctl say ?
Did you this time again mess with systemd config for Mosquitto or Grafana ? You must not.

+ chown --recursive openhab:openhab /etc/openhab /var/lib/openhab /var/log/openhab /usr/share/openhab
chown: cannot access '/var/log/openhab': No such file or directory
+ return 1
+ echo 'FAILED (openhab folders)'

[stm (Stefan)]

‘/var/log’ exists, ‘/var/log/openhabian’ does not exist

As said:
I transferred image to SDcard, made changes to ‘openhabian.conf’
then start Raspi, watched messages of install in Web Interface
when finished did ssh to machine:
started Openhabian-config to install Mosquitto
reboot
started Openhabian-config to install InflixDB/Grafana
reboot
started Openhabian-config to fix permissions

run first config of Openhab via Web Interface (Admin Account, Position)

no other commands, zram config not touched

‘zramctl
NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram3 zstd 450M 16.9M 469.4K 844K 4 /opt/zram/zram3
/dev/zram2 zstd 350M 16.3M 5.2K 96K 4 /opt/zram/zram2
/dev/zram1 zstd 350M 16.4M 10.5K 148K 4 /opt/zram/zram1
/dev/zram0 lzo-rle 450M 4K 86B 12K 4 [SWAP]’

Strange:
MQTT did work after install, did survive one reboot but was gone after reboot after Grafana Install
Grafana did work after Install, but was gone after reboot

[mstormi (Markus Storm)]

@stm

‘/var/log’ exists, ‘/var/log/openhabian’ does not exist

you mean /var/log/openhab ?
what does df -k /var/log say, is it on zram ?
Show /etc/ztab and provide the full output of df.
What is in /opt/zram/zram3/upper (or 1 or 2, whatever the device for /var/log is on your system) and in /opt/zram/log.bind/ ?

Why did you boot twice? reboots are a bad habit they do more harm then good.
Did you mess with systemd config like last time ?

[stm (Stefan)]

yes, you are right /var/log/openhab missing

df -k /var/log
Filesystem     1K-blocks  Used Available Use% Mounted on
overlay3          429960  1168    396536   1% /var/log

cat /etc/ztab
# Use '#' to comment out any line, add new drives with the first column
# providing the drive type and then drive details separated by tab characters.
#
# All algorithms in /proc/crypto are supported but only lzo-rle, lzo, lz4, and
# zstd have zramctl text strings; lzo-rle is the fastest with zstd having much
# better text compression.
#
# mem_limit is the compressed memory limit and will set a hard memory limit for
# the system admin.
#
# disk_size is the virtual uncompressed size approx. 220-450% of memory
# allocated depending on the algorithm and input file. Don't make it much higher
# than the compression algorithm is capable of as it will waste memory because
# there is a ~0.1% memory overhead when empty
#
# swap_priority will set zram over alternative swap devices.
#
# page-cluster 0 means tuning to singular pages rather than the default 3 which
# caches 8 for HDD tuning, which can lower latency.
#
# swappiness 80 because the improved performance of zram allows more usage
# without any adverse affects from the default of 60. It can be raised up to 100
# but that will increase process queue on intense loads such as boot time.
#
# target_dir is the directory you wish to hold in zram, and the original will be
# moved to a bind mount 'bind_dir' and is synchronized on start, stop, and write
# commands.
#
# bind_dir is the directory where the original directory will be mounted for
# sync purposes. Usually in '/opt' or '/var', name optional.
#
# oldlog_dir will enable log-rotation to an off device directory while retaining
# only live logs in zram. Usually in '/opt' or '/var', name optional.
#
# If you need multiple zram swaps or zram directories, just create another entry
# in this file.
# To do this safely, first stop zram using 'sudo zram-config "stop"', then edit
# this file.
# Once finished, restart zram using 'sudo systemctl start zram-config.service'.

# swapalgmem_limitdisk_sizeswap_prioritypage-clusterswappiness
swaplzo-rle200M450M75080

# diralgmem_limitdisk_sizetarget_dirbind_dir
dirzstd150M350M/var/lib/openhab/persistence/persistence.bind
dirzstd150M350M/var/lib/influxdb/influxdb.bind

# logalgmem_limitdisk_sizetarget_dirbind_diroldlog_dir
logzstd200M450M/var/log/log.bind

df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root       30479000 4993020  24205680  18% /
devtmpfs          464768       0    464768   0% /dev
tmpfs             498048       0    498048   0% /dev/shm
tmpfs             498048    7596    490452   2% /run
tmpfs               5120       0      5120   0% /run/lock
tmpfs             498048       0    498048   0% /sys/fs/cgroup
/dev/mmcblk0p1    258095   49508    208587  20% /boot
/dev/zram1        330768     432    305248   1% /opt/zram/zram1
overlay1          330768     432    305248   1% /var/lib/openhab/persistence
/dev/zram2        330768     384    305296   1% /opt/zram/zram2
overlay2          330768     384    305296   1% /var/lib/influxdb
/dev/zram3        429960    1168    396536   1% /opt/zram/zram3
overlay3          429960    1168    396536   1% /var/log
tmpfs              99608       0     99608   0% /run/user/1000

openhabian:/opt/zram/zram3/upper $ ls
auth.log    debug     lastlog   syslog    syslog.2.gz          wtmp
daemon.log  kern.log  messages  syslog.1  unattended-upgrades
openhabian:/opt/zram/zram2/upper $ ls
influxd.pid
openhabian:/opt/zram/zram1/upper $ ls
db4o  mapdb  rrd4j
openhabian:/opt/zram/log.bind $ ls
auth.log  daemon.log  kern.log  messages  syslog    wtmp
btmp      debug       lastlog   private   user.log

In doing first steps in Openhab, I did shut down the Raspi, when not in use - do not have continuously time. Noticed the issue of lost MQTT and Grafana problem.

This time reboot after install was just to reproduce the behaviour.
Besides the steps described, I did not do any operation in openhabian-config nor command line operations like systemctl, create dirs, change files…

Why exactly is it bad to reboot ? Is it bad as well to shut down (and restart) system?

[mstormi (Markus Storm)]

@stm

Why exactly is it bad to reboot ? Is it bad as well to shut down (and restart) system?

Yes. OH is designed to run 24/7. Better get used to it or all sorts of strange stuff might happen.
I suspect your issue is related because it works with others and this seems to be the only difference.
Also I cannot reproduce it.
While debugging it’s bad because it’s destroying states and evidence.

Your ztab looks strange, there’s whitespaces missing. Is that a copy’n paste error or is it really lines without spaces on your system ?
Something is strange about your system. You have /var/log running on zram but it has no openhab or mosquitto directories. These get created on install of oh and mosquitto packages.

I suggest you try installing again. No reboots and no messing with systemd config this time.
And don’t change the hostname.
Double-check again after if the dirs exist. They should exist in /opt/zram/log.bind and /opt/zram/zram3/upper, too.
Check first-boot.log for errors, too.

[stm (Stefan)]

I understand that OH shall run7/24 when in normal operation.
On the other hand, there should be a mode to gracefully shutdown a system - or a robust behaviour for powerfail.

I would not like to have to reinstall the system in case of powerfail, as I do not have a UPS for Raspi (and even UPS seems just for graceful shutdown in case of powerfail)

The ztab was copy and paste from putty.log. (setting printable output) in putty window there are spaces

I will do a reinstall.
No messing with systemd config means not change anything in openhabian.conf ?

Hostname and password change later via openhabian-config ?

How and when set debug to maximum

How to shutdown system ?

Pull plug to simulate powerfail?

[mstormi (Markus Storm)]

@stm

I would not like to have to reinstall the system in case of powerfail, as I do not have a UPS for Raspi

Then get one. openHABian usually keeps running after powerfail but some data is lost which in turn can lead to problems so if you want to be on the safe side, get a UPS.

@stm

On the other hand, there should be a mode to gracefully shutdown a system

Of course, it works to shutdown or reboot.
But it is a bad idea now when debugging.

@stm

No messing with systemd config means not change anything in openhabian.conf ?

No it means NOT to do what read about in the forum (mkdir, override.conf etc). It’s outdated anyway.

@stm

Hostname and password change later via openhabian-config ?

yes when you are done with debugging

@stm

How and when set debug to maximum

Set debugmode=maximum in openhabian.conf.

@stm

Pull plug to simulate powerfail?

NO by no means

[stm (Stefan)]

OK, got it.

[stm (Stefan)]

Situtaion after install:

no zram3

shall I install Mosquitto now via openhabian-config?

[mstormi (Markus Storm)]

Yes.
Does openhab run ?
And no screenshots please.

[stm (Stefan)]

OH runs (I get web loginscreen, frontail shows openhab.log, events.log no red marking)

Openhabian-config wants to update, Branch is set to main, headline says
{2021-08-11T16:17:58-06:00}(49d8999)

after Install of Mosquito: MQTT connect from external machine possible

log.bind has

alternatives.log  bootstrap.log  debug     fontconfig.log  lastlog    openhab  syslog               user.log
apt               btmp           dpkg.log  journal         messages   private  sysstat              wtmp
auth.log          daemon.log     faillog   kern.log        mosquitto  samba    unattended-upgrades

zram2/upper has

apt       daemon.log  dpkg.log  journal   lastlog   mosquitto  samba   syslog.1             wtmp
auth.log  debug       faillog   kern.log  messages  openhab    syslog  unattended-upgrades

/var/log has

alternatives.log  bootstrap.log  debug     fontconfig.log  lastlog    openhab  syslog    unattended-upgrades
apt               btmp           dpkg.log  journal         messages   private  syslog.1  user.log
auth.log          daemon.log     faillog   kern.log        mosquitto  samba    sysstat   wtmp

Installation of InfluxDB and Grafana via openhabian-config
Grafana is reachable from External via Web login

now log.bind

alternatives.log  bootstrap.log  debug     fontconfig.log  journal   messages   private  syslog.1             user.log
apt               btmp           dpkg.log  grafana         kern.log  mosquitto  samba    sysstat              wtmp
auth.log          daemon.log     faillog   influxdb        lastlog   openhab    syslog   unattended-upgrades

and zram2/upper

apt       daemon.log  dpkg.log  grafana   journal   lastlog   mosquitto  samba   syslog.1             wtmp
auth.log  debug       faillog   influxdb  kern.log  messages  openhab    syslog  unattended-upgrades

and /var/log

alternatives.log  bootstrap.log  debug     fontconfig.log  journal   messages   private  syslog.1             user.log
apt               btmp           dpkg.log  grafana         kern.log  mosquitto  samba    sysstat              wtmp
auth.log          daemon.log     faillog   influxdb        lastlog   openhab    syslog   unattended-upgrades

in addition there is a zram3/upper

data influxd.pid meta

and a zram4/upper

apt auth.log daemon.log debug dpkg.log faillog grafana journal kern.log lastlog messages openhab samba syslog

and zram2/upper/openhab and zram4/upper/openhab seem to have partial identical files

openhabian@openhabian:/opt/zram $ ls -al zram2/upper/openhab
total 32
drwxrwxr-x  2 openhab openhabian 4096 Aug 11 12:35 .
drwxr-xr-x 10 root    root       4096 Aug 12 01:18 ..
-rw-rw-r--  1 openhab openhab       0 Aug 11 12:25 audit.log
-rw-rw-r--  1 openhab openhab       0 Aug 11 12:25 events.log
-rw-r--r--  1 openhab openhab     945 Aug 12 01:15 openhab.log
openhabian@openhabian:/opt/zram $ ls -al zram4/upper/openhab
total 36
drwxrwxr-x 2 openhab openhabian 4096 Aug 12 01:16 .
drwxr-xr-x 7 root    root       4096 Aug 12 01:17 ..
-rw-rw-r-- 1 openhab openhab       0 Aug 11 12:25 audit.log
-rw-rw-r-- 1 openhab openhab       0 Aug 11 12:25 events.log
-rw-r--r-- 1 openhab openhab     427 Aug 12 01:18 openhab.log
-rw-r--r-- 1 openhab openhab     945 Aug 12 01:15 openhab.log.2

so far everything seems to work - as did last time before I did reboot…

[mstormi (Markus Storm)]

We’ve just changed a detail.
Uninstall influxDB + Grafana from the command line: apt purge influxdb grafana

Start openhabian-config, have it update (select main branch if not already selected), reinstall Influx+Grafana from the menu.

Remove grafana.service from the line in /etc/systemd/system/zram-config.service (but keep grafana-server.service).

Check if they run, then reboot.

If they don’t start on boot try to start them manually (systemctl start grafana-server.service).

[stm (Stefan)]

OK, thank you.

did uninstall, reciving some warnings

dpkg: warning: while removing influxdb, unable to remove directory '/var/lib/influxdb': Device or resource busy - directory may be a mount point?
Processing triggers for man-db (2.8.5-2) ...
(Reading database ... 53516 files and directories currently installed.)
Purging configuration files for influxdb (1.8.9-1) ...
dpkg: warning: while removing influxdb, unable to remove directory '/var/lib/influxdb': Device or resource busy - directory may be a mount point?
Purging configuration files for grafana (8.1.1) ...
dpkg: warning: while removing grafana, directory '/etc/grafana' not empty so not removed

reinstalled via openhabian-config
Headline says [main]{2021-08-12T11:54:06+02:00}(7f07b20)

Install and removal of grafana.service entry done

processes for /usr/bin/influxd and /usr/sbin/grafana-server running, connect to Grafana Web Interface works

Reboot via sudo reboot

after reboot /var/log/openhab gone

no processes for mosquitto or grafana running, /usr/bin/influxd running

systemctl start grafana-server.service does not start Grafana

/opt/zram is different from before

openhabian@openhabian:~ $ ls /opt/zram
influxdb.bind log.bind persistence.bind zram1 zram2 zram3 zram-config
openhabian@openhabian:~ $ ls /opt/zram/log.bind
auth.log daemon.log kern.log messages syslog wtmp
btmp debug lastlog private user.log
openhabian@openhabian:~ $ ls /opt/zram/zram1/upper
openhabian@openhabian:~ $ ls /opt/zram/zram2/upper
influxd.pid
openhabian@openhabian:~ $ ls /opt/zram/zram3/upper
auth.log daemon.log kern.log lastlog messages syslog wtmp
openhabian@openhabian:~ $

[mstormi (Markus Storm)]

Uninstall mosquitto/influxdb/grafana and then uninstall zram. Re-install zram, then the others.
Or try installing openHABian again from scratch.
There has been a bugfix that may help you.

[stm (Stefan)]

Bugfix unfortunately not helpful.

reinstalled with new image, install mosquitto, Grafana (openhabian Title had date of Aug 13th)
both gone after reboot (/var/log/openhabian as well)