Update to 3.4 can't doit - no space left on devise

  • Platform information:
    • Hardware: RaspberryPi3 2GB
    • OS: Openhabian
    • openHAB version: openHAB 3.3

Hello, when updating openhabian-config tool to upgrade to version 3.4 from 3.3 an error pops up - no space left.

[sudo] password for openhabian:
2022-10-13_16:12:26_CEST [openHABian] Checking for root privileges… OK
2022-10-13_16:12:26_CEST [openHABian] Loading configuration file ‘/etc/openhabian.conf’… OK
2022-10-13_16:12:26_CEST [openHABian] openHABian configuration tool version: [openHAB3]{2022-08-24T11:37:05+02:00}(4a11e45)
2022-10-13_16:12:27_CEST [openHABian] Checking for changes in origin branch openHAB3… OK
2022-10-13_16:12:30_CEST [openHABian] Switching to branch openHAB3… OK
2022-10-13_16:12:30_CEST [openHABian] Adding slightly tuned bash configuration files to system… OK
2022-10-13_16:12:34_CEST [openHABian] Updating Linux package information… OK
2022-10-13_16:12:38_CEST [openHABian] Updating repositories and upgrading installed packages…
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
Calculating upgrade… Done
The following packages have been kept back:
raspberrypi-net-mods raspberrypi-sys-mods
The following packages will be upgraded:
bluez dbus dhcpcd5 isc-dhcp-client isc-dhcp-common libbluetooth-dev libbluetooth3 libdbus-1-3 libuv1
9 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 2,293 kB of archives.
After this operation, 4,096 B of additional disk space will be used.
Get:1 Index of /debian bullseye/main armhf bluez armhf 5.55-3.1+rpt2 [844 kB]
Get:4 Index of /debian bullseye/main armhf dhcpcd5 armhf 1:8.1.2-1+rpt8 [145 kB]
Get:5 Index of /debian bullseye/main armhf libbluetooth-dev armhf 5.55-3.1+rpt2 [221 kB]
Get:7 Index of /debian bullseye/main armhf libbluetooth3 armhf 5.55-3.1+rpt2 [106 kB]
Get:2 Index of /raspbian bullseye/main armhf dbus armhf 1.12.24-0+deb11u1 [219 kB]
Get:3 Index of /raspbian bullseye/main armhf libdbus-1-3 armhf 1.12.24-0+deb11u1 [197 kB]
Get:6 Index of /raspbian bullseye/main armhf isc-dhcp-client armhf 4.4.1-2.3+deb11u1 [296 kB]
Get:8 Index of /raspbian bullseye/main armhf isc-dhcp-common armhf 4.4.1-2.3+deb11u1 [145 kB]
Get:9 Index of /pub/Linux/distributions/raspbian/raspbian bullseye/main armhf libuv1 armhf 1.40.0-2 [119 kB]
Fetched 2,293 kB in 1s (1,605 kB/s)
apt-listchanges: Reading changelogs…
dpkg: cannot write to log file ‘/var/log/dpkg.log’: No space left on device
(Reading database … 55951 files and directories currently installed.)
Preparing to unpack …/0-dbus_1.12.24-0+deb11u1_armhf.deb …
dpkg: cannot write to log file ‘/var/log/dpkg.log’: No space left on device
dpkg: cannot write to log file ‘/var/log/dpkg.log’: No space left on device
dpkg: cannot write to log file ‘/var/log/dpkg.log’: No space left on device
dpkg: cannot write to log file ‘/var/log/dpkg.log’: No space left on device
Unpacking dbus (1.12.24-0+deb11u1) over (1.12.20-2) …
dpkg: cannot write to log file ‘/var/log/dpkg.log’: No space left on device
dpkg: cannot write to log file ‘/var/log/dpkg.log’: No space left on device
Preparing to unpack …/1-libdbus-1-3_1.12.24-0+deb11u1_armhf.deb …
dpkg: cannot write to log file ‘/var/log/dpkg.log’: No space left on device
dpkg: cannot write to log file ‘/var/log/dpkg.log’: No space left on device
dpkg: cannot write to log file ‘/var/log/dpkg.log’: No space left on device
dpkg: cannot write to log file ‘/var/log/dpkg.log’: No space left on device
dpkg: cannot write to log file ‘/var/log/dpkg.log’: No space left on device

Where should I start looking

Your drive or SD card is apparently full. You should probably get a bigger replacement.

do

sudo df -h

this will show the different partitions on your sd card and it’s usage.
Looks like the /var/log space is full and either too small or the space is just being filled up too fast.
The df command will show the sizes of the partitions and their usage.

Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 6.8G 22G 25% /
devtmpfs 455M 0 455M 0% /dev
tmpfs 487M 0 487M 0% /dev/shm
tmpfs 195M 22M 173M 12% /run
tmpfs 5.0M 8.0K 5.0M 1% /run/lock
/dev/mmcblk0p1 253M 50M 203M 20% /boot
/dev/zram2 420M 411M 0 100% /opt/zram/zram2
/dev/zram4 420M 411M 0 100% /opt/zram/zram4
/dev/zram1 323M 153M 146M 52% /opt/zram/zram1
overlay1 323M 153M 146M 52% /var/lib/openhab/persistence
/dev/zram5 420M 411M 0 100% /opt/zram/zram5
overlay5 420M 411M 0 100% /var/log
tmpfs 98M 0 98M 0% /run/user/1000

Ok, I see. How change size ?.

Before increasing the size I would check for the root cause that is responsible that there is no free space.On my pi the size of that area is as well 420MB but 26MB are occupied.
Older logs are normally compressed/zipped that even could mean that the area is full due to recent data.

What is the output of
sudo du -k -s /var/log/* | sort -n

I’m sorry, but I was too busy. This is my results.

openhabian@openhabian:~ $ sudo du -k -s /var/log/* | sort -n
[sudo] password for openhabian:
0 /var/log/zram-config
8 /var/log/comitup.log.2022-10-10
12 /var/log/unattended-upgrades
44 /var/log/auth.log
44 /var/log/mosquitto
344 /var/log/comitup.log.2022-11-01
468 /var/log/messages
948 /var/log/kern.log
1300 /var/log/comitup.log
27176 /var/log/daemon.log
28128 /var/log/syslog
openhabian@openhabian:~ $ sudo df-h
sudo: df-h: command not found

openhabian@openhabian:~ $ sudo df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 7.0G 21G 25% /
devtmpfs 455M 0 455M 0% /dev
tmpfs 487M 0 487M 0% /dev/shm
tmpfs 195M 22M 174M 11% /run
tmpfs 5.0M 8.0K 5.0M 1% /run/lock
/dev/mmcblk0p1 253M 50M 203M 20% /boot
/dev/zram2 420M 411M 0 100% /opt/zram/zram2
/dev/zram4 420M 411M 0 100% /opt/zram/zram4
/dev/zram5 420M 411M 0 100% /opt/zram/zram5
/dev/sda3 30G 28K 28G 1% /storage
/dev/zram8 420M 411M 0 100% /opt/zram/zram8
/dev/zram9 323M 153M 146M 52% /opt/zram/zram9
overlay9 323M 153M 146M 52% /var/lib/openhab/persistence
/dev/zram10 420M 58M 331M 15% /opt/zram/zram10
overlay10 420M 58M 331M 15% /var/log
tmpfs 98M 0 98M 0% /run/user/1000

What can I do ?

In your previous post the /var/log/ related zram/overlay was occupied to 100%
This time it is filled to 15% which should be ok.

I am wondering why there are that many zram entries while for several of them the matching overlay entry is missing.
I also do not see the openhab directory under /var/log.
As in previous post /var/log/ resp. zram/overlay was full to 100% it was not possible to run the update command. This should be ok now according to the information you provided.

Did you do anything which resulted in getting all these orphaned zram ( 2, 4, 5 8 ) entries ?

Yes, updated Zram in openhabian config.

I have the same problem at a update with the space in var/log. Is it really possible to change the size manually?
Update4

Why does your system reqire that much kogging space?

Good question. Maybe it is the Logviewer. I get many Data over the modbus. Or my temperature and humidity of the complette house to store in a graph.

I would say as the logviewer is just used to view the data that is logged it is not the problem.
The data that is being logged could be the problem because of log level and amount of data.
Storing data in graph should be stored in persistence directory shouldn’t it ?
Persistence is 27MB compared to 411 MB in /var/log.
Dig deeper into /var/log to check amount of files and it’s file sizes.
E.g. check how much space is used in /var/log/openhab ( du -s -h /var/log/openhab/ ).

Ok in var/log/openhab is not much

openhabian@openhabian:~ $ du -s -h /var/log/openhab/
28M     /var/log/openhab/

but in mosquitto:

openhabian@openhabian:/var/log $ du -s -h /var/log/mosquitto/
354M    /var/log/mosquitto/

I have many sensors and window contacts in mqtt. I thing thats the Problem.

What do you need to log and which log level is enabled in logging for mosquitto ?
I would suggest just log the error level apart of if you need a higher level.
Are there several files kept in history of the mosquitto folder in /var/log ? In case you do not need the full history that also should be configurable to just keep the current one and n further files.

Sorry I don’t know the log level of mosquitto. How can I show that?
Error level is just enough for me.

Tha are in the folder:

openhabian@openhabian:/var/log/mosquitto $ ls
mosquitto.log  mosquitto.log.1  mosquitto.log.2.gz

mosquitto configuration files can be found in /etc/mosquitto and /etc/mosquitto/conf.d.
/etc/mosquitto/mosquitto.conf is the distributions configuration file. In /etc/mosquitto/conf.d/ you may have added your configuration changes.
In one of the files there should be an entry log_type.

Logging then can have these log levels:

log_type types

Choose types of messages to log. Possible types are: debug, error, warning, notice, information, subscribe, unsubscribe, websockets, none, all.

Defaults to error, warning, *notice *and information. This option may be specified multiple times. Note that the *debug *type (used for decoding incoming/outgoing network packets) is never logged in topics.

Reloaded on reload signal.

The folder /etc/mosquitto/conf.d is empty.

In /etc/mosquitto/mosquitto.conf found I that:

# Place your local configuration in /etc/mosquitto/conf.d/
#
# A full description of the configuration file is at
# /usr/share/doc/mosquitto/examples/mosquitto.conf.example

pid_file /var/run/mosquitto.pid

persistence true
persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d

password_file /etc/mosquitto/passwd
allow_anonymous false

But I read nothing about the log types.

Then the default logging is being used.
Create file /etc/mosquitto/conf.d/log.conf with content

log_type error

ok and now I have to clear the log file at /var/log/mosquitto? Or better to delete the file?

I can see the biggest file is the mosquitto.log.1 with 346M.