Suddenly Disk full

Hi,

I’m running openhab2.3 on a RPi.
All of a sudden the system reports, that my disk is fully used. I hope you could help me with finding the root cause - I’m a Linux newbee:
I’ve got a 32Gb SD and moved all logging and perstistance to my NAS, off from my SD.
Today at 2 o’clock all logging stopped, reporting:

java.lang.RuntimeException: {"error":"engine: error writing WAL entry: write /var/lib/influxdb/wal/openhab_db/autogen/144/_00001.wal: no space left on device"}

which I find strange per se, because my log is external…

[08:03:07] openhabian@openHABianPi:/$ sudo df -h
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root        15G     15G     0  100% /
devtmpfs        481M       0  481M    0% /dev
tmpfs           486M       0  486M    0% /dev/shm
tmpfs           486M     50M  437M   11% /run
tmpfs           5,0M    4,0K  5,0M    1% /run/lock
tmpfs           486M       0  486M    0% /sys/fs/cgroup
/dev/mmcblk0p1   41M     21M   20M   52% /boot

Why can my root grow that much all of a sudden and is there a trick to “purge unused data” which I hope would be the root cause of this error?

How come 15GB are reported as “disk full” when the total size should be 32Gb?

Kurt

First of all, your main (root) partition is 15Gigs. The other partitions on the SD Card have been used for other purposes. It seems that you formatted the card as 16G (even if it has a 32G capacity).

Secondly, your influxdb service is complaining in the log that you posted that it cannot store data anymore (due to 0 available disk space on the root partition)

Maybe? you are storing toooo much data using influxdb persistence?
Check how big is your influxdb storage. (sudo du -sk /var/lib/influxdb/)

Are you sure that you have moved persistence storage to NAS?

Hi,

yes, I moved it.
here’s the output of the command:

48040	/var/lib/influxdb/

Kurt

ok, that’s not so big… this means that influxdb is not the root-cause of your storage problems

post the output of the following commands please:

sudo mount
sudo du -s /var/log/* | sort -nr | head -n10
sudo findmnt

sudo mount:

/dev/mmcblk0p2 on / type ext4 (rw,noatime,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=492508k,nr_inodes=123127,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
tmpfs on /etc/machine-id type tmpfs (ro,mode=755)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mmcblk0p2 on /srv/openhab2-addons type ext4 (rw,noatime,data=ordered)
/dev/mmcblk0p2 on /srv/openhab2-logs type ext4 (rw,noatime,data=ordered)
/dev/mmcblk0p2 on /srv/openhab2-userdata type ext4 (rw,noatime,data=ordered)
/dev/mmcblk0p2 on /srv/openhab2-sys type ext4 (rw,noatime,data=ordered)
/dev/mmcblk0p2 on /srv/openhab2-conf type ext4 (rw,noatime,data=ordered)
/dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)

sudo du -s /var/log/* | sort -nr | head -n10:

[08:34:59] openhabian@openHABianPi:/mnt/DS716$ sudo du -s /var/log/* | sort -nr | head -n10
13816	/var/log/daemon.log
6812	/var/log/syslog.1
5936	/var/log/daemon.log.1
1884	/var/log/syslog
1504	/var/log/kern.log.1
1328	/var/log/daemon.log.3.gz
1172	/var/log/amanda
1148	/var/log/daemon.log.2.gz
912	/var/log/daemon.log.4.gz
432	/var/log/messages

sudo findmnt:

TARGET                           SOURCE                                     FSTYPE   OPTIONS
/                                /dev/mmcblk0p2                             ext4     rw,noatime,data=ordered
├─/dev                           devtmpfs                                   devtmpfs rw,relatime,size=492508k,nr_inodes=123127,mode=755
│ ├─/dev/shm                     tmpfs                                      tmpfs    rw,nosuid,nodev
│ ├─/dev/pts                     devpts                                     devpts   rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ └─/dev/mqueue                  mqueue                                     mqueue   rw,relatime
├─/sys                           sysfs                                      sysfs    rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup               tmpfs                                      tmpfs    ro,nosuid,nodev,noexec,mode=755
│ │ ├─/sys/fs/cgroup/systemd     cgroup                                     cgroup   rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd
│ │ ├─/sys/fs/cgroup/cpu,cpuacct cgroup                                     cgroup   rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
│ │ ├─/sys/fs/cgroup/blkio       cgroup                                     cgroup   rw,nosuid,nodev,noexec,relatime,blkio
│ │ ├─/sys/fs/cgroup/memory      cgroup                                     cgroup   rw,nosuid,nodev,noexec,relatime,memory
│ │ ├─/sys/fs/cgroup/devices     cgroup                                     cgroup   rw,nosuid,nodev,noexec,relatime,devices
│ │ ├─/sys/fs/cgroup/freezer     cgroup                                     cgroup   rw,nosuid,nodev,noexec,relatime,freezer
│ │ └─/sys/fs/cgroup/net_cls     cgroup                                     cgroup   rw,nosuid,nodev,noexec,relatime,net_cls
│ ├─/sys/kernel/debug            debugfs                                    debugfs  rw,relatime
│ └─/sys/kernel/config           configfs                                   configfs rw,relatime
├─/proc                          proc                                       proc     rw,relatime
│ └─/proc/sys/fs/binfmt_misc     systemd-1                                  autofs   rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
├─/run                           tmpfs                                      tmpfs    rw,nosuid,nodev,mode=755
│ └─/run/lock                    tmpfs                                      tmpfs    rw,nosuid,nodev,noexec,relatime,size=5120k
├─/etc/machine-id                tmpfs[/machine-id]                         tmpfs    ro,mode=755
├─/srv/openhab2-addons           /dev/mmcblk0p2[/usr/share/openhab2/addons] ext4     rw,noatime,data=ordered
├─/srv/openhab2-logs             /dev/mmcblk0p2[/var/log/openhab2]          ext4     rw,noatime,data=ordered
├─/srv/openhab2-userdata         /dev/mmcblk0p2[/var/lib/openhab2]          ext4     rw,noatime,data=ordered
├─/srv/openhab2-sys              /dev/mmcblk0p2[/usr/share/openhab2]        ext4     rw,noatime,data=ordered
├─/srv/openhab2-conf             /dev/mmcblk0p2[/etc/openhab2]              ext4     rw,noatime,data=ordered
└─/boot                          /dev/mmcblk0p1                             vfat     rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro

Kurt

I don’t see any NFS mounts… Which guide did you use?
Also: Your /var/log/ dir seems ok… you need to check more subdirs under the root partition for excessive data.

edit: check the output of the following command:

sudo du -sk /srv/openhab2-logs

I would install and use ncdu to find files/folders that use too much space.

4 Likes

Hi,

so, I’ve been fiddling around a bit and found, that my cache was too large, so I cleaned all of the temp and cached items.
new status: System is running again :blush:
Thanks @christoph_wempe for your hint on Ncdu!!

[13:07:27] openhabian@openHABianPi:/$ sudo df -h
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root        15G    2,8G   12G   20% /
devtmpfs        481M       0  481M    0% /dev
tmpfs           486M       0  486M    0% /dev/shm
tmpfs           486M    6,8M  479M    2% /run
tmpfs           5,0M    4,0K  5,0M    1% /run/lock
tmpfs           486M       0  486M    0% /sys/fs/cgroup
/dev/mmcblk0p1   41M     21M   20M   52% /boot

however, there are subsequent issues, where I hope I can get your expert guidance on:

  • I’m still puzzled about @Dim’s statement about:

First of all, your main (root) partition is 15Gigs. The other partitions on the SD Card have been used for other purposes. It seems that you formatted the card as 16G (even if it has a 32G capacity).

Could it be that my backup solution is triggering it?
I’m doing a sudo dd if=/dev/mmcblk0 of=/mnt/DS716/backups/SD-Backup.img. Is an image of a smaller SD not transferrable to a larger (say 32GB to 64GB) without “blocking” the remaining space of the larger SD?
(Hopefully I expressed myself clearly :wink: )

UPDATE: I have solved the below by running sudo /etc/init.d/rpcbind start

  • I have lost the subdirectories of my mounted drive on my NAS. The Folders (namely influxdb, rrd4j, backups) are still present on the NAS, but a ls via the console from the rpi doesn’t show anything besides the log files.
events.log  openhab.log  test.text

How can I restore those subdirectories to be able to allow the openhabianpi to “see” them?

  • I get a lot of those errors in the logs now:
Could not create rrd4j database file '/var/lib/openhab2/persistence/rrd4j/Sunset_Azimuth.rrd': /var/lib/openhab2/persistence/rrd4j/Sunset_Azimuth.rrd (Datei oder Verzeichnis nicht gefunden)

(To answer your question on how I moved the presistance: Write Persistence data to NFS share

Hope you can help!
Kurt

For the curious (I looked it up before I read Christoph’s posting about ncdu) the du command would be:

du -a /home | sort -n -r | head -n 5

Which will show the top five largest files/folders.

How did you set up this RPi? If openHABian, did you have it expand the file system to fill the whole SD card? If Raspbian did you do this from raspi-config?

There isn’t enough information posted here to definitively answer, but usually to mount a remote system requires an entry in /etc/fstab then running the command mount. e.g.

sudo mount /var/log/openhab2

by the way: that guide assumes that you have already mounted a remote NFS file system and then goes on to inform you on what configs to modify to redirect logs + persistence db to that destination. It does not cover the actual mounting. For that, google “NFS client Howto” (or something similar :))

1 Like

Hi,

I’ve used OpenHABian - can I still expand the file system after the install, using openhabian-conf?
I think the problem is, that I made an image, from a smaller SD card and then used a 64Gb one, where I put the “smalled SD Card’s image” on.

Kurt

As I don’t use openHABian I don’t know if there is a command in openhabian-config, but it’s always possible to expand the file system.

Thanks to all for your guidance (once more).

This feels better :wink:

Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root        59G    2,6G   54G    5% /
devtmpfs        481M       0  481M    0% /dev
tmpfs           486M       0  486M    0% /dev/shm
tmpfs           486M    6,7M  479M    2% /run
tmpfs           5,0M    4,0K  5,0M    1% /run/lock
tmpfs           486M       0  486M    0% /sys/fs/cgroup
/dev/mmcblk0p1   41M     21M   20M   52% /boot

Kurt

1 Like

I have a similar problem. My SD card is only 15Gb but all of a sudden I got an out of space error. I expanded the filesystem to be able to use every single bit of the SD card. I use a Cron job to poll bluetooth devices and post the data to openhab (xiaomi temperature and humidity sensors). I thus cleared almost all the lines in the /var/mail/openhabian to get some more free space.

I now have 300Mb free.
Could someone help me understand what is the root cause please, before the SD card gets full again?

Thanks in advance!

See https://www.cyberciti.biz/faq/how-do-i-find-the-largest-filesdirectories-on-a-linuxunixbsd-filesystem/. You need to find what folder(s) are using up the most space. Then you need to figure out why. What is being written to that folder(s)? Why are they not being cleaned up?

Thanks Rich for your help. I did check which are the largest directories but only got the following:

11025760        /var
5854188 /var/lib
4697936 /var/lib/unifi
4675252 /var/lib/unifi/db
3850556 /var/log
1210468 /var/cache
1201636 /var/cache/apt
1141296 /var/cache/apt/archives
819212  /var/lib/mongodb
812420  /var/log/journal

I then tried running the following as per your link:

alias ducks=‘du -cks – * | sort -rn | head’

and got the following:

108416 total
98368 unifi_sysvinit_all.deb
1824 xiaomi-ble-mqtt
1192 libtinyb.so.1
1192 libtinyb.so
708 bluez_5.43-2+rpt2+deb9u2_armhf.deb.2
708 bluez_5.43-2+rpt2+deb9u2_armhf.deb.1
708 bluez_5.43-2+rpt2+deb9u2_armhf.deb
692 Xiaomi-Mijia-Bluetooth-Temperature-and-Humidity-Sensor
680 bt-mqtt-gateway

I don’t see any file or folder which is hogging up space. But I’m quite new to linux so I might not know where to look.

You could do some cleaning such as apt-cache clean but it won’t last for long.
In the end you just have installed too much. That unifi stuff consumes 4,5GB alone.
So the probably most intelligent solution would be to reinstall, else you’ll keep having this sort problem again and again. Omit at least the Unifi stuff, or use a larger SD, anything else is bungling.

Thanks a lot!

I’ve cloned the SD to a 32GB one and expanded the filesystem… works fine at least for now.

I’m just waiting on unifi to issue some hardware and then I’ll remove their controller from the Pi :slight_smile:

@mstormi The command apt-cache clean returns an error
E: Invalid operation clean

Did you by any chance want to refer to apt-get clean instead?

Is there a way to clear the contents of /var/mail/openhabian every so often? I’ve got a script which runs every 5 mins and writes to that file… so it tends to get pretty full too.

sure write a cronjob. but that’s bungling now (means: I’ll leave you alone with that).

Is there a better way?