Openhabian - logs not written, amanda backup not working anymore

Setup, Configuration and Use Installation
1

my setup was stable for a few month but not i have the same (?) issues i had in the summer: frontail log doesn’t show entries, logs are not written to the files. amanda backup doesn’t run anymore…
this is my setup:

###############################################################################
###############  ohab3  #######################################################
###############################################################################
##        Ip = 192.168.178.80
##   Release = Raspbian GNU/Linux 10 (buster)
##    Kernel = Linux 5.10.17-v7l+
##  Platform = none
##    Uptime = 43 day(s). 15:31:48
## CPU Usage = 0% avg over 4 cpu(s) (4 core(s) x 1 socket(s))
##  CPU Load = 1m: 0.09, 5m: 0.07, 15m: 0.07
##    Memory = Free: 1.37GB (36%), Used: 2.41GB (64%), Total: 3.78GB
##      Swap = Free: 0.53GB (100%), Used: 0.00GB (0%), Total: 0.53GB
##      Root = Free: 3.67GB (55%), Used: 3.00GB (45%), Total: 7.00GB
##   Updates = 79 apt updates available.
##  Sessions = 3 session(s)
## Processes = 144 running processes of 32768 maximum processes
###############################################################################

                          _   _     _     ____   _
  ___   ___   ___   ___  | | | |   / \   | __ ) (_)  ____   ___
 / _ \ / _ \ / _ \ / _ \ | |_| |  / _ \  |  _ \ | | / _  \ / _ \
| (_) | (_) |  __/| | | ||  _  | / ___ \ | |_) )| || (_) || | | |
 \___/|  __/ \___/|_| |_||_| |_|/_/   \_\|____/ |_| \__|_||_| | |
      |_|                          3.1.0 - Release Build

looking into sudo journalctl -b i find these errors everyday around midnight:

Dec 12 00:00:01 ohab3 systemd[1]: Starting Rotate log files...
Dec 12 00:00:01 ohab3 logrotate[18890]: error: skipping "/var/log/exim4/mainlog" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell l
Dec 12 00:00:01 ohab3 logrotate[18890]: error: skipping "/var/log/exim4/rejectlog" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell
Dec 12 00:00:01 ohab3 logrotate[18890]: error: skipping "/var/log/exim4/paniclog" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell
Dec 12 00:00:01 ohab3 rsyslogd[611]:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="611" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Dec 12 00:00:02 ohab3 rsyslogd[611]:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="611" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Dec 12 00:00:02 ohab3 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Dec 12 00:00:02 ohab3 systemd[1]: logrotate.service: Failed with result 'exit-code'.
Dec 12 00:00:02 ohab3 systemd[1]: Failed to start Rotate log files.

and this error (around 6:25) everyday:

Dec 12 06:25:02 ohab3 exim4[27942]: ALERT: exim paniclog /var/log/exim4/paniclog has non-zero size, mail system possibly broken

the amanda backup (including mail) worked until dec, 19 (01:00). on the same day later these errors pop up:

Dec 19 14:21:30 ohab3 kernel: EXT4-fs warning (device zram2): ext4_end_bio:349: I/O error 10 writing to inode 35 starting block 101816)
Dec 19 14:21:30 ohab3 kernel: Buffer I/O error on device zram2, logical block 101816
Dec 19 14:21:30 ohab3 kernel: Buffer I/O error on device zram2, logical block 101817
Dec 19 14:21:30 ohab3 kernel: Buffer I/O error on device zram2, logical block 101818
Dec 19 14:21:30 ohab3 kernel: Buffer I/O error on device zram2, logical block 101819
Dec 19 14:21:30 ohab3 kernel: Buffer I/O error on device zram2, logical block 101820
Dec 19 14:21:30 ohab3 kernel: Buffer I/O error on device zram2, logical block 101821
Dec 19 14:21:30 ohab3 kernel: Buffer I/O error on device zram2, logical block 101822
Dec 19 14:21:30 ohab3 kernel: Buffer I/O error on device zram2, logical block 101823
Dec 19 14:21:30 ohab3 kernel: Buffer I/O error on device zram2, logical block 101824
Dec 19 14:21:30 ohab3 kernel: Buffer I/O error on device zram2, logical block 101825
Dec 19 14:21:30 ohab3 kernel: EXT4-fs warning (device zram2): ext4_end_bio:349: I/O error 10 writing to inode 266 starting block 7072)
Dec 19 14:21:35 ohab3 kernel: EXT4-fs warning (device zram2): ext4_end_bio:349: I/O error 10 writing to inode 77 starting block 48670)
Dec 19 14:21:35 ohab3 kernel: buffer_io_error: 1 callbacks suppressed
Dec 19 14:21:35 ohab3 kernel: Buffer I/O error on device zram2, logical block 48670
Dec 19 14:21:35 ohab3 kernel: EXT4-fs warning (device zram2): ext4_end_bio:349: I/O error 10 writing to inode 77 starting block 47296)
Dec 19 14:21:35 ohab3 kernel: Buffer I/O error on device zram2, logical block 47296
Dec 19 14:21:35 ohab3 kernel: EXT4-fs warning (device zram2): ext4_end_bio:349: I/O error 10 writing to inode 347 starting block 7114)
Dec 19 14:21:35 ohab3 kernel: Buffer I/O error on device zram2, logical block 7114
Dec 19 14:21:35 ohab3 kernel: EXT4-fs warning (device zram2): ext4_end_bio:349: I/O error 10 writing to inode 350 starting block 6609)
Dec 19 14:21:35 ohab3 kernel: Buffer I/O error on device zram2, logical block 6609
Dec 19 14:21:35 ohab3 kernel: EXT4-fs warning (device zram2): ext4_end_bio:349: I/O error 10 writing to inode 266 starting block 71682)
Dec 19 14:21:35 ohab3 kernel: Buffer I/O error on device zram2, logical block 71682
Dec 19 14:21:35 ohab3 kernel: EXT4-fs warning (device zram2): ext4_end_bio:349: I/O error 10 writing to inode 266 starting block 71684)
Dec 19 14:21:35 ohab3 kernel: Buffer I/O error on device zram2, logical block 71684
Dec 19 14:21:35 ohab3 kernel: EXT4-fs warning (device zram2): ext4_end_bio:349: I/O error 10 writing to inode 266 starting block 71687)
Dec 19 14:21:35 ohab3 kernel: Buffer I/O error on device zram2, logical block 71687
Dec 19 14:21:35 ohab3 kernel: EXT4-fs warning (device zram2): ext4_end_bio:349: I/O error 10 writing to inode 266 starting block 71689)
Dec 19 14:21:35 ohab3 kernel: Buffer I/O error on device zram2, logical block 71689
Dec 19 14:21:35 ohab3 kernel: EXT4-fs warning (device zram2): ext4_end_bio:349: I/O error 10 writing to inode 266 starting block 71695)
Dec 19 14:21:35 ohab3 kernel: Buffer I/O error on device zram2, logical block 71695
Dec 19 14:21:35 ohab3 kernel: EXT4-fs warning (device zram2): ext4_end_bio:349: I/O error 10 writing to inode 266 starting block 71701)
Dec 19 14:21:35 ohab3 kernel: Buffer I/O error on device zram2, logical block 71701
Dec 19 14:21:35 ohab3 kernel: Aborting journal on device zram2-8.
Dec 19 14:21:35 ohab3 kernel: Buffer I/O error on dev zram2, logical block 65536, lost sync page write
Dec 19 14:21:35 ohab3 kernel: JBD2: Error -5 detected when updating journal superblock for zram2-8.
Dec 19 14:21:35 ohab3 kernel: Buffer I/O error on dev zram2, logical block 0, lost sync page write
Dec 19 14:21:35 ohab3 kernel: EXT4-fs (zram2): I/O error while writing superblock
Dec 19 14:21:35 ohab3 kernel: EXT4-fs error (device zram2): ext4_journal_check_start:83: Detected aborted journal
Dec 19 14:21:35 ohab3 kernel: EXT4-fs (zram2): Remounting filesystem read-only
Dec 19 14:21:35 ohab3 kernel: EXT4-fs (zram2): ext4_writepages: jbd2_start: 3029 pages, ino 266; err -30
Dec 19 14:21:35 ohab3 rsyslogd[611]: file '7' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: file '7' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: file '7' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: file '7' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: file '7' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: file '7' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: file '7' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: file '7' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: file '7' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: file '7' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: file '7' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 14:21:35 ohab3 rsyslogd[611]: file '7' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]

there’s ~8000 more of lines like those, in the middle of some i found this:

Dec 19 14:21:40 ohab3 karaf[692]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : Unable to write to stream /var/log/openhab/events.log for appender EVENT
Dec 19 14:21:40 ohab3 karaf[692]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : An exception occurred processing Appender EVENT
Dec 19 14:21:43 ohab3 karaf[692]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : Unable to write to stream /var/log/openhab/events.log for appender EVENT
Dec 19 14:21:43 ohab3 karaf[692]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : An exception occurred processing Appender EVENT

and the last entries from sudo journalctl -b are:

Dec 19 16:05:01 ohab3 CRON[6746]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 19 16:05:01 ohab3 rsyslogd[611]: file '10' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 16:05:01 ohab3 rsyslogd[611]: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 16:05:01 ohab3 rsyslogd[611]: rsyslogd[internal_messages]: 516 messages lost due to rate-limiting
Dec 19 16:05:01 ohab3 CRON[6747]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Dec 19 16:05:01 ohab3 rsyslogd[611]: file '7' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 16:05:01 ohab3 rsyslogd[611]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 16:05:01 ohab3 rsyslogd[611]: file '7' write error: Read-only file system [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Dec 19 16:05:01 ohab3 rsyslogd[611]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]

no more entries since dec, 19. :man_shrugging:

i haven’t changed anything on my setup for month…

some more infos:

openhabian@ohab3:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.1G  3.1G  3.7G  45% /
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  1.6M  1.9G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda3       7.3G  193M  6.7G   3% /storage
/dev/mmcblk0p1  253M   49M  204M  20% /boot
/dev/zram1      324M  119M  180M  40% /opt/zram/zram1
overlay1        324M  119M  180M  40% /var/lib/openhab/persistence
/dev/zram2      420M  308M   81M  80% /opt/zram/zram2
overlay2        420M  308M   81M  80% /var/log
tmpfs           388M     0  388M   0% /run/user/1000

openhabian@ohab3:~ $ zramctl
NAME       ALGORITHM DISKSIZE   DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram2 zstd          450M 402.6M 68.5M  200M       4 /opt/zram/zram2
/dev/zram1 zstd          350M 172.1M   15M 90.1M       4 /opt/zram/zram1
/dev/zram0 lzo-rle       450M     4K   86B    4K       4 [SWAP]

backup@ohab3:~$ amcheck openhab-dir
amcheck: critical (fatal): Cannot create debug file "/var/log/amanda/server/amcheck.20220101103409.debug": Read-only file system
amcheck: Cannot create debug file "/var/log/amanda/server/amcheck.20220101103409.debug": Read-only file system

openhabian@ohab3:~ $ zramctl --version
zramctl from util-linux 2.33.1
2

I’d guess you hit the disk_size limit of /var/log at times.
Increase it in /etc/ztab and reboot/restart ZRAM.

3

thanks for the hint, i’ve changed disk size from 450 > 900, hopefully this will solve my problem!
is there a way to verify that the disk_size limit was the problem?

happy new year by the way!

1 Like
4

Hello Markus!
i have the same errors again now :frowning:

this is /etc/ztab

# swap  alg             mem_limit       disk_size       swap_priority   page-cluster    swappiness
swap    lzo-rle         200M            450M            75              0               80

# dir   alg             mem_limit       disk_size       target_dir                      bind_dir
dir     zstd            150M            350M            /var/lib/openhab/persistence    /persistence.bind

# log   alg             mem_limit       disk_size       target_dir              bind_dir                oldlog_dir
log     zstd            400M            9000M           /var/log                /log.bind

yes, i know disk_size for /var/log is not 900M (as i mentioned in my post above) but 9000M but that shouldn’t cause any problems, right?

the size of /var/log is

openhabian@ohab3:~ $ sudo du -hs /var/log
[sudo] password for openhabian:
286M    /var/log

and that is ~64% of the allocated mem_limit in ztab. or 3% of the allocated disk_size…

could anything else cause these problems or are you sure that /etc/ztab parameters are causing this?

5

This is no useful description. Don’t expect me to dig through old posts or do anything on a guessing basis. Using the defaults from a new openHABian installation should do.
Playing around is at your own risk so I won’t help there, sorry.

6

i was not “playing around” at all.
i changed the disk_size limit in /etc/zstab at your suggestion and that’s it.
when the error appeared the first time i hadn’t changed anything regarding zram…

i can’t make a fresh start everytime a problem appears… half a year ago i had to start over because of a bug in zram and a problem with amanda installation

don’t get me wrong: i don’t want anybody to guess anything but i just don’t know what information / logs would help. i posted everything i thought maybe could help when opening this thread but i still don’t know if anything contained the infos needed.

7

You directly addressed me after several months with a rather sloppy written request for help but I’m not anyone’s personal hotline - sorry.
From your description I can’t tell what your problem is.
But I know that there are no known problems with using the default values of an unmodified openHABian installation.

8

yes, i addressed you directly, but that’s because in the past you were the only one who was able to figure out / fix problems with zram (and you are the only person besides me in this thread). sorry for that!
i wasn’t aware my post would be considered sloppy, i just didn’t know what else i could/should add…
i don’t consider you or anyone in this great community as my personal hotline and i think i have expressed my gratitude for all the help many times.

neither do i :frowning:
that’s why i posted all the infos in the first place. had the same errors the second time…

what can i say. i have not tinkered with anythink except openHAB related things (things/items, rules, ui …). the problems i described (and you fixed) in the threads linked above were on unmodified openHABian installations and were not known before, so i don’t think it’s fair to assume that i broke the system.

anyway: i rebooted the system and i’ll open a new thread when the problem reappears.