OH3 logging files space overwrite

Setup, Configuration and Use Beginners
1

General question about log files and space management.
Hi All,
I have managed to get Openhabian 3 running on a Raspbarry Pi 3 with a 16GB SD card. I used the auto installer so it is a vanilla build, with bindings for Tapo (power sockets), IPCameras (Generic-ONVIF and FOSCAM) and TPlink for KASA sockets.
Everything is running nicely, with a few rules triggering how they should. On the overview screen I have lots of numbers and status icons, and clicking on the power consumption icons show the default graphs.
My question is… the graphs are building up a set of data per day (logged somewhere) and the openhab log files are building up quickly - many lines a minute. Eventually this will use up all the space available for logging.
So here is the actual question: DOES openhab manage these logfiles and overwrite periodically or when space is short - or do I have to tell it to do so, or do I just need to setup a cron job in linux to delete old files?

All advice/instruction will be gratefully welcomed.

MBD

2

There are different kind of managing the log files.
a) logfile rotation controlled by logrotate ( configuration in /etc/logrotate.d )
b) openhab log files, event log files in /var/log/openhab/ are managed by log4j2.xml

log4j2.xml manages the content ( what is to be logged ) as well the rotation and how many files to keep.

You should not use logrotate for files that are managed by log4j2.xml.
For both you should be able to find documentation on the internet.

3

For the charting it uses Persistence which saves Item states into a database. By default it uses rrd4j which is a fixed size database. It does not grow over time (though space required does increase with the number of Items). With the default settings you can assume 530 KB per Item to store Item states for charting and use in rules and such.

As for the log statements, many lines a minute is nothing. Let’s break it down. Let’s assume on average 150 characters. At 8 bits per character that 150 bytes per line. Let’s go crazy and say there is one line per second. So a full day’s log file will be around:"

150 bytes * 60 secs per min * 60 mins per hour * 24 hours per day = 12,960,000 bytes per day

That’s around 12 MB per day. You have a 16GB SD card but let’s say only 12 GB is free. You have enough space to save logs for 1000 days (a little over 2 3/4 years).

So this is not an immediate concern. But of course it would be foolish to fill up a 16 GB file with logs. And, as @Wolfgang_S indicates, there are many mechanisms to deal with that. OH uses Log4J2 which is configured in $OH_USERDATA/etc/log4j2.xml. The important part are the Appenders.

        <Appenders>
                <!-- Console appender not used by default (see Root logger AppenderRefs) -->
                <Console name="STDOUT">
                        <PatternLayout pattern="%d{HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
                </Console>

                <!-- Rolling file appender -->
                <RollingFile fileName="${sys:openhab.logdir}/openhab.log" filePattern="${sys:openhab.logdir}/openhab.log.%i.gz" name="LOGFILE">
                        <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
                        <Policies>
                                <OnStartupTriggeringPolicy/>
                                <SizeBasedTriggeringPolicy size="16 MB"/>
                        </Policies>
                        <DefaultRolloverStrategy max="7"/>
                </RollingFile>

                <!-- Event log appender -->
                <RollingRandomAccessFile fileName="${sys:openhab.logdir}/events.log" filePattern="${sys:openhab.logdir}/events.log.%i.gz" name="EVENT">
                        <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
                        <Policies>
                                <OnStartupTriggeringPolicy/>
                                <SizeBasedTriggeringPolicy size="16 MB"/>
                        </Policies>
                        <DefaultRolloverStrategy max="7"/>
                </RollingRandomAccessFile>

                <!-- Audit file appender -->
                <RollingRandomAccessFile fileName="${sys:openhab.logdir}/audit.log" filePattern="${sys:openhab.logdir}/audit.log.%i.gz" name="AUDIT">
                        <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
                        <Policies>
                                <OnStartupTriggeringPolicy/>
                                <SizeBasedTriggeringPolicy size="8 MB"/>
                        </Policies>
                        <DefaultRolloverStrategy max="7"/>
                </RollingRandomAccessFile>

                <!-- OSGi appender -->
                <PaxOsgi filter="*" name="OSGI"/>
        </Appenders>

The “Rolling” part means that the files fill up to a given size or condition based on policy. You can see here that it will roll over to a new log file on OH startup and when the log file reaches 16 MB in size. At that point it copies the current file to a backup and zips it up to save space (logs tend to compress very well with around 10x compression (e.g. a 1 MB file will compress to 1 KB)).

You will also see a roll over strategy with a max of 7 meaning it will only save the seven most recent backup files.

So, based on the compression ratios I’m seeing, your logs should never grow beyond 20-25 MB. But even if there were no compression you’d be looking at a max of 256 MB space for OH logs.

4

that would be indeed a very, very ( 10e3 ) well around 10x compression factor. :slight_smile: