[Solution] Logging ERROR and WARN messages into seperate log file and multiple RegexFilters

I want all ERROR and WARN log messages to be stored in a separate file ${OPENHAB_LOGDIR}/critical.log additionally to openhab.log and event.log (those will continue to log these ERROR and WARN messages, too).
In general, my custom log file works and is filled with standard log messages

		<!-- Custom log appender for critical messages -->
		<RollingFile fileName="${sys:openhab.logdir}/critical.log" filePattern="${sys:openhab.logdir}/critical.log.%i.gz" name="CRITICAL">
			<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
			<Policies>
				<OnStartupTriggeringPolicy/>
				<SizeBasedTriggeringPolicy size="16 MB"/>
			</Policies>
			<DefaultRolloverStrategy max="7"/>
		</RollingFile>

I have read the docs and a LOT of good posts on configuring log4j2.xml but I am struggeling with the “logger” section. I can’t find out what and where to do the changes.
Do I need to list all potential log sources and create a reference to my custom appender like this:

<Logger level="ERROR" name="openhab.event.AddonEvent"/>
...
<Logger additivity="false" level="ERROR" name="openhab.event">
	<AppenderRef ref="CRITICAL"/>
</Logger>

or do I need to add a regex expression to the appender? However, if I look into my critical.log, not all messages are collected there.
It would be great if someone can give me a hint.

Furthermore: Can I add more “log sources” myself? How can I get their names?

I’m not 100% certain you can do this. At least not easily.

The loggers are identified by name and the name of the logger is part or all of the fully qualified name of the Class. The names are hierarchical. For example, a logger named “org.openhab” will adjust the settings for every Class whose fully qualified name starts with “org.openhab”.

And there is only one logger with a given name. You have have two <Logger> elements addressing the same logger name. The second one will just overwrite the first one.

I think one way to achieve what you are after would be to add a marker or regex filter to your appender and then add the appender to the root level logger.

However, this smells of an XY Problem. Why do you want these logs copied to a separate file?

Thanks Rich,
as of now it is working nicely when adding a filter to the appender:

<ThresholdFilter level="WARN" onMatch="ACCEPT" onMismatch="DENY"/>

and a reference to all required Loggers.
I am currently testing how to combine this filter with RegexFilter which are supposed to filter out warn/error messages I am not interested in (like: cannot detect master sound level…)
The reason for this separate log file is that I want to feed a MainUI widget which shows the latest warn/error messages.

Edit: I alreadyctried a shell script and logreader binding but I discarded these solutions.

Why not use the Log Reader binding?

:slight_smile: I edited my previous message that I already tested this one.
No, not a good solution, in my opinion.
When logreader logs a warn message then it logs, that it found a warn message which even more clutters my log as this appears also in red colour
The main reason is, I then need to write a rule which gets triggered by logreader and appends the new log message to the items value.
In fact, I am recreating a logic to get a string containing just warn/error messages which provides a logger on a silver plate.
Furthermore, by changing the pattern to

<PatternLayout pattern='["%d{yyyy-MM-dd HH:mm:ss.SSS}","%p","%c{3}",{"%enc{%m}{JSON}"},"%ex{full}"],%n'/>

that gives me a json string (if I added [ ]) so that it can be directly provided to an OH-Repeater

It’s very easy to turn off the logger for a specific binding. If that’s your only problem it’s easily solved. Add a Logger for the logreader binding and set the level to OFF. That blocks all logging from that binding.

It’s not a very complicated rule though. It’s at least an option for others who might be looking to do something similar.

Another approach can be to set up another instance of Frontail configured to only show warnings and errors and use a webview to show that.

Just to answer my own question and to provide a solution to some requests here how to add multiple RegexFilters to the appender:

You can even combine them with other filter types like the following:

<Appenders>
	<RollingFile fileName="${sys:openhab.logdir}/critical.log" filePattern="${sys:openhab.logdir}/critical.log.%i.gz" name="CRITICAL">
		<PatternLayout pattern='["%d{yyyy-MM-dd HH:mm:ss.SSS}","%p","%c{3}","%enc{%m}{JSON}"%ex{NONE}],%n'/>
		<Filters>
			<RegexFilter regex=".*Attempting to send a state update of an item which doesn't exist.*" useRawMsg="false" onMatch="DENY" onMismatch="NEUTRAL"/>
			<RegexFilter regex=".*Cannot determine master volume level - assuming 100.*" useRawMsg="false" onMatch="DENY" onMismatch="NEUTRAL"/>
			<RegexFilter regex=".*whatever.*" useRawMsg="false" onMatch="DENY" onMismatch="NEUTRAL"/>
			<ThresholdFilter level="WARN" onMatch="ACCEPT" onMismatch="DENY"/>
		</Filters>
		<Policies>
			<OnStartupTriggeringPolicy/>
			<SizeBasedTriggeringPolicy size="1 MB"/>
		</Policies>
		<DefaultRolloverStrategy max="7"/>
	</RollingFile>
	...
<Appenders/>

Important is the order of the filter and that either onMatch or onUnmatch is set to NEUTRAL
Furthermore this log file is quick&dirty JSON-stylish in case you might want to load it into a widget.
Thanks again Rich - to me, this solution is more elegant.

EDIT:
For true JSON-based log file simply replace
<PatternLayout .../> with

<JsonTemplateLayout eventTemplateUri="classpath:EcsLayout.json"/>

Hi @Oliver2 ,
I’m trying something similar but simpler (I want to isolate all ERROR log entries to start with into their own file). I’ve altered my /var/lib/openhab/etc/log4j2.xml file with the following new RollingFile appender but it’s not working. Nothing is written to critical.log, although the file is created:-

    <Appenders>
            <!-- Console appender not used by default (see Root logger AppenderRefs) -->
            <Console name="STDOUT">
                    <PatternLayout pattern="%d{HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
            </Console>

            <!-- Rolling file appender -->
            <RollingFile fileName="${sys:openhab.logdir}/openhab.log" filePattern="${sys:openhab.logdir}/openhab.log.%i.gz" name="LOGFILE">
                    <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
                    <Policies>
                            <OnStartupTriggeringPolicy/>
                            <SizeBasedTriggeringPolicy size="16 MB"/>
                    </Policies>
                    <DefaultRolloverStrategy max="7"/>
            </RollingFile>

            <!-- Event log appender -->
            <RollingRandomAccessFile fileName="${sys:openhab.logdir}/events.log" filePattern="${sys:openhab.logdir}/events.log.%i.gz" name="EVENT">
                    <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
                    <Policies>
                            <OnStartupTriggeringPolicy/>
                            <SizeBasedTriggeringPolicy size="16 MB"/>
                    </Policies>
                    <DefaultRolloverStrategy max="7"/>
            </RollingRandomAccessFile>

            <!-- Audit file appender -->
            <RollingRandomAccessFile fileName="${sys:openhab.logdir}/audit.log" filePattern="${sys:openhab.logdir}/audit.log.%i.gz" name="AUDIT">
                    <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
                    <Policies>
                            <OnStartupTriggeringPolicy/>
                            <SizeBasedTriggeringPolicy size="8 MB"/>
                    </Policies>
                    <DefaultRolloverStrategy max="7"/>
            </RollingRandomAccessFile>

            <!-- NEW Critical log file -->
            <RollingFile fileName="${sys:openhab.logdir}/critical.log" filePattern="${sys:openhab.logdir}/critical.log.%i.gz" name="CRITICAL">
                    <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n"/>
                    <Filters>
                            <ThresholdFilter level="ERROR" onMatch="ACCEPT"/>
                    </Filters>
                    <Policies>
                            <OnStartupTriggeringPolicy/>
                            <SizeBasedTriggeringPolicy size="16 MB"/>
                    </Policies>
                    <DefaultRolloverStrategy max="7"/>
            </RollingFile>
            <!-- end of new critial log file>

             <!-- OSGi appender -->
            <PaxOsgi filter="*" name="OSGI"/>
    </Appenders>

Any thoughts on why this isn’t working (logs with ERROR level are happening and being written to the normal log file)?

Thanks
Steve

You have defined a new (rolling) logfile - nothing else.
Now you need to add an <AppenderRef> reference to each <Logger> tag like this:

		<Root level="WARN">
			<AppenderRef ref="LOGFILE"/>
			<AppenderRef ref="OSGI"/>
			<AppenderRef ref="CRITICAL"/>
		</Root>

		<!-- Karaf Shell logger -->
		<Logger level="OFF" name="org.apache.karaf.shell.support">
			<AppenderRef ref="STDOUT"/>
			<AppenderRef ref="CRITICAL"/>
		</Logger>

		<!-- Security audit logger -->
		<Logger additivity="false" level="INFO" name="org.apache.karaf.jaas.modules.audit">
			<AppenderRef ref="AUDIT"/>
			<AppenderRef ref="CRITICAL"/>
		</Logger>
...

Note: It takes 1-2 minutes until log4j2s.xml is loaded and changes become effective.

1 Like

Thank you @Oliver2 - perfect