openHAB - Filtering event logs

[bsarmin (Armin)]

Could you please explain how you did that?

[mrfrh (Martin)]

This is how my item definition looks like:
String RFLinkRaw "RFLink raw data [%s]" <text> { serial="/dev/ttyUSB0@57600,REGEX(([^\"\\r\\n]*[\\S]))" }

Serial binding allows the usage of a regex in the item definition. Took me quite a while to find out this is being treated as a string - therefore the backslash has to be repeated before r / n / S.

[petero (Peter)]

Just a quick note to all that have issues with multiple filters on the same appender. It just wont work. I have been at it for some time now.

For anyone interested in the log4j rabbit hole:
Here are some things that I learned, but still did not make any difference.

• When filters are added to the appender, they are sorted by name (the key): https://github.com/apache/log4j/blob/7be00eed88152dd011a619e8bae5a631235c3f4c/src/main/java/org/apache/log4j/PropertyConfigurator.java#L881
• When checking filters a return that is either Deny or Accept will cause the filter list to stop and not check other filters so multiple filters must use NEUTRAL: https://github.com/apache/log4j/blob/7be00eed88152dd011a619e8bae5a631235c3f4c/src/main/java/org/apache/log4j/AppenderSkeleton.java#L231
• onMismatch is spelled “onMismatch” as this is litterally translated to getOnMismatch which is the setter name (https://github.com/apache/log4j/blob/7be00eed88152dd011a619e8bae5a631235c3f4c/src/main/java/org/apache/log4j/config/PropertySetter.java#L47) not that is seems to matter anyway
• When using xml configuration for log4j the way to have multiple filters are completely different as that will use a CompositeFilter which I cannot find any way to set via the properties file. (https://logging.apache.org/log4j/2.0/manual/filters.html#CompositeFilter)

I have tested with the following filter rules:

## Danfoss Air Updates (because the key is cair this will be the first filter)
log4j2.appender.event.filter.cairupdate.type = RegexFilter
log4j2.appender.event.filter.cairupdate.regex = .*(Solar_PowerTotal).*
log4j2.appender.event.filter.cairupdate.onMatch = DENY
log4j2.appender.event.filter.cairupdate.onMismatch = NEUTRAL

## Frequest updates
log4j2.appender.event.filter.frequpdates.type = RegexFilter
log4j2.appender.event.filter.frequpdates.regex = .*(C_Solar_PowerWest).*
log4j2.appender.event.filter.frequpdates.onMatch = DENY
log4j2.appender.event.filter.frequpdates.onMismatch = NEUTRAL

and frequpdates are not tested (i.e. item updates for C_Solar_PowerWest shows up in the log C_Solar_PowerTotal does not)

[petero (Peter)]

Finally multiple event filters fixed.

log4j2.appender.event.filter.1.type = Filters

log4j2.appender.event.filter.1.a.type = RegexFilter
log4j2.appender.event.filter.1.a.regex = .*(C_Radon_(Level|Updated|Running)|C_Solar_(PowerTotal|PowerEast|PowerWest|TotalYield|DailyYield)|OFFLINE.*10\.13\.0\.70).*
log4j2.appender.event.filter.1.a.onMatch = DENY
log4j2.appender.event.filter.1.a.onMismatch = NEUTRAL

log4j2.appender.event.filter.1.b.type = RegexFilter
log4j2.appender.event.filter.1.b.regex = .*(C_Air_).*
log4j2.appender.event.filter.1.b.onMatch = DENY
log4j2.appender.event.filter.1.b.onMismatch = NEUTRAL

The formal documentation for adding filters is located here: http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PropertyConfigurator.html#method_detail

And it is incorrect!

The correct way is to create a filters attribute and then add new filters under that. The above sample will work, but it will generate an error in the console log file (systemctl status openhab2) that can be ignored

org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : Filters contains invalid attributes "onMatch", "onMismatch" Ignored FQCN: org.apache.logging.log4j.spi.AbstractLogger

I am on the latest milestone build M5 (this changes with the log4j version)

[Schrott.Micha (Michael)]

Hi @jpwise

I just updated to OH 2.5.1 and have seen that I get this error in OH status

systemctrl status openhab2.service

Jan 28 21:48:34 openHABianPi karaf[4032]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : RegexFilter contains an invalid element or attribute "onMismatch" Ignored FQCN:

according to apache.org the last line of the filter should be

log4j2.appender.event.filter.<your-filter-name>.onMismatch = ACCEPT

with lower case “m”. With this the warning is gone.

[rpwong (Russ)]

I now have multiple filters working in OH 2.5.1.2.

Per @petero’s last post, you have to define the filter attribute, and then add filters to it. I defined “Stuff” and then filled it with filter sets for general, astro, ups, wifi, and zwave.

Note that the onMismatch line in each filter set must be NEUTRAL. If it’s set to ACCEPT, the filters will stop evaluating at that point. This stackoverflow link explains it better than I can.

@Schrott.Micha I tried changing the last line to ACCEPT, and my filter immediately stopped working. Going back to NEUTRAL fixed it. I can live with the error in the console log file, which I think is a bug in the version of Log4j2 used in openHAB (see this link).

Honestly, there’s no real benefit to this unless your regex string is extremely long and hard to manage…but it’s satisfying if you like things to be grouped together.

# custom filters for events.log
log4j2.appender.event.filter.stuff.type = Filters

log4j2.appender.event.filter.stuff.general.type = RegexFilter
log4j2.appender.event.filter.stuff.general.regex = .*(has been updated|received command|predicted to|through).*
log4j2.appender.event.filter.stuff.general.onMatch = DENY
log4j2.appender.event.filter.stuff.general.onMismatch = NEUTRAL

log4j2.appender.event.filter.stuff.astro.type = RegexFilter
log4j2.appender.event.filter.stuff.astro.regex = .*(morningNight|astroDawn|astroDusk|nauticDawn|nauticDusk|civilDawn|civilDusk|daylight|noon).*
log4j2.appender.event.filter.stuff.astro.onMatch = DENY
log4j2.appender.event.filter.stuff.astro.onMismatch = NEUTRAL

log4j2.appender.event.filter.stuff.ups.type = RegexFilter
log4j2.appender.event.filter.stuff.ups.regex = .*(UPS_Output_Voltage changed|UPS_Runtime changed|UPS_Runtime_Transformed changed).*
log4j2.appender.event.filter.stuff.ups.onMatch = DENY
log4j2.appender.event.filter.stuff.ups.onMismatch = NEUTRAL

log4j2.appender.event.filter.stuff.wifi.type = RegexFilter
log4j2.appender.event.filter.stuff.wifi.regex = .*(Wemo changed|harmonyhub:hub|LED_|_led changed|_rssi).*
log4j2.appender.event.filter.stuff.wifi.onMatch = DENY
log4j2.appender.event.filter.stuff.wifi.onMismatch = NEUTRAL

log4j2.appender.event.filter.stuff.zwave.type = RegexFilter
log4j2.appender.event.filter.stuff.zwave.regex = .*(Alarm_Raw changed|Lock_Battery changed|Zooz changed).*
log4j2.appender.event.filter.stuff.zwave.onMatch = DENY
log4j2.appender.event.filter.stuff.zwave.onMismatch = NEUTRAL

[Kim_Andersen]

I would love to have an option to turn on/off a filter for the log… As time goes, my tail log is so filled with stuff going on all the time. And sometimes I need to look for something special, it would be great to turn off (temporary) stuff I dont need to look at, (cause I know it´s working fine).

I wonder if this kind of filter would do, and then create a proxy switch to turn on/off several filters? I suspect it´ll have to be done using exec/scripts and commandline to Karaf consol?

[5iver (Scott Rushworth)]

Here you go, Kim…

Change logging level through a UI and access Karaf in a rule

Here is an example of how logging levels can be changed without having to manually enter commands into the Karaf console or editing org.ops4j.pax.logging.cfg. I set this up and it is working in my environment, but the ssh command will need to be modified to work for someone running openHabian or if OH is being run on Windows. The key based authentication is the trickiest bit to setup. Be sure to configure Karaf with the public key and then ssh in manually first to accept the host’s public key. W…

[Kim_Andersen]

Thanks Scott… Seems to be exactly what I´m looking for… Will give it a try soon as my tail log is driven me nuts sometime

[rpwong (Russ)]

@Kim_Andersen

I would love to have an option to turn on/off a filter for the log… As time goes, my tail log is so filled with stuff going on all the time. And sometimes I need to look for something special, it would be great to turn off (temporary) stuff I dont need to look at, (cause I know it´s working fine).

This is a good point for using filter sets. If you want to see the log entries for something specific, it’s easier to find and comment out the filter specific to the device type than editing those entries out of a long regex list and adding them back later.

[Kim_Andersen]

Exactly my point.
Right now I have aprox 15-20 rows of line every 5 seconds. Things are going crazy sometimes

[dj1]

I was wondering if anyone know how to filter the log:tail … i tried the console but it seems to have no effect

[5iver (Scott Rushworth)]

Pipe the output to grep…

openHAB - Filtering event logs

did you try to grep stuff? Not sure if you can use a full RegexFilter on the console. Maybe if you modify the log4j2 config for the console settings? e.g. log:display |grep … or log:tail |grep … https://karaf.apache.org/manual/latest-2.x/commands/shell-grep.html

Is that what you’re looking for?

[dj1]

yea but i was hoping for a permanent solution… i tried pretty much all of the log4j2 settings but nada… there must be a way though since we can exclude specific bindings and levels…

[5iver (Scott Rushworth)]

Paste an example of what you are seeing in the log that you’d like to filer out.

[dj1]

I want to filter anything with amazonechocontrol:smartHomeDevice or other regex compatible lines from log:tail ( i have no issue with log files filtering but that is not what i specifically want )

‘amazonechocontrol:smartHomeDevice:0a4c3487:517c0f67-cdd7-45c7-8dac-3cdcd8114a1b’

and tried this in the org.ops4j.pax.logging.cfg but no effect in karaf ssh…

    log4j2.appender.console.filter.1.type = Filters
    log4j2.appender.console.filter.1.a.type = RegexFilter
    log4j2.appender.console.filter.1.a.regex = .*amazonechocontrol.*
    log4j2.appender.console.filter.1.a.onMatch = DENY
    log4j2.appender.console.filter.1.a.onMismatch = ACCEPT

[grhosso (Juan)]

I quick question.

Is it possible to check out events that have already been filtered? I had a Item filtered and I would like to check out when changed the status yesterday and I do not know if it is stored in another file into the system.

Thanks.

[rlkoshak (Rich Koshak)]

When you filter the logs like this, that log statement is lost. That’s why I’m very much against filtering logs like this except in extreme circumstances. The whole purpose of logs is to preserve information like this so you can go back and see information like this. It’s better to let the log statements be written and use tools available to you to filter the logs based on what’s important to you at that time. Everything continues to be written but you can narrow down what you are looking at to just those lines that are relevant to what you are working on. And then, when something happens and you have to go back and do a forensic analysis, all the logs are there.

You cannot know ahead of time what logs will be relevant so let everything log as normal and search/filter the logs as needed.

[grhosso (Juan)]

yes, you are right.

I will try to minimize the filter.
Thanks.

[jpwise]

@rlkoshak just chucked a link to your thread into the main post. When i wrote this up the intent was to keep stuff I didn’t want ‘out’ of the log, but I can see some users won’t always know the difference between filtering something to prevent it being logged, and filtering a log view to only show what they want. Thx.