openHAB - Filtering event logs

[rpwong (Russ)]

The only other thing I can think of is to go through your cfg file and make sure there aren’t any other lines that are interfering with this. I don’t know how or why that would be the case, though.

[cyb3rko (Niko)]

I used this example for many weeks, but at some point it stopped working from one day to another.
What did I do wrong? Here´s the code:

[ome.event.ItemCommandEvent] - Item ‘wecker_switch’ received command ON
[vent.ItemStateChangedEvent] - temperatur_niko changed from 23.20 to 23.30

.# Custom
log4j2.appender.event.filter.schiss_entfernung_events.type = RegexFilter
log4j2.appender.event.filter.schiss_entfernung_events.regex = .*(received command|temperatur_niko changed from|)
log4j2.appender.event.filter.schiss_entfernung_events.onMatch = DENY
log4j2.appender.event.filter.schiss_entfernung_events.onMisMatch = ACCEPT

[fex (Felix)]

To help others to avoid a similar mistake: these filters help you to match against the log message text - and not with other sections of the log line.

Example:

2019-10-06 17:52:07.131 [vent.ItemStateChangedEvent] - Consumption_BR_Computer changed from 52.920 to 43.120

To filter this line you can match any part of the string ‘Consumption_BR_Computer changed from 52.920 to 43.120’ but if you try to match anything from ‘2019-10-06 17:52:07.131 [vent.ItemStateChangedEvent]’ your filter will not work.

I tried to match against ‘ItemStateChangedEvent’ and it took me a while to find my mistake…

[mrfrh (Martin)]

For me, the filter shows a strange behavior: It works for some of the keywords (TGraph, OpenhabSystem_), but not for “RFLinkRaw”.

My config:
log4j2.appender.event.filter.uselessjunk.type = RegexFilter log4j2.appender.event.filter.uselessjunk.regex = .*(RFLinkRaw|TGraph|OpenhabSystem_).* log4j2.appender.event.filter.uselessjunk.onMatch = DENY log4j2.appender.event.filter.uselessjunk.onMisMatch = ACCEPT

Example log messages which are not being filtered:

2019-10-12 12:00:00.170 [vent.ItemStateChangedEvent] - RFLinkRaw changed from 20;0E;Debug;RTS P1;aaf0f0333f3fcc5566aa;
20;0F;RTS;ID=23f3fcc;SWITCH=01;CMD=STOP;
to 20;10;PONG;

2019-10-12 12:00:23.356 [vent.ItemStateChangedEvent] - RFLinkRaw changed from 20;11;Debug;RTS P1;aaf0f0333f3fcc5566aa;
20;12;RTS;ID=23f3fcc;SWITCH=01;CMD=STOP;
to 20;13;Alecto V5;ID=00d0;TEMP=00d2;RAIN=0000;BAT=OK;

Maybe this is due to the regex filter not working correctly with the new line (“\n”) in the log message?
Can anybody help me?

Edit: Solved by adding a regex filter to the serial binding item RFLinkRaw (to filer out carriage return and new lines).

[bsarmin (Armin)]

Could you please explain how you did that?

[mrfrh (Martin)]

This is how my item definition looks like:
String RFLinkRaw "RFLink raw data [%s]" <text> { serial="/dev/ttyUSB0@57600,REGEX(([^\"\\r\\n]*[\\S]))" }

Serial binding allows the usage of a regex in the item definition. Took me quite a while to find out this is being treated as a string - therefore the backslash has to be repeated before r / n / S.

[petero (Peter)]

Just a quick note to all that have issues with multiple filters on the same appender. It just wont work. I have been at it for some time now.

For anyone interested in the log4j rabbit hole:
Here are some things that I learned, but still did not make any difference.

• When filters are added to the appender, they are sorted by name (the key): https://github.com/apache/log4j/blob/7be00eed88152dd011a619e8bae5a631235c3f4c/src/main/java/org/apache/log4j/PropertyConfigurator.java#L881
• When checking filters a return that is either Deny or Accept will cause the filter list to stop and not check other filters so multiple filters must use NEUTRAL: https://github.com/apache/log4j/blob/7be00eed88152dd011a619e8bae5a631235c3f4c/src/main/java/org/apache/log4j/AppenderSkeleton.java#L231
• onMismatch is spelled “onMismatch” as this is litterally translated to getOnMismatch which is the setter name (https://github.com/apache/log4j/blob/7be00eed88152dd011a619e8bae5a631235c3f4c/src/main/java/org/apache/log4j/config/PropertySetter.java#L47) not that is seems to matter anyway
• When using xml configuration for log4j the way to have multiple filters are completely different as that will use a CompositeFilter which I cannot find any way to set via the properties file. (https://logging.apache.org/log4j/2.0/manual/filters.html#CompositeFilter)

I have tested with the following filter rules:

## Danfoss Air Updates (because the key is cair this will be the first filter)
log4j2.appender.event.filter.cairupdate.type = RegexFilter
log4j2.appender.event.filter.cairupdate.regex = .*(Solar_PowerTotal).*
log4j2.appender.event.filter.cairupdate.onMatch = DENY
log4j2.appender.event.filter.cairupdate.onMismatch = NEUTRAL

## Frequest updates
log4j2.appender.event.filter.frequpdates.type = RegexFilter
log4j2.appender.event.filter.frequpdates.regex = .*(C_Solar_PowerWest).*
log4j2.appender.event.filter.frequpdates.onMatch = DENY
log4j2.appender.event.filter.frequpdates.onMismatch = NEUTRAL

and frequpdates are not tested (i.e. item updates for C_Solar_PowerWest shows up in the log C_Solar_PowerTotal does not)

[petero (Peter)]

Finally multiple event filters fixed.

log4j2.appender.event.filter.1.type = Filters

log4j2.appender.event.filter.1.a.type = RegexFilter
log4j2.appender.event.filter.1.a.regex = .*(C_Radon_(Level|Updated|Running)|C_Solar_(PowerTotal|PowerEast|PowerWest|TotalYield|DailyYield)|OFFLINE.*10\.13\.0\.70).*
log4j2.appender.event.filter.1.a.onMatch = DENY
log4j2.appender.event.filter.1.a.onMismatch = NEUTRAL

log4j2.appender.event.filter.1.b.type = RegexFilter
log4j2.appender.event.filter.1.b.regex = .*(C_Air_).*
log4j2.appender.event.filter.1.b.onMatch = DENY
log4j2.appender.event.filter.1.b.onMismatch = NEUTRAL

The formal documentation for adding filters is located here: http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PropertyConfigurator.html#method_detail

And it is incorrect!

The correct way is to create a filters attribute and then add new filters under that. The above sample will work, but it will generate an error in the console log file (systemctl status openhab2) that can be ignored

org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : Filters contains invalid attributes "onMatch", "onMismatch" Ignored FQCN: org.apache.logging.log4j.spi.AbstractLogger

I am on the latest milestone build M5 (this changes with the log4j version)

[Schrott.Micha (Michael)]

Hi @jpwise

I just updated to OH 2.5.1 and have seen that I get this error in OH status

systemctrl status openhab2.service

Jan 28 21:48:34 openHABianPi karaf[4032]: org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : RegexFilter contains an invalid element or attribute "onMismatch" Ignored FQCN:

according to apache.org the last line of the filter should be

log4j2.appender.event.filter.<your-filter-name>.onMismatch = ACCEPT

with lower case “m”. With this the warning is gone.

[rpwong (Russ)]

I now have multiple filters working in OH 2.5.1.2.

Per @petero’s last post, you have to define the filter attribute, and then add filters to it. I defined “Stuff” and then filled it with filter sets for general, astro, ups, wifi, and zwave.

Note that the onMismatch line in each filter set must be NEUTRAL. If it’s set to ACCEPT, the filters will stop evaluating at that point. This stackoverflow link explains it better than I can.

@Schrott.Micha I tried changing the last line to ACCEPT, and my filter immediately stopped working. Going back to NEUTRAL fixed it. I can live with the error in the console log file, which I think is a bug in the version of Log4j2 used in openHAB (see this link).

Honestly, there’s no real benefit to this unless your regex string is extremely long and hard to manage…but it’s satisfying if you like things to be grouped together.

# custom filters for events.log
log4j2.appender.event.filter.stuff.type = Filters

log4j2.appender.event.filter.stuff.general.type = RegexFilter
log4j2.appender.event.filter.stuff.general.regex = .*(has been updated|received command|predicted to|through).*
log4j2.appender.event.filter.stuff.general.onMatch = DENY
log4j2.appender.event.filter.stuff.general.onMismatch = NEUTRAL

log4j2.appender.event.filter.stuff.astro.type = RegexFilter
log4j2.appender.event.filter.stuff.astro.regex = .*(morningNight|astroDawn|astroDusk|nauticDawn|nauticDusk|civilDawn|civilDusk|daylight|noon).*
log4j2.appender.event.filter.stuff.astro.onMatch = DENY
log4j2.appender.event.filter.stuff.astro.onMismatch = NEUTRAL

log4j2.appender.event.filter.stuff.ups.type = RegexFilter
log4j2.appender.event.filter.stuff.ups.regex = .*(UPS_Output_Voltage changed|UPS_Runtime changed|UPS_Runtime_Transformed changed).*
log4j2.appender.event.filter.stuff.ups.onMatch = DENY
log4j2.appender.event.filter.stuff.ups.onMismatch = NEUTRAL

log4j2.appender.event.filter.stuff.wifi.type = RegexFilter
log4j2.appender.event.filter.stuff.wifi.regex = .*(Wemo changed|harmonyhub:hub|LED_|_led changed|_rssi).*
log4j2.appender.event.filter.stuff.wifi.onMatch = DENY
log4j2.appender.event.filter.stuff.wifi.onMismatch = NEUTRAL

log4j2.appender.event.filter.stuff.zwave.type = RegexFilter
log4j2.appender.event.filter.stuff.zwave.regex = .*(Alarm_Raw changed|Lock_Battery changed|Zooz changed).*
log4j2.appender.event.filter.stuff.zwave.onMatch = DENY
log4j2.appender.event.filter.stuff.zwave.onMismatch = NEUTRAL

[Kim_Andersen]

I would love to have an option to turn on/off a filter for the log… As time goes, my tail log is so filled with stuff going on all the time. And sometimes I need to look for something special, it would be great to turn off (temporary) stuff I dont need to look at, (cause I know it´s working fine).

I wonder if this kind of filter would do, and then create a proxy switch to turn on/off several filters? I suspect it´ll have to be done using exec/scripts and commandline to Karaf consol?

[5iver (Scott Rushworth)]

Here you go, Kim…

Change logging level through a UI and access Karaf in a rule

Here is an example of how logging levels can be changed without having to manually enter commands into the Karaf console or editing org.ops4j.pax.logging.cfg. I set this up and it is working in my environment, but the ssh command will need to be modified to work for someone running openHabian or if OH is being run on Windows. The key based authentication is the trickiest bit to setup. Be sure to configure Karaf with the public key and then ssh in manually first to accept the host’s public key. W…

[Kim_Andersen]

Thanks Scott… Seems to be exactly what I´m looking for… Will give it a try soon as my tail log is driven me nuts sometime

[rpwong (Russ)]

@Kim_Andersen

I would love to have an option to turn on/off a filter for the log… As time goes, my tail log is so filled with stuff going on all the time. And sometimes I need to look for something special, it would be great to turn off (temporary) stuff I dont need to look at, (cause I know it´s working fine).

This is a good point for using filter sets. If you want to see the log entries for something specific, it’s easier to find and comment out the filter specific to the device type than editing those entries out of a long regex list and adding them back later.

[Kim_Andersen]

Exactly my point.
Right now I have aprox 15-20 rows of line every 5 seconds. Things are going crazy sometimes

[dj1]

I was wondering if anyone know how to filter the log:tail … i tried the console but it seems to have no effect

[5iver (Scott Rushworth)]

Pipe the output to grep…

openHAB - Filtering event logs

did you try to grep stuff? Not sure if you can use a full RegexFilter on the console. Maybe if you modify the log4j2 config for the console settings? e.g. log:display |grep … or log:tail |grep … https://karaf.apache.org/manual/latest-2.x/commands/shell-grep.html

Is that what you’re looking for?

[dj1]

yea but i was hoping for a permanent solution… i tried pretty much all of the log4j2 settings but nada… there must be a way though since we can exclude specific bindings and levels…

[5iver (Scott Rushworth)]

Paste an example of what you are seeing in the log that you’d like to filer out.

[dj1]

I want to filter anything with amazonechocontrol:smartHomeDevice or other regex compatible lines from log:tail ( i have no issue with log files filtering but that is not what i specifically want )

‘amazonechocontrol:smartHomeDevice:0a4c3487:517c0f67-cdd7-45c7-8dac-3cdcd8114a1b’

and tried this in the org.ops4j.pax.logging.cfg but no effect in karaf ssh…

    log4j2.appender.console.filter.1.type = Filters
    log4j2.appender.console.filter.1.a.type = RegexFilter
    log4j2.appender.console.filter.1.a.regex = .*amazonechocontrol.*
    log4j2.appender.console.filter.1.a.onMatch = DENY
    log4j2.appender.console.filter.1.a.onMismatch = ACCEPT