Again: Rotating Log

Setup, Configuration and Use Runtime
Tags: #<Tag:0x00007efecbeb2870>
#1

Hi guys,

sorry looks like this question has been asked multiple times but I have never seen any real solution for it.
Running OH2 on Raspbian.
I read using the system-build-in logrotate should not be used I tried to understand the logging configuration but did not get very far…

So the configuration file is /var/lib/openhab2/etc/org.ops4j.pax.logging.cfg
Mine has the followign line in it and as far as I got it herre, it will be rotated once it reaches 16MB

log4j2.appender.out.type = RollingRandomAccessFile
log4j2.appender.out.name = LOGFILE
log4j2.appender.out.fileName = ${openhab.logdir}/openhab.log
log4j2.appender.out.filePattern = ${openhab.logdir}/openhab.log.%i
log4j2.appender.out.immediateFlush = true
log4j2.appender.out.append = true
log4j2.appender.out.layout.type = PatternLayout
log4j2.appender.out.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n
log4j2.appender.out.policies.type = Policies
log4j2.appender.out.policies.size.type = SizeBasedTriggeringPolicy
log4j2.appender.out.policies.size.size = 16MB

What I want to archive is to limit the number of “old” logfiles to 7 and additionally compress them (16MB are compressed only ~200K.

So how do I get this? I read the documentation to log4j2 but as I am not a Java coder I really have issues in understanding all the dependecies there and match them to the above file sniplet…

So anyone here able to tell what I should add/modify in my above configuration to have them to 7 files with compression?

/KNEBB

Modbus stops without any errors
#2

I believe something like:

log4j2.appender.out.strategy.type=DefaultRolloverStrategy
log4j2.appender.out.strategy.max = 7

will keep only 7 previous versions, and to compress using gzip simply change the filePattern option to read:

log4j2.appender.out.filePattern = ${openhab.logdir}/openhab.log.%i.gz
1 Like
#3

https://howtodoinjava.com/log4j2/log4j2-rollingfileappender-example/ shows some options if you want to make logging more sophisticated, e.g. with time based logging or a combined time based/size based approach.

#4

I would really appreciate some very simple, high-level, end-user oriented documentation about the various logs that OH2 creates. After running it for a few months I have several MB of events.log, and 500K of openhab.log, and they are both pretty unwieldy already.
Much of what’s in openhab.log is pure gobbledegook to me. I’m sure it means a lot to a Java developer, but dozens of copies of the same warning, issued every 2 minutes, just makes finding the real issue an exercise in locating a needle in a haystack. What does this even mean?:
[WARN ] [e.californium.scandium.DTLSConnector] - Shutdown DTLS connector on [0.0.0.0/0.0.0.0:48941] executor not terminated in time!
I’m really only interested in keeping a few days’ worth of logs - I don’t even need older stuff archived.
Can someone supply a simple explanation of how to achieve that?

#5

Hi,

yes, looks pretty weird to have such complicated things for such simple requests. I fully agree with you.

However, I chaged my file /var/lib/openhab2/etc/org.ops4j.pax.logging.cfgnow to the following and I am waiting for the next event to see if (and how) it works:

# Rolling file appender
log4j2.appender.out.type = RollingRandomAccessFile
log4j2.appender.out.name = LOGFILE
log4j2.appender.out.fileName = ${openhab.logdir}/openhab.log
log4j2.appender.out.filePattern = ${openhab.logdir}/openhab.log.%i.gz
log4j2.appender.out.immediateFlush = true
log4j2.appender.out.append = true
log4j2.appender.out.layout.type = PatternLayout
log4j2.appender.out.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} [%-5.5p] [%-36.36c] - %m%n
log4j2.appender.out.policies.type = Policies
log4j2.appender.out.policies.size.type = SizeBasedTriggeringPolicy
log4j2.appender.out.policies.size.size = 16MB
log4j2.appender.out.strategy.type=DefaultRolloverStrategy
log4j2.appender.out.strategy.max=7
#6

What about the outdated habit of reading the docs ?

You can - and should tune your log levels, see docs. There won’t be messages every n minutes unless you have a problem with your setup. If they are fundamental such as lack of network connectivity, that’ll trigger all sorts of messages (maybe unrelated in the first place but in fact a consequence) of all sorts of bindings.
But you should only be running bindings you explicitly enabled to run (unless you did stupid things like to use config snippets that you don’t understand) so if there’s not the aforementioned fundamental problem, at least the component it originates from should ring a bell.

#7

That is exactly the problem. The docs are a vast labyrinth of ever-increasing detail. The section on logs that you referred me tells me “Bindings can have the level of detail increased or decreased on a per binding basis and this is done with the Karaf console.” Is the “Karaf console” the same thing as the “Console” that has an entry in the docs? Better go read that… Ah yes, looks like it is. But hang on, now I have to discover which console commands to use to configure the bindings I have?
OK, I’ll try “bundle:list”… Wow. 225 bundles! Three of them relate to “californium”, whatever that is. Even if I persist with this, I’m no closer to knowing what I need to do to keep my log files tidy and trimmed down so that there’s only a few days’ worth of messages. How many hours am I going to have to spend on investigation, reading documents, and understanding detail that is, frankly, irrelevant to my needs?
Now can you see why I am feeling that I need to be a Java developer to understand this?

1 Like
#8

No docs are not a labyrinth unless you make it one by jumping around. They’re pretty hierarchical in fact. And no it does not take hours even if you read all of them (I’m referring to www.openhab.org/docs).

openHAB is a very complex system.
And no I’m not saying docs are comprehensive or close to perfect -BTW that’s an incredibly difficult task to write non-misunderstandable AND correct AND comprehensive docs, to address novices as well as pros -, but they’re not the problem.
The problem rather is that most people to have this feeling like you do are in fact just too impatient and quite some are even too lazy to read and understand docs FIRST.
They start right away with the system, then if they get stuck somewhere they google to find some detail they then fail to properly interpret because that requires basic knowledge of OH concepts which are in the docs that they skipped.
Combine that with the unhealthy habit many users expose to use default configs and snippets of config they found somewhere without understanding what they do. Even I as a longterm pro use don’t know either what “californium” is. But I don’t have it in my system.

In total, those users spend many more hours on getting their basic setup to work than it would take them if they took the recommended route.

1 Like
#9

Markus, I worked in IT for over 35 years. I’ve been a programmer, analyst, system designer, and project leader, and I understand the nature of complex systems. So please don’t leap to conclusions about my levels of patience or willingness to read documentation and understand concepts.
With regard to OH2, I freely admit that I am a mere user. I have home automation needs which I was led to believe that I could satisfy by installing and configuring it. I did that. It is working pretty much as I need it to. But after an unscheduled reboot (power outage) things stopped working, and I had to check logs to discover the cause of the problem. Turned out to be a device which had come up with a different IP address to what it had previously had. I was able to sort it out. It’s working again.
But I noticed that the logs were getting big, and I want to arrange for them to be smaller. For any other system I’d use standard logrotate - but dammit that doesn’t work because of OH2’s Java base. I have zero desire to furtle around in lord who knows how many configuration files in an unfamiliar format - I am well aware of the risks, and I prefer to have the system working.
If it’s not simple to achieve that, I shall not bother.
Thanks for your advice.

2 Likes
#10

Hi,

the question might be: Is it really neccessary to know Java concepts and type structures to change just some minor things?

It is somehow a question of leveling. How deep do I need to dig? In case I want to improve logging for bindings, debugg sessions and so on I obviously need to know everything, indeed. But I bet 80% of OH users just need to be able to change some parameters from time to time (usually called “increase loglevel”).

You guys know what a GUI is for? To make the use of the software easier for everybody. Someone who knows a piece of software very well does not need a nice GUI because he understands the concept of the software. But what about the majority of the others?
So to all who say: hey, read the docs, get known to the concepts, get used to Java: Please ask yourself the question for which target audience is the software designed for? Java developer? System administrators? Code guys? Should we set a minimum level of knowledge (i.e. aware of object oriented programming) before giving the allowance to use OH?

Try to see our point: I was looking to MODIFY (I did not want to re-code it all!) the log settings. As I am in Linux since 1990’s my first attempt was “logrotate” and “rsyslog”. Well, both has been a dead end (unfortunately this is very often the case for Java implementing all of it’s own instead using what is already there…). So I asked for help (and no, this has not been documented anywhere) and was pointed to some futher docs which pointed to log4j. Really? This did not help at all! (Sorry @stefan.oh).
@savoiringfaire posted some code sniplets- but where to put them?
I was happy once I found the file (not in this thread!) where to configure it (I expected it to be somewhere under /etc/, but no! /var/lib… was it). Oh, and the name of the file was “really” self-explaining. I should have guessed org.ops4j.pax.logging.cfg. Makes sens for all people…

Ok, usually all Linux configuration files are well documented an more or less at a minimum level to have the possibility to change items quickly. The comments to the related log entries in the files with 151 lines (!) make such self-explaining statements like:
log4j2.appender.out.policies.type = Policies
Ok, details:

  • log4j2 - is the library used here (why should I know???)
  • appender - seems to be the word for logger? What else might this mean?
  • out - writing data out. Ok. question: is there an in for appender?
  • policies - So I am dealing with policies which are usually somehow globally…
  • type - Why do I need a policy type?
  • Policies - So I have a ploicy of type Policies… well, absolutely clear, indeed.

And I have some double expressions “size.size” yeah. Of course…

Guys, either hide all these items from the users by installing some easy filter (I am not talking about a GUI, a simple text configuration file could be parsed to write all these Java files…) or at least do not tell the guys seeking for help to “read the docs” and get used to Java concepts.

Anyway; I guess my question is answered so I guess we can close it here…

/KNEBB

#11

I will bet 80% never need to change logging. I have applied TRACE to a binding exactly once in five years for fault finding during binding development. Never needed to tinker with everyday logging defaults at all.
But that’s anectodal really; a better measure is how often the query comes on this forum.

#12

We tell people to read the docs when the answer is in there.
The naming hierarchy you need to apply to define the component you want to apply a loglevel to is the same as is used in programming, but that’s about it. It is not a Java concept, just a naming convention.
What’s confusing you is that the program to log (OH) is taking care of log rotation itself instead of using logrotate, and as it’s written in Java, surprise, it uses a java-like config notation to define the parameters to affect that. But again, that’s just a config notion and not a Java concept. Just like other programs to rotate themselves have their own config notion.
And it’s barely anything you would ever need to touch (yeah I know sysadmins hate using defaults, but still). All a normal user would need to change is the log levels, and while you can do that in the log4j2 conf file, too, that is not the recommended method and nowhere written to be done that way.
As a user, you are requested to use Karaf log:… commands as say the docs. So don’t complain please.