Log uncaught exceptions from Karaf console to file?

Is there a way to log uncaught exceptions that show up in the Karaf console to a file? I currently have my logging setup so that everything is logged to a file, but I cannot find a way to log the uncaught exceptions. I’d like to run a script to tail such a log file, and take a snapshot of other logs when an NPE comes up to help in troubleshooting/debugging. Just having a timestamp for the NPE would be very helpful, so that the exception could be correlated to the other logs. I’ve tried configuring the org.ops4j.pax.logging file every way I could think of to do this, but without success. If there is no option for this, I’ll try using ssh user@host | tee -a <file name>, but thought I’d ask the community to see if it’s just a configuration that I haven’t found.

You mean /var/log/openhab2/openhab.log?

It’s there by default. If you have a non installed version the logs go to userdata/logs

So unless you broke something editing the logging config you should have two log files there, openhab.log and events.log.

Thank you for replying, Rich! I have my logging all setup, working well, and highly customized. Everything is going to separate files… EXCEPT for the exceptions that are not caught…

It is these that I would like to output to a log file so that I can monitor them and run a script when they happen to assist in troubleshooting. The image is an example of an NPE that the current Zigbee binding is throwing, and, as has happened many times in the past, a lot of time is being used to try and correlate them to what is actually being caught in the log. The main issue I’m trying to solve is the lack of a timestamp, but I would also be able to automate other tasks when an uncaught exception occurred.

[I cleaned up my first post so that my question is a bit clearer]

You should also take a look at:

OK, I was in a hurry and didn’t really understand your question. I thought this was an area you would have been knowledgeable about, I should have read more closely.

This might also be a good question to post to the ESH forums to reach some developers who may not be OH users as this very much looks like something from the core.

I suspect the problem is the exception is being thrown and caught within library code and perhaps that library isn’t use log4j2 compatible logging. If that is the case the exception would not make it to the code.

Do the exceptions show up in syslog? I’m assuming that you have an installed version running on Linux. If so you can get the logs from there.

journalctl did not show anything logged for the Karaf or OH/Java PIDs.

IIRC, I’ve only seen exceptions in Karaf from the zwave and zigbee bindings, which I’m sure are using log4j2 compatible logging, but these look like exceptions that are not being caught. I came across some minimal documentation on Karaf’s Enhanced OSGi stack trace renderer, but I haven’t found any detail on whether or not it is possible display this information outside of the console. My research has currently led me to look into exception handling (bundle>OSGi>Karaf). I would think that if there wasn’t a default uncaught exception handler implemented, Java would core dump. Or if the bundles have uncaught exception handlers, then the exception would show up in the bundle’s logger and not in an OSGi stack trace. Still researching.

I agree that my post would likely get more response in the ESH forum, or possibly GH. A seasoned OH/ESH developer would be able to answer my question much faster than it would take me to dig through it! My current intent is to understand (and possibly document) the implemented exception logging, so I thought I’d throw a line in the water here first! :fishing_pole_and_fish: :sunglasses:

BTW, I was successful using the following to output everything displayed in the console to a log file, which I can tail in a script to act on the uncaught exceptions, but I’m still curious if there is a simple configuration available that would provide the same:

ssh openhab@localhost -p 8101 | tee -a <file name>

Indeed, but are all of the libraries those bindings use have log4j2 compatible loggers?

It is catching the exception before that point and I suspect that Karaf is set up so there is some root level exception catcher. You wouldn’t want some bundle that isn’t even related to your bundle to be able to bring down the whole OSGI container.

I really don’t know. I’m just guessing.

Here is what I was able to do to log uncaught exceptions from the Karaf console. I use a manual installation and start OH in a screen, so I start using this:

/opt/openhab2/start.sh 2>&1 | tee -a /opt/openhab2/userdata/logs/tee.log

I then start another screen and run this script:

#!/bin/sh
echo "*********************************************"
echo "********                             ********"
echo "********   Running tee log monitor   ********"
echo "********                             ********"
echo "*********************************************"
tail -F -n0 /opt/openhab2/userdata/logs/tee.log | while read line
do
    timeStamp=$(date +'%Y-%m-%d %H:%M:%S.%3N')
    lineContent=`echo $line | grep "Exception"`
    if [ -n "$lineContent" ]; then
        /usr/bin/curl -s -X POST --header "Content-Type: text/plain" --header "Accept: application/json" -d "An exception occurred in OH at $timeStamp" "http://fedora01.rushworth.us:8080/rest/items/SMS_Notification"
        echo "$timeStamp: $line"
    fi
done

When exceptions occur, it will look like this:

*********************************************
********                             ********
********      Monitoring tee.log     ********
********                             ********
*********************************************
2017-12-11 21:40:48.250: Exception in thread "pool-45-thread-11" java.lang.NullPointerException

From there, correlate the thread to the full exception details in Karaf to determine where it came from.

Exception in thread "pool-45-thread-11" java.lang.NullPointerException
        at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
        at com.zsmartsystems.zigbee.ZigBeeNode.setNeighbors(ZigBeeNode.java:510)
        at com.zsmartsystems.zigbee.ZigBeeNetworkMeshMonitor$2.run(ZigBeeNetworkMeshMonitor.java:232)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Then go to the log to see what was going on at the time the exception occurred.