Centralized Logging of openhab via Graylog, Elasticsearch etc

Apps & Services 3rd Party
1

The out-of-the box logging handing of openhabian via frontail works perfectly - if the log entries are only a couple of minutes / hours old. As I was looking for a centralized logging solution to be able to easily find log entries that are a couple of days / weeks old without filling up my openhabian setup, I came to this tutorial which works quite fine Logging to graylog and relies on docker containers for elastic search, redis and graylog as centralized logging handler - all for free.

The solution works fine but as i implemented the log appender via log4j (via syslog - udp) every time I reboot the (seperate) logging instance of graylog the connection breaks up (which should not happen when using UDP should it). I have to reboot openhab2 too to make it work again - which is not a good productive setup.

Anyone able to help?

3

This phrase appears to be entirely content free.

4

Means I receive no more UDP connections from the openhabian system to my graylog instance - thought that is clear. I have no clue where to look up to receive more content.

5

It the packets are being sent but not received the issue appears to be on the graylog side or the internal Docker networking between the applications then.

6

The following is a working log4j2.xml config to log to GrayLog directly without going through syslog.

Create the following Socket appender.

                <!-- Gelf appender -->
                <!-- https://logging.apache.org/log4j/2.x/manual/layouts.html#GELFLayout -->

                <Socket name="GRAYLOG" host="10.10.1.111" port="12201" protocol="tcp" immediateFail="true">
                        <GelfLayout host="argus" compressionType="OFF" includeNullDelimiter="true" includeStacktrace="true">
                                <!-- <KeyValuePair key="additionalField1" value="constant value"/>
                                     <KeyValuePair key="additionalField2" value="${ctx:key}"/> -->
                        </GelfLayout>
                </Socket>

Then add GRAYLOG to each logger you want to forward to GrayLog. For example:

                <!-- Karaf Shell logger -->
                <Logger level="OFF" name="org.apache.karaf.shell.support">
                        <AppenderRef ref="STDOUT"/>
                        <AppenderRef ref="GRAYLOG"/>
                </Logger>

For UDP see the URL in the comment above the appender.

7

Thank you @rlkoshak for providing this! Actually my initial issue was that with the restart of the Graylog server the log4j2 didn’t send any logs to it anymore. Somewhere over the time this worked. So meanwhile I can restart / shutdown the Graylog server anytime and after getting back up running openhab(Ian) still sends the packets.I have used the syslog appender which also works fine. I’ve added some additional fields so I guess I don’t get any more info than you?
Here is an example:
application_name: OpenHAB4
bundle_id: 179
bundle_name: org.openhab.core.io.monitor
bundle_script: ItemStateChangedEvent
bundle_version: 4.1.1
facility: local0
facility_num: 16
level: 6
message: Item ‘waterLastRun’ changed from 2024-04-06T14:26:50.948116167+0200 to 2024-04-06T14:26:57.194128454+0200
process_id: 675
source: openhabian

8

Honestly I finished seeing this up and I’m sitting at “now what?” I don’t actually see how Graylog is going to help me in the long run so might shut it down eventually.

Those look like the fields I get but I’ve so far found watching the logs to be easier and more timely using multitail instead of Graylog so I haven’t looked too closely. Those are definitely the fields I would expect.

9

Graylog is not so straight forward to use and for fast checking the output I think frontail is fine. With graylog I like the following, don’t know if multitail offers these too:

  • be able to search for stuff that happend months ago
  • to define daily reports that inform me of rule excpetions that were thrown in the past day (when I mixed something up or something got broken)
  • to be abler to aggregate several logs from other systems (also from my router)
  • the query language is quite powerful and fast, also for searches over a long period

Cheers, Markus

10

multitail is just the Unix tail command line tool with added features. It’s not a service nor a log aggregator.