Logging to graylog

rbausdorf · August 16, 2017, 2:46pm

I was looking for a way to log from openhab2 into a graylog logging server.
After various tries I got the following solution working:

Enable the syslog listeners in rsyslogd
Add an syslog appender in Karaf to log to the local syslog daemon
Add a rsyslog configuration to propagate log entries to remote graylog server

All tries to directly log from Karaf to graylog where unsuccessful. For the direct connection the GELF-Appender jar is missing in Karaf. The direct connection from the SyslogAppender to graylog did not work perhaps for message format reasons.

Enable syslog listeners in /etc/rsyslogd.conf:
Add or uncomment

    # provides UDP syslog reception
    module(load="imudp")
    input(type="imudp" port="514")

    # provides TCP syslog reception
    module(load="imtcp")
    input(type="imtcp" port="514")

Add an syslog appender in Karaf’s /var/lib/openhab2/etc/org.ops4j.pax.logging.cfg
Change line
log4j.rootLogger = WARN, out, osgi:*
to
log4j.rootLogger = WARN, out, syslog, osgi:*

Add to end of file:

    # Syslog appender
    log4j.appender.syslog=org.apache.log4j.net.SyslogAppender
    log4j.appender.syslog.layout=org.apache.log4j.PatternLayout
    log4j.appender.syslog.layout.ConversionPattern=[%p] %c:%L - %m%n
    log4j.appender.syslog.syslogHost=metis
    log4J.appender.syslog.facility=local0
    log4j.appender.syslog.facilityPrinting=false

It is important to use a well-known log facility to rsyslog, as local0 in this example

Add a rsyslog configuration to propagate log entries to remote graylog server:
Add a file client.conf with following content to /etc/rsyslog.d/

    # An "In-Memory Queue" is created for remote logging.
    $WorkDirectory /var/spool/rsyslog    # where to place spool files
    $ActionQueueFileName queue      # unique name prefix for spool files
    $ActionQueueMaxDiskSpace 1g     # spool space limit (use as much as possible)
    $ActionQueueSaveOnShutdown on   # save messages to disk on shutdown
    $ActionQueueType LinkedList     # run asynchronously
    $ActionResumeRetryCount -1      # infinety retries if host is down
    # Define custom logging templates
    $template GRAYLOGRFC5424,"<%PRI%>%PROTOCOL-VERSION% %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"

    #Using default format for default logging format:
    $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

    # Sending logs that match *.* to graylog1.bausdorf.de via TCP on 514 using GRAYLOGRFC5424 format.
    *.* @@graylog2.bausdorf.de:514;GRAYLOGRFC5424

craigh · August 16, 2017, 11:52pm

@rbausdorf great info. thanks for sharing!

Been wanting to get logs into my graylog, just haven’t gotten around to it.

Moxified · December 2, 2017, 1:33am

This is great and I’m looking to implement. I’m getting an error in my syslog with the following:

Dec 1 19:35:38 OHTest start.sh[445]: log4j:ERROR Could not find metis. All logging will FAIL.
Dec 1 19:35:38 OHTest start.sh[445]: java.net.UnknownHostException: metis: Name or service not known

I did an apt install metis but that didn’t seem to make a difference. I’m on debian 8.9

rbausdorf · December 2, 2017, 11:24am

Metis ist the name of my host. Relpace it with your own hostname in config.

Moxified · December 2, 2017, 2:53pm

Nice. I guess I didn’t read that well. I was tired.

I spent some time with this today and wanted to spit out some of my findings.

You don’t need to feed OH through rsyslog first. You can log directly to graylog server’s syslog input if you like. I didn’t want all the other debian default syslog stuff so instead of modifying how debian syslogs, I just bypassed rsyslog.

If you do use rsyslog, the template is no longer needed as newer rsyslog has it built in:

*.* @graylog.example.org:514;RSYSLOG_SyslogProtocol23Format

Single @ sign if you want to use udp, 2 for tcp.

I also tweaked the appender definition as it was passing through source as localhost and I didn’t care for the message layout. So, if you simply want to take a plain Jane OH on debian and log to graylog on a remote host both root and events:

Change log4j.rootLogger = WARN, out, syslog, osgi:*
Change log4j.logger.smarthome.event = INFO, event, syslog, osgi:*

Append:

# Syslog appender
log4j.appender.syslog=org.apache.log4j.net.SyslogAppender
log4j.appender.syslog.layout=org.apache.log4j.PatternLayout
log4j.appender.syslog.layout.ConversionPattern=%p: %c{2} - %m%n
log4j.appender.syslog.syslogHost=IP/DNS OF GRAYLOG SERVER
log4J.appender.syslog.facility=local0
log4j.appender.syslog.facilityPrinting=false
log4j.appender.syslog.Header=true

The header=true seemed to be the key for getting the proper hostname passed through to graylog. Otherwise you just get source=localhost.

rbausdorf · December 2, 2017, 5:57pm

Thanks for the update. What is your Graylog version ? Mine is Graylog 2.3.1

Moxified · December 3, 2017, 1:25pm

I’m using 2.3.2+3df951e

The template part is related to rsyslog version according to this: https://github.com/Graylog2/graylog-guide-syslog-linux

Thanks again for the post. You got me in the right direction and undoubtedly saved me a bunch of time!

rbausdorf · January 4, 2018, 1:56pm

Since OH 2.2 changed the logging from logj to log4j2 here ist the update for the Karaf logging config in
/var/lib/openhab2/etc/org.ops4j.pax.logging.cfg

Change at the beginning:

# Common pattern layout for appenders
#log4j2.pattern = %d{ISO8601} | %-5p | %-16t | %-32c{1} | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n

# Root logger
log4j2.rootLogger.level = WARN
log4j2.rootLogger.appenderRefs = out, osgi, syslog
log4j2.rootLogger.appenderRef.out.ref = LOGFILE
log4j2.rootLogger.appenderRef.osgi.ref = OSGI
log4j2.rootLogger.appenderRef.syslog.ref = SYSLOG

And append at the end:

# Syslog appender                                                                                                                                                                                                                                         
log4j2.appender.syslog.type = Syslog                                                                                                                                                                                                                      
log4j2.appender.syslog.name = SYSLOG                                                                                                                                                                                                                      
log4j2.appender.syslog.facility = local0                                                                                                                                                                                                                  
log4j2.appender.syslog.host = <your log servers IP or Hostname>
log4j2.appender.syslog.appName = openhab                                                                                                                                                                                                                  
log4j2.appender.syslog.protocol = UDP                                                                                                                                                                                                                     
log4j2.appender.syslog.port = 514

If you want to log the events too, you have to look for the block beginning with

log4j2.logger.events.name = smarthome.event

and change at the events.appenderRefs as for the root logger.

Moxified · January 4, 2018, 4:03pm

Thank you!! I hadn’t gotten around to learning what changed about logging to fix it.

Moxified · January 4, 2018, 10:36pm

I’m not sure what I’m doing differently but simply adding the syslog appender portion at the end alone breaks all logging.

rbausdorf · January 5, 2018, 8:28am

On my first try I had an error which said something like “… Plugin not found …”.
It turned out that the appender properties lines MUST NOT have trailing blanks or tabs. To prove if thats your problem too take a look at /var/lib/openhab2/config/org/ops4j/pax/logging.conf, which seems to be generated on the fly.

If you find an entry like

log4j2.appender.syslog.name="SYSLOG\ "

then you have a trailing-blank-problem.

Another problem I had during upgrade from 2.1 to 2.2 was that I decided to keep my old logging config. But they moved from log4j to log4j2 so my logging was completely broken after the first try.

PS: I upgraded my graylog to 2.3.2 - and I do no longer need to syslog to localhost first. Graylog has obviously fixed something in their Syslog input.

Moxified · January 5, 2018, 4:30pm

Ah yes. That is it. I can see this being an issue for others. If you copy your block of code and paste that… it will create a bunch of spaces. My config file had " / / / / / / / / / " for miles on each entry.

This should be better without the spaces:

# Syslog appender
log4j2.appender.syslog.type = Syslog
log4j2.appender.syslog.name = SYSLOG
log4j2.appender.syslog.facility = local0
log4j2.appender.syslog.host = <your log servers IP or Hostname>
log4j2.appender.syslog.appName = openhab
log4j2.appender.syslog.protocol = UDP
log4j2.appender.syslog.port = 514

nasi_be · April 23, 2018, 8:56pm

Hi,
I know this article is a few months old, but I also tried to send my log messages to graylog, but none of the messages arrived.

My org.ops4j.pax.logging.cfg looks like:

# Common pattern layout for appenders
#log4j2.pattern = %d{ISO8601} | %-5p | %-16t | %-32c{1} | %X{bundle.id} - %X{bundle.name} - $

# Root logger
log4j2.rootLogger.level = WARN
log4j2.rootLogger.appenderRefs = out,osgi,syslog
log4j2.rootLogger.appenderRef.out.ref = LOGFILE
log4j2.rootLogger.appenderRef.osgi.ref = OSGI
log4j2.rootLogger.appenderRef.syslog.ref = SYSLOG

log4j2.logger.events.name = smarthome.event
log4j2.logger.events.level = INFO
log4j2.logger.events.additivity = false
log4j2.logger.events.appenderRefs = event,syslog
log4j2.logger.events.appenderRef.event.ref = EVENT
log4j2.logger.events.appenderRef.osgi.ref = OSGI
log4j2.logger.events.appenderRef.syslog.ref = SYSLOG

# Syslog log appender
log4j2.appender.syslog.type = Syslog
log4j2.appender.syslog.name = SYSLOG
log4j2.appender.syslog.facility = local0
log4j2.appender.syslog.host = graylog
log4j2.appender.syslog.appName = openhab
log4j2.appender.syslog.protocol = UDP
log4j2.appender.syslog.port = 514

(I edited the root-logger and the event-logger and added a log appender to the syslog host.)

After the first start, I got errors in the console and nothing has been logged, but I found the trailing spaces in the config file (like mentioned above).
But after removing the spaces, there were no error messages in the console, but the log messages only appear in the standard log-files and no message is send to the syslog server.

My openhab instance runs in a docker container.

Can anybody see an error in the log-config?
Is there a way to test the config in the karaf-console?

Thanks for the help!
Thomas

nasi_be · April 23, 2018, 9:05pm

Hi,

I found my messages:
It seems that all messages were send to graylog, but there is a timezone-problem so that my messages appeared only 2 hours later.

Now I have to find the wrong timezone-setting…

Kind regards
Thomas

rbausdorf · April 24, 2018, 11:36am

Had that problem too - my servers hardware clock is using UTC, I had to adjust the locale setting either globally or at least for the user which is running openhab.

Moerky · December 27, 2019, 8:52pm

I tried my problem in a separate, new thread but didn’t get any valuable response. I have the same setup, encountered the same problem with time settings on the graylog side. Everything would be perfect if … when the graylog server is restarted openhab(ian) stops sending any UDP packets to the graylog server. It has to be restarted to resend them again. But I see nothing in any (openhabian) logs.

How could I make openhab retry if the connection to the graylog server breaks for any reason?
How could I identify this problem to be notified in advance?

rlkoshak · March 19, 2024, 9:07pm

This came up recently and I wanted to post a working log4j2.xml config that should work with OH 3+.

Create the following Socket appender.

                <!-- Gelf appender -->
                <!-- https://logging.apache.org/log4j/2.x/manual/layouts.html#GELFLayout -->

                <Socket name="GRAYLOG" host="10.10.1.111" port="12201" protocol="tcp" immediateFail="true">
                        <GelfLayout host="argus" compressionType="OFF" includeNullDelimiter="true" includeStacktrace="true">
                                <!-- <KeyValuePair key="additionalField1" value="constant value"/>
                                     <KeyValuePair key="additionalField2" value="$${ctx:key}"/> -->
                        </GelfLayout>
                </Socket>

Then add GRAYLOG to each logger you want to forward to GrayLog. For example:

                <!-- Karaf Shell logger -->
                <Logger level="OFF" name="org.apache.karaf.shell.support">
                        <AppenderRef ref="STDOUT"/>
                        <AppenderRef ref="GRAYLOG"/>
                </Logger>

While the GELF appender does not come with Karaf (as reported above), there is a GELF layout so you can use the Socket appender instead. I’ve tested this to work with GrayLog 5. Don’t forget to enable the GELF TCP Input.

See the URL in the comment above for full docs and a UDP example.