Openhab docker container logs into grafana

Setup, Configuration and Use Runtime
, ,
1
  • Platform information:
    • Hardware: CPUArchitecture/RAM/storage Raspberry Pi 4
    • OS: what OS is used and which version Raspbian OS 64 bit Bullseye
    • openHAB version: 4.0.4
  • Issue of the topic: Openhab container crashes daily, i guess since the update to 4.0.4

Hello together,

for a long time im running openhab within a docker container. Since update 4.0.4 the container crashes daily.

Luckily i have a lot of logging information but unfortunately the log of the container itself is missing. I would like to add these logs to Grafana to go on with the analyzation but i dont know how. Im also using portainer but here i only get a short time information of the container.

Any ideas?

What i found out so far for latest crash:
I startet with the openhab event log, here i found that the last entry was at 2023-12-02 00:21:03.213. After that nothing until restart in the morning.
So i went on the raspberry pi syslog and found a little bit earlier some health check errors for the openhab container. From here on i need further information.

image
image1598×636 107 KB

2

Maybe I’m completely off base but does Grafana even do that? That’s what folks like Chronograf, Spelunk, Graylog, etc are for. Grafana is to plot time series data (unless you are talking about Grafana Cloud Log but if that’s the case you are not running this locally).

Any log analysis approach requires two parts and often there are separate products for each part. First there is log aggregation which collects the logs and moves them to a central repository. Examples of this include Prometheus, Grafana Loki, etc. The second part is the analysis tools that process the aggregated logs.

To answer this question you need to decide on an analyst tool and use what ever that took is compatible with to connect the logs. If you are using Grafana Cloud Log you’d use Grafana Loki.

You are only going to get the information that is already configured to be logged. If you want more from Docker, you’ll need to go to the Docker docs and forums to figure it how to have it generate more logging.

All of this is outside the scope of OH really so you will get better support from other forums closer to what your are trying to do (e.g. the Grafana forums).

1 Like
3

Sorry, i mean Grafana Loki of cause for the logs.

There is a Docker driver client which ships all container logs to loki but its too hard for me to make it run on a raspberry pi. So my hope was there is an similiar easy way.

Maybe the information already logged is enough but i can only see a short time within portainer.

Then I will go to research how to get the logs to understand the cause for the crash. Thanks for your response.

4

So one night later i made it. Indeed no openhab topic but maybe helps someone.

First I had to add the logging to each container

    logging:
      driver: "json-file"
      options: 
        tag: "{{.Name}}"

Then I missed to map the docker logs into my promtail container:

  promtail:
    image: grafana/promtail:2.8.5
    container_name: promtail
    volumes:
      - /var/lib/docker/containers:/var/log/docker/containers:ro

And at least i had to change the promtail config:

## docker logs
- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/log/docker/containers/*/*log
  pipeline_stages:
  - json:
      expressions:
        output: log
        stream: stream
        attrs:
  - json:
      expressions:
        tag:
      source: attrs
  - regex:
      expression: (?P<container_name>(?:[^|]*[^|]))
      source: tag
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      # tag:
      stream:
      container_name:
  - output:
      source: output

And tadaaa, all container logs are available within loki :slight_smile:

So now i can go on with analyzation of openhab keeps crashing :face_with_monocle: