Openhab 4.0.4 crashes almost daily

Setup, Configuration and Use Runtime
,
1
  • Platform information:
    • Hardware: CPUArchitecture/RAM/storage Raspberry Pi 4 2GB
    • OS: what OS is used and which version Raspbian OS 64 bit Bullseye
    • openHAB version: 4.0.4
  • Issue of the topic: Openhab container crashes daily, i guess since the update to 4.0.4. I guess its starting with health check timeouts but dont know why.

Hello together,

for a long time im running openhab within a docker container. Since update 4.0.4 the container crashes daily (already mentioned in another post regarding how to get more log informations). For weeks im struggling how to solve this issue.

Whats happening:
After one or two days openhab stops working. When I try to ssh by hostname its not working anymore, after short time also ssh over IP is dead (could be another DNS issue regarting host/IP). At least ping the IP is still working (tested only once), but i cant access anything else of the web services like grafana, node-red … so I just unplug the pi and its keeps running for another day

Any ideas to this?

What i tried already:
I replaced the microSD card by a SSD storage and also tested another Pi4 without success.

As i now have quite a lot logging information I tried to understand whats going on but i cant as i dont understand some content / dont know where to start.

The Pi CPU statistics (CPU, memory, network traffic) looking quite normal when it crashes.

Heres a log file of the last crash at 20th december around 11:25 when i detected the crash, so the reason may be some time before.

Just filtered out the warnings and erros:

Dec 20 11:17:09 openhab dockerd[615]: time="2023-12-20T11:17:09.536877976+01:00" level=error msg="[resolver] failed to query DNS server: 192.168.178.37:53, query: ;stats.grafana.org.\tIN\t A" error="read udp 172.28.0.6:47399->192.168.178.37:53: i/o timeout"
Dec 20 11:30:50 openhab dockerd[615]: time="2023-12-20T11:30:49.687061474+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 11:30:52 openhab dockerd[615]: time="2023-12-20T11:30:50.531434039+01:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 20 11:30:54 openhab dockerd[615]: time="2023-12-20T11:30:50.571778462+01:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 20 11:31:24 openhab containerd[446]: time="2023-12-20T11:31:23.824559130+01:00" level=error msg="get state for cd547d4b69c75c9cd4032cce8d8534bed13fc9985b327790e41a5ffb0941c6d6" error="context deadline exceeded: unknown"
Dec 20 11:31:25 openhab containerd[446]: time="2023-12-20T11:31:24.484762850+01:00" level=warning msg="unknown status" status=0
Dec 20 11:31:55 openhab dockerd[615]: time="2023-12-20T11:31:54.940636622+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 11:31:55 openhab dockerd[615]: time="2023-12-20T11:31:54.993700361+01:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 20 11:44:39 openhab dockerd[615]: time="2023-12-20T11:44:27.440181361+01:00" level=warning msg="Health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221 error: timed out starting health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221"
Dec 20 11:44:41 openhab dockerd[615]: time="2023-12-20T11:44:27.419911894+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 11:46:39 openhab dockerd[615]: time="2023-12-20T11:46:28.512047981+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 11:48:44 openhab dockerd[615]: time="2023-12-20T11:48:28.687023374+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 11:52:19 openhab dockerd[615]: time="2023-12-20T11:51:24.050138301+01:00" level=warning msg="Health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221 error: timed out starting health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221"
Dec 20 11:52:55 openhab dockerd[615]: time="2023-12-20T11:52:35.749415751+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 11:54:55 openhab dockerd[615]: time="2023-12-20T11:54:45.404841808+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 11:55:08 openhab dockerd[615]: time="2023-12-20T11:55:06.380493148+01:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 20 11:55:20 openhab dockerd[615]: time="2023-12-20T11:55:06.380493111+01:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 20 11:56:24 openhab dockerd[615]: time="2023-12-20T11:56:11.003972285+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 12:12:06 openhab dockerd[615]: time="2023-12-20T12:12:00.219633019+01:00" level=warning msg="Health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221 error: timed out starting health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221"
Dec 20 12:12:50 openhab dockerd[615]: time="2023-12-20T12:12:01.436631633+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 12:16:15 openhab dockerd[615]: time="2023-12-20T12:16:15.176477245+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 12:17:46 openhab dockerd[615]: time="2023-12-20T12:17:41.689144682+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 12:18:13 openhab dockerd[615]: time="2023-12-20T12:18:09.883190246+01:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 20 12:18:16 openhab dockerd[615]: time="2023-12-20T12:18:09.883238467+01:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 20 12:19:11 openhab dockerd[615]: time="2023-12-20T12:19:04.895512509+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 12:20:55 openhab dockerd[615]: time="2023-12-20T12:20:34.959725908+01:00" level=warning msg="Health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221 error: timed out starting health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221"
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49+01:00" level=warning msg="containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header"
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49.681190700+01:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/6.1.21-v8+\\n\"): skip plugin" type=io.containerd.snapshotter.v1
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49.684036052+01:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49.684384700+01:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49.687667515+01:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49.688012441+01:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49.704332052+01:00" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49.705061997+01:00" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
Dec 19 21:13:43 openhab dockerd[615]: time="2023-12-19T21:13:43.800567672+01:00" level=error msg="[resolver] failed to query DNS server: 192.168.178.37:53, query: ;raw.githubusercontent.com.\tIN\t A" error="read udp 172.28.0.4:58311->192.168.178.37:53: i/o timeout"
Dec 19 21:13:43 openhab dockerd[615]: time="2023-12-19T21:13:43.801404512+01:00" level=error msg="[resolver] failed to query DNS server: 192.168.178.37:53, query: ;raw.githubusercontent.com.\tIN\t AAAA" error="read udp 172.28.0.4:39246->192.168.178.37:53: i/o timeout"
Dec 19 21:13:43 openhab dockerd[615]: time="2023-12-19T21:13:43.809005554+01:00" level=error msg="[resolver] failed to query DNS server: 192.168.178.37:53, query: ;grafana.com.\tIN\t A" error="read udp 172.28.0.4:45162->192.168.178.37:53: i/o timeout"
Dec 19 21:13:43 openhab dockerd[615]: time="2023-12-19T21:13:43.809005443+01:00" level=error msg="[resolver] failed to query DNS server: 192.168.178.37:53, query: ;grafana.com.\tIN\t AAAA" error="read udp 172.28.0.4:44270->192.168.178.37:53: i/o timeout"
Dec 20 12:18:02 openhab dockerd[619]: time="2023-12-20T12:18:02.536717730+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d 9bf95ac60f033560b17480290fdcb8b1584d46f7ab638a0f217ee8ba2025baa6], retrying...."
Dec 20 12:18:02 openhab dockerd[619]: time="2023-12-20T12:18:02.770831100+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d 59764f61518cbc8b2b4b06b70ff9313e8debbb08b16c0fe219efe1d74bbac413], retrying...."
Dec 20 12:18:02 openhab dockerd[619]: time="2023-12-20T12:18:02.951950081+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d 4ddc5ba58489a5adfa9928073340e5e3bba12169c8c59e8cb2c5f23858ed887e], retrying...."
Dec 20 12:18:02 openhab dockerd[619]: time="2023-12-20T12:18:02.987723970+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint fce040fad794af1d6beefb8ef5d62647a301b36ef570c253cf7c0a22c3ee996c 8f37e2521727b9b04cf40852288458a0c9f3f448babc5374b1ab21b88672d1fc], retrying...."
Dec 20 12:18:03 openhab dockerd[619]: time="2023-12-20T12:18:03.025468767+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint fce040fad794af1d6beefb8ef5d62647a301b36ef570c253cf7c0a22c3ee996c ba096327f32fb2a505626b231a1b204c07aa4252d132f8569cc3c0e86eb3ecca], retrying...."
Dec 20 12:18:03 openhab dockerd[619]: time="2023-12-20T12:18:03.177840415+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d 30a0faa46cee696051db84e0ddd8b682ade977e366b042aa0497e54b31447699], retrying...."
Dec 20 12:18:03 openhab dockerd[619]: time="2023-12-20T12:18:03.349036118+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d c00eced868bcfdc3037616cb2b6cb71eafc20207240e4137e222c666fb0bd073], retrying...."
Dec 20 12:18:03 openhab dockerd[619]: time="2023-12-20T12:18:03.501318081+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d 4f211bcbddb0b603eb08cfab6b6a7f4fb4aada12ab2bec802af7b5372f4e8151], retrying...."
Dec 20 12:18:03 openhab dockerd[619]: time="2023-12-20T12:18:03.673110914+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d fa317f2af0ec176b9ac0d40f1dd8ee9137aa4021279529ff8ca7b9abaa310e67], retrying...."
Dec 20 12:18:04 openhab dockerd[619]: time="2023-12-20T12:18:04.103430507+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d 2d8e144e1ee6783befb12623710c9b7fa8e448d47b3d4d12e1174959a47a750d], retrying...."
Dec 20 12:18:04 openhab dockerd[619]: time="2023-12-20T12:18:04.145826118+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint fce040fad794af1d6beefb8ef5d62647a301b36ef570c253cf7c0a22c3ee996c ad543728073427645892b4e8a3ab209bd766c26fe3b65c8217e5ec788f8b9a76], retrying...."
Dec 20 12:18:04 openhab dockerd[619]: time="2023-12-20T12:18:04.308071007+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 5cac9c589fa254a31f6f51df24f6796a567942a74007d428a0c1360025cd5696 e428978a057ea3e70a4d556821502b2b25dc8995ca336c5d88859195b70a7574], retrying...."
Dec 20 12:18:04 openhab dockerd[619]: time="2023-12-20T12:18:04.363718951+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d aadb236241f47e3ce9626c78d9ac70aa9ae7ce6b276ae55c2bb24f9c3b6cfed8], retrying...."
Dec 20 12:18:07 openhab containerd[464]: time="2023-12-20T12:18:07.055452968+01:00" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/system.slice/docker-3e7c622707902af4bf704c9e08040ca9b2d054b1687642ed840b552d9b9d3d61.scope/memory.events\": no such file or directory"
Dec 20 12:18:07 openhab containerd[464]: time="2023-12-20T12:18:07.119254079+01:00" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/system.slice/docker-95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221.scope/memory.events\": no such file or directory"
Dec 20 00:47:16 openhab dockerd[615]: time="2023-12-20T00:47:16.981584641+01:00" level=error msg="[resolver] failed to query DNS server: 192.168.178.37:53, query: ;stats.grafana.org.\tIN\t AAAA" error="read udp 172.28.0.6:58243->192.168.178.37:53: i/o timeout"
Dec 20 00:47:16 openhab dockerd[615]: time="2023-12-20T00:47:16.981599289+01:00" level=error msg="[resolver] failed to query DNS server: 192.168.178.37:53, query: ;stats.grafana.org.\tIN\t A" error="read udp 172.28.0.6:44047->192.168.178.37:53: i/o timeout"
Dec 20 12:18:11 openhab containerd[464]: time="2023-12-20T12:18:11.672109669+01:00" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/system.slice/docker-697bff462c1653a15b46bb795d84880019278ff998b898da81fb1cf195f2ce3c.scope/memory.events\": no such file or directory"
Dec 20 12:18:12 openhab containerd[464]: time="2023-12-20T12:18:11.989690946+01:00" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/system.slice/docker-5f7a6d6e0cd629672e4aadc924349e33a2a4a821d5c1694ac600a748610543bb.scope/memory.events\": no such file or directory"
Dec 20 12:18:12 openhab containerd[464]: time="2023-12-20T12:18:12.271412039+01:00" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/system.slice/docker-4c5aaca04395d968aafe852aa74266a661980d478e425c9b14a570f00e280e1a.scope/memory.events\": no such file or directory"
Dec 20 12:18:13 openhab containerd[464]: time="2023-12-20T12:18:13.048685520+01:00" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/system.slice/docker-0a3874b93fc609e87085f9d66c0d7f664d51771495a445cddc7c8ac4f49b1d4b.scope/memory.events\": no such file or directory"
Dec 20 12:18:14 openhab containerd[464]: time="2023-12-20T12:18:14.181712612+01:00" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/system.slice/docker-90e70388b7213afbc82c4d00750ebaa47631361ba689aebc0899c3f90d761385.scope/memory.events\": no such file or directory"
Dec 20 12:18:15 openhab containerd[464]: time="2023-12-20T12:18:15.135720741+01:00" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/system.slice/docker-c01b5597ec83fefb72fee42aff24157edc97f0b10e345bd427791ebf66f69fc4.scope/memory.events\": no such file or directory"
Dec 20 12:18:15 openhab containerd[464]: time="2023-12-20T12:18:15.641296296+01:00" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/system.slice/docker-c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa.scope/memory.events\": no such file or directory"
Dec 20 12:18:16 openhab containerd[464]: time="2023-12-20T12:18:16.253848555+01:00" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/system.slice/docker-17d926e629cdb390e991bc5512e635b9bf830d8feffcbf2736ce2a4bef9d73e8.scope/memory.events\": no such file or directory"
Dec 20 12:18:16 openhab containerd[464]: time="2023-12-20T12:18:16.467263888+01:00" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/system.slice/docker-eb0eae7d01b87ed0109962f47798df620a46fb4cdd4dc91be07c2651e6a8bd07.scope/memory.events\": no such file or directory"
Dec 20 12:18:16 openhab containerd[464]: time="2023-12-20T12:18:16.512502518+01:00" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/system.slice/docker-ebd8e70f9bf9d1697bddca144ae0371f1e55c6b972ed513574b2fdf1a0ba7bd4.scope/memory.events\": no such file or directory"
Dec 20 12:18:16 openhab dockerd[619]: time="2023-12-20T12:18:16.806761092+01:00" level=warning msg="WARNING: No memory limit support"
Dec 20 12:18:16 openhab dockerd[619]: time="2023-12-20T12:18:16.806924962+01:00" level=warning msg="WARNING: No swap limit support"
Dec 20 11:17:09 openhab dockerd[615]: time="2023-12-20T11:17:09.536877976+01:00" level=error msg="[resolver] failed to query DNS server: 192.168.178.37:53, query: ;stats.grafana.org.\tIN\t A" error="read udp 172.28.0.6:47399->192.168.178.37:53: i/o timeout"
Dec 20 11:30:50 openhab dockerd[615]: time="2023-12-20T11:30:49.687061474+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 11:30:52 openhab dockerd[615]: time="2023-12-20T11:30:50.531434039+01:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 20 11:30:54 openhab dockerd[615]: time="2023-12-20T11:30:50.571778462+01:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 20 11:31:24 openhab containerd[446]: time="2023-12-20T11:31:23.824559130+01:00" level=error msg="get state for cd547d4b69c75c9cd4032cce8d8534bed13fc9985b327790e41a5ffb0941c6d6" error="context deadline exceeded: unknown"
Dec 20 11:31:25 openhab containerd[446]: time="2023-12-20T11:31:24.484762850+01:00" level=warning msg="unknown status" status=0
Dec 20 11:31:55 openhab dockerd[615]: time="2023-12-20T11:31:54.940636622+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 11:31:55 openhab dockerd[615]: time="2023-12-20T11:31:54.993700361+01:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 20 11:44:39 openhab dockerd[615]: time="2023-12-20T11:44:27.440181361+01:00" level=warning msg="Health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221 error: timed out starting health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221"
Dec 20 11:44:41 openhab dockerd[615]: time="2023-12-20T11:44:27.419911894+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 11:46:39 openhab dockerd[615]: time="2023-12-20T11:46:28.512047981+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 11:48:44 openhab dockerd[615]: time="2023-12-20T11:48:28.687023374+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 11:52:19 openhab dockerd[615]: time="2023-12-20T11:51:24.050138301+01:00" level=warning msg="Health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221 error: timed out starting health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221"
Dec 20 11:52:55 openhab dockerd[615]: time="2023-12-20T11:52:35.749415751+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 11:54:55 openhab dockerd[615]: time="2023-12-20T11:54:45.404841808+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 11:55:08 openhab dockerd[615]: time="2023-12-20T11:55:06.380493148+01:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 20 11:55:20 openhab dockerd[615]: time="2023-12-20T11:55:06.380493111+01:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 20 11:56:24 openhab dockerd[615]: time="2023-12-20T11:56:11.003972285+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 12:12:06 openhab dockerd[615]: time="2023-12-20T12:12:00.219633019+01:00" level=warning msg="Health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221 error: timed out starting health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221"
Dec 20 12:12:50 openhab dockerd[615]: time="2023-12-20T12:12:01.436631633+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 12:16:15 openhab dockerd[615]: time="2023-12-20T12:16:15.176477245+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 12:17:46 openhab dockerd[615]: time="2023-12-20T12:17:41.689144682+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 12:18:13 openhab dockerd[615]: time="2023-12-20T12:18:09.883190246+01:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 20 12:18:16 openhab dockerd[615]: time="2023-12-20T12:18:09.883238467+01:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 20 12:19:11 openhab dockerd[615]: time="2023-12-20T12:19:04.895512509+01:00" level=warning msg="Health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa error: timed out starting health check for container c884669782ec3da2ca99742811c5d583b7e33f248149d1c2698a20395e1e2cfa"
Dec 20 12:20:55 openhab dockerd[615]: time="2023-12-20T12:20:34.959725908+01:00" level=warning msg="Health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221 error: timed out starting health check for container 95c04878ac0f9e8b4bd543143ec57b34797648d351a78e8e55fc891829cb0221"
Dec 20 12:47:08 openhab dockerd[619]: time="2023-12-20T12:47:08.320911655+01:00" level=error msg="[resolver] failed to query DNS server: 192.168.178.37:53, query: ;stats.grafana.org.\tIN\t AAAA" error="read udp 172.28.0.5:58639->192.168.178.37:53: i/o timeout"
Dec 20 12:47:08 openhab dockerd[619]: time="2023-12-20T12:47:08.320911080+01:00" level=error msg="[resolver] failed to query DNS server: 192.168.178.37:53, query: ;stats.grafana.org.\tIN\t A" error="read udp 172.28.0.5:50808->192.168.178.37:53: i/o timeout"
Dec 20 19:30:04 openhab dockerd[619]: time="2023-12-20T19:30:04.304881407+01:00" level=error msg="[resolver] failed to query DNS server: 192.168.178.37:53, query: ;stats.grafana.org.\tIN\t A" error="read udp 172.28.0.5:39965->192.168.178.37:53: i/o timeout"
Dec 20 19:30:04 openhab dockerd[619]: time="2023-12-20T19:30:04.304879185+01:00" level=error msg="[resolver] failed to query DNS server: 192.168.178.37:53, query: ;stats.grafana.org.\tIN\t AAAA" error="read udp 172.28.0.5:46689->192.168.178.37:53: i/o timeout"
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49+01:00" level=warning msg="containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header"
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49.681190700+01:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/6.1.21-v8+\\n\"): skip plugin" type=io.containerd.snapshotter.v1
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49.684036052+01:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49.684384700+01:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49.687667515+01:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49.688012441+01:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49.704332052+01:00" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
Dec 20 12:17:49 openhab containerd[464]: time="2023-12-20T12:17:49.705061997+01:00" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
Dec 20 12:18:02 openhab dockerd[619]: time="2023-12-20T12:18:02.536717730+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d 9bf95ac60f033560b17480290fdcb8b1584d46f7ab638a0f217ee8ba2025baa6], retrying...."
Dec 20 12:18:02 openhab dockerd[619]: time="2023-12-20T12:18:02.770831100+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d 59764f61518cbc8b2b4b06b70ff9313e8debbb08b16c0fe219efe1d74bbac413], retrying...."
Dec 20 12:18:02 openhab dockerd[619]: time="2023-12-20T12:18:02.951950081+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d 4ddc5ba58489a5adfa9928073340e5e3bba12169c8c59e8cb2c5f23858ed887e], retrying...."
Dec 20 12:18:02 openhab dockerd[619]: time="2023-12-20T12:18:02.987723970+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint fce040fad794af1d6beefb8ef5d62647a301b36ef570c253cf7c0a22c3ee996c 8f37e2521727b9b04cf40852288458a0c9f3f448babc5374b1ab21b88672d1fc], retrying...."
Dec 20 12:18:03 openhab dockerd[619]: time="2023-12-20T12:18:03.025468767+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint fce040fad794af1d6beefb8ef5d62647a301b36ef570c253cf7c0a22c3ee996c ba096327f32fb2a505626b231a1b204c07aa4252d132f8569cc3c0e86eb3ecca], retrying...."
Dec 20 12:18:03 openhab dockerd[619]: time="2023-12-20T12:18:03.177840415+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d 30a0faa46cee696051db84e0ddd8b682ade977e366b042aa0497e54b31447699], retrying...."
Dec 20 12:18:03 openhab dockerd[619]: time="2023-12-20T12:18:03.349036118+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d c00eced868bcfdc3037616cb2b6cb71eafc20207240e4137e222c666fb0bd073], retrying...."
Dec 20 12:18:03 openhab dockerd[619]: time="2023-12-20T12:18:03.501318081+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d 4f211bcbddb0b603eb08cfab6b6a7f4fb4aada12ab2bec802af7b5372f4e8151], retrying...."
Dec 20 12:18:03 openhab dockerd[619]: time="2023-12-20T12:18:03.673110914+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d fa317f2af0ec176b9ac0d40f1dd8ee9137aa4021279529ff8ca7b9abaa310e67], retrying...."
Dec 20 12:18:04 openhab dockerd[619]: time="2023-12-20T12:18:04.103430507+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d 2d8e144e1ee6783befb12623710c9b7fa8e448d47b3d4d12e1174959a47a750d], retrying...."
Dec 20 12:18:04 openhab dockerd[619]: time="2023-12-20T12:18:04.145826118+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint fce040fad794af1d6beefb8ef5d62647a301b36ef570c253cf7c0a22c3ee996c ad543728073427645892b4e8a3ab209bd766c26fe3b65c8217e5ec788f8b9a76], retrying...."
Dec 20 12:18:04 openhab dockerd[619]: time="2023-12-20T12:18:04.308071007+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 5cac9c589fa254a31f6f51df24f6796a567942a74007d428a0c1360025cd5696 e428978a057ea3e70a4d556821502b2b25dc8995ca336c5d88859195b70a7574], retrying...."
Dec 20 12:18:04 openhab dockerd[619]: time="2023-12-20T12:18:04.363718951+01:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b6920b2cbc616104bd4c6a46dd065fb50e1bc6fba1b98278fd738e1413570d1d aadb236241f47e3ce9626c78d9ac70aa9ae7ce6b276ae55c2bb24f9c3b6cfed8], retrying...."
2

Please list the bindings you use. Please tell us if the configuration is file based or UI. How many Things, how many rules. Is anything else running on the PI ?

downgrade? That would isolate if it is something in 4.0.4

3

Thank you for your response. I mostly use file based configuration.
Regarding the installed binding i have no idea how to get the list because

bundle:list

does not work in the docker containers console.

Normally downgrade is easy with docker but i did some changes (specially for java scripts) after the update to 4.0.4, so i have to try how easy it will be to go back. I will give it a try after next crash if i have no more other ideas.

I have some other containers running but no changes for a long time.
(frontail, grafana, influxdb, loki, mosquitto, node-red, node_exporter, portainer, prometheus, promtail, samba, zigbee2mqtt)

4

that is kind of a lot for

only 2 GB… it ran ok on previous version?
Edit sorry is that the log from Docker? Can you get to the openHAB logs?

1 Like
5

Yes thats a lot but i had no issues for a long time. But i have a spare 4gb version which I will use soon.

Unfortunately I have a gap in the openhab logs due to a bug in my compose file (hopefully fixed now). From older events i saw nothing special when it died, just that the logs are (openhab systen & event) just stop the same time the health check starts to fail. I just deactivated the healthcheck complete because I expect its configuration was not complete anyway.

6

OK well I think I’ve seen a couple similar reports of stuff in the milestone discussion
Downgrading to previous version would confirm it’s a buggy in 4.0.4

Please even if it’s off the top of your head list bindings used

1 Like
7

In the main ui goto menu, help and about and then you can click on technical details, then view details, where you get a huge list you can copy and paste here. That is its purpose.

1 Like
8

so i just went back to 3.4.0 like before the update and it looks like its working except the java scripts for diming lights but thats fine for testing.

Normally it should crash within 2 days, we will see.

1 Like
9

See The Console | openHAB. Not the shell, a special Karaf console you ssh to on a different port.

1 Like
10

That all may use a lot of memory especially since you run a 64-bit OS and Docker. You may want to check your kernel logs to see if the Out Of Memory (OOM) killer starts killing processes when it runs out of memory.

1 Like
11

@rlkoshak you mean the karat console right? I dont know yet how to enter it from openhab docker because also

openhab-cli console

Also Not working. But thanks to @matt1 i found a list of Bindings:

bindings:
  - androidtv
  - astro
  - chromecast
  - dwdpollenflug
  - dwdunwetter
  - exec
  - gpstracker
  - homematic
  - http
  - hyperion
  - icalendar
  - icloud
  - mihome
  - miio
  - mqtt
  - network
  - ntp
  - openweathermap
  - playstation
  - remoteopenhab
  - snmp
  - sonos
  - sony
  - spotify
  - systeminfo
  - tr064
  - wifiled

@wborn Good Point, i will Check for Out of memory entries later, thanks. For Sure i will Upgrade to pi4 4gb, maybe i even go to a orangepi 5 with 8gb

1 Like
12

Assuming you have set up the container according to the docs and are using --net=host, you enter it just like it says in the docs:

ssh -p 8101 openhab@localhost

Or you can use

docker exec -it openhab /openhab/runtime/bin/client

Everything that openhab-cli does is represented as a script in /openhab/runtime/bin.

1 Like
13

@rlkoshak great, it worked with ssh.

@wborn really strange that suddenly my grafana data is empty like all before today has been deleted, even the log i added at the beginning of the post. Maybe i have to check the database location and docker compose setting.

However in this log i found something regarding memory:
dockerd warning: no memory limit support
dockerd warning: no swap limit support
Thats what we are looking for?

When i check the pi stats i also got a gap put before that gab it looks like ram slowly rises until around 12:20 crashed

image
image1591Ă—665 123 KB

and of cause in general the ram / swap are running out

14

Yes your dashboard shows it runs out of memory. If it starts killing processes due to lack of memory it will show up as “killed process” in the logging see also:

You could increase the swap size to buy yourself some time until you migrate to something with more memory, see:

However if it continuously swaps memory it can wear out your sdcard quickly.

1 Like
15

@wborn looks like jackpot if i understand correct:

Sun 2023-03-12 05:08:51.085901 UTC [s=8a69ca4fb9e045f8b3e3f9d0c42703aa;i=d0;b=048f7ca9a3e64fcaa0c956d0f60da1d5;m=d326595690;t=5f6acfd7b6e4d;x=30bf993d4a384f28]
    MESSAGE=Out of memory: Killed process 3250 (java) total-vm:4987000kB, anon-rss:1092560kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:3904kB oom_score_adj:0
Tue 2023-06-27 19:03:59.276840 UTC [s=0a241a1eea904cba8b086fb666464281;i=c4;b=115640ac81d6499ba17889292ef4f7ed;m=8cb2ca1072;t=5ff21212edf28;x=2f3dd25547c4278d]
    MESSAGE=Out of memory: Killed process 253286 (java) total-vm:4944624kB, anon-rss:1155348kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:4048kB oom_score_adj:0
Sun 2023-08-06 17:02:26.033648 UTC [s=0a241a1eea904cba8b086fb666464281;i=646b2;b=115640ac81d6499ba17889292ef4f7ed;m=3afa9bb0139;t=60244181fcff0;x=9a8f1c9c802e5ddf]
    MESSAGE=Out of memory: Killed process 1420526 (java) total-vm:4924632kB, anon-rss:938100kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:3580kB oom_score_adj:0
Sat 2023-08-12 18:10:46.316361 UTC [s=9b5fec1d918b471899cae8eb6ecd3bd5;i=165;b=115640ac81d6499ba17889292ef4f7ed;m=429512c9a93;t=602bdbf916949;x=622d61f11160161d]
    MESSAGE=Out of memory: Killed process 1750099 (java) total-vm:5016428kB, anon-rss:978324kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:3852kB oom_score_adj:0
Sun 2023-12-10 09:06:56.616284 UTC [s=543ef944f8a14b3bb3c890eee4d463a3;i=32fb;b=bf824447d3564bc5832f5644e0d4df54;m=1f8a5b7e75;t=60c2423a4b35c;x=c34bdf082dd03781]
    MESSAGE=Out of memory: Killed process 2127 (java) total-vm:5133344kB, anon-rss:1105880kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:3940kB oom_score_adj:0
Tue 2023-12-12 19:00:46.431373 UTC [s=7f9871a523404eadbc3c913792d80d36;i=3820;b=75e629d832614f9ea0f01efbc27ca35a;m=1c5093a434;t=60c54ab07488d;x=357cb725c26a853c]
    MESSAGE=Out of memory: Killed process 2386 (java) total-vm:5147856kB, anon-rss:1104752kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:3980kB oom_score_adj:0
Tue 2023-12-19 10:51:39.240301 UTC [s=0209fe4e7be9410aafc124d2fb3e54e1;i=5d2c;b=d6d7badf054041db8bb4e3074e6b60a2;m=18d9e48a6d;t=60cdaa6b04f6d;x=fab3a1438b2fdc0]
    MESSAGE=Out of memory: Killed process 2402 (java) total-vm:5114456kB, anon-rss:1083268kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:3904kB oom_score_adj:0

So i will just start with my Pi4 with 4gb to see how its going back with OH4.0.4.
Thanks a lot! Hopefully my system keeps running after this hardware upgrade

2 Likes
16

Yes that clearly shows it’s killing openHAB due to lack of memory!

I ran into similar issues when running too many containers on a Pi3 with only 1GB in the past. :wink:

1 Like
17

RPIs do get squirrelly when pushed too hard - I usually keep $ top open to check CPU and RAM usage to determine what to not run and maybe not include in the docker image.

1 Like
18

Was new to me. With the 4gb raspi until now I just come around 2gb used ram.

I will check in grafana how high it will go when it’s running for a longer time.

19

Just be careful how you read the free RAM values. People can panic when they see the free value constantly going down, but this is ok. To learn more read here:

AVAILABLE vs FREE