openHABian / Services freezes

Hi Guys,

i´ve an issue since upgrading to OH2.1.

In details:

Platform = Raspberry Pi 3 Model B Rev 1.2
OS: openHABian v1.3-299(533a314) (NOT ua-netinst)
Services: openHAB2.1, Homegear, Grafana, influxDB
Updates = 0 apt-get updates available.

I´ve used the openHABian Configuration Tool to upgrade OH2 to OH2.1. Fixed permission also with the openHABian Configuration Tool. So everithings look fine.
Now, since a few day i randomly cannot connect to the Services. When i try to connct via SSH, i get the promt for username and password. After entering the password…nothing happens.

I can ping the PI.

Today at 15:18 i´ve restarted the system via power cut.

Here some cutouts from the logs:

openhab.log

2017-07-13 13:58:18.259 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.eclipse.smarthome.model.rule.runtime.internal.engine.RuleEngineImpl@1112b97' takes more than 5000ms.
2017-07-13 13:58:18.378 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.eclipse.smarthome.model.rule.runtime.internal.engine.RuleEngineImpl@1112b97' takes more than 5000ms.
2017-07-13 13:58:19.474 [WARN ] [org.openhab.io.net.http.HttpUtil    ] - Method failed: HTTP/1.1 500 Internal Server Error
2017-07-13 13:58:27.892 [ERROR] [.script.engine.ScriptExecutionThread] - Rule 'Stromverbrauch': java.lang.Number
2017-07-13 15:18:44.554 [INFO ] [marthome.model.script.praesenz.rules] - Jemand ist Zuhause

So it seems, something (or nothing) happend between 13:58 and 15:18.

syslog, dmesg, debug, kern.log, messages didnot really show errors.

Any ideas?
THX
Cyrelian

The ua-netinst is depreciated, see here

and here

Hi sihiu,

sorry my fault, i dont have ua-netinst.
OS: openHABian v1.3-299(533a314) (NOT ua-netinst)

best regards

Hello, I have exactly the same problem - after approx. one day, the openhabian installation freezes. This means: no access via web or app, but I can connect to the RPi via SSH. Although “top” shows the system is almost fully idle, it takes ages to e.g. enter the karaf console (approx. 30-45 seconds waiting for the “passwort” prompt coming up).

All logs I have looked at, don’t show any hint - no error messages, no problems, just regular messages.

The system is completely unstable and unusable since the upgrade to 2.1.

@ThomDietrich: Are we the only two ones who experience this behaviour?

1 Like

But there are “takes more than 5000ms” messages in my log. This is new; earlier I only had them at the beginning for my Harmony Hub - but not during the regular operation:

2017-07-15 14:07:45.930 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.eclipse.smarthome.core.internal.items.ItemUpdater@b38333' takes more than 5000ms.
2017-07-15 14:09:38.777 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.eclipse.smarthome.core.internal.items.ItemUpdater@b38333' takes more than 5000ms.
2017-07-15 14:11:20.654 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.eclipse.smarthome.io.monitor.internal.EventLogger@1746b8' takes more than 5000ms.
2017-07-15 14:11:32.999 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.eclipse.smarthome.core.internal.items.ItemUpdater@b38333' takes more than 5000ms.
2017-07-15 14:12:08.033 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.eclipse.smarthome.io.monitor.internal.EventLogger@1746b8' takes more than 5000ms.
2017-07-15 14:12:29.550 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.eclipse.smarthome.io.monitor.internal.EventLogger@1746b8' takes more than 5000ms.
2017-07-15 14:12:41.111 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.openhab.core.events.internal.EventBridge@f2de2b' takes more than 5000ms.
2017-07-15 14:12:51.533 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.openhab.core.events.internal.EventBridge@f2de2b' takes more than 5000ms.
2017-07-15 14:13:34.581 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.eclipse.smarthome.core.thing.internal.ThingManager@37d1d4' takes more than 5000ms.
2017-07-15 14:13:44.734 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.openhab.core.events.internal.EventBridge@f2de2b' takes more than 5000ms.
2017-07-15 14:14:15.917 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.eclipse.smarthome.core.internal.items.ItemUpdater@b38333' takes more than 5000ms.
2017-07-15 14:14:55.504 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.eclipse.smarthome.core.internal.items.ItemUpdater@b38333' takes more than 5000ms.

I don’t know what could be wrong, but let’s try some things:
first of all, stop OH2.1 to see if the Raspberry becomes more responsive (or the freezes are due to some other process)

sudo systemctl stop openhab2

Then, install htop (a better version of top) and check the situation.

sudo apt-get install htop
sudo htop

check cpu/mem/swap usage (press F6 and sort by percent_cpu and/or percent_mem)

It wouldn’t hurt to stop OH2.1, clean /var/lib/openhab2/cache/ & /var/lib/openhab2/tmp/ and start again to see if things improve.

Ps: I assume that no overclocking is enabled :slight_smile:

1 Like

@Dim, thank you for your hints!
When I start htop I see the following:

What does that mean? Besides openhab, also mosquitto seems to have a big party on my poor RPi…
I’ll try to stop & clean the directories now…

Regarding overclocking: Not active on my RPi

oops… the openHAB process is chewing your rPi’s CPU alive :slight_smile:

interesting… it consumes all of the 4th processing core capacity on the ARM Cortex-A53…

After stopping openhab service, everything calmed down:


Strange: mosquitto is also not active anymore…hmmm

hmm… maybe… a subscribe/publish loop?

Have you configured MQTT Event Bus integration? (/etc/openhab2/services/mqtt-eventbus.cfg)

I haven’t changed anything since the update from 2.0 to 2.1 - and before everything worked fine. I guess, I’ll try to reinstall mosquitto again, just to exclude this as possible source of my problems.

Here’s my full log until everything starts to freeze:

1 Like

quick review: disable MqttPersistenceService
it doesn’t do much and most likely is causing problems… reviewing the log more now

1 Like

Now it’s even worse…LOL (eating up all four cpus)

MQTT persistence is deactivated now; eventbus-CFG is available but not activated:

# Name of the broker as it is defined in the openhab.cfg. If this property is not available, no event bus MQTT binding will be created.
#broker=

# When available, all status updates which occur on the openHAB event bus are published to the provided topic. The message content will 
# be the status. The variable ${item} will be replaced during publishing with the item name for which the state was received.
#statePublishTopic=

# When available, all commands which occur on the openHAB event bus are published to the provided topic. The message content will be the 
# command. The variable ${item} will be replaced during publishing with the item name for which the command was received.
#commandPublishTopic=

# When available, all status updates received on this topic will be posted to the openHAB event bus. The message content is assumed to be 
# a string representation of the status. The topic should include the variable ${item} to indicate which part of the topic contains the 
# item name which can be used for posting the received value to the event bus.
#stateSubscribeTopic=

# When available, all commands received on this topic will be posted to the openHAB event bus. The message content is assumed to be a 
# string representation of the command. The topic should include the variable ${item} to indicate which part of the topic contains the 
# item name which can be used for posting the received value to the event bus.
#commandSubscribeTopic=

did you uninstall the addon?
also remove /etc/openhab2/persistence/mqtt.persist file

I am not sure if this is the root cause of your problems… just making sure

It’s not there:

[16:08:49] openhabian@openHABianPi:/etc/openhab2/persistence$ ls -las
insgesamt 20
4 drwxrwxr-x+  2 openhab    openhabian 4096 Jul 11 20:47 .
4 drwxrwxr-x+ 13 openhab    openhabian 4096 Jul 11 20:58 ..
4 -rw-rw-r--   1 openhabian openhabian  178 Okt 12  2016 exec.persist
4 -rw-rw-r--   1 openhab    openhabian  246 Jan 22 21:40 readme.txt
4 -rw-rw-r--   1 openhabian openhabian  979 Mai 28 22:24 rrd4j.persist
[16:08:53] openhabian@openHABianPi:/etc/openhab2/persistence$

I removed the “mqtt” entry in the persistence section of addons.cfg and in the PaperUI it’s being showed as “not installed”.

what other kind of “interaction” have you configured between openHAB & MQTT?
Items, rules with mqtt action, something else?

Since the mosquitto broker goes crazy also when you fire up openHAB (maybe) there is an issue with that “connection”

can you post the contents of the /etc/openhab2/services/mqtt.cfg file?

CPU is still at 100%?

anything useful inside the mosquitto.log?

That’s the last entries I get on my system before it dies:

2017-07-15 17:12:57.821 [ERROR] [t.mqtt.internal.MqttBrokerConnection] - MQTT connection to broker was lost

Verbindung wurde getrennt (32109) - java.net.SocketException: Connection reset

	at org.eclipse.paho.client.mqttv3.internal.CommsSender.handleRunException(CommsSender.java:153)[197:org.openhab.io.transport.mqtt:1.10.0]

	at org.eclipse.paho.client.mqttv3.internal.CommsSender.run(CommsSender.java:138)[197:org.openhab.io.transport.mqtt:1.10.0]

	at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]

Caused by: java.net.SocketException: Connection reset

	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:115)[:1.8.0_121]

	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)[:1.8.0_121]

	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)[:1.8.0_121]

	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)[:1.8.0_121]

	at org.eclipse.paho.client.mqttv3.internal.wire.MqttOutputStream.flush(MqttOutputStream.java:49)[197:org.openhab.io.transport.mqtt:1.10.0]

	at org.eclipse.paho.client.mqttv3.internal.CommsSender.run(CommsSender.java:117)[197:org.openhab.io.transport.mqtt:1.10.0]

	... 1 more

2017-07-15 17:14:13.915 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.openhab.core.events.internal.EventBridge@140afef' takes more than 5000ms.

2017-07-15 17:14:56.228 [ERROR] [ersey.server.ServerRuntime$Responder] - An I/O error has occurred while writing a response message entity chunk to the container output stream.

2017-07-15 17:14:49.228 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.openhab.core.events.internal.EventBridge@140afef' takes more than 5000ms.

2017-07-15 17:45:53.158 [WARN ] [eclipse.jetty.servlet.ServletHandler] - Error for /rest/events

java.lang.OutOfMemoryError: Java heap space

2017-07-15 17:44:54.063 [WARN ] [eclipse.jetty.servlet.ServletHandler] - 

2017-07-15 17:39:50.604 [WARN ] [eclipse.jetty.servlet.ServletHandler] - Error for /rest/events

java.lang.OutOfMemoryError: Java heap space

2017-07-15 18:14:03.962 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.eclipse.smarthome.core.internal.items.ItemUpdater@383ba9' takes more than 5000ms.

2017-07-15 18:13:35.731 [WARN ] [org.eclipse.jetty.server.HttpChannel] - /rest/events

java.lang.OutOfMemoryError: Java heap space

2017-07-15 18:13:00.452 [WARN ] [org.eclipse.jetty.server.HttpChannel] - Commit failed

java.lang.OutOfMemoryError: Java heap space

2017-07-15 18:12:32.231 [WARN ] [e.jetty.util.thread.QueuedThreadPool] - 

java.lang.OutOfMemoryError: Java heap space

2017-07-15 18:05:05.825 [WARN ] [e.jetty.util.thread.QueuedThreadPool] - 

java.lang.OutOfMemoryError: Java heap space

2017-07-15 18:02:45.429 [WARN ] [eclipse.jetty.servlet.ServletHandler] - Error for /rest/events

java.lang.OutOfMemoryError: Java heap space

2017-07-15 17:58:43.333 [WARN ] [e.jetty.util.thread.QueuedThreadPool] - 

java.lang.OutOfMemoryError: Java heap space

2017-07-15 19:10:53.530 [WARN ] [e.jetty.util.thread.QueuedThreadPool] - Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$3@1603608 in qtp31802071{STARTED,2<=23<=50,i=0,q=24}

2017-07-15 19:10:16.684 [WARN ] [eclipse.jetty.servlet.ServletHandler] - 

2017-07-15 19:09:39.589 [WARN ] [org.apache.karaf.services.eventadmin] - EventAdmin: Exception during event dispatch [org.osgi.service.event.Event [topic=smarthome] {type=ItemStateChangedEvent, topic=smarthome/items/Badezimmer_Feuchte/statechanged, payload={"type":"Decimal","value":"40.80","oldType":"Decimal","oldValue":"39.40"}} | {org.osgi.service.event.EventHandler, org.eclipse.smarthome.core.events.EventPublisher}={event.topics=smarthome, component.name=org.eclipse.smarthome.core.internal.events.OSGiEventManager, component.id=52, service.id=146, service.bundleid=99, service.scope=bundle} | Bundle(org.eclipse.smarthome.core_0.9.0.b5 [99])]

java.lang.OutOfMemoryError: Java heap space

2017-07-15 19:09:11.443 [WARN ] [ore.internal.events.OSGiEventManager] - Dispatching event to subscriber 'org.eclipse.smarthome.io.monitor.internal.EventLogger@5c3544' takes more than 5000ms.

2017-07-15 19:05:15.907 [WARN ] [org.eclipse.jetty.server.HttpChannel] - Could not send response error 500: java.lang.OutOfMemoryError: Java heap space

2017-07-15 19:00:36.970 [WARN ] [org.eclipse.jetty.server.HttpChannel] - /rest/events

java.lang.OutOfMemoryError: Java heap space

The content of my “mqtt.cfg” is quite unspectacular, I guess:

# MQTT-Transport
mqtt_broker.url=tcp://192.168.1.50:1883
mqtt_broker.clientId=OH2RPi
#<broker>.user=<user>
#<broker>.pwd=<password>
mqtt_broker.qos=0
mqtt_broker.retain=false
#<broker>.async=<async>
#mqtt_broker.lwt=""

Next step: I try to deactivate MQTT binding and lets see, if the system is stable without it…

quick check: what is the output of your: java -version command?

1 Like

openjdk version "1.8.0_121"
OpenJDK Runtime Environment (Zulu Embedded 8.20.0.42-linux-aarch32hf) (build 1.8.0_121-b42)
OpenJDK Client VM (Zulu Embedded 8.20.0.42-linux-aarch32hf) (build 25.121-b42, mixed mode, Evaluation)

1 Like