OpenHAB/Raspberry stops working after 2-3 weeks running

Terra · November 27, 2017, 8:09pm

Platform information:
- Hardware: RaspberryPi3
- OS: OpenHabian (latest)
- Java Runtime Environment: OpenHabian (latest)
- openHAB version: 2.1
Issue of the topic:

I have the issue that randomly after 2-3 weeks it seems my OpenHab installations gets terribly slow in showing the webpages and then later on it does not server webpages anymore and the whole RaspberryPi3 cannot be accessed through ssh anymore. I then need to restart the PI by unplugging the usb power.

I am not a Linux or OpenHabian guru, so what and where would I need to check to find the reason and a solution for this issue?
What systemlogs would give me clues what is going on and where would I find them in OpenHabian?

thanks!

kevin · November 27, 2017, 8:58pm

Its probably a memory leak or 2 somewhere, and you could spend ages looking for it / them. Then updates will arrive to either Raspbian or OH that will likely introduce other time based problems to find.
To avoid this I just reboot my Pi’s once a week.with a cron command. I also reboot my ESP8266 devices every 21 - 28 days depending on their load as they too stop responding.
I’ve done this since the OH 1.7 days and it works well

murk_van_rooijen · November 28, 2017, 6:38pm

Interesting solution to a problem I also encounter. Could you share the cron to automatic reboot my Raspberry 3 once a while without a hazzle ?

job · November 28, 2017, 9:22pm

I don’t think scheduled reboots are a solution. If you want to know what’s going on, you may look at your logs.

If you are running openHABian, you should enable frontail, if not already done.

sudo openhabian-config

Afterwards you can review the openHAB logs with pointing your browser to

http://<ip-of-pi>:9001

Then you can track down the problem.

kevin · November 28, 2017, 9:30pm

Sure, I have 2 rules that run, one that sets the uptime to zero after a reboot and then increments it every minute, and another that does the actual reboot. I use the Friendly Uptime in the sitemap so I can see how long its been up for. You could just modify the second rule if you want to reboot it every week at the same time, but as I do a lot of work on it I only reboot after 7 calendar days rather than on a schedule

Items:

String USStartedAt
String USFriendlyUptime

Rules:

import org.joda.time.DateTime
var boolean RebootDue = false

rule "calculate Uptime"
when Time cron "0 0/1 * * * ?" then 	{	
if (USStartedAt.state == NULL)	
{	var DateTime StartedTime = (now)
	USStartedAt.sendCommand(StartedTime.toString)
	logInfo("Timers", "USStartedAt " +USStartedAt.state)
}	else	{
	var DateTime BegunAt = parse(USStartedAt.state.toString)
    Uptime = (now.getMillis() - BegunAt.getMillis())/60000
if (Uptime > 10080)	{	RebootDue = true	}
var String UT
var days = Uptime/1440
Uptime = Uptime - days*1440
var hour = Uptime/60
Uptime = Uptime - hour*60
var min = Uptime
UT = days+ "d "+ hour + "h "+ min + "m"
USFriendlyUptime.postUpdate(UT)
}


rule "Reboot upstairs"
when Time cron "0 20 11 ? * *" then	if (RebootDue)	{	executeCommandLine('"sudo" "reboot"', 5000)	}
end

murk_van_rooijen · December 2, 2017, 10:59am

Thanks a lot Kevin. Probably it will take a while until the Raspberry restarts on its own for the first time, because I am still implementing a lot of new things. At least I can now monitor when I worked for the last time

note: I had a view errors (missing end and var in the rules, below my version)

import org.joda.time.DateTime
var boolean RebootDue = false

rule "calculate Uptime"
when 
    Time cron "0 0/1 * * * ?" 
then 	
if (USStartedAt.state == NULL)	
    {	var DateTime StartedTime = (now)
	    USStartedAt.sendCommand(StartedTime.toString)
	    logInfo("Raspberry reboot", "USStartedAt " +USStartedAt.state)
    }	
else	
    {
	var DateTime BegunAt = parse(USStartedAt.state.toString)
        var Uptime = (now.getMillis() - BegunAt.getMillis())/60000
        if (Uptime > 10080)
        	{	RebootDue = true	 }

        var String UT
        var days = Uptime/1440
        Uptime = Uptime - days*1440
        var hour = Uptime/60
        Uptime = Uptime - hour*60
        var min = Uptime
        UT = days+ "d "+ hour + "h "+ min + "m"
        USFriendlyUptime.postUpdate(UT)
    }
end



rule "Reboot Raspberry"
when 
    Time cron "0 20 11 ? * *" 
then	
    if (RebootDue)	{	executeCommandLine('"sudo" "reboot"', 5000)	}
end

Pezzi42 · December 2, 2017, 11:11am

Hi
Regarding the original issue - have you checked your SD card?
I had similar issues when using raspberry pi and it alsways turned out to be a corrupted SD card. They are just not made for the amount of logs that OH is writing,

I switched to a different machine with a regular hard drive and haven’t had any issues since.

kevin · December 3, 2017, 7:38am

Glad you got it sorted, I must have missed a couple of bits when I edited my original rule which provides reading friendly uptimes on a few other devices too.

kevin · December 3, 2017, 7:39am

Very good advice - I have my logs on tmpfs, and my persistence which I copy onto the file system every time I reboot. Anyone running OH in an SD card is advised to do this, as well as move anything else that changes frequently to tmpfs if possible

Terra · December 4, 2017, 10:11am

Thanks for the hint.
I will try to check the SD card for errors.
I now also configured an automatic reboot every day at 5am, however I am having some issues with persistence because of that I need to work on now, one lamp in my bedroom turns on everyday at 5am after the reset…

If SD card corruption seems to be such a common issue and if OpenHab really writes so much that it destroys SD cards or the filesystem I would expect that the default installation of OpenHabian for the RPi includes some measures to prevent or resolve this, in the end it was created especially for that platform?

Also I have friends running PLEX Media Servers on a RPi and I think they are producing way more reads and writes than openhab…

kevin · December 12, 2017, 9:11pm

The things I know about are log file writes, persistence file writes, and on Linux by default the last file access time is turned on, so every time any file is read it incurs a disk write. On a default installation of OH every state change is published into events.log which can be a lot of disk writes on a busy system.
I managed just under 2 years on my last 32GB card with last file access time turned off and the logs moved to RAM. I’m hoping for longer this time around with persistence in RAM between reboots

RolandS · March 6, 2018, 3:26pm

I have the same issues on 3 different installations. Always a Pi3 is used with openhabian 1.3 or 1.4 and many bindings installed.
After 3 days or up to 3 weeks the Pi stops working, OpenHAB goes offline, even a SSH login is not possible. And today I saw the second time, that the SD card was full. I had this at least once, where I saw this before I shutdowned the Pi, backupped the SD card and started it again and the problem was gone! Again 3.x GB free disk space available! o.O

And today the same, the Pi was running, SSH was possible, but openhab stopped working (502 Bad gateway error of NGINX). And again there was
After some investigation I found out, the problem is the following folder
“/var/lib/openhab2”
Before restarting openhab: 3.3G
After restarting openhab: 125M

After openhab2 restart:
Root = Free: 3.28GB (47%), Used: 3.65GB (53%), Total: 7.25GB

apica · March 8, 2018, 9:41am

Same issue here.

Installed the latest version (2.2.0-1), and with just 16 hours working, the web server page (openhabianpi:8080/start/index) doesn’t load properly.

The log view shows this:

2018-03-08 10:19:53.831 [WARN ] [eclipse.jetty.servlet.ServletHandler] - /start/js/jquery.min.js

java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:219) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:664) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.ops4j.pax.web.service.jetty.internal.ResourceServlet.service(ResourceServlet.java:296) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [31:javax.servlet-api:3.1.0]

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) [88:org.eclipse.jetty.servlet:9.3.22.v20171030]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) [88:org.eclipse.jetty.servlet:9.3.22.v20171030]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [85:org.eclipse.jetty.security:9.3.22.v20171030]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [88:org.eclipse.jetty.servlet:9.3.22.v20171030]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.Server.handle(Server.java:534) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at java.lang.Thread.run(Thread.java:748) [?:?]

Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) ~[79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) ~[79:org.eclipse.jetty.io:9.3.22.v20171030]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:?]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:?]

... 1 more

2018-03-08 10:19:53.860 [WARN ] [org.eclipse.jetty.server.HttpChannel] - 
//openhabianpi:8080/start/js/jquery.min.js

javax.servlet.ServletException: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:88) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.Server.handle(Server.java:534) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at java.lang.Thread.run(Thread.java:748) [?:?]

Caused by: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:219) ~[?:?]

at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:664) ~[?:?]

at org.ops4j.pax.web.service.jetty.internal.ResourceServlet.service(ResourceServlet.java:296) ~[?:?]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[?:?]

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) ~[?:?]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) ~[?:?]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[?:?]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[?:?]

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[?:?]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) ~[?:?]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) ~[?:?]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284) ~[?:?]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) ~[?:?]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) ~[?:?]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) ~[?:?]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[?:?]

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) ~[?:?]

... 13 more

Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) ~[?:?]

at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) ~[?:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:?]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:?]

... 1 more

2018-03-08 10:20:23.747 [WARN ] [eclipse.jetty.servlet.ServletHandler] - /start/css/bootstrap.min.css

java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:219) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:664) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.ops4j.pax.web.service.jetty.internal.ResourceServlet.service(ResourceServlet.java:296) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [31:javax.servlet-api:3.1.0]

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) [88:org.eclipse.jetty.servlet:9.3.22.v20171030]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) [88:org.eclipse.jetty.servlet:9.3.22.v20171030]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [85:org.eclipse.jetty.security:9.3.22.v20171030]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [88:org.eclipse.jetty.servlet:9.3.22.v20171030]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.Server.handle(Server.java:534) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at java.lang.Thread.run(Thread.java:748) [?:?]

Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) ~[79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) ~[79:org.eclipse.jetty.io:9.3.22.v20171030]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:?]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:?]

... 1 more

2018-03-08 10:20:23.798 [WARN ] [org.eclipse.jetty.server.HttpChannel] -     
//openhabianpi:8080/start/css/bootstrap.min.css

javax.servlet.ServletException: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:88) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.Server.handle(Server.java:534) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at java.lang.Thread.run(Thread.java:748) [?:?]

Caused by: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:219) ~[?:?]

at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:664) ~[?:?]

at org.ops4j.pax.web.service.jetty.internal.ResourceServlet.service(ResourceServlet.java:296) ~[?:?]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[?:?]

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) ~[?:?]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) ~[?:?]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[?:?]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[?:?]

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[?:?]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) ~[?:?]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) ~[?:?]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284) ~[?:?]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) ~[?:?]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) ~[?:?]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) ~[?:?]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[?:?]

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) ~[?:?]

... 13 more

Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) ~[?:?]

at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) ~[?:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:?]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:?]

... 1 more

2018-03-08 10:21:24.062 [WARN ] [eclipse.jetty.servlet.ServletHandler] - /start/js/jquery.min.js

java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:219) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:664) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.ops4j.pax.web.service.jetty.internal.ResourceServlet.service(ResourceServlet.java:296) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [31:javax.servlet-api:3.1.0]

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) [88:org.eclipse.jetty.servlet:9.3.22.v20171030]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) [88:org.eclipse.jetty.servlet:9.3.22.v20171030]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [85:org.eclipse.jetty.security:9.3.22.v20171030]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [88:org.eclipse.jetty.servlet:9.3.22.v20171030]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.Server.handle(Server.java:534) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at java.lang.Thread.run(Thread.java:748) [?:?]

Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) ~[79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) ~[79:org.eclipse.jetty.io:9.3.22.v20171030]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:?]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:?]

... 1 more

2018-03-08 10:21:24.106 [WARN ] [org.eclipse.jetty.server.HttpChannel] - 
//openhabianpi:8080/start/js/jquery.min.js

javax.servlet.ServletException: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:88) [191:org.ops4j.pax.web.pax-web-jetty:6.0.7]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.Server.handle(Server.java:534) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [87:org.eclipse.jetty.server:9.3.22.v20171030]

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [79:org.eclipse.jetty.io:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [90:org.eclipse.jetty.util:9.3.22.v20171030]

at java.lang.Thread.run(Thread.java:748) [?:?]

Caused by: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:219) ~[?:?]

at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:664) ~[?:?]

at org.ops4j.pax.web.service.jetty.internal.ResourceServlet.service(ResourceServlet.java:296) ~[?:?]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[?:?]

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) ~[?:?]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) ~[?:?]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[?:?]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[?:?]

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[?:?]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) ~[?:?]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) ~[?:?]

at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284) ~[?:?]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) ~[?:?]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) ~[?:?]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) ~[?:?]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[?:?]

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) ~[?:?]

... 13 more

Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) ~[?:?]

at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) ~[?:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:?]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:?]

... 1 more

(The log is way more extensive…)

But also, the system doesn’t seem to be overloaded:

###############################################################################
################# openHABianPi ################################################
###############################################################################
##             Ip = 192.168.1.141
##        Release = Raspbian GNU/Linux 9 (stretch)
##         Kernel = Linux 4.9.59-v7+
##       Platform = Raspberry Pi 3 Model B Rev 1.2
##         Uptime = 0 day(s). 16:22:43
##      CPU Usage = 0 % avg over 4 cpu(s)  (4 core(s) x 1 socket(s))
##       CPU Load = 1m: 0.00, 5m: 0.00, 15m: 0.00
##         Memory = Free: 0.12GB (14%), Used: 0.82GB (86%), Total: 0.95GB
##           Swap = Free: 0.09GB (100%), Used: 0.00GB (0%), Total: 0.09GB
##           Root = Free: 12.05GB (86%), Used: 1.86GB (14%), Total: 14.53GB
##        Updates = 0 apt updates available.
##       Sessions = 2 sessions
##      Processes = 137 running processes of 32768 maximum processes
###############################################################################

Thank you

BulletProofFool1 · September 4, 2018, 10:51am

Do you have any details on how to set this up?

The-Elk · September 18, 2018, 2:42pm

@RolandS Did you fixed your problem?
I have the same issue since end of april.
My setup:
openhabian 1.3 stable on RASPI 3 B and it runs on SSD without SD card.

Openhab stops, no SSH login possible, the frontail log website is showing last log entry. I also found no logfile with relevant log entries under /var/log/ xxx. I also see no read or write activities on my SSD at this time any more.

openhab stops 20:08

CPU load:

RAM:

SSD:

I only found this strange information in the openhab event.log

2018-09-17 20:08:40.873 [vent.ItemStateChangedEvent] - ZWaveS22W changed from 44 to 45
2018-09-17 20:08:48.074 [vent.ItemStateChangedEvent] - ZWaveS28W changed from 0.669 to 0.564
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   2018-09-17 22:18:01.218 [.ItemChannelLinkAddedEvent] - Link 'ForecastTime-weatherunderground:weather:15da79fae60:forecastToday#forecastTime' has been added.
2018-09-17 22:18:01.222 [.ItemChannelLinkAddedEvent] - Link 'OGSchlafTBattery-max:thermostat:KEQ0566641:KEQ0402531:battery_low' has been added.

My only option is cut the power. Openhabian restarts and works as if nothing had happened, for 1-3 weeks.

BulletProofFool1 · March 19, 2019, 12:12pm

I eventually gave up on my RasPi, installed Linux server on a VirtualBox VM running on a desktop and have no longer had this issue.

lazloman · March 30, 2019, 1:28pm

if you’re running from an SSD card, you may be looking at file system corruption. I had problems like this until I added a monthly reboot with these boot parameters added to
/boot/cmdline.

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=0a79d5a7-02 rootfstype=ext4 elevator=deadline fsck.repair=yes fsck.mode=force rootwait

These two parameters check for file system errors and repairs any it finds:

fsck.repair=yes fsck.mode=force

The-Elk · April 1, 2019, 5:46am

I am not 100% sure, but after deactivating the Alexa Echo Control Binding, my Openhabian instance runs without problems.

@michi : do you know the phenomenon?

lazloman · April 1, 2019, 11:53pm

No, I’ve not seen that.

michi · April 9, 2019, 4:35pm

Yes, I assume you use the released version. This have a memory leak if the internet connection is down. Please use my beta version from here or upgrade to the openhab 2.5 beta.