Too many open files - CLOSE_WAIT connections build up

Setup, Configuration and Use Runtime
1

After some uptime, openhab starts printing a lot of these in the log:
2017-02-09 14:04:02.580 [WARN ] [eclipse.jetty.server.ServerConnector] - java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)[:1.8.0_121] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)[:1.8.0_121] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)[:1.8.0_121] at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)[80:org.eclipse.jetty.server:9.2.19.v20160908] at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)[80:org.eclipse.jetty.server:9.2.19.v20160908] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)[83:org.eclipse.jetty.util:9.2.19.v20160908] at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)[83:org.eclipse.jetty.util:9.2.19.v20160908] at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]

Open files limit for openhab is 4096, which should be enough.

However, netstat show there’s a lot of CLOSE_WAIT connections to openhab:8080. Here a FIN is received, but the connection is not closed by openhab/jetty?

Now special activity other than a web client idling on a sitemap (Basic UI).

Java 1.8.0_121
Openhab 2.1.0.201701292153

Any thoughts?

2

Same problem for me, with the same Java version (problem appeared also with previous Java versions).
@mtryfoss: How do you get the CLOSE_WAIT sockets in netstat? Or are you checking the listening sockets (netstat -l)?

3

Just a simple “netstat -an | grep CLOSE_WAIT”.

I’m not so into java, but a forum post suggested adding the following to /etc/sysctl.conf (followed by a sysctl -p and OH restart):

net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_intvl = 2
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_time = 1800

It seems to help a bit.

4

i have same problem,

net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_intvl = 2
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_time = 1800

this does not fix it.

any ideas ?

java version "1.8.0_131"
Java™ SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot™ 64-Bit Server VM (build 25.131-b11, mixed mode)

latest OH2 Snapshot #934

i have raised openfiles limit
open files (-n) 65535

but does not help

5 
tcp6       1      0 192.168.0.201:58198     192.168.0.184:37904     CLOSE_WAIT  7806/java
tcp6       0      0 192.168.0.201:8080      192.168.0.165:62252     ESTABLISHED 7806/java
tcp6       0      0 127.0.0.1:51328         127.0.0.1:3306          ESTABLISHED 7806/java
tcp6       0      0 192.168.0.201:55260     139.162.171.235:443     ESTABLISHED 7806/java
tcp6       1      0 192.168.0.201:8080      192.168.0.165:62288     CLOSE_WAIT  7806/java
tcp6       1      0 192.168.0.201:38312     192.168.0.49:8080       CLOSE_WAIT  7806/java
tcp6       1      0 192.168.0.201:34348     192.168.0.99:32469      CLOSE_WAIT  7806/java
tcp6       1      0 192.168.0.201:47866     192.168.0.118:8080      CLOSE_WAIT  7806/java
tcp6       1      0 192.168.0.201:36140     192.168.0.119:8080      CLOSE_WAIT  7806/java
tcp6       1      0 192.168.0.201:60314     192.168.0.150:2870      CLOSE_WAIT  7806/java
tcp6       1      0 192.168.0.201:60378     192.168.0.149:2870      CLOSE_WAIT  7806/java
tcp6       0      0 127.0.0.1:8101          127.0.0.1:41350         ESTABLISHED 7806/java
tcp6       0      0 192.168.0.201:38824     192.168.0.187:8008      ESTABLISHED 7806/java
tcp6       0      0 192.168.0.201:8080      192.168.0.165:61940     ESTABLISHED 7806/java
tcp6       1      0 192.168.0.201:8080      192.168.0.165:62365     CLOSE_WAIT  7806/java
tcp6       0      0 192.168.0.201:8080      192.168.0.165:62671     ESTABLISHED 7806/java
tcp6       0      0 192.168.0.201:8080      192.168.0.165:62244     ESTABLISHED 7806/java
tcp6       1      0 192.168.0.201:8080      192.168.0.165:62374     CLOSE_WAIT  7806/java
tcp6       1      0 192.168.0.201:8080      192.168.0.165:62372     CLOSE_WAIT  7806/java
tcp6       0      0 192.168.0.201:8080      192.168.0.165:61850     ESTABLISHED 7806/java
tcp6       1      0 192.168.0.201:47930     192.168.0.98:32469      CLOSE_WAIT  7806/java
tcp6       1      0 192.168.0.201:8080      192.168.0.165:62367     CLOSE_WAIT  7806/java
tcp6       0      0 192.168.0.201:8080      192.168.0.165:62672     ESTABLISHED 7806/java
tcp6       0      0 192.168.0.201:40852     192.168.0.200:8008      ESTABLISHED 7806/java
tcp6       0      0 192.168.0.201:8080      192.168.0.165:62673     ESTABLISHED 7806/java
tcp6       1      0 192.168.0.201:43148     192.168.0.216:33650     CLOSE_WAIT  7806/java
tcp6       1      0 192.168.0.201:8080      192.168.0.165:62347     CLOSE_WAIT  7806/java

current status

habpanel f.e does not respond, i think also habpanel is cousing the issue.

i have about
150 items
30 rules
5-10 clients connected per habpanel
about ~10 charts in habpanel connecting to mysql (persistence)

im trying to find a solution

6

Hy!

I am using habpanel on a tablet. After closing habpanel, there is no error.
The error must be from Habpanel :thinking:

7

I also get this error with Basic UI so i think its a “Server” Error

8

I have the same problem.
Here is a detailed description how I reproduce this error:

link

9

For debugging please provide which files are actually open, the 27 connection are not the complete problem imho.