[Solved] Too many open files

  • Platform information:
    • Hardware: Intel Core i5 VM on QNAP, 2 GB RAM, 40 GB disk
    • OS: Ubuntu 7.3.0-16ubuntu3
    • Java Runtime Environment:
    • openHAB version: 2.4.0 milestone build

I have a bunch of Raspberry Pi cameras which post some status updates through REST to Openhab each minute (CPU temperature, camera status, …). Some days ago I realized that these shell scripts are “stuck” somehow (checking a lock file in /tmp, the script terminates and sends a mail in case the lock file does still exist). Initially I thought the curl post to the rest interface uses too much time and thus added the “–connect-timeout 3” option to these curl commands, however this did not help. About 6-8 hours after restarting openhab (tried both service restart and server reboot) it doesnt respond anymore to REST. Further investigation shows a ton of open files from openhab process, see attached. Which steps can I perform to identify the root cause for these ?

ps -edaf output (764 is the openhab process): https://www.dropbox.com/s/ihw71cdzci95ffm/ps%20-edaf.txt?dl=0

lsof output: https://www.dropbox.com/s/s48cn4egk9ac0gn/lsof.txt?dl=0

lsof -p 764 output: https://www.dropbox.com/s/ajjwjvoyq85hdia/lsof%20-p%20764.txt?dl=0

These are the curl commands I issue for each of my cameras:

curl --connect-timeout 3 --silent --header "Content-Type: text/plain" --request POST --data $MJPEG_STATUS http://KGTec1Openhab2Server:8080/rest/items/`hostname`Status
curl --connect-timeout 3 --silent --header "Content-Type: text/plain" --request POST --data $CPU_TEMP http://KGTec1Openhab2Server:8080/rest/items/`hostname`CPUTemp
curl --connect-timeout 3 --silent --header "Content-Type: text/plain" --request POST --data $WIFI_SIGNAL http://KGTec1Openhab2Server:8080/rest/items/`hostname`WifiSignal

Issue solved - Investigating the log files after another restart showed the following issues which were fixed one by one, after this i dont have this issue anymore:

  • stuck threads from the Miio binding, fixed by uninstalling the Miio binding - however this was not the reason as afterwards the open files error still ocured
  • Z-Wave binding not able to access /usb/ttyxxx due to USB not “mounted” in Qnap VM
  • Login errors from Amazon Echo Control binding asking for re-login