I have a standard openhabian setup up and running with some devices in my house.
Platform information:
Hardware: Raspberry Pi 3 b+
OS: openhabian
openHAB version: 2.3 stable
Recently,i have added some http items to address my hue dimmer switches and sensors.
My http.items has about 50 items with refresh rates ranging from 5000 to 60000. My http.cfg is smaller, about 30 items, with corresponding refresh intervals.
Now my http refresh service is using about 50 percent of the overall CPU time and the Java process iss almost always above 100 percent.
Once I hide the http.items the system goes back to normal. Otherwise I have a very high load average.
Could it be that this comes from items in the http.items file that are not registered in the http.cfg?
If that’s not a possible issue, what can I do to reduce the load by the http binding. Are 50 items already too much?
Some of the items like buttons and motion sensors just need a certain refresh rate (5000 in this case).
I have defined the items as posted in the tutorial on this forum.
just a short answer without a real solution. I am experiencing the same thing in my environment. I will post some test results / numbers from my setup later this days (or tomorrow).
That seems like way more threads polling HTTP than expected. While there may be a bug here and that needs to be pursued, you should also look for ways to reduce that number. For example, if you have multiple Items that can retrieve their value from a single HTTP request, use the cache configuration. The binding will fetch the page only once and the Items will look at that cached version to populate themselves. Do not create a separate cache config or a separate Item that polls the URL themselves. See Comprehensive Wunderground using HTTP Binding Example for an example of using just two HTTP requests to populate a dozen Items.
Okay, thanks for the hint. I am already using those HTTP cache items to reduce the polling. I have the slight feeling that not only the HTTP requests are responsible for the high CPU load but more or less the internal handling of the cached data OR applying the necessary transformations an the data to get the values for the items.
Currently I am using 3 Philips Hue Motion Sensors, 1 Philips Hue Dimmer Switch, and 2 or 3 other URLs to retrieve data from. My setup looks like proposed in this tutorial:
I am really looking forward to switch away from this setup and hope to see the Hue binding to support the sensor API in the near future.
Based on that tutorial I would agree, the problem appears to be coming from either the caching or the transformations. I’d look at the transformations first. Is it feasible to experiment with an alternative like REGEX?
Is the JSON returned by the Hue particularly large? If so the problem might be the continued reparsing of the same JSON for each Item. I also don’t know what the overhead is of spinning up a JS transform for a lot of Items. Maybe that is a problem.
I wish there were easy access to a JSON parser in Rules DSL. If there were one could parse the JSON only once in a Rule and postUpdate all the values to all the Items. That should significantly reduce the overall load on the CPU. But parsing JSON in Rules DSL is a HUGE pain.
I don’t suppose the Hue supports returning the values as XML?
The same is true for my setup! Same tutorial and structure.
Maybe it’s possible to request the whole sensor json partial from hue (it is possible with the web api). Then one could parse the sensors themselves from a single http request. But the JSON would be quite a bit larger I guess.
I can try this on the weekend and see what’s happening. The problim is probably that the sensor adressing will need to go into the transformation script, or am I wrong?
Absolutely. I did. Parsing the JSON string every time is a very painful payload. The JS transformation is the only possible option for my use cases. It allows me not only to access the properties and read the values but additional convert / map them to something different. For example the API returns the presence status as binary (0/1) but I would like to have a valid state for a Switch item like ON/OFF. Or the last change values are returned in GMT timezone. I have to convert it to my local timezone (DST). None of the other transformation services can do that. The only way I can think of is chaining transformations. Which is not possible too.
No chance. It is JSON only.
I tried that too. The result was a very large and uncomfortable rule. And yes, you have to put and evaluate the different sensor ids inside your rule. It is possible, but cost a lot of effort to archive. Another drawback is that you cannot use different polling intervals anymore. Meaning you have to request the whole JSON every time and update all of your items every time. For a real time motion sensor tracking you need at least a refresh frequency of 1 second. Even my tutorial just uses 3 seconds which results in a not very reliable presence detection.
Thanks for the insight and the shared experience! My thread statistics looks similar (50% being http refresh service CPU time).
I can understand that the transformation may be intense with the number of items and amount of content processed. But I have the feeling it has not been like this all the time, has it?
My number of requests has not been raised by much in the last week (just those 3 dimmer switches with 5s refresh rate) and yet, the performance has decreased immensely. Could something else have induced that change? Can we get debug output of the transformation process and the http request itself?
To be honest, I don’t know. For me it was slow the whole time. And as you are still on OH2.3 it must had been introduced before.
Yes, that is possible. You have to enable DEBUG logging for the two services: org.openhab.binding.http and org.eclipse.smarthome.transform.javascript.
There already is a contribution for it. But unfortunately the author got lost - somehow. I pinged him yesterday and to tell him I will start to finalize his work. Hopefully I can finish ib the next weeks.
I asked the question as I was involved in some of the discussion on github. Was wondering. Why it was never finished. Guess a missing programmer makes sense!
currently I am trying to debug a similar problem. How did you obtain the detailed thread CPU usage statistics? All the usual Linux tools I tried show the threads of the openhab process only with the name „openhab“, so that I can not figure out, which binding is causing the problem.