Looking at the code, I can’t quite see how the Core code can make anything block. But, there’s a difference from before, previously it would only process send one event at a time - and would block all sending OH threads in the meanwhile. Now it doesn’t wait, so events might be sent more rapidly. Is “your side” keeping up with the incoming events? If a message must wait more than 10 seconds before being consumed by the other side, the websocket will be disconnected, just like we see.
Since Python doesn’t really do threads, I don’t know how you’ve solved it, but could it be that something is clogging up the reception of the events?
edit (can’t make new posts until somebody else have replied):
I have gotten what seems to be somewhat interesting results. I have added a lot of debug logging on the OH side to see what happens when, and doing this slows everything down. That in itself has almost gotten rid of the disconnects, and it makes the benchmark report success:
+------------------------------------------------------------------------------+
| HABApp |
+------------------------------------------------------------------------------+
Bench events .. done!
| dur | per sec | median | min | max | mean
rtt idle | 2.484s | 20.13k | 0.0us | 0.0us | 24.7ms | 49.7us
async rtt idle | 6.026s | 723.153 | 1.00ms | 0.55ms | 6.00ms | 1.38ms
+------------------------------------------------------------------------------+
| openHAB |
+------------------------------------------------------------------------------+
Bench item operations ... done!
| dur | per sec | median | min | max | mean
create item | 4.203s | 71.377 | 13.0ms | 9.00ms | 69.0ms | 14.0ms
update item | 6.078s | 49.360 | 19.0ms | 6.99ms | 0.370s | 20.3ms
delete item | 2.259s | 132.782 | 7.00ms | 5.97ms | 13.0ms | 7.53ms
Bench item state update ........ done!
| dur | per sec | median | min | max | mean
rtt idle | 15.0s | 55.877 | 11.0ms | 5.96ms | 1.956s | 17.9ms
async rtt idle | 14.8s | 56.243 | 10.0ms | 7.96ms | 2.649s | 17.8ms
rtt idle item | 0.794s | 165.065 | 6.00ms | 3.96ms | 10.00ms | 6.06ms
async rtt idle item | 4.018s | 90.587 | 4.00ms | 2.96ms | 2.308s | 11.0ms
rtt load (+10x) | 7.925s | 7.950 | 67.0ms | 18.0ms | 2.049s | 0.126s
async rtt load (+10x) | 15.9s | 73.275 | 8.00ms | 5.95ms | 2.319s | 13.6ms
rtt load item (+10x) | 0.0us | 0.000 | 0.0us | 0.0us | 0.0us | 0.0us
async rtt load item (+10x) | 14.2s | 64.609 | 4.03ms | 2.95ms | 2.435s | 15.5ms
Cleanup ... complete
There are however still a few disconnects, but they don’t seem to “bother” the benchmark like it otherwise does. If I slowed down the code even more, my guess is that the disconnects could go away completely. This reinforces my suspicion that the problem is that the Python side doesn’t process the messages quickly enough, or not all of them in a timely fashion. So, this whole problem could be the result of OH now sending messages much quicker.
The OH side uses synchronous sends, so that send failures can be caught and logged. Jetty (the web server “engine” handling the low level websocket stuff) doesn’t support a “send queue” for synchronous sending of messages, so they must all be received within 10 seconds of trying to send them, or the connection is closed.
If I don’t care about logging errors, I could send them asynchronously. Jetty does implement a send queue for async sends as far as I understand, which would give “the other side” more time to get their ducks in a row without the connection being dropped. I’m not sure how best to handle error logging in that case though, but I guess it could be worth a try to make sending async to see if the problem goes away - and if that helps, try to figure out how to log errors.
Another possibility is to implement the queue in OH, and have a single thread that is responsible for processing that queue. That is a bit more work.
The question is how “much effort” OH should put into relieving remote endpoints from the burden of prompt processing of messages.
Any insight into how HABApp handles this would be useful, maybe there’s a simple “fix” that could be done there?
