Web Socket Error on AmazonEchoControl binding every 65 seconds

Trinitus01 · June 13, 2020, 9:43pm

Btw lastCommand doesnt work for me too.

memphiz · June 13, 2020, 9:44pm

Could you please provide your compiled jar file?

Regards,
Olli

Trinitus01 · June 13, 2020, 10:07pm

Yes maybe but i would prefer to check if the memory issue is fixed. Im not the provider of the binding so i will not take care of all the features of the binding. And its also possible to use the alexa control script via executeCommandLine in openHAB: https://blog.loetzimmer.de/2017/10/amazon-alexa-hort-auf-die-shell-echo.html?m=1

Harpo3 · June 13, 2020, 10:22pm

Yeah, had to disable the binding, which sucks, as I had a lot programmed for it. Announcements, briefings, lights, etc. Hope this gets fixed quickly by someone. I would if I had the skill.

RedRocket · June 13, 2020, 11:45pm

Just wanted to chime in and say that I’m experiencing the problem as well.

idkpmiller · June 13, 2020, 11:55pm

I have tried unistalling 2.5.5 version of the binding reverting back to 2.5.0_RC1 and 2.5.0_RC3 and they also failed on OH 2.5.5

I think I saw someone had a potential band-aid fix that they had compiled a new jap file and was seeing it improved things. Let’s hope that version comes out as an emergency fix while we await the actual corrective fix from @michi .

J-N-K · June 14, 2020, 7:57am

Exactly. I think that closing the jetty client with unfinished requests is the reason for the crash. The reason for closing jetty is a timeout of 30s that is exceeded (@essjay sent a log that clearly shows that). According to https://github.com/eclipse/jetty.project/issues/1445 it seems that we have to implement a tracker for outstanding requests and wait for them to finish before we close the client to get a graceful shutdown.

MrTBone · June 14, 2020, 7:59am

I have the same memory leak since yesterday 11am (UTC+2)

We were not at home during the weekend hence realized only by today. Plot shows stable 40% availability of RAM before 11am. Running OH 2.5.4 with no changes during the period of this plot.

I have disabled AmazonEchoControl binding now and will see if memory availability is stable as before? But all your comments indicate the same causal reasoning.

Seems that all of us have the same sudden problem at the same point in time. Maybe caused due to change on amazon servers!? At least my system was untouched the last two days, with no impact on memory from my side.

Malte_SmartHome · June 14, 2020, 8:20am

Observed the same memory leak on my maschine.
Restaring the server just works for a few hours, depending on available memory.
Hope this issue can be fixed quickly.

----- Addendum

Observed weird behaviour after disabling the Amazon Echo Control Binding.
Obviously the bindings stops sending requests to the amazon servers, I found no further errors in the log files.

Surprisingly the voice control via Amazon Echo works proper.
Perhaps the deinstallation doesn’t removed all processes?!

Saracen · June 14, 2020, 8:56am

That’s using a different binding - the Amazon Alexa Smart Home Skill.

I’ve got the same problem too - wife isn’t happy. WAF is now zero!

omatzyo · June 14, 2020, 10:09am

Same issues here.

ariela · June 14, 2020, 10:58am

same issue here. Stable 2.5.5.1

Andrea

Trinitus01 · June 14, 2020, 11:04am

I came to the same way and checked the solution on GitHub, but i think (maybe im wrong) it would be the wrong way to implement a tracker for outstanding requests. Because if there is a problem on the amazon server and the request “never” returns, it will never close the webSocketClient properly. It crashes while it tries to close the client but the question is, why the request never returns?!? So i think its better to find the request and check whats wrong and not to wait for the broken request to return. I also tried to set the idle timeout for the client itself to stop the request before the timer tries to close the webSocketClient, but it also ends in an exception.

To the memory test with the modified jar file and the removed timer which try to close the webSocketClient every minute, the memory issue is not present anymore since yesterday.

J-N-K · June 14, 2020, 11:05am

I‘m currently looking into this, but it‘s a bit difficult because I don‘t own any of these devices.

Trinitus01 · June 14, 2020, 11:09am

I think it is only necessary to add an amazon account to the binding and the issue appears.
You should take a look at:

org.openhab.binding.amazonechocontrol.internal.WebSocketConnection

The method:

initPongTimeoutTimer()

starts the timer to close the webSocketClient through the method:

close()

in the same class.

J-N-K · June 14, 2020, 1:59pm

Thanks for that hint. That helped a lot. What happens is:

binding tries connect and immediately schedules a pong timeout
after successfull connect, the pong is acknowledged and a ping is scheduled
when the ping is send, a new pong timeout is scheduled (and acknowleged after the response is received)
when pong times out, the connection is closed (this fails, if no connection was established)
the binding tried to reconnect and we start from the beginning

The problem is that - at least for me - the connection is never established and so this happens every 60s. close on the connection is called twice (by the pong timeout and a second time by the handler), therefore it tries to stop the websocket connection twice, but the connect future is never terminated. This leads to memory and thread leaks.

I have fixed that and now check that the client is only stopped when it was started before, that the connection is only closed once and the connect future is cancelled. This version is neither leaking memory nor threads on my system. Please check if the functionality is still there.

<link removed>

You can directly update from the karaf console, you need to get your bundle number with bundle:list | grep -i amazon which should result in

259 │ Active │  80 │ 2.5.6.202006141340      │ openHAB Add-ons :: Bundles :: Amazon Echo Control Binding

259 is the bundle number on my system (yours will probably be different) and you can update with

bundle:update 259 <link removed>

Please report back if this fixes your issues.

Edit: Removed link, outdated, see below

Trinitus01 · June 14, 2020, 2:07pm

Perfect!

Can you tell me what exactly you changed to fix it? Did you a check on the webSocketClient before closing it? Because my curren version has a lot of changes for multi device announcements (too many request fix aso) and some fixes for an issue in volume tts settings. I dont want to miss these fixes and also want to publish them in another thread. I didnt for now because i didnt know if it is ok for the publisher to release a version out of the release chain (git pull request aso).

J-N-K · June 14, 2020, 2:09pm

Open a PR, it‘s open source. And everybody is encouraged to do changes. If some more confirm that their issues are fixed, I‘ll open a PR and get this into snapshots as soon as possible. Maybe we can get it into 2.5.6

H102 · June 14, 2020, 2:19pm

FYI, I just install the jar file and still getting the web socket error.

Trinitus01 · June 14, 2020, 2:20pm

Ok, ill try to do so. I never worked on a public project, only on projects at work. (own git) We can pull and push but don’t have to do a PR.

This is the stuff i want to merge:

And this was my explanation: