Web Socket Error on AmazonEchoControl binding every 65 seconds

I get this if i try to use tts :S



any ideas? :S

@richaardvark So I posted too soon, those options only made it longer before I hit the OOM errors again and everything stops working, rules, MQTT updates, the works. So last night I had to do this to resolve things:
1)backup my items and rules files
2) remove all the rules and items related to my Amazon Echos
3) Remove and then Force Remove all my Amazon Echo Things
4) Remove and then Force Remove my Amazon Account
5) Uninstall the Amazon Echo Binding
6) Reboot

Since then everything is happy except no more voice control, but WAF is up because now at least motion sensors work to turn on lights and push buttons work to open / close garage!

If you want to adjust your Java memory usage for OH you do it in the /etc/default/openhab2 file. Look for the section like this:

#########################
## JAVA OPTIONS
## Additional options for the JAVA_OPTS environment variable.
## These will be appended to the execution of the openHAB Java runtime in front of all other options.
##
## A couple of independent examples:
##   EXTRA_JAVA_OPTS="-Dgnu.io.rxtx.SerialPorts=/dev/ttyAMA0"
##   EXTRA_JAVA_OPTS="-Dgnu.io.rxtx.SerialPorts=/dev/ttyUSB0:/dev/ttyS0:/dev/ttyS2:/dev/ttyACM0:/dev/ttyAMA0"
##   EXTRA_JAVA_OPTS="-Djna.library.path=/lib/arm-linux-gnueabihf/ -Duser.timezone=Europe/Berlin -Dgnu.io.rxtx.SerialPorts=/dev/ttyS0"

EXTRA_JAVA_OPTS="-Xms400m -Xmx650m"

That Last line is the one I modified as shown.

Note for the Amazon Echo Binding Devs: I noticed that with the binding installed I was having problems with other “Internet-facing” bindings, NTP, Air Quality and Astro binding would complain about not being able to connect to servers or “No Route to Host” would show up next the the Thing in PaperUI and this was right after a reboot of my OH box. Sometimes they would come back online after 20 minutes or so or I would have to edit the Thing, change a parameter (usually I would adjust the Refresh Interval) and save and it would come online.

Andrew

Tried it on my previous RPI3 with Version 2.5.2-1.
Same result…

@richaardvark Amazon Smart Home Devices should not work because webSocket is broken.

@One_Watt The issue doesn’t depend on any item, you can have zero items and the error message will persist.

I don’t have time yet to help I would but can’t. Somebody out there records network traffic and check via wireshark what’s happening?

1 Like

I did several TTS runs

Echo_Living_Room_TTS.sendCommand( '<speak><amazon:effect name="whispered">Lass mich hier raus</amazon:effect></speak>')

and ran tcpdump on openhab host in parallel for about 10 - 15 minutes

tcpdump | grep dp-gw

I did not see any hostname containing the string dp-gw while grepping for amazon returned several server names in eu-west.

1 Like

One more report using OH 2.4: 2.5.5 docker restarts randomly

with that I would conclude
=> it’s independent of OH 2.4, 2.5.X
=> independent of the amazoncontrol binding version ( apart of the OOM problem )

It also was mentioned if the certificate would have been renewed.
I connected to https://dp-gw-na.amazon.com/ via browser and checked the certificate:
Validity: Not Before 1/30/2020, 1:00:00 AM (Central European Summer Time)
=> so that was not changed either

someone on the internet mentioned having problems because his fritzbox protection of minors ( URL/content filter ) was blocking some amazon server. => I think this cannot be the reason ( at least not for the wireshark/tcpdump that was done some posts above as not all requests to that server are blocked.

Ram Swap usage for last 7 days (All time in IST)

Ram Swap Usage for last 24 hr

Ram Swap Usage Post jar fix

In the meantime I also updated to 2.5.6-SNAPSHOT fix, having a bit of a problem now.

My Amazon account already lost the connection twice and was not able to login again, also after reboot it needed some time to get the account online, not sure if I will do a clean installation tomorrow.

Interesting that you guys also lost the things as I did. I checked this morning and had to add them manually again and upgraded to 2.5 afterwards as I thought the new version might be more stable.

Reading through this it might not have been the 2.4 version but the echo binding instead. Will also disable it for now as adding the things was quiet time-consuming.

Running on a raspberrypi zero w for two years and never lost things before.

Strange enough despite the error the binding still seems to work even though the “Last Voice Command” no longer shows up while e.g. TTS works flawless.

It might be this repository that has the iobroker binding, but I can’t find much pointing to something that has been fixed: https://github.com/Apollon77/ioBroker.alexa2

I think I found the culprit. Just doing some verification before I’ll provide a test jar.

Edit: please try link removed, use snapshot, changes are merged. Instructions for usage see above.

14 Likes

I think things got lost because of a regular restart. The broken system (OOM) in written into the json-db. I also lost my things, but there is an automatically backup of the json-db, see the backup folder.

:smiley:

2 Likes

It works, I am getting lastCommand back into OpenHAB! Thank you so much.

1 Like

LGTM

Thanks :slight_smile:

Seems to work also here. Thank you very much!

Br,
Stefan

You are probably right, that the database is corrupted because of the unclean shutdown after the OOM. This should be considered a bug. Could you file an issue in openhab-core repository for that?

this jar got lastVoice working for me too.
but… I did get this error in my log
I couldn’t find any negative consequences of the error, so I’m in no rush to fix it. But I’d be glad to help if anyone wants to.

17:56:22.791 [WARN ] [ntrol.internal.handler.AccountHandler] - handling of websockets fails
org.openhab.binding.amazonechocontrol.internal.HttpException: GET url 'https://alexa.amazon.com/api/activities?startTime=1592258184045&size=10&offset=1' failed: Bad Request
        at org.openhab.binding.amazonechocontrol.internal.Connection.makeRequest(Connection.java:644) ~[bundleFile:?]
        at org.openhab.binding.amazonechocontrol.internal.Connection.makeRequestAndReturnString(Connection.java:499) ~[bundleFile:?]
        at org.openhab.binding.amazonechocontrol.internal.Connection.makeRequestAndReturnString(Connection.java:494) ~[bundleFile:?]
        at org.openhab.binding.amazonechocontrol.internal.Connection.getActivities(Connection.java:938) ~[bundleFile:?]
        at org.openhab.binding.amazonechocontrol.internal.handler.AccountHandler.handlePushActivity(AccountHandler.java:773) ~[bundleFile:?]
        at org.openhab.binding.amazonechocontrol.internal.handler.AccountHandler.handleWebsocketCommand(AccountHandler.java:718) ~[bundleFile:?]
        at org.openhab.binding.amazonechocontrol.internal.handler.AccountHandler.webSocketCommandReceived(AccountHandler.java:706) [bundleFile:?]
        at org.openhab.binding.amazonechocontrol.internal.WebSocketConnection$AmazonEchoControlWebSocket.onWebSocketBinary(WebSocketConnection.java:417) [bundleFile:?]
        at sun.reflect.GeneratedMethodAccessor403.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_192]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_192]
        at org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:70) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCallableMethod.java:72) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onBinaryMessage(JettyAnnotatedEventDriver.java:133) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.websocket.common.message.SimpleBinaryMessage.messageComplete(SimpleBinaryMessage.java:68) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onBinaryFrame(JettyAnnotatedEventDriver.java:125) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:145) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:321) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:226) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:262) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:582) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:428) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:426) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:320) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:158) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:367) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782) [bundleFile:9.4.20.v20190813]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918) [bundleFile:9.4.20.v20190813]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]

I‘m sorry. I already spent much more time in this than I originally planned. I wanted to fix the OOM exception because it affects the whole system. Now I spend around 10 more hours in fixing the websocket connection. I have no idea where to start with your error, so this will need someone else for fixing it.

Good job! To unterstand, it was not a missing part, it was a redundant part?!? I am really interested how this came to your mind OO

Setting jetty to DEBUG shows the requests. It’s a bit hard to see, because jetty is very chatty at DEBUG.

After I found that, I originally thought that amazon is blocking jetty as user-agent and tried to set the user-agent header to a more common Mozilla. To my surprise this resulted in two user-agent headers which is obviously wrong. I then had a closer look at the other headers and found the the binding adds headers that are later added by jetty. After I removed that, the connection succeeded.

I‘m not entirely sure if this should be considered a bug in jetty. If I pass a request to the client, IMO it should not add already existing headers.

3 Likes