Incorrect character encoding in MQTT binding?

wawa79 · February 11, 2018, 7:18pm

Accented characters sent by openHAB MQTT binding appear not to be incoded to utf8 as per MQTT 3.1.1 specification.
For example, calling:

sendCommand(myKodi_artist, "Accentué")

or
sendCommand(myKodi_artist, "Accentu\u00E9")
properly sets the myKodi_artist item
BUT sends a message on MQTT bus where the text is not encoded to utf8.
As a result, listener clients close connection as recommended in the MQTT 3.1.1 specification when a malformed message is received. And this is crashing all my HA system.

Can anyone confirm this point?

glufonec · April 24, 2018, 7:12pm

I have the same problem with Cyrillic letters.

I use the event bus mqtt bridge to connect openhab v1 with openhab v2
and openhab1 does not send the letters properly (Get <?> symbols). Also if I manually publish to a topic by external client, openahb2 is receiving it like ? symbols.

Probably an old bug.

Rene · May 1, 2018, 10:26pm

I have the same problem. For me it affects text I want to send to the AmazonEchoControl (Beta: Openhab2 AmazonEchoControl Binding (controlling alexa from openhab2)) binding via MQTT. Text containing e.g. German umlauts is not received correctly by the MQTT binding.

I opened a bug in git at https://github.com/openhab/openhab1-addons/issues/5569 You might want to add your comments / experience there.

namraccr · May 3, 2018, 1:28am

I think I see where this is going wrong, but if someone can provide a debug log to help confirm, that’d be great…

glufonec · May 6, 2018, 12:06pm

What kind of log do you need? I can help.

But there are just ? for each letter outside basic Latin character set. I suspected that it is the web GUI but it is not.

namraccr · May 6, 2018, 1:25pm

Just the usual openhab.log, with DEBUG enabled for the MQTT bits.

e.g.
log:set DEBUG org.openhab.binding.mqtt
log:set DEBUG org.openhab.io.transport.mqtt

glufonec · May 8, 2018, 9:53pm

there is nothing special logged even with there settings


2018-05-08 23:45:05.112 [vent.ItemStateChangedEvent] - Mreza_Athlon_probe changed from NULL to ON
2018-05-08 23:45:05.618 [vent.ItemStateChangedEvent] - Msg2 changed from NULL to ???????????????? 2 ????????????????
2018-05-08 23:45:05.891 [vent.ItemStateChangedEvent] - Location_Phone2_Mqtt_rabota changed from NULL to OFF
2018-05-08 23:45:06.518 [vent.ItemStateChangedEvent] - Location_Phone2_Mqtt_centar changed from NULL to OFF

Msg2 should become “Кирилица 2 Латиница”

2018-05-08 23:43:39.159 [INFO ] [.core.internal.i18n.I18nProviderImpl] - Locale set to mk_MK, Location set to 41.997346199                                                                999996,21.4279956, Time zone set to Europe/Skopje
2018-05-08 23:43:40.090 [INFO ] [er.internal.HomeBuilderDashboardTile] - Started Home Builder at /homebuilder
2018-05-08 23:43:41.134 [INFO ] [.dashboard.internal.DashboardService] - Started dashboard at http://192.168.5.116:8080
2018-05-08 23:43:41.147 [INFO ] [.dashboard.internal.DashboardService] - Started dashboard at https://192.168.5.116:8443
2018-05-08 23:43:48.526 [INFO ] [el.core.internal.ModelRepositoryImpl] - Loading model 'items.items'
2018-05-08 23:43:54.719 [INFO ] [thome.model.lsp.internal.ModelServer] - Started Language Server Protocol (LSP) service on port 5007
2018-05-08 23:43:56.282 [INFO ] [el.core.internal.ModelRepositoryImpl] - Loading model 'default.sitemap'
2018-05-08 23:44:01.823 [INFO ] [basic.internal.servlet.WebAppServlet] - Started Basic UI at /basicui/app
2018-05-08 23:44:02.184 [INFO ] [assic.internal.servlet.WebAppServlet] - Started Classic UI at /classicui/app
2018-05-08 23:44:02.231 [INFO ] [arthome.ui.paper.internal.PaperUIApp] - Started Paper UI at /paperui
2018-05-08 23:44:02.302 [INFO ] [ui.habmin.internal.servlet.HABminApp] - Started HABmin servlet at /habmin
2018-05-08 23:44:02.557 [INFO ] [panel.internal.HABPanelDashboardTile] - Started HABPanel at /habpanel
2018-05-08 23:44:02.821 [DEBUG] [penhab.io.transport.mqtt.MqttService] - Starting MQTT Service...
2018-05-08 23:44:02.823 [DEBUG] [org.openhab.io.transport.mqtt       ] - ServiceEvent REGISTERED - {org.openhab.io.transport.mqtt.MqttService, org.osgi.service.cm.ManagedService}={service.pid=org.openhab.mqtt, component.name=MQTT Connection Service, component.id=227, service.id=355, service.bundleid=241, service.scope=bundle} - org.openhab.io.transport.mqtt
2018-05-08 23:44:02.859 [DEBUG] [org.openhab.io.transport.mqtt       ] - BundleEvent STARTING - org.openhab.io.transport.mqtt
2018-05-08 23:44:02.897 [DEBUG] [org.openhab.io.transport.mqtt       ] - BundleEvent STARTED - org.openhab.io.transport.mqtt
2018-05-08 23:44:02.896 [DEBUG] [penhab.io.transport.mqtt.MqttService] - Setting last will: [class org.openhab.io.transport.mqtt.MqttWillAndTestament] Send 'OFF' to topic 'ohab/@' retained using qos mode 1
2018-05-08 23:44:02.898 [INFO ] [penhab.io.transport.mqtt.MqttService] - MQTT Service initialization completed.
2018-05-08 23:44:02.911 [INFO ] [t.mqtt.internal.MqttBrokerConnection] - Starting MQTT broker connection 'ohabssl'
2018-05-08 23:44:02.918 [DEBUG] [t.mqtt.internal.MqttBrokerConnection] - Creating new client for 'ssl://192.168.5.8:8883'  using id 'glusec-openhab2' and file store '/var/lib/openhab2/tmp/ohabssl'
2018-05-08 23:44:03.188 [WARN ] [mqtt.internal.MqttPersistenceService] - mqtt-persistence:broker
2018-05-08 23:44:03.191 [WARN ] [mqtt.internal.MqttPersistenceService] - mqtt-persistence:topic
2018-05-08 23:44:03.193 [WARN ] [mqtt.internal.MqttPersistenceService] - mqtt-persistence:message
2018-05-08 23:44:03.528 [DEBUG] [org.openhab.binding.mqtt            ] - BundleEvent STARTING - org.openhab.binding.mqtt
2018-05-08 23:44:03.550 [DEBUG] [.binding.mqtt.internal.MqttActivator] - MQTT binding has been started.
2018-05-08 23:44:03.649 [DEBUG] [org.openhab.binding.mqtt            ] - ServiceEvent REGISTERED - {org.osgi.service.event.EventHandler}={event.topics=openhab/*, component.name=org.openhab.binding.mqtt.MqttItemBinding, component.id=235, service                                                      .id=372, service.bundleid=275, service.scope=bundle} - org.openhab.binding.mqtt
2018-05-08 23:44:03.653 [DEBUG] [org.openhab.binding.mqtt            ] - ServiceEvent REGISTERED - {org.openhab.model.item.binding.BindingConfigReader, org.openhab.binding.mqtt.MqttBindingProvider}={component.name=org.openhab.binding.mqtt.gener                                                                icbindingprovider, component.id=236, service.id=371, service.bundleid=275, service.scope=bundle} - org.openhab.binding.mqtt

glufonec · May 8, 2018, 9:58pm

%D0%BC%D1%99%D1%82%D1%82

This is from MQTT Spy app

Msg1 is exported like that from openhab1 to mqtt
I publish to Msg2 from MQTT Spy to force Cyrillic but again openhab2 does not display that properly.

Msg1 and Msg2 are <?> in openhab2

namraccr · May 11, 2018, 2:21am

These jars implement a first attempt at a fix. If this does not resolve the issue, I’ll need to see your item/sitemap/rule definitions…

org.openhab.binding.mqtt jar
org.openhab.io.transport.mqtt jar

glufonec · May 25, 2018, 7:26pm

The binding does not work at all with OpenHab1.5 so upgraded to 1.8

I post to MQTT topic Cyrillic string. I see it ok in the OH1 console but the item state is not OK, web UI is also garbage (but not <?>).

Then OH1 post it to event-bus mqtt …correctly!

Then OH2 imports it badly but I did not upgrade the binding there…finding the way

There is still something to be fixed in the binding. I see it properly in the OpenHab1 console as the BAT has this at the top

@echo off
chcp 65001

glufonec · May 26, 2018, 9:03pm

Nothing special about my rules or items. I do not process the stings actually.

I publish Cyrillic text to mqtt topic multimedia/top_album_1 and I get wrong results

BUT I did another test, if I change item via REST API, it will get the Cyrrilic OK and the MQTT binding will post it correctly to the MQTT broker

String MusicTopAlbum1	  "[%s]"	<music_title> 	{ mqtt="<[ohabssl:multimedia/top_album_1:state:REGEX((.*))]" }

namraccr · May 27, 2018, 2:35pm

I’ve just posted updated jars (same links as before). Tested in OH 2.1 and appear to be working.

HabPanel:

BasicUI:

glufonec · May 28, 2018, 2:48pm

It works when flow is

MQTT -> OpenHab item

but on exit to the eventbus MQTT topic

MQTT -> OpenHab item -> MQTT event bus ohab/out/state/EarthQuake_1_Text
it is with <?> … now

The exit was working before

I am testing on 1.83 OH