ZNDMWG03LM - Xiaomi Gateway 3

I’m trying to make this device usable for our purposes. I read something using the message published through mosquitto, but it lacks of ble device.

So I took inspiration from the work of AlexxIT for Home Assistant Xiaomi Gateway 3 integration for Home Assistant.
I’ve also changed the firmware https://github.com/serrj-sv/lumi.gateway.mgl03, useful to run custom shell on startup.

My attempt is to convert log data from the “miio_client” program to be readable from mqtt binding of openHAB.

Now ble device write message like this:
log/ble/A4C138E763AE/1371/4102

And zigbee:
log/zb/lumi.158d0004384fb4
Reporting the value in this form
{ "pressure": 100920 }
{ "humidity": 6074}

Raw message are all published in:
log/raw
Like this:
{"method":"_sync.ble_query_dev","params":{"mac":"C4:7C:8D:6B:62:C2","pdid":152},"id":466}

Heartbeat and stat information:
log/heartbeat
log/stat

What I did is only for my few device, but could be useful for someone who wants to use this gateway for reading their. A lot of information is on AlexxIT github, as the key to decode eid.
Command instruction must be sent on usual form.

So what I did on run.sh (once telnet in the gateway):

    #!/bin/sh
    /bin/dropbear -R -B
    #Uccido i demoni sottostanti i client miio
    killall daemon_miio.sh; killall miio_client
    #Carico il server FTP per scaricarsi db o log
    /bin/sftp-server
    #Riavvio il client, filtrando il testo e mappandolo su mqtt 
    (miio_client -l 0 -o FILE_STORE -n 128 -d /data/miio | awk '/miio_client_rpc/{print $0;fflush()}' | sh /data/log_mqtt.sh &)
    #Riavvio il demone di sicurezza
    daemon_miio.sh &

Regarding the “log_mqtt.sh” script:

#!/bin/sh

##Da copiare nel run.sh
#killall daemon_miio.sh
#killall miio_client
#killall inviomqtt.sh
#(miio_client -l 0 -o FILE_STORE -n 128 -d /data/miio | awk '/miio_client_rpc/{print $0;fflush()}' | sh /data/log_mqtt.sh &)
#daemon_miio.sh &

while IFS= read -r line; do
	
	##Debug per vedere cosa filtra l'awk in ingresso
	#iniz_sng="$line"
	#echo $iniz_sng >> /var/log/inviomqtt.log
	
	#Per togliere i caratteri tipo ESC che trovo alla fine della riga e sembra dar problemi al compilatore jshon
	iniz_sng=$(echo $line | sed 's/[^[:print:]]//g' ) 
	
	#Elimino inizio e fine che non ci interessano
	jsonsng=$(echo $iniz_sng | sed 's/.*call=\(.*\)\[0m.*/\1/'  )
	#Nel caso di più json, aggiungo la ; per fare lo split
	jsonsng_tmp=${jsonsng//\}\{/\};\{}
	
	#Istruisco per lo split
	OIFS=$IFS
	IFS=";"
	set $jsonsng_tmp
	IFS=$OIFS
	
	#Ciclo per tutti gli elementi dell'array
	for item
	do
			
			##Debug per leggere il gingolo json in analisi
			#echo $item >> /var/log/inviomqtt_dett.log
			
			#Debug LOG
			mosquitto_pub -t log/raw -m "$item"
			
			method=$(echo $item | jshon -C -e method -u)
			
			#Caso dei dispositivi ZigBee
			if [ "$method" == "props" ]
			then
				sid=$(echo $item | jshon -C -e sid -u)
				params=$(echo $item | jshon -C -e params | sed 's/[][]//g' )
				#id=$(echo $item | jshon -C -e id -u)
				#model=$(echo $item | jshon -e model -u)
				
				#Se il campo è vuoto, per non mandare messaggi inutili 
				if [ ! -n "$sid" ] 
				then
					sid="nd" 
				fi
				
				#Invio il messaggio ZigBee
				mosquitto_pub -t log/zb/$sid -m "$params"
			fi
			
			#Caso dispositivi BLE
			if [ "$method" == "_async.ble_event" ]
			then
				pdid=$(echo $item | jshon -C -e params -e dev -e pdid -u)
				mac=$(echo $item | jshon -C -e params -e dev -e mac -u | sed -e 's/://g')
				evt=$(echo $item | jshon -C -e params -e evt )
				
				if [ ! -n "$mac" ]
				then
					mac="nd"
				fi
				
				if [ ! -n "$pid" ]
				then
					pid="nd"
				fi
				
				if [ ! -n "$evt" ]
				then
					evt=""
				else
					#Pulizia della stringa
					evt_tmp=$(echo $evt | sed 's/[][]//g')
					eid=$(echo $evt_tmp | jshon -C -e eid )
					edata=$(echo $evt_tmp | jshon -C -e edata -u)
					
					if [ ! -n "$eid" ]
					then
					   eid="nd"
					fi
				fi
				
				#Invio in due formati
				mosquitto_pub -t log/ble/$mac/$pdid -m "$evt"
				mosquitto_pub -t log/ble/$mac/$pdid/$eid -m "$edata"
			fi
			
			#Caso HeartBeat - Stato del gateway
			if [ "$method" == "event.gw.heartbeat" ]
			then
				params=$(echo $item | jshon -e params | sed 's/[][]//g' )
				mosquitto_pub -t log/heartbeat -m "$params"
			fi
			
			#Caso statistiche d'utilizzo
			if [ "$method" == "_async.stat" ]
			then
				params=$(echo $item | jshon -e params | sed 's/[][]//g' )
				mosquitto_pub -t log/stat -m "$params"
			fi
			
	done
	
done

One question. I have to read hex data from ble deviced.
To do this, I should convert edata from HEX to byte and then to integer (and then /10).
A little bit difficult on shell.
So how to do this on openHAB ?

AlexxIT, using py, do it on this way:
data = bytes.fromhex(event['edata'])
return {'temperature': int.from_bytes(data, 'little', signed=True) / 10.0}

I’ve made some progress.
It now manages both events and parameters of the ZigBee devices.
Even the BLE devices that interest me consider values ​​in the expected format (decimal).
Hope it can help someone with this great Gateway.
… Maybe there is someone out there who can improve the code :slight_smile:

#!/bin/sh

##Da copiare nel run.sh
#killall daemon_miio.sh
#killall miio_client
#killall log_mqtt.sh
#(miio_client -l 0 -o FILE_STORE -n 128 -d /data/miio | awk '/miio_client_rpc/{print $0;fflush()}' | sh /data/log_mqtt.sh &)
#daemon_miio.sh &

while IFS= read -r line; do

        DATAORA=$(date +'%Y%m%d%H%M')

    ##Debug per vedere cosa filtra l'awk in ingresso
    #iniz_sng="$line"
    #echo $iniz_sng >> /var/log/inviomqtt.log

    #Per togliere i caratteri tipo ESC che trovo alla fine della riga e sembra dar problemi al compilatore jshon
    iniz_sng=$(echo $line | sed 's/[^[:print:]]//g' )

    #Elimino inizio e fine che non ci interessano
    jsonsng=$(echo $iniz_sng | sed 's/.*call=\(.*\)\[0m.*/\1/'  )
    #Nel caso di più json, aggiungo la ; per fare lo split
    jsonsng_tmp=${jsonsng//\}\{/\};\{}

    #Istruisco per lo split
    OIFS=$IFS
    IFS=";"
    set $jsonsng_tmp
    IFS=$OIFS

    #Ciclo per tutti gli elementi dell'array
    for item
    do

                    #Debug per leggere il gingolo json in analisi
                    echo $item >> /var/log/inviomqtt_dett.log

                    #Debug LOG - Invio la stringa grezza
                    mosquitto_pub -t log/raw -m "$item"

                    #Identifico il methodo (discrimina se parametro o evento)
                    method=$(echo $item | jshon -C -e method -u)


                    #Caso dei dispositivi ZigBee - Parametro / valore informativo
                    if [ "$method" == "props" ] || [ "$method" == "_otc.log" ]
                    then
                            sid=$(echo $item | jshon -C -e sid -u)
                            params=$(echo $item | jshon -C -e params | sed 's/[][]//g' )
                            #id=$(echo $item | jshon -C -e id -u)
                            #model=$(echo $item | jshon -e model -u)

                            #Se il campo è vuoto, per non mandare messaggi inutili
                            if [ ! -n "$sid" ]
                            then
                                    sid="nd"
                            fi

                            case "$params" in
                                    *"device_log"*)
                                            #scarto il record - da capire se riporta informazioni utili
                                            ;;
                                    *","*         )
                                            #TO DO - Bisogna individuare le casistiche dove si riportano parametri multipli
                                            ;;
                                    *             )
                                            param_name=$(echo $params | jshon -C -k )
                                            param_value=$(echo $params | jshon -C -e $param_name )
                                            if [ "$param_name" == "temperature" ] || [ "$param_name" == "humidity" ]
                                            then
                                                    param_value=$(echo $param_value 100 | awk '{ print $1/$2 }')
                                            fi
                                            if [ "$param_name" == "pressure" ]
                                            then
                                                    param_value=$(echo $param_value 1000 | awk '{ print $1/$2 }')
                                            fi
                                            mosquitto_pub -t log/zb/$sid/$param_name -r -m "$param_value"
                                            mosquitto_pub -t log/zb/$sid/DATAORA -r -m "$DATAORA"
                                            ;;
                            esac

                    fi



                    #Caso dispositivi BLE
                    if [ "$method" == "_async.ble_event" ]
                    then
                            pdid=$(echo $item | jshon -C -e params -e dev -e pdid -u)
                            mac=$(echo $item | jshon -C -e params -e dev -e mac -u | sed -e 's/://g')
                            evt=$(echo $item | jshon -C -e params -e evt )

                            if [ ! -n "$mac" ]
                            then
                                    mac="nd"
                            fi

                            if [ ! -n "$pid" ]
                            then
                                    pid="nd"
                            fi

                            if [ ! -n "$evt" ]
                            then
                                    evt=""
                            else
                                    #Pulizia della stringa
                                    evt_tmp=$(echo $evt | sed 's/[][]//g')
                                    eid=$(echo $evt_tmp | jshon -C -e eid )
                                    edata=$(echo $evt_tmp | jshon -C -e edata -u)

                                    if [ ! -n "$eid" ]
                                    then
                                       eid="nd"
                                    fi

                                    if [ $eid == 4100 ]
                                    then
                                            edata_sx=$(echo $edata | cut -c1-2 )
                                            edata_nm=$(printf "%d\n" 0x$edata_sx)
                                            edata=$(echo $edata_nm 10 | awk '{ print $1/$2 }')
                                    fi

                                    if [ $eid == 4102 ]
                                    then
                                            edata_sx=$(echo $edata | cut -c1-2 )
                                            edata_dx=$(echo $edata | cut -c3-4 )
                                            edata_nm=$(printf "%d\n" 0x$edata_dx$edata_sx)
                                            edata=$(echo $edata_nm 10 | awk '{ print $1/$2 }')
                                    fi
                            fi

                            #Invio in due formati
                            mosquitto_pub -t log/ble/$mac/$pdid -m "$evt"
                            mosquitto_pub -t log/ble/$mac/$pdid/$eid -r -m "$edata"
                            mosquitto_pub -t log/ble/$mac/$pdid/DATAORA -r -m "$DATAORA"

                    fi



                    #In tutti casi che inizia con "event." potrebbe essere un evento significativo del sensore (tipo movimento persona)
                    case "$method" in
                            "event."*    )
                                    #Caso HeartBeat - Stato del gateway
                                    if [ "$method" == "event.gw.heartbeat" ]
                                    then
                                            params=$(echo $item | jshon -e params | sed 's/[][]//g' )
                                            mosquitto_pub -t log/heartbeat -r -m "$params"
                                    else
                                            sid=$(echo $item | jshon -C -e sid -u)
                                            params=$(echo $item | jshon -C -e params | sed 's/[][]//g' |  sed 's/[ \t]//g' )
                                            params=$(echo $params | tr '\n' ' ' | sed 's/[ \t]//g' )
                                            if [ ! -n "$sid" ]
                                            then
                                                    sid="nd"
                                            fi
                                            if [ ! -n "$params" ]
                                            then
                                                    params="1"
                                            fi
                                            if [ "$params" == "" ]
                                            then
                                                    params="1"
                                            fi
                                            #Retention del messaggio apposto in via temporanea
                                            mosquitto_pub -t log/zb/$sid/$method -r -m "$params"
                                            mosquitto_pub -t log/zb/$sid/DATAORA -r -m "$DATAORA"
                                    fi
                                    ;;
                            *             )
                                    ;;
                    esac


                    #Caso statistiche d'utilizzo
                    if [ "$method" == "_async.stat" ]
                    then
                            params=$(echo $item | jshon -e params | sed 's/[][]//g' )
                            mosquitto_pub -t log/stat -r -m "$params"
                    fi

    done
	
done

Hi @vendrake,

Do you have any updates?

I’ve finally received the gateway and figured out that I have firmware v1.4.6_0043 - factory firmware from 2020.10.

  1. I’ve configured a thing via Xiaomi Mi IO Binding.
    My modelId is lumi.gateway.mgl03 and I’ve configured a switch to ‘Enable Telnet’ which works fine.
  2. I managed to decode a password, login via telnet and turn off the password.
  3. When I run the following commands
killall mosquitto
mosquitto -d
killall zigbee_gw

I can connect to the MQTT from the outside and see some messages.
New messages that need to be decoded appears in gw/BC33ACFFFE518EB5/MessageReceived for me.
@vendrake, what did you do to receive messages to log/raw topic?
Here is the example:

{
“sourceAddress”: “0xA594”,
“eui64”: “0x00158D000288CEE1”,
“destinationEndpoint”: “0x01”,
“clusterId”: “0x0000”,
“profileId”: “0x0104”,
“sourceEndpoint”: “0x01”,
“APSCounter”: “0x12”,
“APSPlayload”: “0x182F0A0000100”,
“rssi”: -38,
“linkQuality”: 248
}

This is a door sensor, and I need to check APSPlayload’s last nibble: 1 - open, 0 - close.

I configured the rule that enables Telnet on thing’s state update - that’s ok.

UPDATE:

It’s possible to login via telnet and run commands above via the rule on raspberry.

  1. Install telnet client if it’s not yet installed

sudo apt-get install telnet

  1. Create sh script with the following text:

#!/bin/bash
HOST=‘gateway-IP-address’

(
echo open “$HOST”
sleep 2
echo “admin”
sleep 2
echo “killall mosquitto”
sleep 2
echo “mosquitto -d”
sleep 2
echo “killall zigbee_gw”
sleep 2
echo “exit”
) | telnet

Save it ‘xiaomi_enable_mqtt.sh’

  1. Make it executable

chmod +x xiaomi_enable_mqtt.sh

  1. In a rule use the following code:

executeCommandLine(Duration.ofSeconds(60), “/home/openhabian/xiaomi_enable_mqtt.sh”)

Hope it helps someone to save some time.

Now the task is how to easily parse different messages from different sensors.

To sum up everything:

  1. @vendrake, what did you do to receive messages to log/raw topic? Seems that it’s possible to receive data in different topics
  2. Where did you find any info on decoding eid?
  3. Is it possible to create many Things based on one MQTT topic that detect their state based on some conditions, like “only update this item state if eid=‘12345’ and take the state as a last symbol from APSPlayload”?
1 Like

Hi @dexter,
I’m working with few devices, so what I’ve done is enough for me (for now…).

Maybe you don’t want to lose warranty, but if you install serrj-sv firmware you get a lot of improvement :slight_smile:
https://github.com/serrj-sv/lumi.gateway.mgl03/tree/main/firmware

  1. @vendrake, what did you do to receive messages to log/raw topic? Seems that it’s possible to receive data in different topics

From the “miio_client”. Take a look on run.sh script: I filter for “miio_client_rpc” string.

  1. Where did you find any info on decoding eid?

Here (for ZigBee):
https://github.com/AlexxIT/XiaomiGateway3/blob/master/custom_components/xiaomi_gateway3/core/utils.py
And here (for BLE):
https://github.com/AlexxIT/XiaomiGateway3/blob/master/custom_components/xiaomi_gateway3/core/bluetooth.py

  1. Is it possible to create many Things based on one MQTT topic that detect their state based on some conditions, like “only update this item state if eid=‘12345’ and take the state as a last symbol from APSPlayload”?

I think it’s possible. I’m not so smart on shell script.
Now I’m doing it from openHab items update.

For the door sensor, the “status” value let me know if the door is open or close.
Event trigger a change of status.

status = "close"
chip_temperature = 11
DATAORA = 202102181735
no_close = 0
event.open = 1
event.keepalive = 1
event.close = 1
event.no_close = 60
send_all_cnt = 1
send_fail_cnt = 0
lqi = 232
battery = 60
voltage = 2965

Thank you for your reply.

Can you please describe what exactly you have done after installation of a custom firmware on the Gateway? I will install it later when I need an offline bluetooth. As for now - I an fine with what I have.

As for my findings - yes, it is possible to listen to the only channel and receive messages for different items. Here is the example for my three door sensors:

Thing mqtt:topic:Sensors "Sensors" (mqtt:broker:XiaomiGateway3Connection)  {
  Channels:
    Type contact : FrontDoorChannel  [ stateTopic="gw/BC33ACFFFE518EB5/MessageReceived" , 
                                      transformationPattern="REGEX:(.*device_uid_1.*)∩JSONPATH:$.APSPlayload∩JS:get_last_symbol_check_length.js",
                                      OPEN="1", CLOSED="0"]         
    Type contact : GarageDoorChannel  [ stateTopic="gw/BC33ACFFFE518EB5/MessageReceived" , 
                                      transformationPattern="REGEX:(.*device_uid_2.*)∩JSONPATH:$.APSPlayload∩JS:get_last_symbol_check_length.js",
                                      OPEN="1", CLOSED="0"] 
    Type contact : DeckDoorChannel  [ stateTopic="gw/BC33ACFFFE518EB5/MessageReceived" , 
                                      transformationPattern="REGEX:(.*device_uid_3.*)∩JSONPATH:$.APSPlayload∩JS:get_last_symbol_check_length.js",
                                      OPEN="1", CLOSED="0"] 
  }

where ‘get_last_symbol_check_length.js’ just a simple javascript code which checks the message length and returns the last symbol

(function(v) {
    if(v.length != 16)
        return null;     
    return v[v.length-1];
})(input)

This works only for messages that are sent when an event happens (message length is 16 symbols). This doesn’t work for messages that are sent when the device is idle - and I suppose there is much more info in them (84 symbols).

There is also an interesting topic on the gateway zigbee/send which I think contains all the info with the battery. I will check it and post an update.

I’ve created a new binding for this gateway:

1 Like