[SOLVED] TTN TheTingsNetwork MQTT Parsing

Michael_Stjerna · March 27, 2018, 2:25pm

Hi

I’m struggling to get the metadata out of a MQTT message I’m subscribing to.

This works just fine:

Number  ELT1_externalTemperature        "ELT1_externalTemperature [%.1f]"       <temperature>           (gLoRa)          {mqtt="<[ttn:starman/devices/a81758fffe034049/up/externalTemperature:state:default]"}
Number  ELT1_humidity                   "ELT1_humidity [%.1f]"                  <humidity>              (gLoRa)          {mqtt="<[ttn:starman/devices/a81758fffe034049/up/humidity:state:default]"}
Number  ELT1_pressure                   "ELT1_pressure [%.0f]"                  <pressure>              (gLoRa)          {mqtt="<[ttn:starman/devices/a81758fffe034049/up/pressure:state:default]"}
Number  ELT1_temperature                "ELT1_temperature [%.1f]"               <temperature>           (gLoRa)          {mqtt="<[ttn:starman/devices/a81758fffe034049/up/temperature:state:default]"}
Number  ELT1_vdd                        "ELT1_vdd [%.1f]"                       <battery>               (gLoRa)          {mqtt="<[ttn:starman/devices/a81758fffe034049/up/vdd:state:default]"}

But I want to catch the
“metadata” -> time, airtime
“metadata”-> “gateways” -> gtw_id, rssi, snr

From this message rawformat I get with MOSQITTO from bash:

mosquitto_sub -h eu.thethings.network -t 'starman/devices/a81758fffe034049/up' -u '465749' -P 'ttn-account-v2.1234' -v

starman/devices/a81758fffe034049/up
{       "app_id":"starman",
        "dev_id":"a81758fffe034049",
        "hardware_serial":"A81758FFFE034049",
        "port":5,
        "counter":5,
        "payload_raw":"AQD1AgwDAwQ9Bw3uDADnDQAUAA+k5g==",
        "payload_fields":{
                "digital":0,
                "externalTemperature":23.1,
                "humidity":12,
                "pressure":1025.254,
                "temperature":24.5,
                "vdd":3566,
                "x":3,
                "y":4,
                "z":61
        },
        "metadata":{
                "time":"2018-03-27T13:28:44.857209839Z",
                "frequency":867.1,
                "modulation":"LORA",
                "data_rate":"SF7BW125",
                "airtime":77056000,
                "coding_rate":"4/5",
                "gateways":[{
                        "gtw_id":"eui-46584258c00001d0",
                        "timestamp":4163345947,
                        "time":"",
                        "channel":3,
                        "rssi":-37,
                        "snr":10.5,
                        "rf_chain":0}]
        }
}

vzorglub · March 27, 2018, 2:49pm

You need the JSONPATH transformation
Check it is installed
Then

For example, the first one:

Number  ELT1_externalTemperature        "ELT1_externalTemperature [%.1f]"       <temperature>           (gLoRa)          {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.payload_fields.externalTemperature)]"}

Note the topic and the JSONPATH expression
And the same for the others, you just need to change the jsonpath expression

You can paste the json in http://jsonpath.com/ and play with the jsonpath expression to extract whatever value you need.

Good luck

Michael_Stjerna · March 27, 2018, 6:15pm

Hi @vzorglub

I think you missed my comments, the first section works just fine as it is already!

This is what I have a hard time to access:

“metadata” -> time, airtime
“metadata”-> “gateways” -> gtw_id, rssi, snr

Michael_Stjerna · March 27, 2018, 6:18pm

@vzorglub but I see your comments there with the JSON:
up:state:JSONPATH($.metadata.time)]"}

I will try this first thing tomorrow when I have access to the OpenHAB server again…

vzorglub · March 27, 2018, 9:33pm

metadata time: $.metadata.time
metadata airtime: $metadata.airtime
metadata gateways gwt_id: $.metadata.gateways[0].gtw_id
metada gateways rssi: $.metadata.gateways[0].rssi
metada gateways snr: $.metadata.gateways[0].snr

Note the [0] at the end of gateways: gateways[0]

Good luck

Michael_Stjerna · March 28, 2018, 8:15am

@vzorglub

Bah, still struggeling.
The JSON parsing works fine on the webpage.
It’s the MQTT string that’s the issue…
I have installed the “JSONPath Transformation” from the PapperUI

OpenHAB don’t like this syntax…

String  ELT1_meta_time                  "Metadata Time: %s]"            {mqtt="<[ttn:starman/devices/a81758fffe034049/up/state:JSONPATH($.metadata.time)]"}

2018-03-28 10:11:16.647 [ERROR] [el.item.internal.GenericItemProvider] - Binding configuration of type 'mqtt' of item 'ELT1_meta_freq' could not be parsed correctly.

org.eclipse.smarthome.model.item.BindingConfigParseException: Configuration 'ttn:starman/devices/a81758fffe034049/up/state:JSONPATH($.metadata.frequency)' is not a valid inbound configuration: Configuration requires 4 or 5 parameters separated by ':'

Michael_Stjerna · March 28, 2018, 8:21am

Closer now

String  ELT1_meta_time                  "Metadata Time: %s"            {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.time)]"}

Gives me the output:
JSONPATH($.time)2018-03-28T08:19:39.726715

vzorglub · March 28, 2018, 8:21am

{mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.time)]"}

Michael_Stjerna · March 28, 2018, 8:25am

@vzorglub May thanks!

Everything work as expected

String  ELT1_meta_time                  "Metadata Time: [%s]"                   {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.time)]"}
String  ELT1_meta_freq                  "Metadata Frequency: [%s]"                      {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.frequency)]"}
String  ELT1_meta_datarate              "Metadata DataRate: [%s]"                       {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.data_rate)]"}

vzorglub · March 28, 2018, 8:27am

Cool
Glad I could help

vzorglub · March 28, 2018, 8:30am

Could you mark the thread as solved, please?
Thanks

Michael_Stjerna · March 28, 2018, 8:33am

Give me 10min, cleaning up the files to leave a full documentation for other users!

Michael_Stjerna · March 28, 2018, 9:04am

Items file & Sitemap for LoRaWAN sensor attached to TTN “The Things Network”

lora.items

//LoRaWAN Sensor Data
Number  ELT1_externalTemperature        "ELT1_External Temp [%.1f]"     <temperature>           (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up/externalTemperature:state:default]"}
Number  ELT1_humidity                   "ELT1_humidity [%.1f]"          <humidity>              (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up/humidity:state:default]"}
Number  ELT1_pressure                   "ELT1_pressure [%.0f]"          <pressure>              (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up/pressure:state:default]"}
Number  ELT1_temperature                "ELT1_temperature [%.1f]"       <temperature>           (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up/temperature:state:default]"}
Number  ELT1_vdd                        "ELT1_vdd [%.1f]"               <battery>               (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up/vdd:state:default]"}
Number  ELT1_pos_x                      "ELT1_pos_x [%.1f]"             <niveau>                (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up/x:state:default]"}
Number  ELT1_pos_y                      "ELT1_pos_y [%.1f]"             <niveau>                (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up/y:state:default]"}
Number  ELT1_pos_z                      "ELT1_pos_z [%.1f]"             <niveau>                (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up/z:state:default]"}

//LoraWAN Meta Data
String  ELT1_meta_time                  "Metadata Time: [%s]"           <time>                  (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.time)]"}
String  ELT1_meta_freq                  "Metadata Frequency: [%s]"      <settings>              (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.frequency)]"}
String  ELT1_meta_datarate              "Metadata DataRate: [%s]"       <settings>              (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.data_rate)]"}
String  ELT1_meta_airtime               "Metadata AirTime: [%s]"        <settings>              (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.airtime)]"}
String  ELT1_meta_coding_rate           "Metadata CodingRate: [%s]"     <settings>              (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.coding_rate)]"}


//LoraWAN GW1 information
String  ELT1_gw_id                      "GW ID: [%s]"                   <text>                  (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.gateways[0].gtw_id)]"}
String  ELT1_gw_channel                 "GW Channel: [%s]"              <qualityofservice>      (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.gateways[0].channel)]"}
String  ELT1_gw_rssi                    "GW RSSI: [%s]"                 <network>               (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.gateways[0].rssi)]"}
String  ELT1_gw_snr                     "GW SNR: [%s]"                  <network>               (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.gateways[0].snr)]"}
String  ELT1_gw_rf_chain                "GW RF-Chain: [%s]"             <qualityofservice>      (gLoRa)         {mqtt="<[ttn:starman/devices/a81758fffe034049/up:state:JSONPATH($.metadata.gateways[0].rf_chain)]"}

lora.sitemap

sitemap lora label="Lora" {
    Frame {
        Text label="Elsys ELT-2" icon="sensor" {
			Frame label="Sensor Data" {
                Default item=ELT1_temperature label="Temperature"
				Default item=ELT1_humidity label="Humidity"
                Default item=ELT1_pressure label="Pressure"
				Default item=ELT1_externalTemperature label="External Temp"            
                Default item=ELT1_vdd label="VDD"
                Default item=ELT1_pos_x label="Position-X"
                Default item=ELT1_pos_y label="Position-Y"
                Default item=ELT1_pos_z label="Position-Z"
			}
			Frame label="LoRaWAN MetaData" icon="network"{
                Default item=ELT1_meta_time label="PacketTime"
				Default item=ELT1_meta_freq label="Frequency"
				Default item=ELT1_meta_datarate label="DataRate"
				Default item=ELT1_meta_airtime label="AirTime"
				Default item=ELT1_meta_coding_rate label="CodingRate"
			}
			Frame label="LoRaWAN GateWay information" icon="qualityofservice"{
                Default item=ELT1_gw_id label="ID"
				Default item=ELT1_gw_channel label="Channel"
				Default item=ELT1_gw_rssi label="RSSI"
				Default item=ELT1_gw_snr label="SNR"
				Default item=ELT1_gw_rf_chain label="RF-Chain"
			}			
        }
    }
}

bernd_d · November 20, 2019, 8:10pm

As of Nov 18th the separate “field” values have been disabled (for performance reasons) by TTN. Who knows when they will be reactivated…
You can now fetch each field data through the “payload_fields” field by using a JSONPATH transformation like for the metadata information.