Dynamically change mapping between thing and item?

Hi,

I think about how to solve the following problem in an elegant way.

Current situation and implementation:
I have 6 physical off-grid solar/battery inverters. There are 17 things defined for each of these inverters (e.g. current PV power, current load, temperature, serialNumber, mode, etc.) - a total of 102 things. Each thing is connected to one item (1-to-1 mapping between thing and item). The things are created with the MQTT binding where any topic results in a separate thing.

I have 3 inverters running in parallel building L1 (named L1A, L1B, L1C) and the 3 others in parallel are building L2 (named L2A, L2B, L2C) of the 120V/240V US split phase system.

The names of all items and all things contains these name as part of the thing/item name to make it easy for me to identify the physical inverter because of the naming.

The MQTT topics have this typical form (just an example):
solar_assistant/inverter_2/device_mode/state

The inverter number is part of the topic (here it’s inverter_2). All six inverters result just in topic parts in the form inverter_<n> where runs from 1 to 6.

I’ve just used inverter_1 to inverter_3 to map them to L1A, L1B and L1C and the inverter_4 to inverter_6 to L2A, L2B and L2C.

The problem:
The sender of the MQTT messages changes the mapping after each reboot. E.g. in the previous situation the inverter_2 was really the physical inverters L1B but after reboot the MQTT sender (SolarAssistant) may assign the physical inverter L2C e.g. to inverter_2. There is no chance to get this problem fixed on the MQTT sender side.
As a result I have items for all inverters L1A, L1B, L1C, L2A, L2B and L2C but have no idea which physical inverter is really behind these values and it changes after every reboot of the MQTT sender (SolarAssistant).

The idea:
One of the items of each inverter is the serial number of the physical inverter. So in general it’s possible for me to identify the physical inverter by looking at the value of the item serialNumber. But I really want to map the items for L1A always to the (same) physical inverter L1A etc.
If it would be possible to decouple the thing from the item and make the assignment dynamically, depending of a specific value (serial number) of one of the MQTT values, it would be great.

How to achieve this?
I have some possible ideas in my mind, but I don’t know if one of them is useful or which may be the best. Maybe I can make all existing items to virtual items and create a rule to feed these virtual items to do the mapping in the rule. But for that I need to map the 102 existing things to one special item which may trigger the rule on any change (I think this is not possible in OH?). The rule would be able to do the dynamic mapping depending on the serial number.
An other possibility would be to get completely rid of the existing things and do an “raw” MQTT subscribe (I think this is possible in OH) in a rule to all inverter messages to do the dynamic mapping in a rule.
I don’t know if one of the OH transformation mechanisms may help in this situation?

Any ideas how to achieve this in a nice way?

This is quite doable using jruby’s dynamic thing/item builder. You’d just have to wait until the serial number is received then create the items (using code) accordingly.

You can also pre-create the items and only change the links dynamically.

I’ve never used JRuby scripting. Do you refer to the Dynamic Generation of Rules? As far as I understand, it only refers to rules not things/items.

OK, I just found it for things and items.

Does a similar approach/possibility exists for JavaScript? I was able to find a rule builder but not thing/item builder in the JS library?

Yes that’s it

I’m not familiar with js. I’m guessing you’ve checked the relevant docs?

I think that or dynamically changing the item links is the only sensible way without having to deal with all the edge cases.
How do you receive the data? Is it a big json? Depending on the data structure I’d pick an approach. If you like python it’s also possible in HABApp.

1 Like

Not really solving your problem, but could perhaps reduce it a bit: why do you have 17 separate Things for each inverter? You could have one Thing for each inverter, with 17 Channels instead. That way they would be grouped in a more logical way, possibly making the relinking of the Items to the Channels easier.

1 Like

Sorry, I’ve made a mistake while describing my situation. I’m already using just ONE MQTT Thing with 102 Channels. You’re right, I may change this to 6 Things with 17 Channels each to separate the inverters better on the logical side.

That’s one approach and it might be the least likely to feed some values to the wrong Item during the transition.

Yes, you can use wild card subscriptions. For example foo/bar/# will subscribe to all message to all topics under foo/bar.

This would be done by creating an event Channel on an MQTT Thing or even on the MQTT Bridge with a wild card subscription like described above. That Channel can directly trigger a rule. See MQTT Event Bus [4.0.0.0;4.9.9.9] for an example.

Probably not.

I worry that with this approach though there will be a brief period between when the change is detected and all the Items are relinked where updates will go to the wrong Items. Maybe the updates don’t come in that fast after the change is detected?

I’ll try to find a solution with JavaScript to dynamically relink the items.

I receive the data as simple MQTT values. Here are some examples:

...
solar_assistant/inverter_1/pv_power/state 2908
solar_assistant/inverter_3/battery_voltage/state 54.1
solar_assistant/inverter_5/device_mode/state Solar/Battery
...

Because the transition occur only after reboot of SolarAssistant (MQTT sender), it may happen only 3-4 times per year after updating SolarAssistant. So I will be able to live with the possibility of just a couple of wrong values.

Btw. There was a mistake in my previous posting, I’m using just one thing with 102 channels.

I know about these wildcard subscriptions but unfortunately I can only specify a + for a single level wildcard and/or # for a multi level wildcard, but I’m not able to have wildcards inside a topic level itself. So I need at least 6 subscribes like this (which should not be a problem):

solar_assistant/inverter_1/#
solar_assistant/inverter_2/#
solar_assistant/inverter_3/#
solar_assistant/inverter_4/#
solar_assistant/inverter_5/#
solar_assistant/inverter_6/#

I can’t use just solar_assistant/# because SolarAssistant also send me data for all 12 batteries which starts with solar_assistant/battery_<n>/....

I’ll keep this in mind as a plan B in case the other route doesn’t work.

Ok, I think I will try the following approach:

a) Changing the Thing configuration to have 6 different Things, one for each inverter with 1 Channel.
b) The Channel will contain the multi level wildcard like e.g. for inverter 1 it will be
solar_assistant/inverter_1/#
c) Define an Item for the Channel of the inverter Thing (6 times for all six inverters)
d) Implement a rule with the ItemStateChangeTrigger trigger of all 6 (MQTT wildcard) Items to receive all updates.
e) Implement the logic to map the SolarAssistant inverter number to the physical inverter L1A to L2C depending on the value of the received serialNumber field AND sendin topic, which contains the SolarAssistant inverter number implicitely.

I’ve just done a short test and found a problem with such a wildcard based MQTT item. I’m not able to find out which real topic was used to send the updates, I only see the values. I’ve tried different item properties like, type, name, label, state, rawState, groupNames, tags, etc. but non of them contains the sending topic of the value.

Any ideas how to identify the sender topic of a MQTT wildcard channel linked to an item?

Why is that ia problem? Just ignore the messages from the topics you don’t care about.

However, at least in OH 4.2 (maybe earlier, I don’t know when this was added) there is an optional payload property. Unfortunately, the name of the property and the description leaves some ambiguity.

An optional condition on the value of the MQTT topic that must match before this channel is triggered.

Maybe that can be used to ignore the battery topics.

In a rule you’d just need something like the following (JS Scripting, assuming # is the separator):

var payload = event.event.split('#'); // payload[0] is the topic, payload[1] is the message
if(payload[0].includes('inverter_')) {
  // process the message (payload[1])
}

Any message from the battery topics simply get dropped.

Or, if you add the check as a rule script condition for a managed rule

event.event.split('#')[0].includes('inverter_');

then in the action you don’t need to check. The actions won’t run unless inverter is in the topic.

That’s why I was careful to say above to use an event Channel, not one of the State Channels.

And note that you cannnot add an event Channel to a Generic Thing, only the Broker. But when you define it, you get to set a separator character.

Then set Thing Channel trigger on the rule and the value of event.event will be <topic>#<message> (assuming you chose # as the separator character).

All of this is discussed in the MQTT EvenBus example I linked to above.

Thanks, I didn’t think about filtering the data later outside of MQTT…

I’ve try to work through all the information and links you’ve provided, but I still struggle to find the string which contains the topic and the value.

My Thing configuration is (I’ve set trigger=true to get the channel events in the JavaScript rules and configured the separator to “#”):

Bridge mqtt:broker:mymosquittobroker "MQTT Mosquitto Broker (MQTT Broker)" @ "office" [ host="localhost", port=1883, secure=false, clientID="mymqttbridge", qos=0 ] {
...
//TEST  
  Thing topic SolarAssistantInv1 "SolarAssistantInv1" @ "rvgarage" { 
    Channels:
      Type string : states1 "Inverter1 states" [stateTopic="solar_assistant/inverter_1/+/state", trigger=true, separator="#"]
  }
...
}  // end of mqtt bridge

The generated code view for the channel in the GUI:

UID: mqtt:topic:mymosquittobroker:SolarAssistantInv1
label: SolarAssistantInv1
thingTypeUID: mqtt:topic
configuration: {}
bridgeUID: mqtt:broker:mymosquittobroker
location: rvgarage
channels:
  - id: states1
    channelTypeUID: mqtt:string
    label: Inverter1
    description: null
    configuration:
      retained: false
      postCommand: false
      trigger: true
      formatBeforePublish: "%s"
      stateTopic: solar_assistant/inverter_1/+/state
      separator: "#"

The JS rule which should receive the topic#value but just with logging output to better understand what the different properties contains:

rules.JSRule({
  name: "JS MQTT test rule",
  id: "JS_MQTT_test_rule",
  description: "MQTT test rule",
  triggers: triggers.ChannelEventTrigger("mqtt:topic:mymosquittobroker:SolarAssistantInv1:states1"),
  execute: (event) => {
    console.log("MQTT test", "event                      : " + event);
    console.log("MQTT test", "event.eventClass           : " + event.eventClass);
    console.log("MQTT test", "event.status               : " + event.status);
    console.log("MQTT test", "event.eventType            : " + event.eventType);
    console.log("MQTT test", "event.triggerType          : " + event.triggerType);
    console.log("MQTT test", "event.raw                  : " + event.raw);
    console.log("MQTT test", "event.payload              : " + event.payload);
    console.log("MQTT test", "event.payload.class        : " + event.payload.class);
    console.log("MQTT test", "event.receivedEvent        : " + event.receivedEvent);
    console.log("MQTT test", "event.receivedEvent.payload: " + event.receivedEvent.payload);
    console.log("MQTT test", "event[0]                   : " + event[0]);
    console.log("MQTT test", "event.event                : " + event.event);
    console.log("MQTT test", "event.payload[0]           : " + event.payload[0]);
    console.log("");
  }
});

Here is an example of the logging output:

MQTT test event                      : [object Object]
MQTT test event.eventClass           : org.openhab.core.thing.events.ChannelTriggeredEvent
MQTT test event.status               : undefined
MQTT test event.eventType            : triggered
MQTT test event.triggerType          : ChannelEventTrigger
MQTT test event.raw                  : undefined
MQTT test event.payload              : [object Object]
MQTT test event.payload.class        : undefined
MQTT test event.receivedEvent        : 2097
MQTT test event.receivedEvent.payload: undefined
MQTT test event[0]                   : undefined
MQTT test event.event                : undefined
MQTT test event.payload[0]           : undefined

event.event seems not existing
event.receivedEvent just contains the value 2097 in this example (but not the topic)
event and event.payload looks like an array with two objects but when I try to log them with event[0] and event.payload[0], both are undefined.

I’m still doing something wrong, but I don’t understand where the error is.

What does the serial number look like in the mqtt message? What’s the topic?

Does the serial number get sent before all the other stuff?

A typical serial number message topic with it’s value (just a number with 14 digits):

solar_assistant/inverter_1/serial_number/state 96xxxxxxxxxxxx

The serial number is a message with the retain flag set so each new subscribe will receive it right at the beginning.

The serial number message is not always the very first message, so the reassignment may take a short time and the other messages needs to be ignored until the serial number of all 6 inverters have been received and processed.

What happens when you restarted the mqtt broker? Would the inverters re-send the serial number to the broker? How much can we rely that we’ll get the serial number, without waiting too long (e.g. max an hour, not three months).

Post your current things/items/channels here. I’ll try to write a jruby script that does the relinking for you if you like.

Thanks a lot for your offer to implement the relinking in jruby. I still have hope to implement it in JavaScript and I would prefer to use the same language for all my rules/scripts (I never used Ruby before) if possible. But I would come back to your offer in case I would not be successful.

I thought about storing the current (last) valid mapping between the logical SolarAssistant inverter number and real physical inverter in a virtual item to make it persistent. This would guarantee at least that I will have the last correct mapping independent of a restart of SolarAssistant, MQTT broker or OpenHAB.
The idea is, as soon as I receive a (new) serial number topic, I’ll ignore all other messages until all 6 serial numbers have been received to be able to do a complete relinking.

That’s not the right type of Channel. Here is what a managed Broker Thing with a working event Channel looks like.

UID: mqtt:broker:broker
label: Mosquitto MQTT Broker
thingTypeUID: mqtt:broker
configuration:
  lwtQos: 2
  publickeypin: true
  clientID: something
  hostnameValidated: true
  lwtMessage: OFFLINE
  birthRetain: true
  secure: false
  certificatepin: true
  password: mind your business
  protocol: TCP
  qos: 2
  mqttVersion: V5
  host: somewhere
  enableDiscovery: false
  birthTopic: openhab/status
  keepAlive: 60
  birthMessage: ONLINE
  lwtTopic: openhab/status
  shutdownRetain: true
  reconnectTime: 60000
  port: 1883
  shutdownTopic: openhab/status
  shutdownMessage: OFFLINE
  lwtRetain: true
  username: some user
location: Global
channels:
  - id: dadsoh_out
    channelTypeUID: mqtt:publishTrigger
    label: Dad's OH EB out topics
    description: ""
    configuration:
      stateTopic: dadsOH/out/#
      separator: "#"

Notice:

  1. the Channel is defined on the Broker Thing (as described above), not a separate Generic MQTT Thing. This is not a trigger Channel on a Generic MQTT Thing. It’s something different.

  2. The channelTypeUID is mqtt:publishTrigger, not mqtt:string.

Until you get this part right I can’t comment on the rule. Though I do know that for file based JS rules the event Object gets recreated as a JS Object as opposed to the raw Java Object. Maybe the name is different from event.event. The docs indicate it might be receivedEvent. But that’s not going to include the MQTT topic until you correctly define the Channel.

My problem was that I didn’t fully understand the difference between a trigger channel defined on the MQTT Bridge vs defining it on a MQTT Thing and that’s the channel type publishTrigger is only available on a MQTT Bridge channel.

Now the event.receivedEvent contains the expected <topic>#<value> string.

Here are the the details:

things file:

...
Bridge mqtt:broker:mymosquittobroker "MQTT Mosquitto Broker (MQTT Broker)" @ "office" [ host="localhost", port=1883, secure=false, clientID="mymqttbridge", qos=0 ]

// global MQTT Bridge Channel Triggers
{
  Channels:
    Type publishTrigger : states1 "Inverter1" [stateTopic="solar_assistant/inverter_1/+/state", separator="#"]
}

// other MQTT Things with their Channels and/or Thing-Triggers
...

The optional configuration option trigger=true is not required here on the bridge channel. Maybe it’s only required if a trigger channel is defined on a MQTT Thing itself?

JavaScript rule:

rules.JSRule({
  name: "JS MQTT test rule",
  id: "JS_MQTT_test_rule",
  description: "MQTT test rule",
  triggers: triggers.ChannelEventTrigger("mqtt:broker:mymosquittobroker:states1"),
  execute: (event) => {
    console.log("MQTT test", "event.receivedEvent       : " + event.receivedEvent);
    console.log("MQTT test", "event.receivedEvent topic : " + event.receivedEvent.split('#')[0]);
    console.log("MQTT test", "event.receivedEvent value : " + event.receivedEvent.split('#')[1]);
    console.log("");
  }
});

example output:

MQTT test event.receivedEvent       : solar_assistant/inverter_1/pv_power_2/state#1967
MQTT test event.receivedEvent topic : solar_assistant/inverter_1/pv_power_2/state
MQTT test event.receivedEvent value : 1967

Now I can start to implement the dynamic mapping mechanism in the rule(s).

Thanks a lot for your help!

Correct, that’s only required to create a trigger Channel on a Generic MQTT Thing. You can only create an mqtt:publishTrigger on the Broker Thing so there’s no need to a separate property. They all are “trigger=true”.

Thank you for the confirmation!