Filtering out rubish modbus readouts

[xsherlock (Maciej Eckstein)]

Hi all,
So I do that new setup of OH4.3 with a brand new 50kW Bluesun/Solis hybrid inverter.
I read out parameters from modbus using the RS485 to TCP converter, not hammering it to much every 5sec about 20 registers.

I noticed that once in a while I get a rubish value from it. I’m not sure what is at fault. OH, converter or the inverter itself.

The example here is Energy production tooday, (so it resets on midnight)

Those are last 2 days , and you see 9 times it did send some rubish.
The table below is 1 min window with max agregation to see that it sent 1390 and then retuned to proper value of 4 (threre is gain * 0.1 transform on it)

Number inverter_PV_Today_energy "PV_Today_energy [%.1f kWh]"  <energy> 					{ channel="modbus:data:remoteTCP:PV_energy:PV_Today_energy:number" [ profile="modbus:gainOffset", gain="0.1 kWh", pre-gain-offset="0" ]}

Have you ever seen such a behaviour? Maybe it is known Modbus poller problem? should I add some timeouts?
How to remedy that? If that cant be fixed at the poller I can imagine I can write a rule that will check previous value and make some calculation given that those values can only increment by some extent every 5-10 seconds apart.
But that is hell intensive on DB I guess. Anyone have some other idea?

here is a poller setup just for reference:

Bridge modbus:tcp:remoteTCP [ host="192.168.1.199", port=502 ]  {
   Bridge poller Battery1 [ start=33134, length=18, refresh=5000, type="input" ]  {
			 Thing data Battery1_Current		[ readStart="33134", readValueType="int16" ]
			 Thing data Battery1_Direction		[ readStart="33135", readValueType="uint16" ]
			 Thing data Battery1_SOC 			[ readStart="33139", readValueType="uint16" ]
			 Thing data Battery1_Voltage 		[ readStart="33141", readValueType="uint16" ]
			 Thing data Battery1_Current_BMS	[ readStart="33142", readValueType="int16" ]
   }
   
   Bridge poller Load_Total_Power [ start=33148, length=10, refresh=5000, type="input" ]  {
	         Thing data Load_Total_Power [ readStart="33148", readValueType="uint16" ]
			 Thing data Battery_Total_Power [ readStart="33149", readValueType="int32" ]
			 Thing data Grid_Total_Power [ readStart="33151", readValueType="int32" ]
   }
   
   Bridge poller PV_energy [ start=33029, length=24, refresh=5000, type="input" ]  {
	         Thing data PV_Total_energy 			[ readStart="33029", readValueType="int32" ]
			 Thing data PV_Current_Month_energy 	[ readStart="33031", readValueType="int32" ]
			 Thing data PV_Last_Month_energy		[ readStart="33033", readValueType="int32" ]
			 Thing data PV_Today_energy				[ readStart="33035", readValueType="int16" ]
			 Thing data PV_Current_Year_energy		[ readStart="33037", readValueType="int32" ]
			 Thing data PV_Last_Year_energy			[ readStart="33039", readValueType="int32" ]
			 
   }
}

[rlkoshak (Rich Koshak)]

@xsherlock

Have you ever seen such a behaviour?

I don’t use modbus so not yes specifically. But it’s not unusual for sensors to periodically throw a bogus reading every now and then.

@xsherlock

How to remedy that? If that cant be fixed at the poller I can imagine I can write a rule that will check previous value and make some calculation given that those values can only increment by some extent every 5-10 seconds apart.
But that is hell intensive on DB I guess. Anyone have some other idea?

Use one of Basic Profiles - Transformation Services | openHAB or a script transform profile (e.g. Filter Profile) to filter out those erroneous measurements based on what ever criteria you think makes the most sense.,

There won’t be any DB issues because the state is filtered out before it ever gets to the Item.

[xsherlock (Maciej Eckstein)]

Ok great, many thanks for the clue, filtering before it gets to the influxdb is the way to go.

however I hit a minor bump , I already have the profile="modbus:gainOffset" in that item,
What would be the correct syntax to have both in the same item, I try this:

Number inverter_PV_Today_energy "PV_Today_energy [%.1f kWh]"  <energy> 	{channel="modbus:data:remoteTCP:PV_energy:PV_Today_energy:number" [ profile="modbus:gainOffset", gain="0.1 kWh", pre-gain-offset="0" , profile="basic-profiles:state-filter", conditions="$DELTA < 2" ]}

And while it does not throw the error I see in log:

No ProfileFactory found which supports profile 'basic-profiles:state-filter' for link 'inverter_PV_Today_energy -> modbus:data:remoteTCP:PV_energy:PV_Today_energy:number'

---

oops ignore basic-profiles was not installed
lets see now if both profiles execute in the same time

[jlikonen (Jari Likonen)]

Which rs485/tcpip converter do you use? I have been using Moxa MGate converters and they have been very reliable. What is the distance between the converter and the inverter? What is the baud rate in the converter? You could perhaps try to decrease the baud rate and see if that has any effect?

[rlkoshak (Rich Koshak)]

I’m pretty sure you can’t use more than one profile at a time. You’ll probably have to use a script transform profile to do the math and do the filtering in one script.

[xsherlock (Maciej Eckstein)]

@jlikonen

I use USR-DR302, A cheapo from amazon that I have seen in some post here were ok.

Tried a waveshare before that did not work nice.

I have some Moxa’s Modbus to MQTT installs for industrial applications and also find them ultra reliable, however they are very expensive.

The cable is barely 2m. And it is 9600 bauds. that is preset in inverter and cant change it.

@rlkoshak

Indeed, adding the basic-profiles:state-filter killed the modbus:gainOffset functionality so I have to do the single script… Oh jesus I have no knowleage of the JS at all I’m the python guy, but lets try…

[jlikonen (Jari Likonen)]

OK, I don’t think the speed is the issue because I’m using e.g. 57600 in my installations without any issues. So, it could be that your converter is causing the problems. You can find rather cheap Moxa units in Ebay. I have one Moxa Mgate 3170 which I don’t need so if you are interested in it please PM me.

[rlkoshak (Rich Koshak)]

@xsherlock

Indeed, adding the basic-profiles:state-filter killed the modbus:gainOffset functionality so I have to do the single script… Oh jesus I have no knowleage of the JS at all I’m the python guy, but lets try…

If you have Jython installed you can use Python 2.7. Rules DSL, jRuby, and Blockly are all also available.

[xsherlock (Maciej Eckstein)]

Ok, not to many examples in other languages I’m working basing on your Filter Profile example. Have some questions as your script is used in Main UI and not sure if I have all the functions available

First I would use in items file as profile
[profile="transform:JS", toItemScript="divide_by_10_and_checkDelta.js" ]

Is that right place?

Then I see you do following

  const curr = Quantity(data);
  const last = cache.private.get(id, () => curr);
  const delta = curr.subtract(last);

data is the incoming value ,yes?
id is the name of the item that I need to call from cache? to get the last (previous) value.
can I pass it in the item definition like below?

[profile="transform:JS", toItemScript="divide_by_10_and_checkDelta.js?id=inverter_PV_Today_energy" ]

Then my function would look like:

(function(data,id) {
	var current = parseFloat(data) / 10;
 	const curr = Quantity(current);
	const last = cache.private.get(id, () => curr);
	const delta = curr.subtract(last);
 	if(delta.greaterThanOrEqual(2) || delta.lessThanOrEqual(-2)) {
		console.debug('Threshold exceeded, bogus value ' + curr);
		return null;
  	}
	else {
	cache.private.put(id, current);
	return current;
	}
})(input,id)

I don’t understand what Quantity() does, just copied it from your example, can it be ommited?

Does that make sense?

[rlkoshak (Rich Koshak)]

@xsherlock

Is that right place?

It’s not a wrong place. there isn’t a “right” place. file based configs and managed configs are both supported.

@xsherlock

data is the incoming value ,yes?

There is a JS convention to put a floating function like this into a self executing anaymous function. It’s not strictly required here (we are not running in a browser with a dozen third party libraries and the like) but all the examples you’ll see use this convention so I follow it as well.

The data to be transformed gets passed into the transform as a String called input. In order avoid confusion, I use a different name in the argument to the self calling function. That’s this part:

(function(data, id, threshold) {
  ...
})(input, id, threshold)

The (function(data,id,threshold){ }) part is defining the self calling function and the (input, id, threshold) part is calling that function. As you can see, input is passed as the data argument to the function.

I probably should have been consistent and use a different name for id and thershold but I was in a hurry when I wrote that post.

@xsherlock

id is the name of the item that I need to call from cache? to get the last (previous) value.
can I pass it in the item definition like below?

You see two other arguments. these are passed in to the script (see JavaScript Scripting - Automation | openHAB).

3. Passing parameters is also possible by using a URL like syntax: JS(<scriptname>.js?arg=value). Parameters are injected into the script and can be referenced like variables.

Not only can you pass the id and the threshold, you must. Note that the id may be anything you want it to be but it must be unique across all uses of this transformation across all of OH. So just the Item name may or may not be sufficient.

@xsherlock

I don’t understand what Quantity() does, just copied it from your example, can it be ommited?

See JavaScript Scripting - Automation | openHAB

It’s what you use when your data is a number with units. Quantity will also work with values that do not have units so in my version I use Quantity for everything so that the transform handles values with unit and without units.

If you know this transform will never include units, do not use Quantity. But you should also use the regular operations.

(function(data,id) {
 	const curr = Quantity(data);
	const last = cache.private.get(id, () => curr);
	const delta = curr.subtract(last);
 	if(delta.greaterThanOrEqual(Quantity("2 A")) || delta.lessThanOrEqual(Quantity("-2 A"))) {
		console.debug('Threshold exceeded, bogus value ' + curr);
		return null;
  	}
	else {
  	  cache.private.put(id, current);
	  return current;
	}
})(input,id)

Without Units

(function(data,id) {
 	const curr = parseFloat(data) / 10;
	const last = cache.private.get(id, () => curr);
	const delta = curr - last;
 	if(delta >= 2 || delta <= -2) {
		console.debug('Threshold exceeded, bogus value ' + curr);
		return null;
  	}
	else {
  	  cache.private.put(id, curr);
          // Don't you need to do some math here to replecate the offset profile operations?
	  return curr;
	}
})(input,id)

If you want to use this to add units to the value, first set the unit metadata on the Item to the units you want to use (e.g. “kA”).

(function(data,id) {
 	const curr = Quantity(data + "A"); // use what ever unit the value is comming from modbus as
	const last = cache.private.get(id, () => curr);
	const delta = curr.subtract(last);
 	if(delta.greaterThanOrEqual(Quantity("2 A")) || delta.lessThanOrEqual(Quantity("-2 A"))) { // use what ever units you want to use for the comparison, they do not need to match the units of curr
		console.debug('Threshold exceeded, bogus value ' + curr);
		return null;
  	}
	else {
  	  cache.private.put(id, current);
	  return curr;
	}
})(input,id)

curr remains in what ever unit it was modbus passed. However, OH will automatically convert it to kA since that’s what you used by setting the unit metadata. You don’t have to do the divide by 10, just use units and set the unitmetadata to the units you want to use.

[jimtng]

You could solve this by creating a profile in JRuby:

• Install jrubyscripting automation addon
• Create <CONF>/automation/ruby/div10_filter.rb (you could put this file in a profile subdirectory under ruby if you wish to further organise the files but that’s optional). Note the file name can be anything here. It isn’t used as an identifier for your profile. The argument to the profile method below determines the profile name.

@previous = {}

profile(:div10filter) do |event, item:, callback:, state:|
  next unless event == :state_from_handler

  new_data = state.to_f / 10
  if (prev_data = @previous[item])
    delta = (prev_data - new_data).abs
    next if delta > 2 # discard
  end

  @previous[item] = new_data
  callback.send_update(new_data)
end

Your item:

Number inverter_PV_Today_energy "PV_Today_energy [%.1f kWh]"  <energy> 					{ channel="modbus:data:remoteTCP:PV_energy:PV_Today_energy:number" [profile="ruby:div10filter"]}

The main benefit of this approach is you don’t need to add a unique id in the profile config for each item, thus making it easier to use the profile.

If you’d like to customise the delta for each use of the filter, you could add an argument too, so the item will look like Number Xxxx {channel="xxx" [profile="ruby:div10filter", max_delta=2]}. The script would need to be modified slightly to use this configuration. Also if max_delta is not specified, you could have a default delta.

[xsherlock (Maciej Eckstein)]

Many thanks for detailed explanation and examples, I’m now running Rich solution and will see if that populates DB in a nice way

Just to make the solution complete I’m making a note that my coresponding item looks the following

 Number inverter_PV_Today_energy "PV_Today_energy [%.1f kWh]" <energy> { channel="modbus:data:remoteTCP:PV_energy:PV_Today_energy:number" [profile="transform:JS", toItemScript="divide_by_10_and_fix.js?id=PV_Today_energy"]}

The ruby solution is indeed more elegant however Ruby looks just alien to me, I would not be able to fix a single coma in that code.

Thanks