Modbus unable to read uint32 registers on the same data poller

pjpeartree · September 28, 2021, 11:17am

Hi all,

I have a smart meter, and I can connect to it using the serial port, using openhab3 (openhabian 3.1 on rpi 4b)

Also, I can successfully read several registers on the same Poller for value types “uint16”. ( Check powermeter.things voltage and current data things)

Sadly, I am not able to do the same for “uint32” value types. Just if I create an individual poller for each register I can read the uint32 values.

Bridge modbus:serial:powermeter [ port="/dev/ttyAMA0" ,id=1 , baud=9600, stopBits="1.0", parity="none", dataBits=8, encoding="rtu" ]{

Bridge poller instantaneous [ start=108, length=2, refresh=5000, type="input" ]{    
        Thing data voltage      [ readStart="108", readValueType="uint16", readTransform="JS(divide_by_10.js)"]
        Thing data current      [ readStart="109", readValueType="uint16", readTransform="JS(divide_by_10.js)"]
    } 

Bridge poller instantaneouspower [ start=121, length=4, refresh=5000, type="input" ]{    
        Thing data power_import [ readStart="121", readValueType="uint32"]
		Thing data power_export [ readStart="122", readValueType="uint32"]
    } 
}

I have tried with several length values, without luck.

The smart meter documentation shows the following information:

Data type definition:

I have tried to read the same input registers with mbpoll with the following result:

openhabian@openhabian:~ $ mbpoll -m rtu -a 1 -b 9600 -d 8 -P none -v -s 1 -t 3:int -c 1 -1 -r 122 -o 10 -0 -B  /dev/ttyAMA0
debug enabled
Set rtu stop bits=1
Set function=3
Set format=int
Set number of values=1
Set start reference=122
Set timeout=10
Set device=/dev/ttyAMA0

Opening /dev/ttyAMA0 at 9600 bauds (N, 8, 1)
Set response timeout to 10 sec, 0 us
Protocol configuration: Modbus RTU
Slave configuration...: address = [1]
                        start reference = 122, count = 1
Communication.........: /dev/ttyAMA0,       9600-8N1
                        t/o 10.00 s, poll rate 1000 ms
Data type.............: 32-bit integer (big endian), input register table

-- Polling slave 1...
[01][04][00][7A][00][02][50][12]
Waiting for a confirmation...
<01><04><06><00><00><02><7D><03><63><B1><EA>
Message length not corresponding to the computed length (11 != 9)
Read input register failed: Invalid data

I would like to avoid individual pollers by the register. The Smart meter does not like too many requests.

Thank you
Pedro

rossko57 · September 28, 2021, 11:24am

The docs are misleading/wrong in some way.
If register x0079 is two consecutive 16-bit words, it would be x0079 and x007A
The following double register cannot be x007A as well.

You’ll have to experiment a bit to find out what is really going on.

EDIT - another way to interpret the table is
121 - power sum high word
122 - power sum low word
leading to

Bridge poller instantaneouspower [ start=121, length=2, refresh=5000, type="input" ]{    
        Thing data power_sum [ readStart="121", readValueType="uint32"]
    }

Regarding your mbpoll error -
Bear in mind many devices will simply reject attempts to read what they know are 32-bit word pairs if you start at the wrong odd/even address, and/or if try to read odd quantity of words instead of multiple of 2.

pjpeartree · September 29, 2021, 10:22am

The plus and the minus signs in the documentation are the power import (+) and the power export(-) instantaneous amounts.

I also have a tasmota modbus-> MQTT gateway, that can read the same registers with the following request (via script):

→ r010400790003

The response decoder is this:

1,01040aUUuuUUuu@i4:1,Active Power Import ,W,ActPow,16
1,01040axxxxxxxxUUuuUUuu@i4:1,Active Power Export ,W,ActPowExp,16
1,01040axxxxxxxxxxxxxxxxUUuu@i4:1000,Power Factor ,pu,PowFac,19

Can the Smart Meter, have physical registers with 32bit sizes on the specified addresses?

rossko57 · September 29, 2021, 10:56am

Modbus protocol is strictly 16-bit input registers.
How we, the users or product designers, choose to interpret those (numbers, characters) or group them together ( 2 bytes, 64-bit words) is entirely up to us.

Bottom line; so far as Modbus goes (and the OH binding, and mbpoll) you can’t have a 32-bit register pair at 120 and a different 32-bit pair at 121, because you only get 16 bits at 120 and another 16 bits at 121.

I’ve no idea what any of that means.
If you do, convert it to mbpoll and make that work.

Your actual device is still a secret, means you get a lot of guesses here.

Beware “off by one” addresses; some software and docs choose to number registers starting at 1, but the protocol address for that is 0 (as used in openHAB)

pjpeartree · September 29, 2021, 11:28am

I can read the voltage and the current based on the document addresses, without any problem. I am assuming that the document addresses are ok.

If I read these two registers in independent pollers:

I can read the values, and they are look good;

Same for Energy registers:

rossko57 · September 29, 2021, 11:39am

Sure, but they are not correct.

Please paste text not screenshots to make it easier to highlight what you’re doing.

Poll 121 length 2, get 16-bit registers 121 and 122 back. Contents say X and Y.
Assemble into 32-bit word XY

Poll 122 length 2, get 16-bit registers 122 and 123 back. 122 Contents we already know is Y, 123 is say Z.
Assemble into 32-bit word YZ

They can’t both be right. Stuff can look right when there is zeros involved, because all zeros look alike.

Any chance of revealing the secret meter model?

pjpeartree · September 29, 2021, 12:11pm

Sure, the top top secret meter model is; sagemcom s212

I also have taken a photo with the total power import to check if the value is correct:

and looks like…

Playing around with mbpoll:

Requesting the 2 registers from 22:

[01][04][00][16][00][02][90][0F]

Response:
<01><04><08>|<00><33><59><AA>|<00><08><41><6C>|<82><60>

Requesting the 2 registers from 23:

[01][04][00][17][00][02][C1][CF]
Response:
<01><04><08>|<00><08><41><6C>|<00><00><2D><7E>|<92><65>

Requesting the 4 registers from 22:
[01][04][00][16][00][04][10][0D]

Response:
<01><04><10>|<00><33><59><AA>|<00><08><42><08>|<00><00><2D><7E>|<00><00><05><86>|<A5><F0>
Message length not corresponding to the computed length (21 != 13)

rossko57 · September 29, 2021, 1:12pm

This device is not Modbus compliant.

First transfer, you request 2 registers (4 bytes) and get 8 data bytes back.

Third transfer is similar, you request 4 regs and get back 16 bytes instead of 8.

As it happens, knowing the secret model did not help, after all. I can’t find a proper manual for it online.
I don’t suppose it has a configuration option for “ordinary modbus” versus “super special nearly modbus”?

pjpeartree · September 29, 2021, 4:14pm

The EMI - HAN protocol specification can be found here: https://www.e-redes.pt/sites/eredes/files/2020-07/DEF-C44-509.pdf

This is from the grid operator. They have several Meters devices and all of them use the above specification.

rossko57 · September 29, 2021, 6:56pm

Okay, well unfortunately here we are working with a Modbus binding, not an EMI binding.
As it says in the document -

The communication protocol used in this interface is Modbus with some specificities.

EMI is customised, modified from Modbus. It’s not Modbus.

EMI breaks the Modbus specification by not keeping to 16-bit registers - for an example in the doc, see section 5.4.1.1 where one input register (x04) is requested, and six data bytes are returned - instead of the standard two. They’ve just invented a 48-bit register.

Alright, so you got what you got, can we make it work? Maybe.
A strict Modbus master would just throw away responses like that -“That’s not what I asked for !”.

But I think the openHAB binding is more forgiving though - just so long as the response packet is a valid Modbus structure, we don’t check that there are actually the expected number of bytes for that last query, just check that we have the byte count correct for that packet.

We can map EMI to Modbus registers with care.
EMI 32-bit reg 121 maps to two regular Modbus 16-bit registers.
EMI 32-bit reg 122 maps to two regular Modbus 16-bit registers.
From your experiments, the EMI device is using the requested poll address not as a regular Modbus address but as its own internal register reference. That’s a good thing, otherwise the address mapping just would not work.

So, if we send a regular Modbus poll request for “address” 121, length 2, we get back two of the EMI 32-bit registers, 8 bytes. From regular Modbus point of view, this looks as though we have got 16-bit registers 121 to 124.
Modbus 121 + 122 correspond to EMI 121
Modbus 123 + 124 correspond to EMI 122

Note that if we change the address we poll for, it will also change the mapping of the target registers.
For example if we poll 122 length 1, now Modbus 122 + 123 corresponds to EMI 122.
We’ll have to work out the correspondence by hand for each poll type.

First draft of openHAB Thing

Bridge poller instantaneouspower [ start=121, length=2, refresh=5000, type="input" ]{    
      Thing data power_import [ readStart="121", readValueType="uint32"]
      Thing data power_export [ readStart="123", readValueType="uint32"]

would fetch what appears to be four Modbus registers 121-124

But that won’t work. openHAB will say “don’t be silly with readStart="123", you only asked for length=2, that’s 121-122 so you can’t look at 123 ! Configuration error.”

We have to fetch more data than we need so that openHAB thinks we have the asked for enough Modbus registers.- same but with length=4 this time.

Bridge poller instantaneouspower [ start=121, length=4, refresh=5000, type="input" ]{    
      Thing data power_import [ readStart="121", readValueType="uint32"]
      Thing data power_export [ readStart="123", readValueType="uint32"]

Now we’ll get twice the length back we really want, but what we did ask for does include the Modbus addresses that we really want to look at.

This will only work as long as the EMI device will let us ask for what it thinks are EMI registers 123 and 124. If it rejects requests because those registers don’t exist, you will just have to reshape your poller start/length to get acceptable queries.
Try it and see.

ssalonen · September 29, 2021, 7:13pm

This is no longer true. Indeed the binding is strict and checks the response against the request.

This was introduced to detect corrupted and invalid data we saw with some modbus serial gateways (mixing up requests and responses, fixed by gateway reboot) (original issue [modbus] Handle invalid slave responses gracefully · Issue #7159 · openhab/openhab-addons · GitHub)

rossko57 · September 29, 2021, 7:35pm

Ah, that ruins the plan then.

Do I understand correctly that the invalid data caused an error when it was too short? That opens up the possibility to accept “too long but otherwise valid” data (correct byte count, correct CRC), perhaps?

Although it does seem to be working for OP

Must say I’ve never seen anything like this EMI variation. It just seems to break the rules for no apparent gain or purpose.

ssalonen · September 30, 2021, 3:47am

Correct. Too short data is problematic indeed (data things referring to data that is not there).

The check is now both ways though: too long / too short both yield an error.

pjpeartree · September 30, 2021, 9:36am

Hi,

Thanks for all your effort to make the problem clear. I have now understood the issue.

Based on your explanation, I have made some additional tests.

Test 1:

Bridge poller instantaneouspower [ start=121, length=4, refresh=5000, type="input" ]{    
      Thing data power_import [ readStart="121", readValueType="uint32"]
      Thing data power_export [ readStart="123", readValueType="uint32"]
}

Openhab gives a CRC error:

Try 1 out of 3 failed when executing request (ModbusReadRequestBlueprint [slaveId=1, functionCode=READ_INPUT_REGISTERS, start=121, length=4, maxTries=3]). Will try again soon. Error was I/O error, so reseting the connection. Error details: net.wimpi.modbus.ModbusIOException I/O exception: IOException CRC Error in received frame. 0 bytes of payload () with invalid CRC of 01 84  [operation ID f6135395-82fd-4533-afb6-aa88c6dee4f7]

The same request on mbpoll:

[01][04][00][79][00][04][20][10]
Waiting for a confirmation...
<01><84><02><C2><C1>
ERROR Illegal data address
Read input register failed: Illegal data address

Documentation error description:

ILLEGAL DATA ADDRESS: The data address received in the query is not an allowable address for the server (or slave). More specifically, the combination of the reference number and transfer length is invalid. For a controller with 100 registers, a request with offset 96 and length 4 would succeed a request with offset 96 and length 5 will generate exception 02.

This makes sense since your explanation, we are trying to fetch an invalid data address. But I guess that the EMI device just throws this error because the data type of the contiguous address is different:

Test 2:

Energy poller:

Bridge poller energy [ start=22, length=4, refresh=5000, type="input" ]{    
      Thing data import [ readStart="22", readValueType="uint32"]
      Thing data export [ readStart="24", readValueType="uint32"]
}

Openhab does not throw any error. I can use the same poller to fetch 2 EMI registers.

The same request on mbpoll give us:

-- Polling slave 1...
[01][04][00][16][00][04][10][0D]
Waiting for a confirmation...
<01><04><10><00><33><70><D0><00><08><49><A8><00><00><2D><7E><00><00><05><86><A3><C1>
Message length not corresponding to the computed length (21 != 13)
Read input register failed: Invalid data

Looks that Openhab isn’t being strict with the data length.

Knowing this EMI device limitations I was able to adapt my .things and reduce the number of pollers from 14 to 9.

My current .things:

Bridge modbus:serial:powermeter [ port="/dev/ttyAMA0" ,id=1 , baud=9600, stopBits="1.0", parity="none", dataBits=8, encoding="rtu" ]{

// Instantaneous EMI Registers
Bridge poller instantaneous [ start=108, length=2, refresh=5000, type="input" ]{    
        Thing data voltage      [ readStart="108", readValueType="uint16", readTransform="JS(divide_by_10.js)"]
        Thing data current      [ readStart="109", readValueType="uint16", readTransform="JS(divide_by_10.js)"]
    } 

Bridge poller power_import [ start=121, length=2, refresh=5000, type="input" ]{    
        Thing data power_import [ readStart="121", readValueType="uint32"]
    } 

Bridge poller power_export [ start=122, length=2, refresh=5000, type="input" ]{    
        Thing data power_export [ readStart="122", readValueType="uint32"]
    } 

Bridge poller power_factor [ start=123, length=1, refresh=5000, type="input" ]{    
        Thing data power_factor [ readStart="123", readValueType="uint16", readTransform="JS(divide_by_1000.js)"]
    } 

Bridge poller frequency [ start=127, length=1, refresh=30000, type="input" ]{    
        Thing data frequency    [ readStart="127", readValueType="uint16", readTransform="JS(divide_by_10.js)"]
    } 

Bridge poller active_tariff [ start=11, length=1, refresh=5000, type="input" ]{    
        Thing data active_tariff [ readStart="11.1", readValueType="uint8"]
    } 

// Totals  EMI Registers

Bridge poller energy [ start=22, length=4, refresh=5000, type="input" ]{    
      Thing data import [ readStart="22", readValueType="uint32"]
      Thing data export [ readStart="24", readValueType="uint32"]
}

Bridge poller tariff_energy_export [ start=45, length=6, refresh=5000, type="input" ]{    
        Thing data tariff_1 [ readStart="45", readValueType="uint32", readTransform="JS(divide_by_1000.js)"]
        Thing data tariff_2 [ readStart="47", readValueType="uint32", readTransform="JS(divide_by_1000.js)"]
        Thing data tariff_3 [ readStart="49", readValueType="uint32", readTransform="JS(divide_by_1000.js)"]
    }

Bridge poller tariff_energy_import [ start=38, length=6, refresh=5000, type="input" ]{    
        Thing data tariff_1 [ readStart="38", readValueType="uint32", readTransform="JS(divide_by_1000.js)"]
        Thing data tariff_2 [ readStart="40", readValueType="uint32", readTransform="JS(divide_by_1000.js)"]
        Thing data tariff_3 [ readStart="42", readValueType="uint32", readTransform="JS(divide_by_1000.js)"]
    }

}

rossko57 · September 30, 2021, 11:26am

Oh good lord, what a horrible device. I’ll guess it is returning a mixed payload with the different register types.
“Get me 4 registers please”
“Sure, here are 4 bytes, 4 bytes , 2 bytes, 2 bytes, and the CRC”

I think you’ve done very well to work around these issues

What openHAB version, out of interest?
I would caution that an update might one day stop all this working if it gets stricter about expected/received.

rossko57 · September 30, 2021, 11:51am

I’m not sure the best way to accommodate this bonkers standard going forward, Sami.
This EMI-HAN protocol does seem to be a smartmeter standard for at least one Portuguese supplier, it is going to crop up again.

I had thought that just “over long” responses would be the problem, but this EMI includes byte-size registers too. So I reckon some combinations will give an “under size” response as well, sending us just one byte for each requested ‘register’.

As you know failure to detect under-, over-sized messages are a nasty problem in other circumstances, we will not want to do away with that checking.

I’m reluctant to suggest yet another obscure configuration option, but it may be necessary here.
I’m suggesting a default action to check expected response lengths should be retained.
But a new poller parameter (checkresponselength=false or something?) regresses to checking only the CRC against the byte count actually in the message.

I think then it would be fiddly but possible for the user to build ‘standard’ poller and data Things to access every HAN register with that.

Or would a binding extension like the solar ones be better suited, is that able to deviate from standard at the low level required?

pjpeartree · September 30, 2021, 12:03pm

runtimeInfo:
version: 3.1.0
buildString: Release Build
locale: en-PT
systemInfo:
configFolder: /etc/openhab
userdataFolder: /var/lib/openhab
logFolder: /var/log/openhab
javaVersion: 11.0.12
javaVendor: Azul Systems, Inc.
javaVendorVersion: Zulu11.50+19-CA
osName: Linux
osVersion: 5.10.60-v7l+
osArchitecture: arm
availableProcessors: 4
freeMemory: 59948752
totalMemory: 194772992

splatch · September 30, 2021, 2:43pm

I recently wrote a profile to manage a bit different, yet in some ways similar issue. I had two registers which returned combined value. One was a MWh part and later kWh part of total energy consumption. From modbus perspective it was split in two uint16 registers.
To avoid creation of two items just to sum these values I read it as uint32 and then do basic bit manipulation to create first and second part of number.

For your use case solution could be similar, yet your device returns more data than it should. If your meter uses 32 bit per address instead of standard 16 any modbus client will break out. If it would return properly aligned data then you can read two registers of yours as 64 bit number (4x16 in proper modbus). Then it would have a chance to work. Data parsing in terms of modbus is positional. Have a look below.

Read request - registers 1-2 for emi and 1-8 for modbus (depending how you count)

+--------------+-----+-----+-----+-----+-----+-----+-----+-----+
| position     |  1  |  2  |  3  |  4  |  5  |  6  |  7  |  8  | 
+--------------+-----+-----+-----+-----+-----+-----+-----+-----+
| value (hex)  |  A  |  B  |  C  |  D  |  E  |  F  |  A  |  B  |
| modbus offset|     0     |     2     |     4     |     6     |
| modbus uint32|     AB    |     CD    |     EF    |     AB    |
| emi-han uint3|          ABCD         |          EFAB         |
| emi-h. offset|           0           |           2           |

From this point of view you increase registers by 4 and not by 2 as for normal modbus.
You still need to take care about scaling/transforming value. This trick will work only if CRC is fine and response data block is aligned with offset logic defined by modbus protocol.

ssalonen · September 30, 2021, 3:56pm

You are right – I was too hasty when reading the code. Indeed, we allow more data than requested. From java docs:

Check that number of bits/registers/discrete inputs is not less than what was requested.

According to modbus protocol, we should get always get always equal amount of registers data back as response.

With coils and discrete inputs, we can get more since responses are in 8 bit chunks.

However, in no case we expect less items in response.

This is to identify clearly invalid responses which might cause problems downstream when using the data.

https://github.com/openhab/openhab-core/blob/338204341b76754f3656f459e9017ab22b246091/bundles/org.openhab.core.io.transport.modbus/src/main/java/org/openhab/core/io/transport/modbus/internal/ModbusManagerImpl.java#L179-L181

pjpeartree · September 30, 2021, 4:10pm

I hope that my current configuration, can survive to future updates