Binding For Brutlech GEM Whole-home Energy Monitor?

I guess my question had more to do with how you are setting up the topics in btmon. As far as I can tell you can only setup one topic like “/house/energy” and then btmon publishes “/house/energy/01000813_ch1_w” as the the topic and some payload value. Or if you have mapped the channels then it might be something like this “/house/energy/main_w”. In either case though my power “main_w” or energy “main_wh” is always under the topic of “/house/energy”. What I am trying to figure out is how you managed to get the different topics for power, energy and volts like below

house/power/#
house/energy_total/#
house/volts/#

I am assuming I am missing something really obvious.

If I had it setup like you do it would have made it much easier to figure out the telegraf portion of moving this data around.

ah, ok. I misunderstood what you were looking for.

I use my btmon.py configuration to map from the physical Device/ChannelId of the GEM/ECM-1240, such as 01099999_ch1_w, into a set of logical names for the various time-series.

These names all have a common prefix (mqtt_base_topic) of “house” and all live in a btmon.py configuration file section that looks like the example below.

Other parts of this btmon.py configuration are used to push my Solar data to PVO and, at one point, I was pushing to SEG.

Anyhoo, when this is run the topic names are mapped before the data is published. Without a map, the default behavior is it construct the topic-name using the mqtt_base_topic, the Brultech DeviceId and the ChannelId.

Incoming value that might otherwise default to:

house/01099999_volts
house/01099999_ch1_w

when mapped, it’ll come out with topic names more like:

house/volts/volts
house/power/bath

This will (hopefully) let me sub-out the source with another if I need to change out a device down the line.

Overall, that part of the btmon.py config looks like:

[mqtt]
mqtt_out = true
mqtt_upload_period = 30
mqtt_host = yourMQTTHostHere
mqtt_base_topic=house
mqtt_user = yourAccountHere
mqtt_passwd = yourPasswdHere
mqtt_map = 01099999_volts,volts/volts,01099999_ch1_w,power/bath,01099999_ch2_w,power/upstairs_hall,01099999_ch3_w,power/hood,01099999_ch4_w,powe
r/furnace,01099999_ch5_w,power/master_bath,01099999_ch7_w,power/dryer,01099999_ch8_w,power/mains,01099999_ch9_w,power/disposal,01099999_ch10_w,p
ower/dishwasher,01099999_ch11_w,power/living,01099999_ch12_w,power/family,01099999_ch13_w,power/garage,01099999_ch14_w,power/solar,01099999_ch15
_w,power/unused15,01099999_ch16_w,power/unused16,01099999_ch17_w,power/kitchen_lights,01099999_ch18_w,power/sunpower,01099999_ch19_w,power/garag
e_lights,01099999_ch20_w,power/stairs,01099999_ch21_w,power/bedroom23,01099999_ch22_w,power/master,01099999_ch23_w,power/laundry,01099999_ch24_w
,power/microwave,01099999_ch25_w,power/kitchen_island,01099999_ch26_w,power/wine_fridge,01099999_ch27_w,power/main_fridge,01099999_ch28_w,power/
kitchen_outlets,01099999_ch29_w,power/unused29,01099999_ch1_wh,energy_total/bath,01099999_ch2_wh,energy_total/upstairs_hall,01099999_ch3_wh,...

My “mqtt_map” section is a lot longer, so I’ve clipped it so it’ll display well.

Ok I got it all working now. The part that I was missing was I had added an extra forward slash in the mqtt_map parameter:

I had mqtt_map = 01099999_volts,**/**volts/volts
instead of
mqtt_map = 01099999_volts,volts/volts

my telegraf setup was actually right to begin with and then when things weren’t working I assumed I had that setup wrong when it was really what I had in btmon that was wrong.

Off to start making some charts with Grafana now.

In a post above you provided a sample of your telegraf file and said you had 5 sections that looked similar. Did you mean you had 5 different “input.mqtt_consumer” plugins setup in your telegraf file or that you had defined 5 different topics like this?

topics = [
     "/house/volts/#",
	 "/house/power/#",
	 "/house/energy_total/#",
	 "/house/dif_energy/#",
        ]

When I setup more than one “input.mqtt_consumer” plugin in a single telegraf config file it doesn’t play nice and cause a bunch of issues so I just setup multiple topics which seems to be working fine.

I have 5 sections. It doesn’t really matter as long as it can gather the different bits correctly :slight_smile:

Mine is wasteful as I’ll have multiple concurrent connections.

Ok, that is what I thought you did.

I have a different question for you. I noticed you have solar, which I do to. In grafana are you showing your household consumption and if so how are you getting that value? Basically I would like to be able to show total house hold usage, which would be the mains minus the solar production, except that would be a calculated point and I haven’t figured out how to write a query to do it. Maybe a better option would be to have BTMON do it and supply it as another data stream. Along with that a stream showing net solar to the grid would be nice to.

I know that EMONCMS does show net solar to the grid and total usage for a house but I am not sure if that is calculated by EMONCMS or is provided by BTMON.

The raw elements I do in a table, akin to:

SELECT last("value") - first("value") 
FROM "house-energy-total" 
WHERE ("topic" = 'house/energy_total/mains' 
OR "topic" = 'house/energy_total/oven' 
OR "topic" = 'house/energy_total/ev' 
OR "topic" = 'house/energy_total/ac') 
AND $timeFilter GROUP BY "topic"

Where mains is the total of my subpanel, and sourced from the GEM. The other values are from my primary panel, and sourced from my ECM-1240

This can be simplified to show a SingleStat

I’ve been meaning to upgrade to InfluxDB 1.2, as that’ll give me the option to aggregate an aggregate. Then I can get the net production value more easily. I suspect I can do it directly, but I’ve not tried.

This complexity is only because I’m sourcing data from two Brultech devices at different time points.

So ultimately what I am trying to do is something like this, which is from EMONCMS but in Grafana

What I trying to figure out right now is how to get a chart like above with the actual household energy use, almost as if I didn’t have solar.
When I am producing a lot of solar energy (yellow) I want to be able to show the household consumption (green-blue) along with the net solar. Basically the yellow plus greenblue areas should add up to how much solar I am currently producing.

What I think this means for a query is I need to subtract the solar energy topic from the mains energy topic for every time interval that has been recorded. This is not the right query language but what I want to do is basically this

SELECT mean("value") FROM "Home" WHERE "topic" = '/house/power/main' AND $timeFilter GROUP BY time(1m) fill(null) 

minus

SELECT mean("value") FROM "Home" WHERE "topic" = '/house/power/solar' AND $timeFilter GROUP BY time(1m) fill(null)

Just an update on my questions above.

Apparently the way I have my data in influxdb makes it not possible to get what I want. See this thread I started here

@guessed Do you think it would be possible to make the BTMON.py MQTT uploader use tags and be in the following format or at least have the option to be in this format
AllingtonServ,tag1=tagValue house_power_main= 2323.419

Basically what I have been told is in order to do math on data like I describe above I need to write that data as fields.

Hi, I am busy setting up my 4 x GEM’s with ethernet (2 per panel), and I’d like them to post directly to a cloud hosted service.

Is it possible to use one of the various GEM packet formats and post directly to an instance of OpenHAB?

Less ideal, replace the GEM ethernet module with a Pi (2 x GEM serial to Pi) and go serial to Pi to OpenHAB (maybe run btmon on the Pi)?

Hello,

is there any update on this?

What is the easiest way to get the GEM working with openhab?

thanks
Daniel

MQTT using bmon is still the way to go. It works well, so I think there’s no big incentive to develop a specific binding.

Is there an explanation somewhere how to set that up`

thanks
Daniel

In short, unless you already have MQTT setup;

  1. Install a MQTT broker somewhere e.g. Mosquitto
    1a. Maybe download MQTTexplorer or similar tool to see “whats going on”
  2. install btmon, on a linux box (maybe possible also on windows, but I have no knowledge about that)
    2a. Check that btmon is populating the broker (using MQTTexplorer)
  3. Install MQTT binding in openhab, make it connect to your broker

Maybe you also need to install a MQTT client for btmon, looks like!
Create things and items in your favorite way.

both btmon and MQTT binding is quite good in terms of documentation. First topic and the rest of this thread will probably give you the answer to any question. Not sure if the btmon version in first post is the latest one. Especially if you have python 3 installed. mtools/btmon3.py at influxdb · BenK22/mtools · GitHub

Ok, I installed mosquitto, started btmon, and it seems to post stuff? (MQTTexplorer shows lots of messages, however I cannot find the content/payload, only that there are messages?).

I added the MQTT bridge in openhab, and it is shown as online.

Now how do I correctly add things (for other bridges I just pushed “scan” when adding new things and they showed up).

Thanks
Daniel

Sorry, I was wrong, it does not work on btmon side.

I get the error “Exception in MQTTProcessor: ‘ch1_a’”. Any ideas what that could be?

thanks
Daniel

So which version of btmon did you end up with? And which python version do you have installed? Did you install paho as well?

Yes I installed paho.

btmon 4.0.0 (from the link above), python 3.9.2

python3 btmon3.py --ip --ip-mode=client --ip-host=10.0.0.6  --ip-port=80 --print --device-type=gem

works as expected:

root@openhab:~/btmon# python3 btmon3.py --ip --ip-mode=client --ip-host=10.0.0.6  --ip-port=80 --print --device-type=gem
2021/12/10 15:01:51 btmon: 4.0.0
2021/12/10 15:01:51 python: 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110]
2021/12/10 15:01:51 platform: linux
2021/12/10 15:01:51 device type: gem
2021/12/10 15:01:51 device list: ['']
2021/12/10 15:01:51 packet format: gem48ptbin
2021/12/10 15:01:51 schema: counters
2021/12/10 15:01:51 SOCKET: timeout: 60
2021/12/10 15:01:51 SOCKET: server host: 10.0.0.6
2021/12/10 15:01:51 SOCKET: server port: 80

2021/12/10 15:01:59: Serial: 01011343
2021/12/10 15:01:59: Voltage:  241.80V
2021/12/10 15:01:59: Ch01:   -128.619799KWh (   54W)
2021/12/10 15:01:59: Ch02:   -326.071099KWh (   68W)
2021/12/10 15:01:59: Ch03:   -488.204104KWh (  272W)
2021/12/10 15:01:59: Ch04:      0.328359KWh (    8W)
2021/12/10 15:01:59: Ch05:     -0.002141KWh (    0W)
2021/12/10 15:01:59: Ch06:     -0.002517KWh (    0W)
2021/12/10 15:01:59: Ch07:     -0.000008KWh (    0W)
2021/12/10 15:01:59: Ch08:     -0.000216KWh (    0W)
2021/12/10 15:01:59: Ch09:   -143.683807KWh (   68W)
2021/12/10 15:01:59: Ch10:   -468.186038KWh (  214W)
2021/12/10 15:01:59: Ch11:    -10.443799KWh (    0W)
2021/12/10 15:01:59: Ch12:     -0.000008KWh (    0W)
2021/12/10 15:01:59: Ch13:     -0.000006KWh (    0W)
2021/12/10 15:01:59: Ch14:     -0.009339KWh (    0W)
2021/12/10 15:01:59: Ch15:     -0.000004KWh (    0W)
2021/12/10 15:01:59: Ch16:     -0.000006KWh (    0W)
2021/12/10 15:01:59: Ch17:     -0.000004KWh (    0W)
2021/12/10 15:01:59: Ch18:     -0.001485KWh (    0W)
2021/12/10 15:01:59: Ch19:     -0.000005KWh (    0W)
2021/12/10 15:01:59: Ch20:     -0.000061KWh (    0W)
2021/12/10 15:01:59: Ch21:     -0.000010KWh (    0W)
2021/12/10 15:01:59: Ch22:     -0.000063KWh (    0W)
2021/12/10 15:01:59: Ch23:     -0.000008KWh (    0W)
2021/12/10 15:01:59: Ch24:     -0.000009KWh (    0W)
2021/12/10 15:01:59: Ch25:     -0.000004KWh (    0W)
2021/12/10 15:01:59: Ch26:     -0.000124KWh (    0W)
2021/12/10 15:01:59: Ch27:     -0.000008KWh (    0W)
2021/12/10 15:01:59: Ch28:     -0.000015KWh (    0W)
2021/12/10 15:01:59: Ch29:     -0.000005KWh (    0W)
2021/12/10 15:01:59: Ch30:     -0.000063KWh (    0W)
2021/12/10 15:01:59: Ch31:     -0.000007KWh (    0W)
2021/12/10 15:01:59: Ch32:     -0.004349KWh (    0W)
2021/12/10 15:01:59: p1:               0
2021/12/10 15:01:59: p2:               0
2021/12/10 15:01:59: p3:               0
2021/12/10 15:01:59: p4:               0
2021/12/10 15:01:59: t1:        0.000000
2021/12/10 15:01:59: t2:        0.000000
2021/12/10 15:01:59: t3:        0.000000
2021/12/10 15:01:59: t4:        0.000000
2021/12/10 15:01:59: t5:        0.000000
2021/12/10 15:01:59: t6:        0.000000
2021/12/10 15:01:59: t7:        0.000000
2021/12/10 15:01:59: t8:        0.000000

however

python3 btmon3.py -c config.cfg

with my config.cfg

[source]
device_type = gem
ip_read = true
ip_host = 10.0.0.6
ip_port = 80
ip_mode = client
ip_poll_interval = 5
[mqtt]
mqtt_out = true
mqtt_upload_period = 5
mqtt_host = 127.0.0.1
mqtt_clientid = btmon-gem1

I get

2021/12/10 15:03:08 btmon: 4.0.0
2021/12/10 15:03:08 python: 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110]
2021/12/10 15:03:08 platform: linux
2021/12/10 15:03:08 device type: gem
2021/12/10 15:03:08 device list: ['']
2021/12/10 15:03:08 packet format: gem48ptbin
2021/12/10 15:03:08 schema: counters
2021/12/10 15:03:08 SOCKET: timeout: 60
2021/12/10 15:03:08 SOCKET: server host: 10.0.0.6
2021/12/10 15:03:08 SOCKET: server port: 80
2021/12/10 15:03:08 SOCKET: poll interval: 5
2021/12/10 15:03:08 MQTT: mqtt:127.0.0.1:1883?clientid=btmon-gem1
2021/12/10 15:03:08 MQTT: user: <not-specified>
2021/12/10 15:03:08 MQTT: tls: <not-specified>
2021/12/10 15:03:08 MQTT: topic: /house/energy
2021/12/10 15:03:08 MQTT: qos: 0
2021/12/10 15:03:08 MQTT: retain: False
2021/12/10 15:03:08 MQTT: will: <not-specified>
2021/12/10 15:03:08 MQTT: upload period: 5
2021/12/10 15:03:08 MQTT: map:
2021/12/10 15:03:19 Exception in MQTTProcessor: 'ch1_a'

The last line (Exception in MQTTProcessor: ‘ch1_a’) repeats every few seconds.

So it looks like there’s an issue in the MQTT connection or publishing. Maybe the broker does not accept anonymous connections, or something like that? I think mosquitto defaults changed on this some time ago, if I’m not mistaken.

I allowed anonymous connections, and there is nothing I can see that would link to this error in mosquitto.log