Binding For Brutlech GEM Whole-home Energy Monitor?

Add-ons Bindings
, ,
21

The House*Solar Items (above) illustrate my Solar Panels, which are attached to one of the Brultech GEM Power inputs. All of my channels look like that, but I kept it short/simple here because it’s quite long once you have all the channels entered.

On the btmon.py side of things, there are similar “aliases” that map from the default/internal channel names (eg. ch23_w) to these more readable forms. That’s not strictly required, but I wan tto use MQTT directly, and outside of openHAB, so I wanted the names to be readable.

Example of how to do that level of mapping is inline in the extension code that I’ve proposed to Matthew Wall.

eg. https://github.com/mrguessed/btmon/blob/mqtt/src/btmon.py#L277

All of this requires that you have an MQTT Broker around, and I use mosquitto.

There are a few pages on how to get this going, and if you leave out TLS (and use un/pw auth) it’s very simple to get going - esp on an rPi). In my case, it’s currently running on the same node as my openHAB instance (an ODroid C1+)

PS: As you go through this, feel free to put together a Wiki page of pointers. We can then edit it together and/or correct/clarify things as needed.

22

Mark,

Your work is great. This enables me to move beyond emoncms and add events.

I was able to get my ECM-1240 data to OpenHAB in a few hours. One item I noted after running overnight:

My water heater would not update when it is off. The script determines there is no payload to send, so nothing is sent. The openHAB bus receives no update, therefore the water heater looks like it never turns off, just changes wattage in between cycles.

I went to line 3114 and commented out the following:

 def _add_msg(self, packet, channel, payload):
     #  if not payload:
      #     return

While this may not be an ideal fix, this will always send a payload even when the channel payload is zero.

Regards,

Paul

23

@peliosis,
I’ll check out your mod tomorrow and see how it impacts the other feeds.

For your case, was it just the primary feeds that dropped off, or the delta (*_dwh) feeds also?

One other option you could use would be to add persistence to the OH Items and then use the historical-data functions to see when you got a last update. It’s a bit round-the-houses, but it’s handy for finding out when you haven’t received updates from some entity.

24

@sipvoip,
Posting this PM response publicly, since the information might be useful to others also:

My gemsub.cfg file looks like:

[source]
device_type = gem
reverse_polarity = true
ip_read = true
ip_host = 192.xxx.xxx.xxx
ip_port = yyyy
ip_mode = client
ip_poll_interval = 90
[pvo]
pvo_out = true
pvo_api_key = xxxxeieioxxxxxx
pvo_system_id = xxxxxx
pvo_generation_channel = ch14 
[mqtt]
mqtt_out = true
mqtt_upload_period = 90
mqtt_host = xxxxx.mynetwork.com
mqtt_clientid = xxxxxxxxx
mqtt_user = xxxxxxx
mqtt_passwd = xxxxxxx
mqtt_map = 01054321_volts,volts,01054321_ch1_w,bath_w,01054321_ch2_w,upstairs_hall_w,01054321_ch3_w,hood_w,01054321_ch4_w,furnace_w,01054321_ch5_w,master_bath_w,01054321_ch7_w,dryer_w,01054321_ch8_w,mains_w,01054321_ch9_w,disposal_w,0100
0175_ch10_w,dishwasher_w,01054321_ch11_w,living_w,01054321_ch12_w,family_w,01054321_ch13_w,garage_w,01054321_ch14_w,solar_w,01054321_ch17_w,kitchen_lights_w,01054321_ch18_w,sunpower_w,01054321_ch19_w,garage_lights_w,01054321_ch20_w,stair
s_w,01054321_ch21_w,bedroom23_w,01054321_ch22_w,master_w,01054321_ch23_w,laundry_w,01054321_ch24_w,microwave_w,01054321_ch25_w,kitchen_island_w,01054321_ch26_w,wine_fridge_w,01054321_ch27_w,main_fridge_w,01054321_ch28_w,kitchen_outlets_w
,01054321_ch1_wh,bath_wh,01054321_ch2_wh,upstairs_hall_wh,01054321_ch3_wh,hood_wh,01054321_ch4_wh,furnace_wh,01054321_ch5_wh,master_bath_wh,01054321_ch7_wh,dryer_wh,01054321_ch8_wh,mains_wh,01054321_ch9_wh,disposal_wh,01054321_ch10_wh,di
shwasher_wh,01054321_ch11_wh,living_wh,01054321_ch12_wh,family_wh,01054321_ch13_wh,garage_wh,01054321_ch14_wh,solar_wh,01054321_ch15_wh,unused1_wh,01054321_ch17_wh,kitchen_lights_wh,01054321_ch18_wh,sunpower_wh,01054321_ch19_wh,garage_li
ghts_wh,01054321_ch20_wh,stairs_wh,01054321_ch21_wh,bedroom23_wh,01054321_ch22_wh,master_wh,01054321_ch23_wh,laundry_wh,01054321_ch24_wh,microwave_wh,01054321_ch25_wh,kitchen_island_wh,01054321_ch26_wh,wine_fridge_wh,01054321_ch27_wh,mai
n_fridge_wh,01054321_ch28_wh,kitchen_outlets_wh,01054321_ch29_wh,unused2_wh,01054321_ch1_dwh,bath_dwh,01054321_ch2_dwh,upstairs_hall_dwh,01054321_ch3_dwh,hood_dwh,01054321_ch4_dwh,furnace_dwh,01054321_ch5_dwh,master_bath_dwh,01054321_ch7
_dwh,dryer_dwh,01054321_ch8_dwh,mains_dwh,01054321_ch9_dwh,disposal_dwh,01054321_ch10_dwh,dishwasher_dwh,01054321_ch11_dwh,living_dwh,01054321_ch12_dwh,family_dwh,01054321_ch13_dwh,garage_dwh,01054321_ch14_dwh,solar_dwh,01054321_ch15_dwh
,unused1_dwh,01054321_ch17_dwh,kitchen_lights_dwh,01054321_ch18_dwh,sunpower_dwh,01054321_ch19_dwh,garage_lights_dwh,01054321_ch20_dwh,stairs_dwh,01054321_ch21_dwh,bedroom23_dwh,01054321_ch22_dwh,master_dwh,01054321_ch23_dwh,laundry_dwh,
01054321_ch24_dwh,microwave_dwh,01054321_ch25_dwh,kitchen_island_dwh,01054321_ch26_dwh,wine_fridge_dwh,01054321_ch27_dwh,main_fridge_dwh,01054321_ch28_dwh,kitchen_outlets_dwh,01054321_ch29_dwh,unused2_dwh,01054321_t1,garage_temperature
[openenergymonitor]
oem_out = true
oem_url = http://xxxxxx.mynetwork.com/emoncms/input/post.json
oem_token = aaaaaaaaeieieieoaaaaaaaaaaaaa
oem_node = 2

and I run it like:

/usr/bin/python /usr/local/share/btmon/btmon.py -c /usr/local/share/btmon/gemsub.cfg

My main openHAB machine is also running MQTT (mosquitto) and Emoncms, so those entries are just pointing back to that machine (for now)

BTW, for that CLI response you were seeing:

0x41 0x6c 0x69 0x76 0x65
A    l    i    v    e
25

What config if any off default did you need to do to the GEM?

[root@maggie ~]# ./btmon.py -c gem.cfg  --debug
2015/11/12 11:24:22 btmon: 3.1.2
2015/11/12 11:24:22 python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)]
2015/11/12 11:24:22 platform: linux2
2015/11/12 11:24:22 polarity is reversed
2015/11/12 11:24:22 device type: gem
2015/11/12 11:24:22 device list: ['']
2015/11/12 11:24:22 packet format: gem48ptbin
2015/11/12 11:24:22 schema: counters
2015/11/12 11:24:22 buffer size: 120
2015/11/12 11:24:22 SOCKET: timeout: 60
2015/11/12 11:24:22 SOCKET: server host: 10.71.129.83
2015/11/12 11:24:22 SOCKET: server port: 8000
2015/11/12 11:24:22 SOCKET: poll interval: 10
2015/11/12 11:24:22 MQTT: mqtt:localhost:1883?clientid=btmon
2015/11/12 11:24:22 MQTT: user: btmon
2015/11/12 11:24:22 MQTT: tls: <not-specified>
2015/11/12 11:24:22 MQTT: topic: /house/energy
2015/11/12 11:24:22 MQTT: qos: 0
2015/11/12 11:24:22 MQTT: retain: False
2015/11/12 11:24:22 MQTT: will: <not-specified>
2015/11/12 11:24:22 MQTT: upload period: 90
2015/11/12 11:24:22 MQTT: map: 
2015/11/12 11:24:22 packet format is GEM48PTBinaryPacket
2015/11/12 11:24:22 using collector PollingSocketClientCollector
2015/11/12 11:24:22 using 1 processors:
2015/11/12 11:24:22   MQTTProcessor
2015/11/12 11:24:22 setup PollingSocketClientCollector
2015/11/12 11:24:22 setup MQTTProcessor
2015/11/12 11:24:22 SOCKET: waiting for 10 seconds
2015/11/12 11:24:32 opening socket connection to 10.71.129.83:8000
2015/11/12 11:24:32 sending request 1 to device 
2015/11/12 11:24:32 SOCKET: sending ^^^APISPK
2015/11/12 11:24:32 waiting for data from device 
2015/11/12 11:24:32 reading 1 of 1 packets
2015/11/12 11:24:32 read failed: expected START_HEADER0 0xfe, got 0xd
2015/11/12 11:24:32 sending request 2 to device 
2015/11/12 11:24:32 SOCKET: sending ^^^APISPK
2015/11/12 11:24:32 waiting for data from device 
2015/11/12 11:24:32 reading 1 of 1 packets
2015/11/12 11:24:32 read failed: expected START_HEADER0 0xfe, got 0xa
2015/11/12 11:24:32 sending request 3 to device 
2015/11/12 11:24:32 SOCKET: sending ^^^APISPK
2015/11/12 11:24:32 waiting for data from device 
2015/11/12 11:24:32 reading 1 of 1 packets
2015/11/12 11:24:32 read failed: expected START_HEADER0 0xfe, got 0x43
2015/11/12 11:24:32 3 requests failed for device 
2015/11/12 11:24:32 closing socket connection
2015/11/12 11:24:32 buffer info:
2015/11/12 11:24:32 processing with MQTTProcessor
2015/11/12 11:24:32 SOCKET: waiting for 10 seconds
2015/11/12 11:24:42 opening socket connection to 10.71.129.83:8000
2015/11/12 11:24:42 sending request 1 to device 
2015/11/12 11:24:42 SOCKET: sending ^^^APISPK
2015/11/12 11:24:42 waiting for data from device 
2015/11/12 11:24:42 reading 1 of 1 packets
2015/11/12 11:24:43 read failed: expected START_HEADER0 0xfe, got 0xd
2015/11/12 11:24:43 sending request 2 to device 
2015/11/12 11:24:43 SOCKET: sending ^^^APISPK
2015/11/12 11:24:43 waiting for data from device 
2015/11/12 11:24:43 reading 1 of 1 packets
2015/11/12 11:24:43 read failed: expected START_HEADER0 0xfe, got 0xa
2015/11/12 11:24:43 sending request 3 to device 
2015/11/12 11:24:43 SOCKET: sending ^^^APISPK
2015/11/12 11:24:43 waiting for data from device 
2015/11/12 11:24:43 reading 1 of 1 packets
2015/11/12 11:24:43 read failed: expected START_HEADER0 0xfe, got 0x43
2015/11/12 11:24:43 3 requests failed for device 
2015/11/12 11:24:43 closing socket connection
2015/11/12 11:24:43 buffer info:
2015/11/12 11:24:43 processing with MQTTProcessor
2015/11/12 11:24:43 SOCKET: waiting for 10 seconds
2015/11/12 11:24:53 opening socket connection to 10.71.129.83:8000
2015/11/12 11:24:53 sending request 1 to device 
2015/11/12 11:24:53 SOCKET: sending ^^^APISPK
2015/11/12 11:24:53 waiting for data from device 
2015/11/12 11:24:53 reading 1 of 1 packets
2015/11/12 11:24:54 read failed: expected START_HEADER0 0xfe, got 0xd
2015/11/12 11:24:54 sending request 2 to device 
2015/11/12 11:24:54 SOCKET: sending ^^^APISPK
2015/11/12 11:24:54 waiting for data from device 
2015/11/12 11:24:54 reading 1 of 1 packets
2015/11/12 11:24:54 read failed: expected START_HEADER0 0xfe, got 0xa
2015/11/12 11:24:54 sending request 3 to device 
2015/11/12 11:24:54 SOCKET: sending ^^^APISPK
2015/11/12 11:24:54 waiting for data from device 
2015/11/12 11:24:54 reading 1 of 1 packets
2015/11/12 11:24:54 read failed: expected START_HEADER0 0xfe, got 0x43
2015/11/12 11:24:54 3 requests failed for device 
2015/11/12 11:24:54 closing socket connection
2015/11/12 11:24:54 buffer info:
2015/11/12 11:24:54 processing with MQTTProcessor
2015/11/12 11:24:54 SOCKET: waiting for 10 seconds
26

In case anyone else runs into this, I had to go into the GEM and change the packet format from 0 to 4.

27

Is there a way to have it even update channels with 0 use? I have some channels like my well and septic pumps that correctly show data when they are running, but often they go from say 3000 watts down to 0 and I don’t get updates for 0 so OpenHab still shows them running.

28

@sipvoip,
This is probably the issue @peliosis references above. It will impact the Power (*_w) type items, it looks like the delta-power (*_dwh) Channels transition to 0 correctly:

2015-12-18 02:32:47 - HouseDEnergyEV state updated to 65.864
2015-12-18 02:33:17 - HouseDEnergyEV state updated to 65.922
2015-12-18 02:33:48 - HouseDEnergyEV state updated to 65.855
2015-12-18 02:34:19 - HouseDEnergyEV state updated to 68.11
2015-12-18 02:34:50 - HouseDEnergyEV state updated to 65.569
2015-12-18 02:35:21 - HouseDEnergyEV state updated to 65.948
2015-12-18 02:35:52 - HouseDEnergyEV state updated to 5.011
2015-12-18 03:00:10 - HouseDEnergyEV state updated to 0.0
2015-12-18 03:14:38 - HouseDEnergyEV state updated to 0.0
2015-12-18 03:19:17 - HouseDEnergyEV state updated to 0.0

I’ll need to get back to this in the New Yr.

The first priority will be getting the Brultech GEM Wiki page together so others can benefit from the work so far. Once that’s in place, I’ll take a stab at this one…

29

I investigating moving my brultech GEM monitoring from my Vera into OH2 - will this solution also work on OH2? I’m guessing the discussion is centered around OH1…

30

Since it uses MQTT and that binding looks compatible with 2.0 you should be good to go.

31

OK, so the short list of things to do, to get this working is something along the following lines?

  1. Set GEM to output format 4.
  2. Install mosquitto
  3. Install pip
  4. Install paho-mqtt using (3)
  5. Download btmon.py from Marks link above
  6. Copy binding.mqtt + io.transport.mqtt from oh18 package into oh2 addons folder.
  7. Add “mqtt:mymqtt.url=tcp://localhost:1883” to runtime.cfg
  8. Put a gemsub.cfg in oh2/conf/services

Did I miss anything (I’m sure I did!) :slightly_smiling:

32

May want to also add clientId in openhab.cfg such as:

mqtt:mosquitto.url=tcp://localhost:1883
mqtt:mosquitto.clientId=openhab

I have 3 GEMs, my first config looks like:

[source]
device_type = gem
reverse_polarity = true
ip_read = true
ip_host = 10.71.129.83
ip_port = 8000
ip_mode = client
ip_poll_interval = 8
[mqtt]
mqtt_out = true
mqtt_upload_period = 15
mqtt_host = localhost
mqtt_clientid = btmon-gem1

If you have more then one just make a config file for each, they can talk on the same port. Also, as Paul mentioned there is an issue with getting the 0 value back. You can go to line 3114 in btmon.py and comment out the following:

 def _add_msg(self, packet, channel, payload):
     #  if not payload:
      #     return

It’s a hack, but it is working for me, if you get stuck, let us know!

P.S. You will want to assign static IPs via DHCP or use static on your GEMs.

33

OK, finally I had time to look into this… :slight_smile:
I think I have it mostly right, but the last thing - openhab seeing my data…
I have decided to start with Volts, then do the rest. Probably it is just me being a openhab newbie.
This is on oh2. I guess maybe I am missing a mapping between things and mqtt?

I have set-up mosquitto, and both btmon and openhab seams to be connecting;
1455378322: New connection from 127.0.0.1 on port 1883. 1455378322: New client connected from 127.0.0.1 as micke.1455378322141 (c1, k60). 1455378325: New connection from 127.0.0.1 on port 1883. 1455378325: New client connected from 127.0.0.1 as btmon-gem1 (c1, k60). 1455378325: Client btmon-gem1 disconnected. 1455378341: New connection from 127.0.0.1 on port 1883. 1455378341: New client connected from 127.0.0.1 as btmon-gem1 (c1, k60). 1455378341: Client btmon-gem1 disconnected.
For some reason I get my linux user (micke) instead of openhab, but this might not be an issue.

My runtime.cfg holds these related lines;
mqtt:mymqtt.url=tcp://localhost:1883 mqtt:mosquitto.clientId=openhab

btmon publishes lines along this;
2016/02/13 16:51:12 MQTT: len=218, msgs=[{"topic": "/house/energy/01000590_volts", "retain": false, "qos": 0, "payload": 226.7},

In openhab log, the only related lines are
16:45:22.137 [INFO ] [penhab.io.transport.mqtt.MqttService] - MQTT Service initialization completed. 16:45:22.137 [INFO ] [t.mqtt.internal.MqttBrokerConnection] - Starting MQTT broker connection 'mymqtt' - nothing related to gem updates.

So in my openhab items file, I have put
Number HouseVolts "House Volts [%f]" <energy> (GPersist) {mqtt="<[home:/house/energy/01000590_volts:state:default]"}

And finaly, my btmon config;
[source] device_type = gem ip_read = true ip_host = 10.1.200.7 ip_port = 8000 ip_mode = client ip_poll_interval = 8 [mqtt] mqtt_out = true mqtt_upload_period = 15 mqtt_host = localhost mqtt_clientid = btmon-gem1

34

OK, after doing more searches and test, I ended up creating a mqtt.cfg in conf/services, and moved settings from runtime.cfg.
My new mqtt.cfg
broker.url=tcp://127.0.0.1:1883 broker.clientId=openhab
Then I changed the row in items file into
Number HouseVolts "House Volts [%f]" <energy> (GPersist) {mqtt="<[broker:/house/energy/01000590_volts:state:default]"}

And now I get the volt in openhab log. :slightly_smiling:

I guess I had mixed the config files up. :blush:

1 Like
35

It’s been a while since I’ve updated here, and I’ve made a few changes:

  • a) The data is now in-house and I no longer push data to SEG.
  • b) I’ve moved the system, including openHAB, to a virtualized Ubuntu box (on a Intel NUC 6i5) running Proxmox.
  • c) All data is persisted to a local InfluxDB.
  • d) Data loaded from MQTT using InfluxData’s Telegraf module. This also provides CPU/utilization ifor the openHAB VM.
  • e) I’m graphing the data using a Grafana dashboard

grafana-energy.png1242×543 80.6 KB

1 Like
36

My Brultech-MQTT work has been merged into the main repo by Matthew, so you can now get the code from here:

So I’ve dropped my local github repo copy (I’ll fix links above, where I can)

There’s also a pending PR to add native InfluxDB support, for those that want to push data more directly:

37

Thanks for all your hard work and also for recommending GEM, I have 3 of them in my new house and have been very happy with the setup.

1 Like
38

I got a PM for a little more detail, so here’s the specifics on the settings in InfluxData’s Telegraf, to pull this data into InfluxDB.

There are drivers now that’ll push directly, but I still prefer these to come via MQTT.

In /etc/telegraf/telegraf.d/mqtt.conf …

I have sections like the following, one for each of:

  • house/power/# - energy from my GEM & ECM-1240 via btmon.py
  • house/energy_total/# - energy from my GEM & ECM-1240 via btmon.py
  • house/volts/# - voltage data from my GEM & ECM-1240 via btmon.py
  • house/temperature/# - mixed temperature data from openHAB (Nest, Weather) and btmon (GEM Temperature data)
  • house/controller/# - general openHAB data streams (Switches, etc)

Here’s a sample of what one of these 5 sections looks like in the Telegraf configuration:

[[inputs.mqtt_consumer]]
  # Running on localhost, put the host/port here if it's elsewhere, along with any TLS config
  servers = ["localhost:1883"]
  qos = 0

  topics = [
    "house/energy_total/#"
  ]

  metric_buffer = 100000
  # Replace these with the MQTT Account to use in your env
  username = "<yourMQTTAccount>"
  password = "<yourMQTTPassword>"

  name_override = "house-energy-total"
  data_format = "value"
  data_type = "float"

NB: In btmon.py my mqtt_base_topic setting is “house” instead of “/house”, so change the prefix as needed for your specific environment.

39

Thank you for posting this. It makes sense. A question though. Are you running different instances of btmon for each topic? As far as I can tell you can only set one topic in btmon.

Adam

40

I have one GEM and one ECM-1240. Each is running on a separate btmon.py process, one direct-attached serial, and one over Wifi, with both processes running on the same rPi.

Each of these is setup to write to the same metric tree under MQTT tree.