ZigBee network map when using only the ZigBee Binding

ndasyras · September 5, 2019, 6:16pm

Hi fellow openhabians,
In this post I would like to share a method I am using to create a ZigBee network map while using the ZigBee Binding. As far as I know, the capabiity to create ZigBee network map exists when using Zigbee2MQTT but not when using only the ZigBee Binding, which is my case.

The problem:
Create, update and display in the Sitemap a ZigBee network map when using only the ZigBee Binding.

Input:
Two inputs can be used with similar results

The Things file (epoch–org.eclipse.smarthome.core.thing.Thing.json) from /var/lib/openhab2/jsondb/backup . I am using the backup file just to be on the safe side and not to work on the actual file used by the system. This is the input that I will use further down in the script.
Get all the Things information via the REST API e.g http://openhab_ip/rest/things. In that case substitute .value.properties.xxx with .properties.xxx in the examples bellow (e.g. use .properties.zigbee_routes instead of .value.properties.zigbee_routes)

Tools needed:

jq : Used to parse the json file used as input and get the information needed
sed : Used for clean-up and formating of the input
Graphviz (the dot command) : Used to create the graph

Execution:
I have created the following script which is called periodically from crontab and creates a png image with the ZigBee network map. The script creates three input files. One with zigbee nodes information, one with routing information and one with the zigbee neighbors. The three files are then concatenated into one and a dot file is created. The dot file is then used to create the network map image which is stored in /etc/openhab2/html/ so that it can be displayed as an item in the Sitemap. The “strict digraph” graphing method is used to avoid duplicated links from being shown in the map.

Script

#   
# !/bin/bash
#

dir=/home/openhabian/zigbee_map

#copy Things backup file to working directory
cp `find /var/lib/openhab2/jsondb/backup -name *org.eclipse.smarthome.core.thing.Thing.json | tail -1` $dir/Thing.json

#extract zigbee nodes information, the entry for the coordinator with id 0 is added manually at the end of the command
jq '.[] | select(.value.properties.modelId != null) | .value.properties.zigbee_networkaddress + ",[shape=record,label=^{" + .value.properties.modelId + "|" + .value.properties.zigbee_networkaddress +"|"+ .value.properties.zigbee_logicaltype +"}^]"' $dir/Thing.json | sed -e 's/ /./g' -e 's/"//g' -e 's/,/ /g' -e 's/\./_/g' -e 's/\///g' -e 's/\^/"/g' -e '1s/^/0 [shape=record label="{CC2531|0|COORDINATOR}"]\n/' > $dir/zigbee_things

#extract zigbee nodes routing
jq '.[] | select(.value.properties.zigbee_networkaddress != null) |  select (.value.properties.zigbee_routes != "[]") | .value.properties.zigbee_networkaddress, .value.properties.zigbee_routes' $dir/Thing.json | sed 's/},{/\n/g' | grep -v INACTIVE | grep -v UNDERWAY | sed -e 's/"//g' -e 's/\\//g' -e 's/\[{//g' -e 's/}]//g' -e 's/next_hop/-> /g' -e 's/destination/ -> /g' -e 's/://g' -e 's/://g' -e 's/,//g' -e 's/stateACTIVE/ ;/g' | awk '/^[0-9]/{val=$1;print;next} NF{sub(/^ +/,"");print val,$0;next} 1'| grep ";" > $dir/zigbee_routes

#extract zigbee nodes neighbors
jq '.[] | select(.value.properties.zigbee_networkaddress != null) |  select (.value.properties.zigbee_neighbors != "[]") | .value.properties.zigbee_networkaddress, .value.properties.zigbee_neighbors' $dir/Thing.json | sed -e 's/},{/\n/g' -e 's/"//g' -e 's/\\//g' -e 's/\[{//g' -e 's/}]//g'| cut -d , -f2 | sed 's/address:/ -> /g' | awk '/^[0-9]/{val=$1;print;next} NF{sub(/^ +/,"");print val,$0;next} 1'| sed 's/$/ ;/g' | grep ">" > $dir/zigbee_neighbors

#create concatenation file and add date and time as label
echo "label=\"`date +%d-%m-%Y\ %H:%M`\";" > $dir/zigbee_map.temp

#concatenate all inputs
cat $dir/zigbee_things >> $dir/zigbee_map.temp
cat $dir/zigbee_routes >> $dir/zigbee_map.temp
cat $dir/zigbee_neighbors >> $dir/zigbee_map.temp

#add beggining and end of dot file
sed -e '1s/^/strict digraph G {\n/' -e '$ a }' $dir/zigbee_map.temp > $dir/zigbee_map.dot

#graph map
dot -Tpng $dir/zigbee_map.dot -o $dir/zigbee_map.png

#copy map file to html server location
cp $dir/zigbee_map.png /etc/openhab2/html/zigbee_map.png
    
exit

Item

Group    Zigbee_Network    "Zigbee Network Map"    <zoom>

Sitemap

Group item=Zigbee_Network label="Zigbee Network Map" icon="zoom" {
          Image url="http://<openhab_ip>/static/zigbee_map.png"
        }

NoneWhereTo · September 6, 2019, 6:41am

That’s great! When I’m off the business trip back at home I need to implement this. Do you maybe already have a picture?

ndasyras · September 6, 2019, 7:10pm

@NoneWhereTo Here is an example ZigBee network map image

NoneWhereTo · September 11, 2019, 1:33pm

Hi Nikolaos,

I tried your script, but it fails in the command line. Somehow the copy cp operations does not work. I get these results:

: not foundngmap.sh: 4: zigbeebindingmap.sh:
: not foundngmap.sh: 6: zigbeebindingmap.sh:
cp: cannot create regular file '/home/openhabian/zigbee_map'$'\r''/Thing.json'$'\r': No such file or directory
: not foundngmap.sh: 9: zigbeebindingmap.sh:
: Directory nonexistent: zigbeebindingmap.sh: cannot create /home/openhabian/zigbee_map
/Thing.json: No such file or directorypenhabian/zigbee_map
: not foundngmap.sh: 12: zigbeebindingmap.sh:
: Directory nonexistent: zigbeebindingmap.sh: cannot create /home/openhabian/zigbee_map
/Thing.json: No such file or directorypenhabian/zigbee_map
: not foundngmap.sh: 15: zigbeebindingmap.sh:
: Directory nonexistent: zigbeebindingmap.sh: cannot create /home/openhabian/zigbee_map
/Thing.json: No such file or directorypenhabian/zigbee_map
: not foundngmap.sh: 18: zigbeebindingmap.sh:
: Directory nonexistent: zigbeebindingmap.sh: cannot create /home/openhabian/zigbee_map
: not foundngmap.sh: 21: zigbeebindingmap.sh:
: Directory nonexistent: zigbeebindingmap.sh: cannot create /home/openhabian/zigbee_map
: Directory nonexistent: zigbeebindingmap.sh: cannot create /home/openhabian/zigbee_map
: Directory nonexistent: zigbeebindingmap.sh: cannot create /home/openhabian/zigbee_map
: not foundngmap.sh: 26: zigbeebindingmap.sh:
: Directory nonexistent: zigbeebindingmap.sh: cannot create /home/openhabian/zigbee_map
: not foundngmap.sh: 29: zigbeebindingmap.sh:
/zigbee_map.dot't open /home/openhabian/zigbee_map
: not foundngmap.sh: 32: zigbeebindingmap.sh:
cp: cannot stat '/home/openhabian/zigbee_map'$'\r''/zigbee_map.png': No such file or directory
: not foundngmap.sh: 35: zigbeebindingmap.sh:

The directory zigbee_map is there as I created it manually. Also the json files are in the original location, though they are starting with an epoch timestamp but this is handled by the * operator. Do you know what’s wrong with the cp line? (I copied your script without modification).

Christian

Update 1:
Maybe permission issue. I’ll look into it and get back

Update 2:
Changed the directory to html where openhab should have rw access to, but still the same errors come up

curlyel · September 11, 2019, 1:51pm

Would you consider raising an issue at the openhab2-addon for the zigbee binding pointing to your work?

Would be nice, if the zigbee binding would provide such an network overview at least in the admin UI(s). Probably ping @chris for that.
He did something quite similar for Zwave in the HABmin UI…

Maybe he could even provide this as a channel (channel-type image) of the coordinator thing?

chris · September 11, 2019, 2:29pm

The ZigBee binding does produce this information - I think that is what is being used in this script?

The UI needs to take the information that the ZigBee binding creates, and turn this into the map. I did do this a few years back in PaperUI for a commercial user of the binding and in theory I could probably publish this if I can find the code now

ndasyras · September 11, 2019, 5:11pm

Hi @NoneWhereTo ,
It does look like a permission error. I suggest to check the permissions on the /home/openhabian/zigbee_map dir. Could it be possible that you might have created the script directory with one user and running the script with another (e.g. one with openhab user and the other with openhabian user)?
So, I would sugget to check the owner and permissions of the script directory and that the user running the script has access to the script directory and the jsondb/backup directory. Also, I would suggest to try running the commands of the script one by one in a terminal to find which is the first part that it is not working (e.g. just do a plain copy of one org.eclipse.smarthome.core.thing.Thing.json file in the script directory and rename it as Things.json and then try the rest of the commands in the script).

BR,
Nikos

ndasyras · September 11, 2019, 5:16pm

Hi @chris and @curlyel,
Indeed the information I use as input is produced by the ZigBee binding and it would be great if such a network map could be incorporated in the binding so that it could be displayed in the BasicUI or any other place.
I am not familiar with binding development so I have to ask, would it be possible to re-use what I have done in the binding or it has to be written from scratch? If the former is the case then it would be great and I would be happy to follow @curlyel sugestion.

BR,
Nikos

chris · September 11, 2019, 5:27pm

Just for info, this is what I did for the customer a few years back -:

Clicking on a node shows the node properties. The graph is updated as information is updated, and nodes can be dragged around (they have a kind of gravity function to align them).

PaperUI theming is the Eclipse SmartHome theme.

If I can find the source to this I will see if I can make it available - someone might want to take it and add it to PaperUI (or the next-gen PaperUI).

giuseppeiannello · May 3, 2020, 7:45pm

This script worked very well for me, but I needed additional info, so here’s my solution.
Major differences: work directly with the OpenHAB API to get the most up-to-date info, no more sed, detailed route/neighbor info, extensive color coding, use circo instead of dot for visualization, simpler to extend.

#/bin/bash

dir=$(mktemp -d)

wget -q http://localhost:8080/rest/things -O $dir/things.json

# Replace with the "strict" version if it's too dense.
echo "digraph G {" >> $dir/network.dot
#echo "strict digraph G {" >> $dir/network.dot

echo "edge [decorate=true]" >> $dir/network.dot
echo "node [shape=box]" >> $dir/network.dot
echo "0 [label=<0<br/>COORDINATOR> color=blue]" >> $dir/network.dot

jq -r '.[] |
  select(.thingTypeUID == "zigbee:device") |
  .properties.zigbee_networkaddress + " " +
  "[" +
    "label=<" +
    .properties.modelId + "<br/>" +
    .properties.zigbee_networkaddress + "<br/>" +
    (.properties.zigbee_macaddress // .configuration.zigbee_macaddress) + "<br/>" +
    .properties.zigbee_logicaltype + "<br/>" +
    .label + "<br/>" +
    .location +
    ">" +
    "color=" +
    if .statusInfo.status == "ONLINE" then "green"
    elif .statusInfo.status == "OFFLINE" then "orange"
    elif .statusInfo.status == "UNKNOWN" then "red"
    else "black" end +
  "]"
' $dir/things.json >> $dir/network.dot

nodes=$(jq -r '.[] |
  select(.properties.zigbee_networkaddress != null) |
  .properties.zigbee_networkaddress
' $dir/things.json)

for node in ${nodes}; do
  jq -r --arg node $node '.[] |
    select(.properties.zigbee_networkaddress == $node) |
    select(.properties.zigbee_routes != "[]" and .properties.zigbee_routes != null) |
    .properties.zigbee_routes
  ' $dir/things.json |
  jq -r --arg node $node '.[] |
    $node + " -> " +
    if .destination != .next_hop then .next_hop + " -> " else "" end +
    .destination +
    " [" +
    "color=" +
    if .state == "ACTIVE" then "green"
    elif .state == "DISCOVERY_UNDERWAY" then "orange"
    elif .state == "DISCOVERY_FAILED" then "red"
    elif .state == "INACTIVE" then "blue"
    else "yellow" end + " " +
    "]"
  ' >> $dir/network.dot

  jq -r --arg node $node '.[] |
    select(.properties.zigbee_networkaddress == $node) |
    select(.properties.zigbee_neighbors != "[]" and .properties.zigbee_neighbors != null) |
    .properties.zigbee_neighbors
  ' $dir/things.json |
  jq -r --arg node $node '.[] |
    $node + " -> " + .address + " " +
    "[" +
      "fontcolor=" +
      if .joining == "ENABLED" then "green"
      elif .joining == "DISABLED" then "black"
      elif .joining == "UNKNOWN" then "red"
      else "blue" end + " " +
      "label=<" +
      .lqi + "/" + .depth +
      ">" +
    "]"
  ' >> $dir/network.dot
done

echo "label=\"`date +%Y-%m-%d\ %H:%M`\";" >> $dir/network.dot

echo "}" >> $dir/network.dot

/usr/bin/circo -Tpng $dir/network.dot -o /etc/openhab2/html/zigbee_network.png

rm -rf $dir

Example graph

Example graph (strict mode)

Suy · August 12, 2020, 8:34pm

I only get a box wtih COORDINATOR…
probably something is changed.
No routes and no neighbors are found

Suy · August 12, 2020, 9:03pm

probably something is changed.
No routes and no neighbors are found

ndasyras · August 13, 2020, 6:23am

Hi Stefan,
The script is working for me using openhab 2.5.4 on openhabian. Let us try to troubleshoot why it is not working for you. I suggest the following steps:

Check in Paper UI that in your Zigbee Things properties you can see neighbors and routes
Run the jq commands without any post-processing with sed and grep and see if you get any output
e.g.

zigbee nodes information
jq '.[] | select(.value.properties.modelId != null) | .value.properties.zigbee_networkaddress + ",[shape=record,label=^{" + .value.properties.modelId + "|" + .value.properties.zigbee_networkaddress +"|"+ .value.properties.zigbee_logicaltype +"}^]"' Thing.json
zigbee nodes routing
jq '.[] | select(.value.properties.zigbee_networkaddress != null) | select (.value.properties.zigbee_routes != "[]") | .value.properties.zigbee_networkaddress, .value.properties.zigbee_routes' Thing.json
zigbee nodes neighbors
jq '.[] | select(.value.properties.zigbee_networkaddress != null) | select (.value.properties.zigbee_neighbors != "[]") | .value.properties.zigbee_networkaddress, .value.properties.zigbee_neighbors' Thing.json

* Where Thing.json is your backup copy of the latest Things backup file from /var/lib/openhab2/jsondb/backup

Suy · August 13, 2020, 9:12am

Maybe I found the problem.
I use the HUE binding(for my huebridge) and the DECONZ binding(for my deconz usb stick) for my zigbee devices.
There’s also a zigbee binding. Maybe you are using the zigbee binding?

ndasyras · August 13, 2020, 9:36am

Indeed, this method is for the Zigbee binding as noted in the topic’s title and the 1st paragraph of the initial post.

In this post I would like to share a method I am using to create a ZigBee network map while using the ZigBee Binding.

Suy · August 13, 2020, 11:18am

my fault

abol3z · August 15, 2020, 2:27pm

The current implementation shows only the details of devices of type zigbee:device. I’ve added a small fix to also show devices with static thing definition.
I’ve also added an option to draw the graph with dot since it might be cleaner looking for some people

#/bin/bash

dir=$(mktemp -d)
#dir=./zigbee_map

wget -q http://localhost:8080/rest/things -O $dir/things.json

# Replace with the "strict" version if it's too dense.
#echo "digraph G {" >> $dir/network.dot
echo "strict digraph G {" >> $dir/network.dot

echo "edge [decorate=true]" >> $dir/network.dot
echo "node [shape=box]" >> $dir/network.dot
echo "0 [label=<0<br/>COORDINATOR> color=blue]" >> $dir/network.dot

jq -r '.[] |
  select(.thingTypeUID | startswith("zigbee:")) |
  .properties.zigbee_networkaddress + " " +
  "[" +
    "label=<" +
    .properties.modelId + "<br/>" +
    .properties.zigbee_networkaddress + "<br/>" +
    (.properties.zigbee_macaddress // .configuration.zigbee_macaddress) + "<br/>" +
    .properties.zigbee_logicaltype + "<br/>" +
    .label + "<br/>" +
    .location +
    ">" +
    "color=" +
    if .statusInfo.status == "ONLINE" then "green"
    elif .statusInfo.status == "OFFLINE" then "orange"
    elif .statusInfo.status == "UNKNOWN" then "red"
    else "black" end +
  "]"
' $dir/things.json >> $dir/network.dot

nodes=$(jq -r '.[] |
  select(.properties.zigbee_networkaddress != null) |
  .properties.zigbee_networkaddress
' $dir/things.json)

for node in ${nodes}; do
  jq -r --arg node $node '.[] |
    select(.properties.zigbee_networkaddress == $node) |
    select(.properties.zigbee_routes != "[]" and .properties.zigbee_routes != null) |
    .properties.zigbee_routes
  ' $dir/things.json |
  jq -r --arg node $node '.[] |
    $node + " -> " +
    if .destination != .next_hop then .next_hop + " -> " else "" end +
    .destination +
    " [" +
    "color=" +
    if .state == "ACTIVE" then "green"
    elif .state == "DISCOVERY_UNDERWAY" then "orange"
    elif .state == "DISCOVERY_FAILED" then "red"
    elif .state == "INACTIVE" then "blue"
    else "yellow" end + " " +
    "]"
  ' >> $dir/network.dot

  jq -r --arg node $node '.[] |
    select(.properties.zigbee_networkaddress == $node) |
    select(.properties.zigbee_neighbors != "[]" and .properties.zigbee_neighbors != null) |
    .properties.zigbee_neighbors
  ' $dir/things.json |
  jq -r --arg node $node '.[] |
    $node + " -> " + .address + " " +
    "[" +
      "fontcolor=" +
      if .joining == "ENABLED" then "green"
      elif .joining == "DISABLED" then "black"
      elif .joining == "UNKNOWN" then "red"
      else "blue" end + " " +
      "label=<" +
      .lqi + "/" + .depth +
      ">" +
    "]"
  ' >> $dir/network.dot
done

echo "label=\"`date +%Y-%m-%d\ %H:%M`\";" >> $dir/network.dot

echo "}" >> $dir/network.dot

# draw graph using dot or circo
/usr/bin/circo -Tpng $dir/network.dot -o ./zigbee_network.png
#dot -Tpng $dir/network.dot -o ./zigbee_network.png

rm -rf $dir

Gauthier_Folzan · November 20, 2020, 5:15pm

If someone is interested, here is a pure python version of the script without any file creation or copy but you need to install graphviz python interface

#!/usr/bin/python3
import json
import sys
from graphviz import Digraph

def main(argv):
    if(len(sys.argv) < 2):
        print('missing input file')
        sys.exit(2)
    inputfile = str(sys.argv[1])
    outputfile = 'zigbee_map'
    if(len(sys.argv) >= 3):
        outputfile = str(sys.argv[2])

    file = open(inputfile)
    json_string = file.read()
    file.close()
    dot = Digraph('graph',comment='Zigbee map',graph_attr={'rankdir':'LR'},node_attr={'shape': 'record'},format='png',filename=outputfile)
    json_dictionary = json.loads(json_string)
    for key in json_dictionary:
        if(key.find("zigbee:")>=0):
            zigbee_logicaltype=json_dictionary[key]["value"]["properties"]["zigbee_logicaltype"]
            zigbee_networkaddress =json_dictionary[key]["value"]["properties"]["zigbee_networkaddress"]
            label = json_dictionary[key]["value"]["label"]
            if(zigbee_logicaltype=="COORDINATOR"):
                zigbee_macaddress=json_dictionary[key]["value"]["properties"]["zigbee_macaddress"]
                modelId =""
            else:
                zigbee_macaddress=json_dictionary[key]["value"]["configuration"]["properties"]["zigbee_macaddress"]
                modelId =json_dictionary[key]["value"]["properties"]["modelId"]
            dot_label=str()
            dot_label+=zigbee_logicaltype
            dot_label+="|"+label
            dot_label+="|"+modelId
            dot_label+="|{adress |"+zigbee_networkaddress+"}"
            dot_label+="|{mac |"+zigbee_macaddress+"}"
            dot.node(zigbee_networkaddress, dot_label)

            zigbee_networkaddress =json_dictionary[key]["value"]["properties"]["zigbee_networkaddress"]
            zigbee_neighbors_json =json_dictionary[key]["value"]["properties"]["zigbee_neighbors"]
            zigbee_neighbors_list = json.loads(zigbee_neighbors_json)
            for item in zigbee_neighbors_list:
                dot.edge(zigbee_networkaddress, item['address'], label=item['joining'], color='blue')
            zigbee_routes_json =json_dictionary[key]["value"]["properties"]["zigbee_routes"]
            zigbee_routes_list = json.loads(zigbee_routes_json)
            for item in zigbee_routes_list:
                dot.edge(item['next_hop'], item['destination'], label=item['state'], color='black')
    dot.render()

if __name__ == "__main__":
   main(sys.argv[1:])

Do you have any reason to use a backup or a request and not directly the jsondb file /var/lib/openhab2/jsondb/org.eclipse.smarthome.core.thing.Thing.json ?

I was able to produce a map in /tmp/zigbee.png with the command
./zigbee_map.py /var/lib/openhab2/jsondb/org.eclipse.smarthome.core.thing.Thing.json /tmp/zigbee

StefanH · February 15, 2021, 5:10pm

Hi all,

I’m about to move from zigbee2mqtt to the openHAB zigbee binding as my new coordinator isn’t supported by z2m. Great to see the activity in this thread

I’m using openHAB 3.0.1 and I wonder (before I try to run the script) if v3 is supported? In the main UI I see the values below in blue, for both my xiaomi zigbee power plug (ROUTER) and contact sensor (END_DEVICE)

ndasyras · February 16, 2021, 4:45pm

Hi Stefan,
I have not migrated to OpenHAB 3 yet so I have not tried any of the methods mentioned here on the new version. Nevertheless, you could try to check whether the files or the REST API endpoints are available.