I am hoping to hear from people who either are using the serial binding or who have tried the binding and found it did not work for their use case and went another way. What does the serial binding need or lack? What do you use and would kick/cry/scream if the binding no longer supported or offered a better alternative way to work? Some changes are needed to make it compatible with V2 concepts so a direct drop in replacement is not possible so it is an opportunity to introduce changes when you still have a choice of using V1 or V2 addons to slowly move across.
It appears to lack a lot for my use case as I have >50 serial devices on the same bus all talking to each other at very high speed. Similar to DMX, I do this for low latency speedy changes to lights and more.
I plan to give it a month or two for people to respond before any coding starts and in this time I hope to get the IpCamera binding merged to learn what wrong things I am doing that need to change before starting another binding. I will post some ideas in post 2 of this thread and update it as the ideas are improved with peoples feedback.
Quick overview of proposed changes and will add more detail of how the binding will work as time allows to generate ideas, suggestions and feedbackā¦
THINGS:
SerialPort: Will be a bridge thing that will takes all childs OFFLINE if the port is not reachable. All below things must select a bridge. Also this will help if you move from windows to linux as you only need to update the bridge to point to the serial port and all childs will follow.
RawString: Will be a thing with a single string channel that behaves like the old binding. Used to view or send serial data untainted.
Switch: Specify two strings/packets that will trigger the channel to move ON or OFF. Maybe a check box to flip the switch on any incoming data.
Dimmer: Specify a packet format that must match and which byte that will contain the needed value. Also specify Min and Max values so the binding can rescale a 0-255 value automatically to a Slider that uses 0-100 values. Thing may also have a string channel to output the raw matched packet.
Example packets to show how wide a range of formats you can get from devices: If you have a different format please post an example.
Marantz Surround Receiver SR 5004
Ascii Based
Turn on @PWR:2\n
Turn off @PWR:1\n
Hex based is common
Controller ID,ZONE, Fan Speed, CRC
eg ā0x33,0x02,0x25,0xFFā
Minor suggestion - call it rawChars or ,something, to discourage the idea that it should be immediately viewable. Iām guessing user can make or select channels from this in different encodings.
Please make sure that transforms can be applied between binding and channel. This has different effects and different uses to profiles (acting between channel and Item)
A very simple Switch implementation sounds useful.
I am dubious about mapping more complex types like Dimmer , trying to force openHAB model onto a chaotic serial world. Scaling etc. is easily done with transforms. I would suggest aim to provide more generalized function building blocks.
Example, put that āpacket format must matchā into the general Thing. That would be good if it could cope somehow with separately specified chunks e.g. āheaderā + ādevice addressā. Some kind of chaining I guess.
Many serial formats include checksum/CRC techniques. Naturally there are few common standards. Iām not sure whether it is worth trying to build in any such function, given variations. Maybe something to append outgoing / analyse incoming according to one of a few selected common formats. Could be part of āpacket formatā.
If weāre error checking, we need a channel to report errors! lastReadError, lastwriteError dateTime channels are simple? user can catch updates. Should treat as transient, not catastrophe.
Maybe also stats from Bridge and/or Thing - packet counts
Low-level details for Bridge I guess - specify a minimum gap time between our TXs (does that imply buffering or queuing?), separately a minimum listen-for-idle or turnaround time between RX and TX. Params useful in shared bus environment.
Sort of linked - a timeout for incoming data to determine end of RX
Example serial data network - the old Pelco-D CCTV camera control. Fixed format, CRC, addressable devices. Iām not suggesting this should be directly implemented but thinking about the tools needed should be helpful.
I was thinking String to be consistant that it has a string channel so it makes sense to me to call it String in the name of the thing type. Then Switch because it is for switches and so onā¦ Yes I was thinking of making the thing have options so that its format can change into various formats that people may want. Only 1 channel to keep the CPU load low. Transforms should be supported to give backwards compat with what people already have.
Can you provide me a link to some examples of this in use for turning say a 8bit value into 0-100 percent for a dimmer? I find the official docs on this topic to be lacking what I need to fully understand it and be able to create my own. The fact the map file contents were left out of the examples āto be simpleā really frustrates me as only half the example is there to look atā¦
There are protocols that use the same 8 bits for multiple sensors, so a way to grab a start and an end and have that scale nicely all automatically is what I am aiming for and without the need to create map files, edit item files and also setup a binding. My opinion is it should be all in the one place with each option having a helpful description right there to guide you. Iām guessing by having the planned options in the binding, nothing stops you from not using them and still using transforms if that is what you wish. Openhab has the ability to mark options as advanced so a more simple view is shown at first and then with a click you get the advanced stuff shown should you wishā¦
I have a very loose concept which is not fully developed yet so ignore the example it is only to try and explain the concept. Basic idea was to allow wildcards on bytes. IE
deviceID,sensor1Value,sensor2Value,CRC
12,*,Value,*
You ask it to match it based on the device ID in this case 12, and the length of the packet also needs to match, then it will āmatch all valuesā for the wildcard marked bytes. The Dimmer would then react to the 3rd byte marked as the value. This was why I was considering a String channel on the Dimmer thing type so you can do your own check on the rest of the packet and it only updates when the packet matches what you have asked for, compared to the rawString thing type. If the packet has a changing value in the second byte it does not block the packet from matching as it used a wildcard.
I agree, I was planned on using the wildcard solution explained above for these use cases and leaving CRC and other standards for a future update. My own protocol requires certain things but since it is part of my security system I will not discuss the protocol at any detail on a public forum
Ah yes I had forgotten that some protocols use a packet counter that increments, the wildcard allows for that and the String channel to pass the packet onto openhab.
Yes that is something that could be implemented and I have done this in my EspMilightHub binding so as not to crash a esp8266 device with the amount of data that Openhab can send. Iāll make an edit to record this as it will be needed and Iāll have to consider what options to give.
You can be sure something like this is implemented, as mentioned I have >50 devices which all can RX and TX down the same RS485 bus. Ability to send ACK packets will be considered too depending on what the contents of the ACK needs to be, ie CRC
Yes that is exactly what I am after at the moment, examples of the packets that people wish to use that perhaps are difficult to implement at the moment. I donāt see any reason why that wont work with what I have in my head. The idea is to pass the fully matched packet back as a string you can use in rules should you wish to do checksum calcs and then you can pass the value onto the Openhab control if it passes, or you could just ignore the checksum and make an assumption it was fine.
The other thing I value highly is what formats are most useful with transforms, rules and other methods?
One nice thing mqtt2 does it it also sets the Items liked to those childrenās channels to UNDEF, indicating that we donāt know what state the device is in. Itās a great way to detect and react to errors in rules.
Again taking an idea from mqtt2, would chaining transforms be useful here?
From a high level, I know there is a serial communications bundle that a lot of other bindings use. Would this binding also use that shared bundle? Not knowing the details I canāt push either way, but it seems like a good idea.
I donāt use serial so all of my experience comes from helping others. If there is any way you can make the simple cases simpler to configure it might help users get started more easily. How that looks I cannot say.
Yes I have seen Hilbrand recommend the DSMR binding as a place to start and that is my plan and I checked it out a few weeks back to see what it usedā¦ I do have concerns over the rxtx library I believe it used as it does not have a good reputation for being stable and the project has had no updates of any kind in 6 years. Does not fill me with confidence, but yes I will try it first as having working examples to learn from will speed things up.
Thanks for the tips, will have to check this and the second suggestion out as I have a lot to learn still.
With the UNDEF idea, in the modbus binding we found it best to have that as a configurable option.
For transforms, see modbus again for an example (the raw data there is 16-bit word). Read and write are separate, think scaling.
CRC - I understand put it off till later and ignore for now but you canāt do that for writing. Iād suggest at least planning how to dovetail into writing.
I may have misunderstood your thing structure.
I was imagining -
A Bridge, specifying serial port. There may be more than one of these.
A umm station Thing, specifying a particular packet structure, device id, etc. Potentially several of these.
Each Thing has many channels, e.g. raw payload, simple switch, selected parts of payload. This is where transforms would apply. In my view, it is these channels that have the flavour of Dimmer or Sensor etc.
I suppose that summarizes as
Bridge = hardware
Thing = protocol
Channel = data
Using writing to Pelco as an example, envision setting up Bridge for COM5 9600 etc.
Set up station thing for pelco protocol, camera 7
Set up another station for pelco, cam 9
Incoming data gets routed via channels to separate Items, by way of each thing analyzing the packet (broadcasts may be llowed as defined by protocol)
To write, OH commands a channel, the thing formats, transforms,adds id, adds CRC, passes to Bridge.
EDIT - a structure as above, with multi stations sharing a bus, make obvious it would be desirable to define āprotocolā or āpacket structureā or whatever you call it in a shareable way. Perhaps an external file, info encoded as you like but JSON might work.
I am not going to take that route and create a nightmare where if someone wants X protocol I need to spend hours on a protocol I donāt even own. Protocols should go into their own bindings in my opinion and the serial binding should be a fall back GENERIC binding to use when nothing else better is implemented. There would be hundreds if not thousands of serial protocols in use and if someone wants to give up the rest of their lives implementing them they are welcome.
I guess my proposal would be summarised as:
Bridge = hardware serial port which we agree on.
Thing = Different Filter types. A way to filter out the data you want from a stream that comes from the bridge and convert it to the data you really want. The name of the thing is what the filter is designed to give the end user.
Channel = data that is filtered and already in the format Openhab needs with no need to transform it.
We could say there are two groups of serial devices, 1 that only have a single sensor or a small number of controls like a Fan speed controller. It is stupid to create a separate binding for this, which is where the serial binding best fits.
The second type of serial devices like Pelco that have a large number of control ability say more than ten channels from a single device. This is where a separate binding should be created with a better or more standard use of the THING concept. However for people who only wish 2-3 controls it should be simple to use the serial binding to get what they wantā¦
Example a surround amp with 50 different controls defined but we only have 1 user of Openhab that is happy if they can get control over 3 of them, ie POWER, INPUT and VOLUME
Of course nothing is stopping someone from spending countless hours creating 50 things for the 50 controls, I just believe someone will be happy that what they need is possible very quickly to get the 3 basic controls.
Perhaps a thing type could be defined to filter out 8 switches from a byte. Each bit goes to its own switch channel and it also has a string channel to display the full packet with.
Does this make sense? Should it be called āSerial Filter Bindingā or something other than Just Serial? Have a better idea how a simple and flexible binding can be created that does not require huge up keep?
Absolutely, poor choice of word by me - letās call it āformatā which might describe the shape and size of some packet in terms of āmatch this headerā, ānext is three bytes of dataā, ānext one dataā. A sort of REGEX essentially.
I think that would have you reinventing wheels. Applying transforms at channel level is already implemented and understood, and gives users lots of flexibility e.g. I want data ābananaā represented by a CLOSED contact in OH.
Following from earlier example, the three bytes of data arrive here as they are, ready for transform.
In general, yes I agree, keep it simple. If possible and practical, design in structure to allow people to make their own knobs and whistles.
I think Pelco makes a pretty good example.
I am NOT suggesting that you write Pelco protocol into a general Serial binding!
But the packet has address, header, multi data payload, and error check sections.
Iām really not sure at all how to go about it, but a configurable means to divide up and match or route such chunks is I think what you plan?
Yes that is the idea to use āThingsā to filter/parse the incoming serial data until it matches a specified format. Being able to do easy strings like ābananaā to more advanced packets of bytes like Pelco.
Yes you get to choose from something like the following:
MATCH (match only exact value user gives)
IGNORE (match all values)
ROUTEx (Send data1 to the channel1, send data2 to the channel2ā¦)
The exact how and finer details I am still working it all out as I like to see what packet formats people are wanting to use and sketch an idea up based on this information. CRC and checksums can be added later, but the IGNORE feature which is also needed for multiple other reasons could be used.
If there is an elegant way to add 50 Pelco devices to Openhab, I would love to learn and see an example for my own knowledge. I feel that as Openhab moves towards a graphical UI it does not hurt to have another way to do things.
Comment - me harping on the CRC thing is not about providing an immediate solution. For sure, we can ignore on read.
If anyone is interested, I imagine they will be able to channels of raw packet topped and tailed, channel of checksum field, run their own script.
Iām thinking about write, where you cannot control the targetās expectations.
Say the Thing is assembling a packet e.g. templated fixed fields + data from assorted channels, it is ādifficultā for some external user script to calculate CRC type addon across an internal binding creation.
Iām saying plan a means to deal; top of head, aÄŗlow a user script to be insertable in the process. Rather like a transform between the Thingās completed assembly and the Bridge.
Thatās different to channel transfoms massaging e.g. ON into x00FE
Yes I agree reads are easy(ish) compared to writes.
Writes with CRC could be handled in multiple ways including using a full string created and sent to the RawString thing type, the same way the current binding can be used to do this already. I donāt see an issue with that request.
If the CRC/checksum is a common method used by multiple protocols it should be added into the binding in my opinion and not in a script. Sometimes turning a script into a built in Java function is only 1-2 hours of work if the framework inside the binding has been created to be flexible.
The other thing that can be a blocking issue for controlling a device is the requirement for ACK(nowledge) messages. Some device will keep spamming the serial bus thinking the packet was lost and needed to be re-transmitted if Openhab does not respond with the correct ACK packet.
These are the kinds of use cases I am looking for right now so I can develop the idea into something more concrete and create an empty binding with no code behind the controls for people to look at. But I simply do not have the time to do this for at least 2 months.
The ACK point is important. Iām guessing some variants will also reflect some of the data or a sequence number, blah.
Again I think about writing - where OH could expect an ACK. Like CRC, simplest is to ignore but we ought to think how it might be implemented and/or provide a mechanism for user to implement. The rules of the game may require awaiting an ACK before sending next packet. I suppose an actual implementation might toggle a channel on the Thing āawaiting ACK / all clearā. How to manage a NACK - eek! Also must have a timeout.
In truth, both read and write ack-ing could be managed reasonably well outside of the binding. Maybe consider that sort of thing as āprotocolā business, best left to external code (or a dedicated binding like Modbus).
Speaking of Modbus, you may find it worth a peek at that binding. It is an apparently simple protocol which spawns complexity at the interface to openHAB because of the way it gets used.
But it will include all the elements weāve discussed, not suggesting reinventing this but look for building blocks.
wildcards to receive values āāwould be great. e.g. assign a temperature value to an item via the MySensors serial protocol:12;6;1;0;0;[VALUE]\n
Item Example:
Number Temp_1 { serial="SerialDeviceThing","12;6;1;0;0;[VALUE]\n" }
to set VALUE to Item Temp_1 if this pattern is received
Thatās the general idea, except it will be things and channels. Youād somehow set up a kind of template in a thing 12;6;1;0;0;[VALUE]\n and have the value available in a channel to link to a Number thing.
I have a custom version of 2.4 that supports /r as a delimiter between messages, so it can use readLine (name ?). This makes the latency for each message to be much smaller. It would be good to see this as an option in the official binding.
Why did I do this? I use a arduino as an IR receiver, sending messages back via serial. The arduino is there to handle the unusual pulse signature of B&O ir codes. The delay in the stock bindings polling system makes it feel very slow when using up/down/left/right type actions.
Any update on the status of this migration? Iām using serial communication between an Arduino and a rPi for lights and buttons. I can test the binding if necessary.
Nothing has happened yet, been busy with other projects but this is still on the to do list. Finding spare time with a baby is difficult and now I have bush fires in my area that puts more strain on my āspareā time.
but you canāt do that for writing. Iād suggest at least planning how to dovetail into writing.