I am trying to use a Mill device (not supported by the Mill binding) using the HTTP binding. I’ve managed to get many things working, but I’ve reached a case that I don’t know how to solve.
The device API requires that you send a GET request that contains a parameter in the body as JSON. In “theory” this should be straight forward, something like this:
Type number : set-temperature-normal "Set temperature normal" [
stateExtension="set-temperature",
stateTransformation="JSONPATH:$.value",
stateContent="{\"type\": \"Normal\"}",
mode="READONLY"
]
This doesn’t work, the content sent is null. The documentation for the binding (HTTP - Bindings | openHAB) says about stateContent: “Content for state requests (if method is PUT or POST)”.
So, it seems like this is intended behavior. I don’t understand why though, and if there is another way to achieve this, it has alluded me this far. Is this done out of some “correctness principle” where GET requests aren’t supposed to have a body and should use URL parameters instead, or is it a practical/technical reason for this? If the former, it’s not very helpful to enforce correctness on a binding when the devices we’re trying to operate don’t adhere to this. With this, I mean that it’s pretty futile for me to try to contact the manufacturer and make them change the design of their API because openHAB doesn’t consider it the correct way to do it.
Am I missing something and there’s a way around this, or does this effectively prevent using the HTTP binding for this device? If so, does anyone have a suggestion for a workaround?
Although request message framing is independent of the method used, content received in a GET request has no generally defined semantics, cannot alter the meaning or target of the request, and might lead some implementations to reject the request and close the connection because of its potential as a request smuggling attack (Section 11.2 of [HTTP/1.1]). A client SHOULD NOT generate content in a GET request unless it is made directly to an origin server that has previously indicated, in or out of band, that such a request has a purpose and will be adequately supported. An origin server SHOULD NOT rely on private agreements to receive content, since participants in HTTP communication are often unaware of intermediaries along the request chain.
So while it isn’t forbidden, it’s strongly discouraged for both clients and servers to use a body in a GET request. The correct solution would thus be that the device manufacturer changes the API (don’t see why they couldn’t just use query string parameters instead). But as you say, it’s probably not feasible to convince them of that.
Personally I don’t think OH should violate the standard just because someone else does, but if you can convince the maintainers to change it it’s not my decision.
Alternatively, you could probably use your own script and call that via the exec binding instead of using the http binding.
I’m pretty sure asking them to change it is completely futile, it would also create a headache for existing implementations where a firmware upgrade would break functionality.
I fail to see how this is “breaking the standard”, this isn’t a web server on the Internet exposed to all kind of abuse, it’s a local device, without much security to begin with (HTTPS with self-signed can be enabled). With HTTPS enabled, it relies on an “API key” for all access, apart from that it’s wide open. So, I have a hard time to see the potential security implications.
Using the EXEC binding is what I’d call very impractical, there are loads and loads of “getters” and “setters” in this API, and doing all that in ugly curl syntax is what I’d call unmaintainable.
I’ve never understood the reason for enforcing such “correctness” in software like openHAB that is all about getting things to talk to each other. If you develop a web server, sure, go ahead and ban it - but as a “client”, you must do whatever the “server” demands.
If correctness is that important, there could be a binding configuration allowGetContent with a big warning of how dangerous this is in the documentation, just like with ignoreSSLErrors.
I very much doubt that this is the only manufacturer that makes such an implementation mistake. Embedded device developers typically aren’t “web standard” experts, and are probably happy when they’ve made something that works. openHAB should be able to manage these devices without having the user jumping through too many hoops.
Just to clarify, this has nothing to do with security, standard protocols are for interoperability between different devices and vendors, which gets harder if someone decides to deviate from the standard.
The problem here is that you are using a generic binding to do something specific, that the generic binding doesn’t support. The way openHAB is “getting things to talk to each other” is through different bindings, that can support either standard or proprietary (and even undocumented) communication protocols.
If there is a Mill binding already, but it doesn’t support this specific device, it’s that binding which should be modified to add support, not the generic HTTP binding.
and might lead some implementations to reject the request and close the connection because of its potential as a request smuggling attack (Section 11.2 of [HTTP/1.1])
That’s why I mentioned security, because if it’s just about following the standard to be pedantic, I really think doing it is meaningless.
I know that you can create custom bindings, but there are a number of challenges with that. Foremost, it’s a lot of unnecessary work if it could be done with a generic binding, it requires maintenance, code review, testing and general bureaucracy. Also, most users are unable to write a binding, but they can easily create or copy/modify a configuration for an existing binding. It is also my impression that more and more devices come with WiFi and some API instead of using some proprietary, licenced standard like ZWave etc. The WiFi enabled chips like ESPx are dirt cheap, so this is a very cheap way for the manufacturers to make their devices “smart”. As such, I expect this to just increase in relevancy.
I’ve already been done that road, the author doesn’t have any “Gen3” devices which has a local API. Previous generations only has “cloud functionality”, and the binding is a hacked thing that tries to use the “cloud” without proper documentation that can break at any time. The author said that I was free to extent the binding to also support local operation, but that’s where the general problems with openHAB comes in: I can’t contribute to openHAB because the CLA requires that you declare your full name and e-mail publicly - which I find unacceptable. In addition to that, all the “official” bindings are in one repository, which means that there’s a very slow turnaround and even if the CLA problem didn’t exist and I did modify the binding, there would probably be months before I could use it. I only intend to use a few days on getting this device working, so that’s not an option anyway.
As it looks now, I only see one option, and that is to write an “unofficial” binding that does this. But, the likelihood of others finding and downloading/installing it manually is slim, so that would be quite a lot of work just so that I can manage two devices. My plan was to figure out the .things and .items definitions and then post them on this forum so that others would at least have a starting point. That doesn’t work if I have to make my own binding.
That is just one motivation to why it should be avoided, some intermediary servers might drop the request for security reasons, which causes the interoperability to suffer. That is always a risk if you deviate from a common standard: interoperability cannot be guaranteed with all other parties, so things may break. If you only intend for your own products to communicate, you are of course free to do as you like (which could very well be the case here: Mill have no interest in supporting third parties to integrate with their products, so they don’t care).
It this means that more and more devices start using established standards, but deviate from them in different ways, it will be no better than with different proprietary protocols IMHO. We would still need as many bindings to compensate for everyone’s various quirks.
But beside all that, I’m just another user/contributor stating my opinion, and don’t have any say in the matter, so it’s not me you need to convince. Create a feature request for the HTTP binding, and someone might decide it’s worthwhile to do.
This is a major reason why the Zigbee binding for OH doesn’t support nearly as many devices as zigbee2mqtt does. All these vendors add their own non-standard stuff to the Zigbee protocol which required special handling. The developers of the zigbee binding cannot keep up.
@Nadahar it may seem “pedantic” to you because it’s inconvenient to you. But deviating from established standards imposes a real and in some places a significant continuing burden on the mantainers. As soon as you deviate from a standard you are now in custom territory and custom takes a whole lot more work over the long run.
But you have lots of options, not all of which require a modification to OH:
submit an issue to see if the maintainers are willing to take on the work of developing and maintaining this deviation from the standard (assuming it’s possible, the HTTP binding uses the standard Java HTTP libraries which themselves may not support a GET with a body like this).
1 plus submitting a PR to go with it adding the feature you desire. Again, you get to use the modifications from the moment you make them. You don’t have to wait for a review and the changes being merged.
Modify the existing add-on. Again, you get to use it as soon as you complie in your changes. You don’t have to wait for a merge for it to be usable for you or even others.
Create a new add-on. You can publish the add-on to the Marketplace to make it discoverable and installable by others.
In your rule, use the raw Java HTTP libraries yourself. You have access to all of Java in rules. You could publish the rule as a tempalte to the Marketplace to make it discoverable and installable. This of course assumes that the raw Java HTTP lets you create a GET that violates the standard.
Use jRuby or Jython for rules. I know with Jython you can install and import HTTP libraries (is requests still around?) and use them directly in your rules. I suspect jRuby supports HTTP stuff natively. Again, this presupposes that these allow crafting a GET request that deviates from the standard.
Use executeCommandLine or the Exec binding and curl.
Use HABApp which is 100% pure Python 3.
Use an external script or service which push/pulls events to OH over the REST API.
Excuse me, but this simply isn’t true. I’ve been a developer for a UPnP/DLNA media server for many years, and if I had been following this philosophy, almost NO device would have worked at all. Yes, in an ideal world the manufacturers would follow the standards properly, but as long as we live in the real world, we have to deal with reality. There are so many exceptions from the standards that it’s mind-boggling in TVs, game consoles, receivers etc. What good is it to make a software that rigorously follows the standards but can’t work with any of them? My approach when developing such things is to try to be as flexible as possible while maintaining some semblance of “order” and system. I completely agree that a patchwork of “hacks” is useless and unmaintainable, but there are many “levels” between purism and anarchy.
In addition, this doesn’t even deviate from the standard as I read the above quote. It is in fact “allowed” as I see it, why wouldn’t cases like this qualify when the standard says this?
A client SHOULD NOT generate content in a GET request unless it is made directly to an origin server that has previously indicated, in or out of band, that such a request has a purpose and will be adequately supported.
I was hoping there was a workaround or some other way when I created this thread. Asking for somebody to implement something generally isn’t very quick and is often futile. People often come up with all kind of arguments for why things should stay as they are. My experience is that if you want to have any hope of getting something implemented, you better make a PR that actually does what you need. Then you might get lucky and get it merged. But, that’s not possible in this case because of the CLA.
Impossible because of the CLA.
Impossible because of the CLA.
This is an interesting possibility if there are no CLA-like roadblocks. Any idea where I can find the “small print” for publishing things to the Marketplace in this way?
Again, this doesn’t violate any standard, and I’ve written many both HTTP clients and servers using several Java frameworks (the “native” one that I don’t remember what they call, the Apache HTTP library, Netty at least), and I can’t remember ever having met such a problem.
I know this can be solved using rules, but I see that has “hacky” and unmaintainable. I want to have things simple, define Things and Items like “normal” and be done with it. The reason is that otherwise I won’t remember all the quirks when I want to modify something in the future.
Technically, probably a possibility (you generally don’t know until you try and bump into perhaps some other limitation with that binding), but something I’d really rather avoid both because curl syntax is terrible, and because I see it as very wasteful to have to spawn a process every time the device has to be polled.
Also probably viable, but not what I had in mind. It seems like a lot of extra stuff that has to happen (it communicates using the openHAB REST API from what I understand), and I just hate any format/language where whitespace is part of the syntax. I don’t think I’ve written python since 2005 for that reason. Running scripts/external processes to achieve something like this is just “wrong” in my view, and it sounds like it would be just as much work (if not more) as making a binding in Java. I’m comfortable with Java, so I’d rather write a binding.
An important aspect for me is that if I am to invest the time needed to implement most of the calls in Mill’s “local API”, it should be usable by others as well. It just doesn’t seem like it’s worth the effort if I’m the only one that will ever use it. While the API isn’t in any way huge, it’s substantial enough that it will take some hours to implement it regardless of which method is used:
If this is the rules that still apply, I see no major roadblocks from my point of view:
But, my binding might end up being rejected because of one of these points:
Publishing add-ons that compete directly with an official add-on is not allowed; by “compete” we mean “forks” or alternative implementations offering similar functionality.
Your contribution must be significantly different from other submissions (no copycats with minor changes). You are encouraged to work together as a community instead of proliferating multiple confusingly similar versions of the same thing.
Whether this “competes” with the official add-on is a question of how you interpret it. The official add-on doesn’t handle the local API of Mill devices, but it does handle Mill devices. The CLA prevents me from modifying the existing binding.
Before undertaking this work, I’d need to know that it won’t be rejected. How would I go about getting clarity about that?
That’s not the case here though. We have a binding that, based on evidence reported thus far, doesn’t work with one device. No one else over the decade that OH has been developed has reported a problem or asked for an HTTP GET with a body.
But none of that matters. No one here is saying that OH can not and will not support a GET with a body. We are just explaining why it isn’t supported now.
You do not have to submit anything back to the OH projec. 3 means you make your changes and continue on your merry way and do not submit the changes back to the OH project. The OH project is not going to change the CLA for valid legal reasons which means you’ll never be able to contribute to the project. But it’s open source. You can make your changes to your copy of the repo and use it from day one.
There is no license restrictions to the Marketplace. Marketplace posting are by definition not a part of the openHAB project. The big rules are: don’t duplicate existing OH addons or bundles already posted to the marketplace. That’s pretty much it. For the users of marketpalce stuff it’s caveat emptor.
There is also the Smarthome/J marketplace which doesn’t have these restrictions (but I’m sure there are other restrictions) and you can create your own marketplace if you want.
Yes, that’s the list of rules for the Marketplace.
I don’t think it competes if it implement devices that the official add-on doesn’t support. Very few add-ons have ever been rejected and so long as you are not just completely replacing the behavior of the existing binding or basing your add-on on the existing binding there shouldn’t be a problem.
You are posting an addon to support devices that no exisitng add-on supports.
I’m not the king of the marketplace so I can’t 100% guarantee it would be rejected or not, but in general we try to be as accepting and flexible as possible. The developer of the existing binding would probably have to complain before anyone would take a second look at it.
I was talking about the principle. In my opinion, it’s much better to make the “generic” tools as flexible as practically possible than to be pedantic and reject everything that is not completely “mainstream” as a “corner case” that needs a separate implementation. That doesn’t mean that the generic tools can handle all cases, but the more they can handle, the less device-specific implementations are needed and as a result, less code maintenance is required in total.
This doesn’t just apply to one device. This API is, as far as I can understand, used on all devices from this manufacturer that are of “generation 3”. That, and the fact that I think it’s probably very easy to remove this limitation in the HTTP binding, and the fact that I think it’s very likely that there are other cases where this is useful, leads me to think that the best solution would be if the binding allowed it.
But, discussing this here is moot I’m sure, as those that make this decision aren’t part of the discussion.
I know that the project will probably never change the CLA, and I know the reason such CLAs are used. I have no problem with the content of the CLA either, only the unreasonable demand that the name and e-mail mush be published in public. I reject the idea that the name and e-mail must be made public for the validity of the agreement. I would have no problem signing an agreement like this in any form that didn’t make my private data public. I also know that openHAB didn’t “invent” this, other projects use a similar system. It’s a shame that his has kind of become the de-facto standard way of doing CLAs - as I see it, it’s done out of lazyness more than anything. I’d also say that I question how “binding” that really is, anybody can write a name and an e-mail address in a commit - it doesn’t in any way prove that this information is real.
I also very much doubt that I’m the only one that cares about privacy and is stopped by this. I might be the only one that’s vocal about it, but to me it’s quite clear that such CLAs that require people to share personal details in public hinders collaboration and contributions in general. I think it’s sad more than anything.
That doesn’t seem likely:
@ysc Maybe you could give a quick input regarding whether you think such a binding would be problematic on the Markedplace? I don’t want to do the work if it will be rejected.
And it’s moot because you are the first person who has encountered the problem, first who want’s it changed, and are unable to submit a PR for it. We’ll have to wait for another motivated person who has this problem before anything is changed.
It’s probably worth filing an issue. Who knows, someone may take it on.
I am not @ysc but I can second what @rlkoshak already wrote.
The Binding you want to publish will not be identical to the existing one, as it adds a completely new feature - local communication. I don’t see any reason not to publish it.
Thanks for the feedback @rlkoshak and @hmerk. I’ve started looking at creating a binding, but the OH Eclipse installation is what I would call challenging. There are obvious version/dependency issues at work here, and I’m currently at my third attempt (I first tried 2024-03, then 2021-09 and now 2023-09) and it takes hours and hours to build everything to see if the errors finally resolve.
But, assuming that this will finally work out, I’ve started to look/think about how the binding should work. As a part of that, I’m considering the concept of discovery. As far as I know, these devices don’t broadcast their existence in any way, no mDNS or UPnP or similar mechanisms. That means that the only way I can think of to “discover” them automatically would be to contact/scan all local IPs and try some of the API commands if it responds on port 80.
This is what I’d consider quite “crude” and “rude” behavior. The other alternative is that there is no auto-discovery and that items must be configured manually for the binding to be aware of them. I assume that the situation with no broadcast isn’t unique to these devices, and I’m wondering what is considered the best approach?
Discussing the binding might be considered highly irrelevant for this topic, so maybe this thread should be split or a new one created. If so, where?
You are absolutely right, scanning the complete IP subnet for devices reacting on API commands is a no go. It is not uncommon that devices cannot be auto discovered, thats why we have many Bindings without an implemented discovery service. But this is fine.
Just responding since I was mentioning but @rlkoshak and @hmerk have already said most of it - and despite the rules having been posted by me, I’m not the marketplace king either Arguably the principle is there’s no king, only the OH Foundation hosts the marketplace so some guidelines were in order, so I’ll just give my opinion:
The spirit of this “non-compete” rule, I think, is mainly to avoid “dissident” forks or reimplementations of official bindings (like “HTTP Binding - Nadahar’s version”) that come mainly because of disagreements with the official version’s code owner, or just because you didn’t bother trying to collaborate. It can come off as disrespectful.
If you are unable - rather than unwilling - to contribute to the official version, then that’s another story, and I don’t think there’s anything preventing you from publishing this alternative version. It’s probably best to make it prominent what it offers more and ideally your reasons why there has to be an alternative version.
One example (not involving anything official but still):
I made the original version, and was asked for consent to have this alternative version published since I was reluctant to have the parameter added “by default”, and I said yes, so no problem having the 2 versions cohabiting. The purpose of the addition is clearly stated in the title and description.
This part of the rules:
When you base your contribution on someone else’s work, make sure you obtain permission and/or at the very least properly credit the original author
is probably more important than the “non-compete” rule.
Arguably the principle is there’s no king, only the OH Foundation hosts the marketplace so some guidelines were in order, so I’ll just give my opinion: