Channel Templates

Add-ons Bindings
41

This is almost the very definition of “perfect”.

Also, it’s not lying or misleading if you are up front with how it works.

Because you cannot perfectly detect when a version has changed, users must do it manually.

It is an objectively better end user experience to be notified that a new version is available periodically when one is not than for the user to not be notified at all under an circumstances. Therefore yes, the perfect is the enemy of the good here.

OK, that doesn’t bother me as much then. However, I wouldn’t want this to hold up being able to do Thing templates overall. If it’s too hard or too controversial I’d rather see that dropped and have to manually define the parameters like we have to everywhere else rather than have no templates at all.

Note, if there is no existing registry to store this in, that doesn’t mean a new registry couldn’t be created to store this information in. I personally do not see why it’s not possible to add a field to the JSONDB entry for rule templates, widgets, transformations, and block libraries to record the install time. Indeed I see that bundles are a problem but none of the issues you’ve brought up so far applies to these other entities. As far as I can tell, those JSONDB files are only written to when you actively add or remove.

And these Thing templates will need a brand new registry anyway, so even if you can only do this for Thing Templates now it’s a start.

42

Think again. Check your JSONDB. If that was the case, this whole thing wouldn’t be a problem. Basically, the logic of how this is handled is flawed, and instead of solving the logic, ways have been found to have a “backup way” to detect that an add-on is installed. But that method is different for each type, and has no way to embed extra information. This “backup detection” has allowed the handling of the “primary information table” to remain broken.

43

Basically, I think it’s pretty clear that we won’t agree on the “update available” part. I’ve put a lot of work into making versioning work already, and there are unresolved challenges, but that doesn’t mean that I haven’t found solutions to a lot of the details. My goal is to have all this resolved and work universally, not to “monkey patch” something to make something work under the best of circumstances for a particular situation only. Those “solutions” are mostly what ends up being the obstacles that make it very difficult, or impossible, to get things to work properly, so I won’t agree to creating new obstacles.

But, it might not matter what I think anyway, so let’s not get that hung up on this point. I think we’re pretty much aligned on the rest of this.

44

I have. I don’t see anything unexpected. None of the jsondb files I’m talking about here appears to be changing in ways that prevent storing the install timestamp as an additional parameter. Bundles are another issue with the marketplace JSON file but rule tempaltes, transformations, block libraries, and UI templates are not changing for me except when I add or remove something. Or I’m not seeing these changes.

45

We’re talking about different things. I’m talking about the table that holds information about installed marketplace add-ons, not those that hold rule templates, transformations and whatnot. Specifically org.openhab.marketplace.json. This is the bugged one, and this is where it would need to be stored.

46

I know. But marketplace add-ons are not the only thing posted to the marketplace and they are barely relevant to this discussion. There are more widgets posted to the marketpalce than there are add-ons last time I looked at the counts. Well more than half of the marketplace is made of non-add-ons.

And if this (Thing Templates) goes anywere, the Thing Templates should work like rule tempaltes, ui widgets, block libraries and transformation instead of touching org.openhab.marketplace.json.

No matter what, the approach to detecting and dealing with version updates for all these will be at least somewhat different from what needs to be done for add-ons because they are already implemented differently. I see no reason why the brokeness of how add-ons are handled should prevent addressing these, particularly given they make up the majority of the marketplace.

47

This is a misunderstanding. Everything on the marketplace are add-ons, I’m not talking about bundles. Widgets, rule templates, block libraries, they are all “add-ons” in this context. And they all initially appear in this table, and should be retained there were it not for the mentioned bugs.

When I say add-on, I mean what is handled as an Addon in core, which includes everything that can be installed from the marketplace. org.openhab.marketplace.json is the one table common for all of these, the other tables you refer to are actually “providers”, they “are” the “managed providers” that host elements that have nothing to do with the marketplace. The exceptions are marketplace_ruletemplates.json, which exists because there is no managed rule template provider, and possibly org.openhab.marketplace.transformation.json, which I’m not sure how is used.

But the general principle remains: Even if there are “dedicated providers” for the marketplace, that’s not where the information about installed marketplace add-ons live. This information is supposed to be in org.openhab.marketplace.json, which is seriously bugged.

edit: Try to install a rule template or a widget, and check org.openhab.marketplace.json immediately afterwards, and you will see that it will be there, until the table is wiped because of one of the bugs.

48

But except for bundles, all of these have data stored in another registry. Why can’t a timestamp be added there? That’s what I’m saying. Yes, org.openhab.marketplace.json is buggy but marketplace_ruletemplates.json, org.openhab.marketplace.transformation.json do not appear to be. It seems UI widgets and Blockly are mixed in with the locally defined stuff in uicomponents_ui_widget.json and uicomponents_ui_blocks.json and they are not buggy in the same way.

Even if org.openhab.marketplace.json wasn’t buggy, it seems like the timestamp for when these were installed makes sesne to add to these other files instead or in addition to.

Frankly, I’ve found myself missing the fact that it’s hard to find when an entity was created and when it was last edited already so I’d be happy if such a timestamp were added everywhere for everything really. It’s in the top three reasons why I use git in the first place.

49

Because they are directly serialized versions of the objects that are stored there, which is deserialized directly into Java objects when “retrieved to storage”. To store something new there, you would have to change all the involved components, e.g. RuleTemplates, RootUIComponents etc., and this field would be system-wide, i.e. show up for all instances of these, not only limited to the marketplace, and it would show up in the REST endpoints, YAML formats etc.

There is no “extra storage” in the provider registries, they only store the elements they provide. And, to change that, you’d have to change the very core provider/registry logic that the whole system is built on.

So, instead of trying to jam it in there, it’s better to find another place to store extra information. I had the exact same problem for versioning information, and in my branch, I’ve implemented a separate JSON that is stored either as a file next to the JAR, or in separate “tables”. But, the whole solution is messy, and I’m not happy with it and don’t consider it a “satisfactory solution”. There’s a lot to keep track of here, keeping that extra information “in sync” with everything that happens with the add-ons themselves, how to handle when the extra info is missing etc.

I’ve sort of ended up concluding that it’s “less work” to actually fix org.openhab.marketplace.json than to try to get this “extra info” storage to work reliably. But, it’s not straight forward to “fix” org.openhab.marketplace.json either, because there are some logical problems that must be solved in addition to pure bugs.

Whatever the best solution is, you get into the same jam as I’ve been in before when trying to store extra information, and touching many of these things is “scary” because any (new) bugs could have pretty essential consequences.

Only having a timestamp doesn’t achieve much, if you can’t trust it. It might be possible to somehow amend JsonStorage itself to store timestamps with everything it stores, but you quickly get into some issues there as well, unless you do the same for Storage itself, and as a consequence, any other Storage implementation. Because, if you don’t, it will be awkward to get access to the information, and you will need the code to handle both that the information exists and that it doesn’t.

Even if you modify the storage system to include timestamps, these won’t “line up” with those generated by the forum, with timezones, local RTC drift and all that.

50

Let me also add that I’m not saying that any of these problems are unsolvable. But, I ended up giving up, at least temporarily, but that has more to do with “the OH process” than that the problems themselves are unsolvable:

I identify a problem where I see that something must be changed to make it work properly. It will have some consequence, and somebody will probably be unhappy with some aspect of it. I don’t feel that I have the necessary insight to make the decision alone, so I seek to discuss it, primarily with developers that understand the code in question. Whether I try to do this on the forum or on GitHub, chances are that there is little feedback to be had, and if I get some, it’s usually generic without taking the concrete challenges here into account.

So, that takes me nowhere. I end up just having to decide what I think is best, and then start implementing that. This can be a lot of work, and I have no idea if even the core “decision” I’ve made will ultimately rejected. Then, after spending a month or two to implement this, I’ll have to submit a PR. If the problem is somewhat complicated, nobody is likely to want to review it. So, it may end up never being reviewed, or it could go many months or a year, before I suddenly have to answer for all the details of my reasoning. Which I just can’t do at that point, because it’s no longer fresh in my mind. And, if I can’t explain it properly, somebody is likely to want to change it, at least parts of it. So, there is another large undertaking to change what I’ve already done. Once that’s completed, it’s rinse-and-repeat, waiting a number of months before maybe somebody will look into it.

I never know if there even is a goal at the “end of this road”, and I have no idea how many times I’ll have to change everything along the way. Doing hard, labor-intensive things, under this “regime” just isn’t that tempting. Which is why I tend to stall when I’m faced one of these decisions that I feel don’t have an “obvious answer”. And, experience has shown, that even when I think that the answer is “obvious”, others might very much disagree… so even going with these is “risky”.

51

I don’t see that as a problem. Like I said, knowing when something was created has it’s uses outside of the marketplace.

But it’s neither here nor there. You are likely the only one to try to implement something and so it doesn’t really matter what I think about it.

52

But, it wouldn’t work for this purpose, since the timestamps wouldn’t be comparable to those issued by the forum anyway. They would also be updated if you saved e.g. a widget from the editor. And, introducing such a field for all these classes would probably be more work (when you take all the little details that follow with it into account) than to actually fix org.openhab.marketplace.json. org.openhab.marketplace.json would have one huge “benefit” in this context, and that is that it could store the actual marketplace timestamp that was present when the add-on was installed, so it would actually make sense to compare against.

But, fixing org.openhab.marketplace.json would also remove one of the larger obstacles blocking my work making the marketplace versioned, so if that was solved, I think I’d rather look at what the remaining obstacles are and if there’s a way to solve them. I know that one of the obstacles has to do with KAR files, but that is bugged today as well, and could just be “ignored” and remain broken I guess. I think those two are the two main obstacles blocking versioning.