Next generation design: A Paper UI replacement proposal

[rlkoshak (Rich Koshak)]

@David_Graeff

There is. A profile is a “configuration” on a Link .

Ah, I was looking in the wrong spot. Thanks for pointing me to the right location.

@David_Graeff

This UI should just be for “Setup & Maintenance”. Sitemaps are for the layout of the “Control” part already. A separate sitemaps layout app might make sense or sitemaps should be editable by all apps that can visualize sitemaps, like the iPhone, android and classicUI app.

One could make the same argument for Rules and Scripts. Sitemaps are required for the phone apps and support for building them needs to be in some UI if we are ever going to get to the possibility for UI only users (to all the text users, I’m not saying to get rid of text based sitemaps).

It’s unclear to me how much long term support there will be for sitemaps in their current form anyway. It uses the same Xtext based parser so if we are going to get to where we can drop those libraries something is going to have to replace the file format at the very least. If it weren’t for the fact that the phone apps are based on them I’d be fine with deprecating them when .items and .things files as they exist now are deprecated. But it will be pretty odd to have a system where everything can be done through a UI but if you want to use the official phone app you have to manually create text configs.

@David_Graeff

I have no idea how persistence works. I have never used it. But I know that there is a REST interface and I agree that it deserves a place in the UI somewhere. Maybe as sub-page within “Maintenance”.

That sounds like a reasonable. While there is a REST interface to interact with persistence there is no REST API I can find to actually configure the persistence strategies. Basically a REST end point to set:

Strategies {
  default = everyChange
}
Items {
        gChart*,gIndoorTemps,gIndoorHumidity,vIsCloudy : strategy = everyUpdate
        gHistory*: strategy = everyChange
}

Save every update to members of gChart, and the values of gIndoorTemps, gIndoorHumidity and vIsCloudy. Save every change to members gHistory.

or

Strategies {
  default = everyChange
}
Items {
        * : strategy = everyChange, restoreOnStartup
}

Save every change to all Items and restore all Items on startup.

It looks like everything else is possible but I see no way to set the strategy.

Also note that one can define their own strategy using a cron expression. For rrd4j everyMinute is a requirement.

Strategies {
    // for rrd charts, we need a cron strategy
    everyMinute : "0 * * * * ?"
}

Items {
    // persist items on every change and every minute
    * : strategy = everyChange, everyMinute
}

Maybe the ability to configure the strategies is under one of the other REST endpoints?

[David_Graeff]

Could you sketch a rough interface for persistence, so that I get an idea?
Seems like the strategy and an item selection, based tags or regex matching is required at least.

@rlkoshak

It’s unclear to me how much long term support there will be for sitemaps in their current form anyway.

There was a PR that changed the storage for sitemaps to jsondb and added REST endpoints as well.
As far as I remember it wasn’t completely finished. But for OH3 it can be picked up as a starter, I guess.

[hilbrand (Hilbrand Bouwkamp)]

All persistence implementations are openHAB 1 implementations (probably explains why you didn’t know them ) I did some fiddling with implementing rrd4j as oh2, but never finished it as I had the impression nobody was looking for it. I might have some code laying around somewhere.
I guess it can be implemented with a specific trigger and some kind of persistence action. But don’t know if that would be the correct approach, architectural wise.

[rlkoshak (Rich Koshak)]

@David_Graeff

Could you sketch a rough interface for persistence, so that I get an idea?

Let me ponder because it seems like there are two parts and perhaps they should be represented separately.

The first part is defining the strategies themselves. Here we have a few predefined strategies (everyChange, everyUpdate, restoreOnStartup) and perhaps some predefined time based ones (everyMinute, everyHour, everyDay) but then we need the ability to create a custom strategy. Cron expressions are used for now.

It seems, assuming the core can support it, that we should be able to define the strategies separately from the next part. Then we can select them from a list.

The next part is mapping Items/Groups to a strategy for a given persistence engine. For example, above map gChart*,gIndoorTemps,gIndoorHumidity,vIsCloudy to the everyUpdate strategy on the InfluxDB persistence engine. I map all my Items to the everyChange and restoreOnStartup strategy on the mapDB persistence engine.

So maybe the UI isn’t so hard.

I gotta run but will post a drawing here later today or tomorrow. My fear though is, as Hilbrand says, persistence is OH 1 through and through and there is no REST endpoint yet to support setting the strategies.

[vossivossi]

@hilbrand

All persistence implementations are openHAB 1 implementations

As persistence is really one of the very strong features of OH it seems funny to see that there are only 1.x implementations. Apparently there was nothing to improve?

[David_Graeff]

@hilbrand

All persistence implementations are openHAB 1 implementations

Can’t we just declare the OH1 persistence interface to be the OH2 one and add some REST endpoints . Those are basically services, aren’t they. So they just need a service description xml file. But yeah I really have no idea, never seen the code.

[hilbrand (Hilbrand Bouwkamp)]

Its probably because, If I’m correct, there is no oh2 way to configure them and no specifications that I know off on how to do it the oh2 way. Which would explain why there are no oh2 implementations. If that is in place it can be implemented as oh2 , which does include copying most of the internals of the presistence implementations as there is probably not much to improve.

[5iver (Scott Rushworth)]

@hilbrand

Its probably because, If I’m correct, there is no oh2 way to configure them and no specifications that I know off on how to do it the oh2 way. Which would explain why there are no oh2 implementations.

There is an ESH MapDb persistence bundle.

@hilbrand

which does include copying most of the internals of the presistence implementations as there is probably not much to improve.

There are also a number of OH2 item types that are not supported (Player, Image, Location, etc.).

[unparagoned]

@David_Graeff

A big concern at the moment is that we have a dual system of configuration. You either use the UI or files. And you cannot change to the other option without a massive time investment.

Please have a look at how I see this solved. Every thing/item/rule/timer has a “storage association” (aka filename). The backup service will consider those for exporting to several storage units (aka files).

Every thing/item/rule/timer has a list, grid and textual view mode. The latter one allows find/replace, batch editing. For the format I have settled on YAML for now. Please have a look and tell me if you like that syntax. Maybe there is a better suited format that I have not yet considered.

The current way openhab is kind of special and very confusing. Paper UI items and things are completely different to the config items. Since bindings use the paper ui setup, but if you want to have any non-trivial setup it’s really confusing to try and work with the two completely separate setups. The config files should mirror the paperui items, and vise versa. It took me months to understand why paperui wouldn’t let me edit some config items.

I used YAML with homeassistant and hated it. I thin the main issues was that it’s all based on whitespace which is hard coming from a c background, and the fact I didn’t have an editor which raised any issues. You also had to restart oa after any changes so it was just too much of a pain in the ass. But if there is proper vscode support for the syntax then it shouldn’t be so bad.

[rlkoshak (Rich Koshak)]

@unparagoned

The current way openhab is kind of special and very confusing. Paper UI items and things are completely different to the config items. Since bindings use the paper ui setup, but if you want to have any non-trivial setup it’s really confusing to try and work with the two completely separate setups. The config files should mirror the paperui items, and vise versa. It took me months to understand why paperui wouldn’t let me edit some config items.

This is exactly what is being proposed.

@unparagoned

I used YAML with homeassistant and hated it. I thin the main issues was that it’s all based on whitespace which is hard coming from a c background, and the fact I didn’t have an editor which raised any issues. You also had to restart oa after any changes so it was just too much of a pain in the ass. But if there is proper vscode support for the syntax then it shouldn’t be so bad.

I use YAML for Ansible all the time and PyCharm at least manages the white space warnings but I find they are rarely needed as the proper white space gets inserted automatically on every newline. Over all, YAML is chosen because it is easier to work with than JSON but I believe whatever gets implemented will support multiple formats should someone choose to implement and add-on to support it. Of all the structured markup langauges I am familiar with (JSON, XML, and YAML), YAML is by far the easiest for a person to work with so it is probably the best choice for now.

Nowhere will we find it acceptable to have to restart OH to make a config change and that is not being proposed here. We will have an import/export function that doesn’t require a restart. So that should not be a fear.

[rlkoshak (Rich Koshak)]

Here is a quick mock up of one way it could look. This is just one option and it may not be the best. I split it up by strategy. One could also split it up by database or I’m sure other ways as well.

I’m not sure what filters would make sense.

The root of the issue is there needs to be a mapping between three things:

• Database that the states get saved to
• Strategy used that determines when states get saved (every change, every update, periodically based on a cron expression, restore on startup). It gets complicated because the user needs to be able to define their own strategy, though we should offer the most common ones by default (everyMinute, everyHour, everyDay). That is what the cron dialog is all about.
• The Items that get saved to the selected database using the selected strategy. It gets complicated with Groups because one can save all members of a Group (in the .persist file we use *), or save the state of the Group itself. This makes the selection complicated.

But as I think is pretty clear from the rest of the discussion, there doesn’t appear to be anything in the core at this time (though maybe there is given Scott’s link). But either I’m missing it or there is no REST end point to set these strategies.

[bartus (B K)]

I like the idea of splitting up the strategy configuration and DB/Strategy assignment, but would it be easier to just add the DB/Strategy assignment to the Item confiugration screen? That way, the persistence could be assigned at time of creation/configuration of the item (which is something of an after-thought and a dedicated secondary step in the current OH1/OH2 realm), it would be more intuitive to users that Items need direct persistence assignment (until recently, I was confused myself that the “*” symbol in .persistence files isn’t a wildcard that lets me assign the same strategy to multiple items), and we could even show that items are already persisted (as part of a group), but still allow the user to select a secondary/direct persistence strategy/DB.

[rlkoshak (Rich Koshak)]

@bartus

but would it be easier to just add the DB/Strategy assignment to the Item confiugration screen?

As I was making this that did come to mind. My only problem with that approach is handling the “All Items” case. I’d have to manually add something to all my Items which seems excessive. Or we treat “All Items” as a special case and can select it where we create/edit the strategy. So I opted for consistency in this case. But like I said, this is one way and probably not even the best. As long as we have a way to handle the “All Items” special case I don’t really have a strong opinion either way.

I suspect the whole persistence subsystem needs a review and rework. Maybe something better will come up.

@bartus

That way, the persistence could be assigned at time of creation/configuration of the item (which is something of an after-thought and a dedicated secondary step in the current OH1/OH2 realm), it would be more intuitive to users that Items need direct persistence assignment (until recently, I was confused myself that the “*” symbol in .persistence files isn’t a wildcard that lets me assign the same strategy to multiple items), and we could even show that items are already persisted (as part of a group), but still allow the user to select a secondary/direct persistence strategy/DB.

This is actually exactly why I created Design Pattern: Group Based Persistence. You then just assign the Group to the Item to choose which persistence strategy get’s used for that Item.

[ssalonen]

Regarding state of persistence in openHAB2, perhaps this short thread is of interest: Developing a persistence addon (not a binding)

I personally converted dynamodb persistence to be compatible with openHAB2 but still using compat layer since the new APIs seemed “raw”… For details check the discussion in the thread

[David_Graeff]

Alright that means I’ll come up with a UI and we try to extract necessary functionality out of that study to design a REST interface. That allows later on to create an API bundle. And at that point people can individually work on persistence bundles that implement that API.

For oh3 we need one working implementation of course.

[hilbrand (Hilbrand Bouwkamp)]

Just some crazy idea: why not persist the state of all items by default onEveryChange when persistence is configured? (I understand it requires some additional logic for rrd4j, but that would be an presistence specific implementation detail)

[5iver (Scott Rushworth)]

By the time we get to OH 3, we should have a way to view and edit Item metadata. You can currently add and remove it though the REST API, but you cannot view it. It would be nice to view and edit metadata in the Item details.

[rlkoshak (Rich Koshak)]

@hilbrand

Just some crazy idea: why not persist the state of all items by default onEveryChange when persistence is configured? (I understand it requires some additional logic for rrd4j, but that would be an presistence specific implementation detail)

There are a ton of use cases for Persistence and that “strategy” doesn’t fit with all of them. For example:

• sometimes we need to save when an Item is updated, not just changed
• to generate some charts for slow changing Items sometimes it is necessary to save the Items current state periodically whether or not it changed or else the charting software will not draw the line on the graph
• at least one of the databases requires an every minute strategy just to work (rrd4j)
• sometimes you only want to save the state after some filter is applied to the data to get rid of bad readings (we need a Rule for this)

Not to mention that saving every change of every Item in every database will be a TON of writes (sorry SD card users) and generate a huge amount of useless data stored in the database.

It really is a shame that Persistence has been ignored up to this point because it is a core feature that many of us absolutely depend upon for telemetry (i.e. charts), alerts, calculations, and to dive logic.

Here is a good concrete example for why we need more than one strategy to support. Let’s say we have a slow changing Item, maybe a Switch, that we want to chart. I do this myself to chart whether the fan or the heater is on on the same chart as my indoor temperatures.

In order for the chart to work properly, I must have data stored in the database at least every 10 minutes or so. Otherwise when I zoom in too far the line goes away because there are not enough points to generate the chart.

But, I also have some logic that needs to know when the last time the Item changed was. I can’t get that information using the every 10 minute strategy because the lastUpdate really means the timestamp of the most recent value stored in the database. With an every 10 minute strategy, that timestamp is not guaranteed to be the time when the Item changed. And we can’t use previousState(true) because that will give us the time when the Item changes to the previous state. There is no way to determine when the Item changed states to its current state. So we would need to store the state of this Item using two different strategies for the two different use cases.

@5iver

By the time we get to OH 3, we should have a way to view and edit Item metadata. You can currently add and remove it though the REST API, but you cannot view it. It would be nice to view and edit metadata in the Item details.

This will be increasing important for the NGRE because there is no way to set global variables in it. So metadata is going to have to pick up some of that slack.

[ssalonen]

To add to the list you mentioned, even though the SD card writes is not an issue with external persistence services (e.g. Dynamodb), persisting all may have a very detrimental effect to openHAB performance.

Since persisting actually does take some time, all internal queues eventually keep on growing, making the system unoperable.

[oblaise]

I will apologize beforehand if this was already discussed/proposed I don’t have the time currently to read the full thread.

For having a single way of storing items/things/binding configuration : cannot we simply use the current syntax (parser already exist) and store config for the different elements in file using same syntax? No need to have a new parser, easy way to go from file to paper UI and vice-versa.