New tool: a purely JS parser for item definitions

Development
1

Hi all,
I experimented with Nearley, a JS parser library and ended up doing a grammar for items definitions, you may preview it in this small web app on codesandbox linked below and are invited to test it and fork/improve it; if it proves useful I intend to include as a developer tool in the UI (see below). Let me know what you think!

openHAB Items Parser

Preview: https://ic0w3.sse.codesandbox.io/

Source: https://codesandbox.io/s/openhab-items-parser-ic0w3

This app uses Nearley with a custom grammar (in src/assets/items.nearley) to parse openHAB items definitions into JS objects.

You can paste or write your definitions from your own .items files or snippets found in the docs or community.openhab.org in the code editor, the parsing occurs in real time and the input happens in your browser (it doesn’t leave your computer). The results should be directly compatible with the expected format of the REST API PUT /rest/items endpoint as the request body. If there’s a syntax problem, the error and explanation will be displayed instead.

This tool will be integrated into the new default UI due to ship with openHAB 3, as a “developer tool”, and will be able to call the REST API to actually add or update the items. Similar tools may be provided for things and other objects.

The intended goal is to ease the migration of items from textual definitions to managed objects in the JSON DB fully editable in the UI, keeping the current syntax as a source for definitions because it’s widespread in the docs, the forums, and existing deployments. It can also be used to validate the syntax of item definitions quickly.

MIT License

7 Likes
2

Hey Yannick,

that sound that in openHAB3 textual configurations are not longer available. Could you tell us more about that.

Thanks
Thomas

3

No, nobody is saying that. There’s been some discussion about Xtext being one of the reasons why openHAB is stuck with Java 8 but AFAIK there are no plans to discontinue the textual definitions. As I said there’s too much references to this syntax to simply get rid of it. I personally intend to move away from it though and manage my items directly with the new UI in the future, except when it’s not possible (OH1 bindings), hence this tool.

1 Like
4

Thanks for your fast feedback. So i misinterpreted your word.

Unfortunately there is not much information about version 3 around here in the forum or i didi not find it. So every bit is interesting for me.

Thomas

5

I think we are stuck with Java 8 because of the OH1 compatibility layer. OH2 accepts text config too and, I believe, Kai has said text config will not vanish.

6

No the OH1 compatibility layer is not blocking on java 8. It all compiles for java 11. But switching to java 11 might mean a consequence for our java 8 compatibility.

7

can we stay on topic here gents :wink:

Nice work Yannick!

8

Based on my experience with Jython Drop-in Replacement for Expire 1.x Binding, at least for the expire binding, the oh 1.x binding config shows up as Item metadata. The REST API for dealing with metadata isn’t that great, but I think there is enough there where you could manage even items bound to 1.x bindings.

Anyway, this tool is very timely. I’ve just started looking at how to migrate my Items to JSON. I’ll definitely be giving this tool a try. Thanks!

9

Actually I saw your topic on the subject and am completely on board :slight_smile: - having all objects stored in one place (the JSON DB) has a clear advantage, but there must be a way to mass-insert things and items and leverage the existing syntax that you encounter everywhere around here, with simple copy-paste mechanics to define similar objects.

It’s definitely getting in the new UI :wink:

yKJHAfJNn2
yKJHAfJNn2.gif1723×822 974 KB

8 Likes
10

Awesome Yannick!

11

Any place we can sample this and possibly make feature suggestions?

1 Like
12

I gave the tool a quick try and there is one area that is a little disappointing.

Unfortunately, it appears that setting the channel information as metadata is not sufficient to cause OH to reestablish the Links between the Items and the Thing Channels.

For example, I copied the following Items into the tool:

Group:String PubItems

Switch vDad_LR_Motion          (PubItems) { channel="zwave:device:dongle:node2:sensor_binary" }
Number vDad_LR_Lum             (PubItems) { channel="zwave:device:dongle:node2:sensor_luminance" }
Switch vDad_LR_Alarm           (PubItems) { channel="zwave:device:dongle:node2:alarm_burglar" }
Number vDad_LR_Sensor_Battery  (PubItems) { channel="zwave:device:dongle:node2:battery-level" }
Switch vDad_Heartbeat        (PubItems)

NOTE: These Items are on a remote OH instance so there is no need for categories or labels on these Items.

The generated JSON is as follows:

[
    {
        "type": "Group",
        "groupType": "String",
        "name": "PubItems"
    },
    {
        "type": "Switch",
        "name": "vDad_LR_Motion",
        "groupNames": [
            "PubItems"
        ],
        "metadata": [
            {
                "key": "channel",
                "value": "zwave:device:dongle:node2:sensor_binary"
            }
        ]
    },
    {
        "type": "Number",
        "name": "vDad_LR_Lum",
        "groupNames": [
            "PubItems"
        ],
        "metadata": [
            {
                "key": "channel",
                "value": "zwave:device:dongle:node2:sensor_luminance"
            }
        ]
    },
    {
        "type": "Switch",
        "name": "vDad_LR_Alarm",
        "groupNames": [
            "PubItems"
        ],
        "metadata": [
            {
                "key": "channel",
                "value": "zwave:device:dongle:node2:alarm_burglar"
            }
        ]
    },
    {
        "type": "Number",
        "name": "vDad_LR_Sensor_Battery",
        "groupNames": [
            "PubItems"
        ],
        "metadata": [
            {
                "key": "channel",
                "value": "zwave:device:dongle:node2:battery-level"
            }
        ]
    },
    {
        "type": "Switch",
        "name": "vDad_Heartbeat",
        "groupNames": [
            "PubItems"
        ]
    }
]

The channel links from the .items file is being picked up and added to the Item as metadata.

I PUT this to the rest/items end point and got a good response saying that all the Items were created. So far so good.

But when I look in PaperUI, the Items are no longer Linked to Channels. When I look in jsondb, the org.eclipse.smarthome.core.thing.link.ItemChannelLink.json is empty

{}

So this is already really far along in making it easy to move .items files to JSONDB. But it would be complete if it also were able to reestablish the Links, or generate some JSON I can put to the rest/links/{itemName}/{channelUID} end point.

Unfortunately, it looks like that end point doesn’t support a JSON array so we are still stuck updating the Items one-by-one. :frowning: As the tool matures, it would be awesome if there were a button at the bottom to “make it so” and the tool makes the REST calls for you to:

  • create the Items
  • recreate the Links
  • does any other endpoint stuff needed like resetting up the Profiles if necessary

This raises another question. How are Profiles handled? I don’t actually use any Profiles but I wonder if posting them as metadata will work or if that get’s handled differently like the channel does.

Finally, one gotcha that may crop up happens when people create Items in .items files but establish the Links in PaperUI. In this case deleting the .items file removes the Item but doesn’t remove the Link so OH ends up in this confused state where it thinks the Item still exists. This may break the ability to create/update the Things using the REST API for those Items until the Links are removed. So it is important to remove the Links first before removing the Items in .items files and then recreate the Items using the generated JSON. And for now, recreate the Links.

13

How would that happen? I thought OH read the text files into the JSONDb anyway. Deleting the text should clean up all the associated parts too
A typographical error in that same .items file deletes ALL those items from the JSONDb so that could remove links that are not restored after a typo is corrected


14

No, it doesn’t. It reads the JSONDB into memory. Then it reads the .items files into memory. .items files never get saved to the JSONDB. If they did there would be no need for Yannick’s tool in the first place. It provides a semiautomated way to move your Items from .items files to JSONDB.

One of the big problems with .items files et al is that it means that OH has two independent sources for truth. And that can cause problems like the one I describe.

Not if you created the Items in .items files but Linked them to Channels in PaperUI or the REST API. In that case the Link is separately stored and maintained in the JSONDB from the Items which is stored and maintained in the .items file. And OH isn’t smart enough to “clean up” the links when you remove the Item from the .items file.

There is no problem when you maintain the Link in the .items file (i.e. channel=“blah:blah:blah”). But when you establish the Link in PaperUI instead, you end up with two sources of truth for that one Item. The Item itself is defined in the .items file and the Link is defined in JSONDB.

No, those Items never see the JSONDB. The typographical error prevents OH from loading them into memory, that is all. But it wouldn’t prevent OH from loading the Link that exists in JSONDB into memory resulting in a zombie Item.

It’s stuff like this that is driving users like Yannick and myself to get configs out of the .items et al files and into one single source of truth.

1 Like
15

This was the intent :slight_smile: At this stage it is merely a way to try out the items syntax grammar in Nearley and display the parsing results into a JSON representation, but it should evolve into something than can import those results to the JSONDB with the push of a button, recreating the links, metadata and anything related.
The thing is, currently it would involve potentially many requests to the API to recreate the links, it would be IMO better, of course, if the endpoint accepting the list of items could handle those metadata arrays and do the right thing in the backend. But a client-only implementation with no API changes involved is also possible.

The profiles are configured at the link level so they can be defined when the link is created (currently PUT /links/{itemName}/{channelUID}.

You’re right, it’s kind of frustrating, same thing about items happily referencing a parent group even if it doesn’t exist. Again some checks can be put in place in the UI but it’s probably better to treat it as a bug - just delete the links when the item is deleted.

That’s the intended goal of this initiative: making it easier for users who want a single source for truth, and still be able to copy-paste snippets of items syntax they encounter here or in the docs. That syntax has a more-or-less 10-year legacy, so in all honesty I’m very doubtful it can be replaced easily. New users especially will get confused if the examples they see everywhere are not relevant anymore.
It could be complimented however by a newer & better supported syntax like YAML.

As I mentioned elsewhere, what could be very beneficial in the long term is to move from watching for file changes to a Kubernetes-styles declarative object management approach, meaning you would be able to say with an API client, like a command-line tool: “apply this bunch of folders and files to this openHAB instance” - modifying objects which exist and creating those which don’t exist (object deletions i.e. in the DB but not in the files anymore however would still have to be done by hand). You could choose the existing syntax or a new one if you wish, not only for items, but also things, persistence strategies and so on. That way you can keep the benefits of having your configuration in source files (backups, version control
) but get all your objects in one place.

4 Likes
16

As a matter of fact I set up a temporary Jenkins job to check if it could sucessfully build the UI: https://ci.openhab.org/job/sandbox-openhab-ui/lastSuccessfulBuild/
Temporary as in, it will not stay there after the code is moved to the openHAB organization.

You can download the “www.zip” build artifact and extract it in html then go to http://‘openhab’:8080/static/www/.
Just so we’re clear, those builds are still alpha level so don’t bother expecting things working outside the settings section. Even there, some functionality related to the inbox and discovery is still to be implemented. You can try building a semantic model from scratch with the model editor, and defining new NGRE rules. If you have a proper semantic model you will hopefully be able to try the other tabs on the home page. But all the end-user screens are mockups or read only. You’ll also find this tool in the developer tools section and change the theme in the about screen.
In all honesty, If you encounter a bug, that’s pretty much to be expected - a look at the browser’s console is always handy. And as for feature requests, frankly I have a lot on my plate already, you can suggest but no guarantees I’ll address them. I don’t work on this full time :slight_smile:

3 Likes
17

Understood. I am not a Java programmer but perhaps i can assist with tidbits here & there.

I may be interested in looking at developing a Z-Wave network viewer similar to the one in HABmin. I have one from another project that looks promising. I just need to develop & “plug in” to our API, hopefully.

18

This is something I would like very much. That way, I could have a Jenkins job that creates all the JSON stuff, and just restarts the docker image with those files mounted somewhere


But on the other hand, you would be basically spinning up a nearly complete OpenHAB instance without starting the server. I am not fully sure but I think it would also need all the bindings so it can ask them about their specific item syntax, no?

19

You would need the API server to be up to do that indeed . But for that particular use case I think you can directly edit the JSON DB files, if you do it right since you’re offline it could work reliably.

For 2.x bindings the syntax is basically well-defined, most of the configuration is on the Things and you simply link channels to items eventually with a profile. For 1.x bindings, I’m not sure how hard it would be to move away from the current approach.

20

I completely agree. I strongly suspect that as more and more advanced users move in this direction, it will be more than just UIs that heavily use the REST API. Providing support for bulk updates will be useful beyond this tool.

I’m totally fine with letting the user’s handle it manually. But we need to make sure the docs are clear that they do need to handle this case. It has been awhile since the last time I had to help someone deal with zombie Items caused by the Links hanging around, but suspect that there will be a huge number of new users who need help as people discover this tool.

I used the tool to move the Items on my remote OH instance and even as it exists now it saved me a ton of time. I’ll need to wait for the support for Links before I try to apply it to my main OH instance. But then I’ll be ready to be one of your first testers. :wink: