Migrating .items / .things files to UI - My Plan - Feedback Wanted

Setup, Configuration and Use Migration
, ,
1

Hi all,

What I have:
I am using OpenHAB 3.2.0 on a QNAP NAS in the Container Station (i.e. docker), migrated more than a year ago from a 2.5.x instance. I have all my configuration in text files (.things, .items, .rules, etc.). For some Items defined in the text files, I have added Metadata in the UI. There are around 90 Things from various bindings (Gardena, Miele, Homematic, Modbus, Daikin, iCalendar, HTTP, etc.) and around 680 Items in around 13 different text files each (~13 for Things, ~14 for Items). I created the semantic model for locations in a separate .items file (i.e. my house, floors, rooms hierarchy) and added semantic tags to a lot of items in the .items file.

What I want to do:
Now I want to make the step from the text file based configuration to the UI based configuration. The first phase would be to migrate Things and Items into the UI. Once done, I’ll think about how to bring all my rules in (I have lots of them, and some are quite complex). So, first focus is on Items and Things. I’ve read a number of threads here but haven’t found an “universal guide to migrate text files to UI”.

I came up with the following plan on paper and would love to get your thoughts about it, and any issues that you would anticipate with it. Or any ideas for a much better plan.

Basic thinking behind my plan is that I have a not-so-small OH installation that I really want to seemlessly continue to work, and I do not have the bandwidth to spend 30 hours in 3 days in a row to brute-force re-install everything manually in the UI. So I need an approach where I can migrate the Things and Items step by step, if needed over weeks and months, without impacting the existing installation and data collection, until I completed the process and all .items / .things files are gone.

Here my plan:

1. First Migrate all Things
I would start with migrating Things step-by-step until I have all of them UI based. Once finished with all Things, I would then start migrating Items. Any concerns with this sequence?

How to migrate Things? Here an idea, not sure if it works:
I take the first Thing, open it in the UI, go to the “Code” tab, copy the code to a text editor. Then I go to the corresponding .things file and remove the Thing from the file. It will disappear from the list but should appear in the Inbox (right?). I add it from the Inbox with “Add as a Thing”. I open the new Thing in the list, go to the “Code” tab, and replace the code with what I copied before. Now the thing should be configured correctly and identical to the previous Thing from the text file and it is my first UI-configured Thing. Then, rinse and repeat with all other Things.

Does this make sense? Any concerns? Any more efficient way to do it? Again, it is important to me not to have any interruptions for my OpenHAB setup (besides the 2-3 minutes that it takes to re-discover and config each Thing again as described). Any impact for the Items related to the Thing Channels, etc (besided not receiving / sending any data for those 2-3 minutes)?

2. Build Semantic Model for House
Once all Things are in the UI, the next step would be to load the Semantic Model for locations (house, floors, rooms) that I have in a separate .items file. I would use the “Add Items from Textual Definition” functionality to copy the content of that .items file and load it. I would do this while the .items file still exists. Once it is added in the UI, I would rename the corresponding .items file to e.g. .old.

Why do I want to do it that way? Because I hope that this way OpenHAB never really loses the location semantics (I think the text file definition overrides any UI definition). I am not sure what happens to the 680 Items, that are often tied to a room, when the room definitions all disappear for a moment. I’m afraid OpenHAB then suddenly ignores all 680 Items because they use an invalid group (e.g. gLivingRoom tagged with [“LivingRoom”]). And some seconds later all location semantics comes back … and what happens to the Items? Are they reloaded? Or do I need to touch every singe .items file so it is reloaded? Will OpenHAB manage the shock of losing all 680 Items in case this happens or am I risking unstability or strange behaviour (often it’s funny things like these that start corrupting your setup)? Any experience with this? Any better way to do it?

3. Load Items File by File
Last step of “Phase 1” would be to transfer Items step by step via “Add Items from Textual Definition” feature. I would first start with single items, just to get the feel for it and to explore any issues. Then probably continue with groups of Items in a file, until the whole file has been loaded successfully. Then, repeat for the next .items file etc. until all are UI based and the .items files are empty.

Is it better to load the Items via “Add Items from Textual Definition” while the .items file still exists, and then rename the file to .old, or is it better to first remove the item from the file (or rename the whole file) and only then copy the definition into the “Add Items from Textual Definition” feature?

As far as I could read in a number of threads, the “Add Items from Textual Definition” has its own challenges.

  • Does the issue with patterns still exist? If yes, I’d probably load the Items as they are and then manually correct in the UI where I used patterns
  • Will I be able to keep the existing UI-defined Metadata of the Item when I remove it from the .items file (and thus it disappears shortly from the UI) and then load it via “Add Items from Textual Definition”? Will the UI-entered Metadata appear again for the “imported” Item?
  • Are the Semantic Tags imported correctly through this functionality?
  • Any other known limitations of this functionality?

Any feedback, ideas, concerns are very welcomed. Would be great if we could work out together a little “Guide to seamlessly migrate text file based Things and Items into the UI”.

2

I don’t think it will work exactly like this.

What I recommend is to pick one binding first and remove all of those things from the .things files. Pay attention to the IDs of the Bridge Things (e.g. MQTT Broker Thing, Zwave Controller Thing, etc.).

Next go to the Inbox, select the binding, and if the binding has a Bridge Thing, you’ll need to manually create that. Here is where that ID comes in because if you make the ID the same as it was in the text file, that Thing and all the Things that fall under that Thing will end up with the same IDs which will save some work later as you won’t need to relink your Items.

Once you have the bridge thing, return to the Inbox, select the binding, and now run a scan. If all is well all the devices for that add-on will be discovered and you can accept them from the inbox.

If you forgot or were unable to use the same ID for the Bridge Thing, you’ll need to go to your Items and update the Channel Links with the new IDs.

That won’t work. You’ll need to rename the .items file first or else it will complain that those Items already exist.

You won’t lose anything. The semantic model is implemented using Groups, Group membership, and tags. Nothing will explode if the Groups disappear for a small period of time. the Items will still be there and work as always.

Same as above, you’ll need to remove the Item or the whole .items file first or else you won’t be able to create the Items in the UI.

You can load the whole file in one go though using “Add Items from Textual Definition”.

Probably because there is a fundamental difference in how managed Items handle state display (instead of being defined on the Item’s label, it’s defined as State Description metadata on the Item). You could theoretically add that State Description metadata before importing the Items from the text definitions. It’d look something like { channelid="blah:blah:blah", stateDescription=""["pattern"="%.1 °F] }.

It’ll probably be easier to do it that way than to do it after the fact. The “pattern” field of the stateDescrption metadata takes the same thing you’d put between the in a text Item definition’s label.

Yes. Recently I even argued on a GitHub issue to make sure this remains the case. Item metadata is stored separately from the Item itself so it’ll still be there when the Item is recreated.

It’s just Group membership and Item tags. Both have been around forever. There will be no problems there.

If you have Items that are not yet part of the semantic model (and don’t feel like all of your Items should be part of the model), I find it to be much easier to recreate the Items using “Create Equipment from Thing” as that will create the Equipment Group with semantic tag, position it in the Location, and if you set the semantic tags for all the Point Items, and create the Links to the Channels all in one go.

This could be less work than trying to apply the tags and update the Group membership by hand which is pretty cumbersome in MainUI.

If you are worried about uptime of your home automation, I recommend doing this migration one of two ways:

  1. Create a separate instance of OH on another machine (or on the same machine using different ports) and do most of your work there. You will be limited by hardware access (only one instance can access USB devices at a time) but you can get a good long way down the road before you ever have to touch your old instance.

  2. Use periods of processing. Shut down the old instance and bring up a new instance while you are working. Then switch back when you are done for the day. You’ll have brief periods of down time but won’t break everything until you are done with the switch over.

I also recommend instead of going “breadth first” consider going depth first. Migrate all the Things, Items, and Rules(?) for one binding. Then move on to the next and so on.This will give you the end-to-end experience which might inform how you do later bindings. For example, it would be a pain to get to transitioning your rules only to discover that if you had only done X differently with your Items it would be much easier. That way you won’t have to go back and change all your Items.

If you have a number of similarly but manually configured Things (e.g. Generic MQTT Things, HTTP Things, KNX, Modbus, etc.) make use of the code tab or the API Explorer (under Developer Tools).

  1. Manually configure the one Thing and test that it works.

  2. Copy the code
    a. if using MainUI open the code tab and copy the YAML
    b. if using the API Explorer, query for the Thing and copy the JSON

  3. Paste, edit, submit
    a. if using MainUI, manually create a new Thing of the same type, fill in the “metadata” (Name, ID, etc.), open the code tab, paste the YAML and modify it as necessary. Save.
    b. go to the add thing end point, paste the JSON, edit, and submit. This way is a little faster because you can edit/submit without the extra steps of manually creating the Thing and filling in the metadata, you just need to edit the JSON in place and submit. But JSON is harder to deal with.

2 Likes
3

Thank you so much for sharing your experience and ideas also on this topic, Rich, it is very much appreciated.

About the Things:
Hmm, I was afraid you’re going to write something like this :sweat_smile: . I saw your input in another thread about the Things. “Emotionally” I did not feel comfortable with it, although I understand the reasoning and it makes sense, and still I was trying to think about a different approach. But I agree that especially with Things that have a bridge that does discovery, etc., your approach seems to be the one to follow. I have a number of Things where this should be feasible, also with my self-applied constraints of uptime, and I will start with those accordingly (and will also transfer then the related Items as well, as you recommended, to learn from the process).

The two most challenging and critical parts, however, are two devices polled via Modbus. For each device, there is a Modbus TCP bridge Thing, within that Thing there is one or multiple Bridge Poller Things, and within those are many Things for each Bridge Poller that connect individual data point each (around 30 per device, i.e. per Modbus TCP Bridge).

My understanding is that there is no discovery for the Modbus binding, it’s all manual configuration. The working approach I see here would be the bottom-up approach, and I think it is basically what you described in your posting as well (I’d be using the MainUI, not the API Explorer). Means, first transfer manually each individual data point Thing within the same Bridge Poller Thing into the UI. Means, copy the existing “code” from the UI to an editor, remove the individual data point Thing from the .things file, then manually create it in the UI and copy back the “code” section in the UI. Then I should have that data point working again. Then, repeat for each individual data point Thing for the same Bridge Poller.

Once I transferred all data points within a Bridge Poller, I can transfer this Bridge Poller Thing in the same way to the UI. Once done, I repeat the whole process for all other Bridge Pollers within the same Modbus TCP Bridge Thing. Once I have transferred all Bridge Poller Things, the only Thing left in the .things file would be the Modbus TCP Bridge Thing for that device. As a final step, I would bring that Modbus TCP Bridge into the UI in the same way, and the .things file is empty. Then repeat the exercise for the other device.

If this approach works, then I could avoid any significant downtime and would hardly lose any data that is polled from these two Modbus devices. Would that approach work?

About the Items:
Thank you for the hint that I first need to remove / rename the .items file before using the “Add Items from Textual Definition” feature. And great to hear that nothing should “explode” when location sematics / groups, etc. disappear for a moment. Fantastic to learn also that I will not lose any UI-defined Metadata for Items, this is a relief :slight_smile: . And great to hear that the Semantic Tags are imported correctly, very cool.

For the patterns, ok this helps, thank you, I will try and see what will work best for me, but I understand my options now. It’s not too critical as it’s mostly cosmetic.

Last but not least, thank you for your thoughts around having two instances. I will not do this in my situation because for me there’s a too high risk that I’m messing it up. There are too many dependencies to other services, etc. where I do not feel that I have sufficient understanding in analyzing and mitigating any risks that come with it to be comfortable with the approach. I’d rather accept a bit more downtime for some parts vs. handling two instances on the NAS server. If you feel that my approach for the Things and Items, described in this posting based on your input, would generally make sense and work, then I would feel comfortable with starting the process step by step, and could take my time until everything is in the UI.

4

I took the step and tested the process for two bindings with less critical components (Miele Cloud binding, Homematic binding). Where necessary (Homematic) I copied the “code” section of the Things first into a text file, then renamed the .things file, so all related Things disappeared. Manually registered the Homematic Bridge in the UI, then copied back the “code” section of the bridge, and it was online again. By that, it discovered the other Things, which I added in the UI one by one, also copying back the “code” section for each, and all were online again as well.

Then I simply renamed the .items file and loaded the content through “Add Items from Textual Definition”. Everything worked again immediately and had all semantic tags and Metadata. So it was a fast and smooth process, you guys did a really good job with the “Add Items from Textual Definition” feature, I like it!

I also already started to appreciate the advantages of having things and items in the UI, when I added another Point. Really cool and easy!

So the last part where I’d appreciate your expertise is the one for the Modbus devices, as described in the previous posting of this morning. If you think my approach will work, then I’m good to go with those as well.

5

It may make sense to keep those in .things files.

I think you are better off avoiding down time by doing the migration on a separate instance of OH or using periods of processing rather than trying to do it in place on your production instance. Since the modbus is connecting through TCP you should be able to have two instances of OH connected at the same time.

When you are ready to switch over, you can then activate persistence on the new one for external and copy over the persistence files from the old to the new for internal databases to get that nearly unbroken set of historic data.

Beyond that, I’m actually not certain what will happen to your Things if you recreate the Bridge last. That might very well wipe out all the sub-Things. If it doesn’t wipe them out, you might have to go in and change each of their settings to use the new Bridge.

The registries in OH like the Things registry have a lot of features that try to keep the registry clean without orphans and stuff that doesn’t make sense (e.g. a Thing without it’s required bridge) so you’ll need to experiment with that to see what happens.

If the Thing was discovered, what are you “copying back”?

1 Like
6

Thank you very much for your additional thoughts!
Oh, very good point, recreating the modbus bridge last might indeed clean out the rest belonging to that bridge. Hmm, I may create a small test bridge with a poller and some data points in there to verify how it behaves. If it does anything I don’t want, I may need to keep the modbus Things in the .things file, indeed.

Regarding the “copying back” the “code” for Things, I can give a specific example. For the Homematic Bridge, I first went to the “code” page and copied it into a text file. It has all the necessary configuration for the Thing:

UID: homematic:bridge:ccu
label: Homematic Bridge
thingTypeUID: homematic:bridge
configuration:
  binCallbackPort: 9126
  xmlCallbackPort: 9125
  cuxdPort: 8701
  socketMaxAlive: 900
  installModeDuration: 60
  callbackRegTimeout: 120
  timeout: 15
  hmIpPort: 2010
  factoryResetOnDeletion: false
  wiredPort: 2000
  discoveryTimeToLive: -1
  gatewayType: ccu
  callbackHost: <IP1 ...>
  groupPort: 9292
  gatewayAddress: <IP2 ...>
  unpairOnDeletion: false
  rfPort: 2001
  bufferSize: 2048

Then I renamed the .things file. Next, I manually added the Bridge Thing again in the UI, with the default name proposed by OpenHAB. Of course, needed to set the same ID (“ccu”). Once the Thing was created, I opened the Thing in the UI, went to the “code” page and replaced the existing lines with the ones I copied above. This is much faster than manually trying to set all Ports, IPs, etc. in the corresponding fields manually, plus it avoids mistakes that may cost you lots of troubleshooting time. Worked smoothly.

7

OK, so you are talking about the Bridge Thing. It sounded like you were scanning and automatically discovering Things and then overriding that with something you’ve copied and pasted from elsewhere which is unnecessary. Everything should be there in the discovery already.

8

Ah, sorry for the confusion. No I’m not overriding anything that was discovered, I’m just re-applying the configuration of the discovered Thing, which goes easier for me by copy&pasting the “Code” lines instead of manually entering all the values into the individual fields.

Thank you again very much for your extensive help, Rich, I think I have now everything (for the moment :wink: ) to walk through my journey of bringing the text files into the UI. I may get back to the community latest when I tackle the .rules files … but first I’ll focus on Things and Items.

9

To migrate your Things, I would perhaps plug the little script that I wrote and shared: Ohadm - A simple script to manipulate openHAB objects and sync them to YAML files

If you get it working and do a ohadm fetch things you will end up with all your things as separate YAML files in a subfolder.
Then you can update them with ohadm update thing <thingUID> but you will have to remove the non-managed (i.e. defined in a .things file) first, or rename the file and change the UID part to duplicate it.

For Items, I would suggest to first comment out your Locations tree in your .items file, then use the Add from Textual Definition feature in the UI using the contents of that very file to re-add the items as managed items. The Locations tree is probably the easiest to migrate because it doesn’t involve any links.

For the rest of the Items it’s perhaps best to redefine them from things with the UI.

3 Likes
10

I forgot about your script! I need to remember that.

11

Cool, this is a great script, thank you very much for pointing me to it. I’ll check if I can get it working.

At the moment, I’m progressing very well with my manual approach. I already migrated Things and Items of several bindings into the UI. For standard “Things” it’s quite easy to do, actually.

Meanwhile, I had an idea about how I can bring all the Modbus Things and Items into the UI. I’m building a parallel structure of my existing Modbus TCP Things and Modbus Poller Things in the UI, polling the same device, but having a different ID. Then I manually add in the UI all the Modbus Data Things related to a Poller Thing. Once done, in the Items file I change the channel for the Items related to that Poller Thing to the new Poller Thing. Once done, I can remove this Poller and the related Modbus Data Things from the .things file. Rinse and repeat for all other Pollers. This allows a seamless step-by-step migration.

I already did it for the first Poller Thing and it worked. It’s a lot of manual work and it will take some time, but it is seamless and I have control of every step.

12

Quick feedback:
I managed to transfer all my Things and Items into the MainUI, including the two Modbus devices. My items and things folder under conf is completely empty now.

The non-Modbus devices were quite easy. For the Modbus devices migration, my idea mentioned in the previous message worked perfectly. I had less than one minute of data collection downtime in total. It was a bit of a boring click-and-copy/paste session for around 3-4 hours in total for both Modbus devices but hey, I only do that once, so no problem. Really happy that I did it. Next steps would be to tackle the Rules, although I may not want to migrate all rules, I guess (having 64 rules, some of them quite long and complex).

Thanks again Rich, Yannick, for all your help, wouldn’t have done it without your support.