Migrating Thing.json from OH2.5 to OH4.1

Matej_Kotnik · May 29, 2024, 11:32pm

Hi

I have ran into remote access troublewith OH2, and now i decided to stage the OH4 setup and cant cleanly migrate yet to OH4.

I have had OH 2.5 for legacy devices for many years. now it looks like, openhab cloud doesn’t support OH 2.5 so much it is offline all the time, it reconects on restart, it likes to go offline and stay so, but it was somewhat reliable before that cloud upgrade on may 24, needed monthly check the save button before also tho.
What could be wrong?

So i tried out new OH4.1 on debian 12. I am familiar with it from my remote property where i run OH4 on Pi 4 already, i was smart there to only use config files, and it is running fine there, remote access and all.

Here at home i have can of worms as the system is older, and i learned on UI and than i have some Things in UI and others in config files, other items and stuff i have already in config files.

The Luxtronik binding changed completely, and that Thing.json file changed with new OH.

I have can of worms with the Thing.json file org.eclipse.smarthome.core.thing.Thing.json (261.9 KB) that holds all the things from paper UI, it is stupid large like 10k lines. It effectively will not work on OH4 and i don’t know how to correct its syntax or sth to be inline with OH4.

I know that new OH has different filename but contents seem slightly of so they don’t work, and i cant figure out what off.

I tried to transfer some things with ChatGPT to mqtt.things and they work fine, but it is so humongous that chat GPT cries in between it literally crashes the webpage, i use 4.0 version and some SoftwareArchitect GPT, it was brilliant until half way where it started acting up.

It is like 40 devices that need to be transferred, loads of channels.

I have option sync items from OH 2.5 to 4.1, i may settle with this option, but OH should support upgrade path without those compatibility issues on essential configs really.

I tried the remote openhab binding, and i get it to online for bridge and the thing that i wrote in also comes online, but i cant get any channels to appear on OH4 under that remote thing, even if i update channels value or move a switch, as it says it detects only command channels.

Seems like all of OH2 of channels appeared in new OH4 json org.openhab.core.thing.Thing.json (872.7 KB) over the remote binding but non are visible over UI even when thing like mqtt:topic:vtic5 is online, has nochanels visible in OH4 UI.

Recomendations appreciated.

Thanks
Matej

rlkoshak · May 30, 2024, 3:48am

That specific binding says in the docs that it discovery. So why not just run a scan from Settings → Things → + and accept the discovered Things?

You’d probably need to upgrade 2.5 to 3.0, then 3.1 and so on so that all the breaking changes to the managed Things get applied at each step. You might be able to take bigger steps (e.g. 3.4 to 4.2) but it all depends on what changed in the bindings you used. During the upgrade process often there are scripts and processes that will modify your managed configs to adjust for breaking changes.

But sometimes the bindings change so drastically that they require recreation of the Things. When Things can be discovered, that’s the best way to recreate them. It’s almost always going to be less work to just discover those Things anew on the new instance. When this is the case it will be covered in the list of breaking changes in the release notes of each major release. But because you are jumping so far you’ll have to review the release notes for all of them between 3.0 and 4.1.

Of course, unmanaged things in .things files will need to be manually updated. But reviewing the docs for the individual bindings should tell you what to change. The changes will usually be relatively minor with changed names of some parameters, removal of some, new parameters, stuff like that.

It does.

Managed Things usually get modified during the upgrade process. But the upgrade process only occurs during an upgrade. You can’t just copy the files from an old version to a new one.
In rare circumstances the binding will change a whole lot requiring recreation of the Things. But every case of this I know of over the years has been with bindings that support automatic discover of Things so it’s just a matter of deleting and rediscovery of the Things. If you are careful with the Thing UIs and make them the same as the old ones you don’t even need to relink the Items.
If you chose unmanaged Things (i.e. .things files), well you made the choice to manage the Things yourself. So it’s up to you to modify them manually when there are breaking changes. You can’t have your cake and eat it too. There really hasn’t been any changes to the .things file formats between OH 2.5 and 4.2 so any breaking changes there are going to have been implemented by the binding authors.
As OH continues to mature, upgrades are handled better and better. But hardly any software will handle seamless upgrades from two full release versions back. OH 2.5 was released four years ago. That’s a lot of development to account for.

My recommendations are as follows:

For those bindings that support discovery, simply rediscover the Things. When creating the Bridge (where required) or accepting the Things, take care to set the Thing UID to be the same as the old Things and your Items will work withou needing to be relinked,
If you have any bindings that do not support discovery, if you don’t have many Things to deal with, I’d simply manually recreate them. If you do have a lot of Things to manually create, then consider doing an in place upgrade of OH through the major releases (i.e. 2.5 to 3.4 to 4.1). That will apply any automated upgrades to your managed Things.
For Things in .things files you’ll either have to recreate them as managed Things through rediscovery or manual creation through the UI or do the upgrades/modifications yourself manually. That’s kind of what it means for them to be unmanaged.

Matej_Kotnik · May 30, 2024, 11:34am

Hi Rich, Thank You

My main portion of devices are generic MQTT. If i understand right, they wouldn’t be upgraded in json as they ar not managed by the binding.

I spoted some diferences in json, i than changed class but i guess mqtt:topic:MQTT_Bridge:ad36aea45c vs mqtt:topic:razdelilec_4vt_1 is mising bridge part, i fixed both.

Not sure what else is also important, that i could use multirename tool to substitute.

Seems like OH is ignoring tje json file even on restart. i see thet error

024-05-30 13:52:41.058 [ERROR] [re.storage.json.internal.JsonStorage] - Couldn't deserialize value 'org.openhab.core.storage.json.internal.StorageEntry@5ce83fe3'. Root cause is: java.lang.IllegalStateException: Expected STRING but was BEGIN_OBJECT at path $.bridgeUID
2024-05-30 13:52:41.059 [ERROR] [re.storage.json.internal.JsonStorage] - Couldn't deserialize value 'org.openhab.core.storage.json.internal.StorageEntry@6746da91'. Root cause is: java.lang.IllegalStateException: Expected STRING but was BEGIN_OBJECT at path $.bridgeUID
2024-05-30 13:52:41.060 [ERROR] [re.storage.json.internal.JsonStorage] - Couldn't deserialize value 'org.openhab.core.storage.json.internal.StorageEntry@714b2bb9'. Root cause is: java.lang.IllegalStateException: Expected STRING but was BEGIN_OBJECT at path $.bridgeUID
2024-05-30 13:52:41.061 [ERROR] [re.storage.json.internal.JsonStorage] - Couldn't deserialize value 'org.openhab.core.storage.json.internal.StorageEntry@469325fa'. Root cause is: java.lang.IllegalStateException: Expected STRING but was BEGIN_OBJECT at path $.bridgeUID
2024-05-30 13:52:41.062 [ERROR] [re.storage.json.internal.JsonStorage] - Couldn't deserialize value 'org.openhab.core.storage.json.internal.StorageEntry@5b990dec'. Root cause is: java.lang.IllegalStateException: Expected STRING but was BEGIN_OBJECT at path $.bridgeUID
2024-05-30 13:52:41.063 [ERROR] [re.storage.json.internal.JsonStorage] - Couldn't deserialize value 'org.openhab.core.storage.json.internal.StorageEntry@adad65c'. Root cause is: java.lang.IllegalStateException: Expected STRING but was BEGIN_OBJECT at path $.bridgeUID
2024-05-30 13:52:41.064 [ERROR] [re.storage.json.internal.JsonStorage] - Couldn't deserialize value 'org.openhab.core.storage.json.internal.StorageEntry@22e4225d'. Root cause is: java.lang.IllegalStateException: Expected STRING but was BEGIN_OBJECT at path $.bridgeUID

The bridge class is diferent but i need to findout what is it.

I considered moving all mqtt things to mqtt.things config file as said before, but it is a lot of work.

I may try to upgrade a clone of OH2 i have in VM to see how other things in josn upgrade but i guess mqtt needs fixing manually right?

I tried to find any obvious anouncements of changes but nothinhg is directlly ponting to json fixes nedded.

Here is the comparison

The new json MQTT device

  "mqtt:topic:MQTT_Bridge:ad36aea45c": {
    "class": "org.openhab.core.thing.internal.ThingStorageEntity",
    "value": {
      "isBridge": false,
      "channels": [],
      "label": "testni",
      "bridgeUID": "mqtt:broker:MQTT_Bridge",
      "configuration": {},
      "properties": {},
      "UID": "mqtt:topic:MQTT_Bridge:ad36aea45c",
      "thingTypeUID": "mqtt:topic"
    }
  },

The sinpet of old json MQTT device

 "mqtt:topic:razdelilec_4vt_1": {
    "class": "org.eclipse.smarthome.core.thing.internal.ThingImpl",
    "value": {
      "label": "Razdelilec 4VT - 1",
      "bridgeUID": {
        "segments": [
          "mqtt",
          "broker",
          "mosquitto_broker"
        ]
      },
      "channels": [
        {
          "acceptedItemType": "String",
          "kind": "STATE",
          "uid": {
            "segments": [
              "mqtt",
              "topic",
              "razdelilec_4vt_1",
              "state"
            ]
          },
          "channelTypeUID": {
            "segments": [
              "mqtt",
              "string"
            ]
          },
          "label": "State JSON",
          "configuration": {
            "properties": {
              "stateTopic": "Vticnice/tele/Razdelilec_1/STATE"
            }
          },
          "properties": {},
          "defaultTags": []
        },
        {
          "acceptedItemType": "Switch",
          "kind": "STATE",
          "uid": {
            "segments": [
              "mqtt",
              "topic",
              "razdelilec_4vt_1",
              "power0"
            ]
          },
          "channelTypeUID": {
            "segments": [
              "mqtt",
              "switch"
            ]
          },

Cheers Matej

rlkoshak · May 30, 2024, 2:35pm

If they are managed (i.e. created through the UI) then they would be upgraded. If they are in .things files, they would not be upgraded.

You really are unlikely to be successful manually editing the JSONDB files to upgrade manually. It will be less work and more likely to lead to success to do in place upgrades as described above so that the upgrade scripts can make the adjustments for you.

Beyond that you can try to look at an OH 4 Generic MQTT Thing’s JSON compared to an OH 2.5 Generic MQTT Thing’s JSON and look at the differences.

Do not edit the JSONDB while OH is running. OH doesn’t watch this file, only loads it during startup, and it will overwrite your changes the next time it needs to write to the file.

Yep, it’s really easy to mess up the JSONDB file. That’s why manually editing it is not recommended.

At this point you need to fix what ever you broke. But if you have your 2.5 configs on a 2.5 instance, then upgrade to 3.4 your MQTT Things will be upgraded for you. There’s nothing you need to do. Then upgrade to 4.1 and the Things will again be upgraded for you. When you are working with managed Things, except in certain rare circumstances, the upgrade process will fix your managed Things automatically. But just dropping a 2.5 config on a 4.1 instance of OH bypasses those upgrade processes.

I’m not sure what I’m looking at here, but the new JSON looks somewhat reasonable. Something’s wrong with it though given the error. Or maybe there’s something wrong with the bridge.

Matej_Kotnik · May 30, 2024, 2:40pm

That sounds great, then i will proceed with upgrading the old OH in spare VM.

Matej_Kotnik · June 3, 2024, 11:46pm

Hey Rich, Thanks

I have great news, 4.1 with config of 2.5 is cruising along in VM.

Amazing guidance for migrationfrom from 2.5 to 3.4 and than 4.1.

The fresh openhab with old configs was a potato mine field, endless issues, influx db was also messy, default port 8086 was busy. It refused to keep config file, the documentation is hard to folow, to set it permanent port. sudo nano /etc/influxdb/conf.toml http-bind-address = ":8087" Was not permanent.

So i opted for inplace upgrade of OH and OS with LTS in clone VM, and am happy that most of things work.
Regarding JSON migration, upgrade scripts did the magic

Only here binding changes it needs fixing of items or rules, i dont like that bindings break old functionality, mainly Luxtronik.

Now Choosing where to put new OH4?

For test i thrown VM on my workstation.

But the main OH server is prodesk with baremetal Pepermint install, i picked backup and made VM from it.

I want to run OH4 alongside OH2 for few months to get seteled in, where to run it 24/7?

The VM is on vmdk so VMware or Vbox compatible.

What supervisor is good and compatible?
And what host i choose for that upgraded OH4, same prodesk or new mini PC?

It runs cpu quite hard like 25-40% on 4 cores, i5 4590T (4000 pasmark) or 20-30% 4 vcores on R5 2400g workstation.
It transcodes video of 4 cams, lots of db pooling, 260 rules, 80 things, 1920 items.
Its kind of crazy amount of items and rules, pending a cleanup of ingest rules.

On prodesk I have Vbox in pepermin linux, need to learn auto start tho,maybe load it there, but than prodesk may sound like a racket?

I ve heard of proxmox, and truenas also has hypervisor, need dedicated hardware for homelab to experiment there.

Or is it simpler to go baremetal
N100 mini pcs are interesting option for host w 5600 pasmark, is there enough umph?

What is recomended for headless/remote managed mini server.

Cheers, Matej

rlkoshak · June 4, 2024, 1:55pm

For the most part that’s going to be based on what you know and what your preferences are. I used to use VMWare until they stopped supporting my hardware and moved to Proxmox. I probably would choose VMWare right now because of all the shenanigans that have happened post Boradcom takeover. But that’s all personal preference.

If it will run a standard Linux distro and it supports hardware passthrough it will work with OH. If not you need to find something that does.

Simplest is to go openHABian. If you know what containers are and how what they are good for, Docker is probably second simplest. Then bare metal.

Anything outside that is going to be driven by requirements outside of openHAB itself. We cannot answer that for you.

As for umph, a RPi 3 has plenty of CPU to run a typical OH install. However, the move to Java 17 increased the memory requirements so an RPi 4 with 4 GB RAM is kind of the new recommended minimum. RAM is the the primary limiting factor with OH which will consume around 2 GB when running 64-bit Java. If you’ve got enough RAM, a mini PC is going to be more than powerful enough, as will any of the scenarios you’ve outlined.

The official recommendation is an RPi 4 running openHABian and nothing else. But not everyone has a homelab so this recommendation may not be the best for you.

Matej_Kotnik · June 4, 2024, 6:30pm

Thnks Rich,

I really like our conversations, i admire your dedication and politeness, i feel like talking to my tech savy relative. i wonder how that noone else chimes in, do i ask so niche questions?

Yeah vmware website is acting up, download links broken, broadcom nonsens.
I dont know, Is vmware server free?

I use vmware workstation player all the time.

If my synology would be a rocket with n100 or i5 i would run vm there, it supports VMWare vmdk.

I prefer not to change OS, cause of all periferall scripts and services that are set up there.

I wish free vmware server with autostart on linux. So it would all run reliablly, but not sure where to get instalation.

This datasheet sounds too good to be true.
server_datasheet.pdf (111.9 KB)

I run some docker containers, i am new there, but i consider them homelab, not mision critical like OH may be.

I have virtualbox on linux on HP prodesk already, how can i set it to autostart? I know it can run vmdk.

I am thinking of minimal changes to make it work reliablly on prodesk,

then add another mini pc for a hypervisor like proxmox for homelab when needed.

Cheers
Matej

rlkoshak · June 4, 2024, 6:39pm

For awhile that was in doubt. I think Workstation and the bare bones hypervisor are free, at least for now.

Matej_Kotnik · June 4, 2024, 9:53pm

That was true, i even found guide.

Oh dear, looks like vmware has been ghoasted.
No downloads just contact sales, and some acount required to log in.

I will proceed with a vbox with custom service to autostart, hopefully it will work.
This is a generic autostart service i found

[Unit]
Description=Autostart VirtualBox VM
After=network.target vboxdrv.service
Requires=vboxdrv.service

[Service]
User=yourusername
Group=yourusername
ExecStart=/usr/bin/VBoxManage startvm "YourVMName" --type headless
ExecStop=/usr/bin/VBoxManage controlvm "YourVMName" acpipowerbutton
Restart=always
RestartSec=60

[Install]
WantedBy=multi-user.target

splatch · June 7, 2024, 6:58am

Hey,
Doing migration from OH 2.5 to 4+ will not work, because there were many steps in between which updated storage. Storage migration is rather basic and handles major releases.
In order to move forward you should first run OH 3, then 3.5, then 4.0. Just to let migrations implemented in these versions run. You need to copy jsondb.
With regard to channel updates, they might be migrated automatically only from OH 4 onwards.

Matej_Kotnik · June 7, 2024, 9:37pm

Hi, thanks

I did upgrade to 3.4 and than to 4.1.3 and regarding json it all went propperly, as recomended by Rich. I didn’t feel like updating main setup but first i forgot i can spin up clone and update, i onl, thought of starting clean.

The new OH4 is still not fully operational. Some odd Mqtt devices are still offline and Luxrtonik Heatpump is having many agregated items broken.

Here i have to learn some rule improvements and time formatting. So i can fix it without endless errors.

The Luxtronik binding changed, so much, didnt keep same time formatting for runtimes of heatpump devices, and status formatting is diferent. I did talk about legacy comatibility on luxtronik thread in OH3 times, but only got it so far that it talks properly to my Heatpump.

before they were strings like hhhh:mm:ss, now they are number in seconds, so anoying that one decides to change default channel values.

I want to display in hhhh:mm:ss, as it was before, like on Heatpump controller, i would like to apply formatting like hhhh:mm:ss in the items definition for items that i only display.
I cant find a working formatting notation.

I also have rules that broke as they have wrong input, they expect string hhhh:mm:ss, They than agregate runtime in 24 in hh:mm and in % and number of power cycles in a day. They also supply the data to thermostat and alert rules so i suddenly see lots of errors.

Those rules below worked fine with OH 2.5 Luxtronik binding.
Toplotne_Izgube.rules.txt (6.8 KB)
StatusTC.rules.txt (4.1 KB)
Toplotna_Preracun_Energij.rules.txt (4.2 KB)
ToplotnaCrpalka_Diagnostika.rules.txt (3.6 KB)

And they provide me that usefull control panel in OH2.5

rlkoshak · June 8, 2024, 3:05pm

If you use Number:Time Items you can use DateTime formatting state description patterns to get hh:mm:SS for display.

This change is actually a good one because you can’t do math with a String nor can you control how it’s displayed.

Here is the state description for my UPS runtime.

%1$tH:%1$tM:%1$tS

In your rules, you use them as you would any other quantity type to do comparisons and math.

Total.postUpdate(Total.state as QuantityType<Time> + Current.state as QuantityType<Time>)

if(Curr.state as QuantityType<Time> < 60 | s)

Matej_Kotnik · June 10, 2024, 2:16pm

That is a neat solution for formating.

The caveat is that heatpump returns runtime since installed so it cant get handled by that format.

Indeed the numbers format is good for math so my first solution is calculated fractions in rules, to display readable string.

Do rules support writing reusable functions or such to reduce redundant comands?
I would write conversion once and pass items to it, than it would write results to output items.
Here is example of rules to optimize.
ToplotnaCrpalka_Diagnostika.rules.txt (6.5 KB)

These are runtimes since installed, they get agregated for daily runtime from DB also.

We like to keep an eye on the heatpump backup heater, and heatpump runtime and starts in winter to make sure it isn’t acting up, i have some notifications in my heating rules.

Actually my control system twice detected a fan issue before it totally failed, first relay and than bearings.
OH actally feels like the heatpump buddy keeping it in check.

Regarding the migration of devices things are now operational, with only some errors in log and cleanup on my roadmap for heavy DB agregate operations.

rlkoshak · June 10, 2024, 2:38pm

In OH 4.2 the formatting of Number:Time has been fixed so it can handle long durations (durations over 31 days).

If it’s just for display, it should be in a transformation.

Rules DSL, no, not really.

All the rest of the languages do, though it’s a bit of work and separate skill to do so for Blockly.

But if you had it as a transform (e.g. a JS Transformation) you could call that transform from a rule.

val formatted = transform('JS', 'formatSeconds.js', MyTimeItem.state.toString)

You can use the transform in your label:

Runtime [JS(formatSeconds.js):%s]

or as a State Descriuption

JS(formatSeconds.js):%s

Matej_Kotnik · June 10, 2024, 9:20pm

Yes that transform JS looks like the most flexible and clean solution, i will look into it.