ESXi 6.5 and Zwave Help

[rlkoshak]

I'm migrating my home to an ESXi 6.5 server. Most everything seems to be working just fine except I cannot get my zwave dongle to work.

I've plugged the dongle directly into the host machine, no hubs or other things to get in the way.

I've configured the USB passthrough by adding it as a new USB device to the VM and it shows up just fine.

The zwave binding is able to connect to the device and communicated with it at least well enough to download the nodes and add the Things to the inbox. However, the Things never resolve to their device type. They remain unknown.

All of the nodes currently added to the controller are mains powered.

Open Zwave Control Panel seems perfectly able to interrogate and communicate with the controller on this same VM. At least there are no errors and everything appears as it should.

I've moved the controller back to the old laptop and after a little bit longer than expected it was able to resolve all of the devices.

Has anyone experienced the same problem and solved it, or have any advice for how to get this to work?

[chris]

You probably need to look at the debug logs to find out why it's not resolving the things. I've seen issues with VMs and serial ports under OH in the past - of course I've no idea if that's your problem, but there are a number of threads around about that.

[Dim]

The zwave binding is able to connect to the device and communicated with it at least well enough to download the nodes and add the Things to the inbox. However, the Things never resolve to their device type. They remain unknown.

Hey @rlkoshak !
Rebuilding your OH2 platform?

I found something relevant here:

ZWave: Timeout while sending message. Requeueing

I have been having very similar problems recently but I have made some changes to my system that have reduced the frequency of the problem considerably. I have openHAB 2.0 running on Ubuntu 16.10 as a VM on VMware ESXi 6.0u3. I use USB passthrough to passthrough an Aeotec Zwave stick S2, an RFXtrx433e (433MHz), a CUL (can't remember the exact version) (868Mhz) and a bluetooth stick. The issue would manifest itself by a complete failure of Zwave comms via the Aeotec Zwave USB stick S2. Ever…

[rlkoshak]

Rebuilding your OH2 platform?

My old laptop ran low on RAM so I bought a cheap desktop server to run ESXi. I choose poorly though and bought one that doesn't support VT-D (or, as I'm now reading, it is just disabled in the BIOS and I've not checked in which case I'm going to be mad at days of lost effort making this all work again) so I'm stuck on 32-bit OS's and therefore can't use Docker containers like on my old setup. Moving to a server running ESXi was always in the plan. I moved to Docker containers on my laptop initially to make this transition easier. The best-laid plans of mice and men often go awry.

Thanks for the link. I searched the forum of course but apparently, my searching skills are deficient. Wish I had seen that posting before I got everything else set up. Grumble. Now I need to reinstall the ESXi and rebuild my VMs. Thankfully I'm using Ansible to build everything so I don't have to figure everything out again and it won't take but a few hours. It's still cheaper than buying a new stick.

Back to the old drawing board. Grumble.

@chris, of course, after rebuilding this virtual machine a dozen times the dongle seems to be working properly, just when I've enabled trace level logging. Maybe the logging is slowing it down just enough to make it work properly. At least to the point where it is recognizing all the mains powered devices right out of the gate. Gotta love those heisenbugs.

I'll come back with logs if I start seeing problems with other devices or the problem returns when I rebuild this thing on the previous verison of ESXi per Dim's link.

Time is still really short for me these days so it may be a few days.

[rlkoshak]

Well shoot. I managed to mess everything up. But I did learn the following.

• Virtualization was disabled in my BIOS. Waaaaaaa! I could have been using 64-bit and docker all this time
• I installed 6.0 only to discover that it won't let me create version 7 hardware VMs which are required for USB passthrough. I don't know if the other posts on this forum recommending the earlier version have a paid version of ESXi or what but for now, I'm stuck on 6.5 or nothing.
• Even though the nodes did show up in PaperUI this last time, they eventually timed out and went offline. I might not be able to host OH on a VM like I planned.
• When I reinstalled ESXi, somehow I became unable to attach a really important VMDK to one of my VMs. I might be doomed if I can't figure that one out. Gaaaa!

[smar]

Has anyone experienced the same problem and solved it, or have any advice for how to get this to work?

I have it working fine on 6.5 by disabling the new usb driver vmkusb introduced in this version, and reverting to the legacy usb driver - https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2147650.

SSH into esxi, and run the command esxcli system module set -m=vmkusb -e=FALSE. Then reboot. That should resolve the time out issue.

(I have done this on two different machines, each now running for a couple of months on esxi 6.5 without any issues).

[rlkoshak]

That is really good to hear. Are you using an S2 or a Gen5 controller (or something else entirely)? It gives me hope that I can get everything deployed the way I want it rather than needing to buy a new Pi just to run OH.

I should have just reverted to the legacy driver in the first place to see how it worked. The thread Dim linked to had the person reporting that it made things better but didn't solve the problem so I decided that since I needed to rebuild my VMs as 64-bit anyway I may as well go whole hog and rebuild everything.

I forgot that I had a lot of important data on a VMDK on the machine which I can't seem to get attached to a new VM and I didn't realize that the free 6.0 version (or it has something to do with my hardware, I don't know) won't let me create 7+ version VMs so I can use USB passthrough. I definitely leapt before looking and landed in a big pile of poo. That is what one gets when they can only spend 15 minutes at a time working on this stuff.

Step 1 is to try and recover that data. Step 2 is to reinstall yet again with 6.5 (I may need to do this before I can do Step 1). Then for the rebuilding. Luckily the rebuilding will take the least amount of work. I have it all scripted through Ansible so in a couple of commands I can have it back.

If you've any experience working with vmdks on the host (I made the mistake of making the vmdk larger than any of my spare HDs so can't transfer it off of the host) I'm all ears.

Thanks for the bit of hope though.

[smar]

One of my installs is using a zwave.me controller, and the other an Aeotec z-stick Gen5. I'm not sure why the other thread reported that they continued to have problems, but certainly in my case the timeout issues resolved immediately after disabling the new driver (after spending a few days trying everything to figure out what had gone wrong following my upgrade to 6.5!).

What specifically is the issue you are facing with your VMDK file? If you have rolled back to 6.0, you will not be able to use the VMs created under 6.5 as they are on a newer 'hardware' version (if I recall, 6.5 uses VM Machine hardware version 13, whilst 6.0 uses 11). If this is the case, then as you guessed, first step is to install 6.5, and then try to re-attach the VMDK.

[rlkoshak]

It's weird actually. The first VM I tried to rebuild is an OpenMediaVault server. I created a brand new VM (it would only let me create a Version 6 VM). I had two vmdks attached to the old VM and when I tried to "Add Existing Disk" to the new one and selected the big vmdk from the old server it was treating it like I wanted to create a new disk rather than attaching the existing one.

At this point is was midnight and I probably was both a little tired and a little panicked so I very likely could have done something stupid. I don't know. There is one folder with files that will be very difficult to recover and another that will take a loooong time to recreate so it is worth my spending some time to see if I can get at the data.

I found some links for how to fix a corrupt disk descriptor file which could be my problem. Honestly, I'm hoping that I'll just install 6.5 and reregister the old VM and the disk will be there. Fingers crossed.

As an aside, what do you think of the zwave.me controller? Are you using the z-way binding or zwave binding? If I have to purchase a new controller I can consider that way.

[smar]

The VMDK should just attach - again, I have done this many times, and in fact, also used to have an OMV install as well. One thing you could do if you are worried about the data is to create a new VM for OMV, get it up and running and then attach the old VMDKs as additional drives. Another question - is your large VMDK a -flat file? If so, the other should be the descriptor. Also when you say it was treating the existing VMDK as a new disk, is that at boot up or when you added the VMDK?

I've had the zwave.me controller since I started out with zwave, back with OH 1.6 days. It has and continues to work well. It has very good range, reaching from my garage all the way to my loft (according to the the z-wave network diagram in HABmin). As I have the cheaper (£25) unit, it doesn't come with a z-way license. As such, I haven't tried the z-way binding (beyond installing it and finding out that I needed to buy a separate license!). In any event, Chris' z-wave binding does everything I need, and works robustly. For the money, you can't go wrong with the zwave.me controller. (I have on occasion also used it with an open-zwave control panel running in a VM for testing etc - again, works fine every time).

EDIT: Also what do you mean when you say:

it would only let me create a Version 6 VM

You should only be able to create version 13 machines on esxi 6.5. e.g. here's a screenshot from one of my VMs:

[rlkoshak]

The VMDK should just attach - again, I have done this many times

So have I. Which is why this was so troubling.

One thing you could do if you are worried about the data is to create a new VM for OMV, get it up and running and then attach the old VMDKs as additional drives.

What seemed to work was upgrading to 6.5 and reattaching the old VM. The drive was there and I'm able to get at the files. I am now moving the data off and plan on starting over completely from scratch. Given this weirdness I don't trust that drive, not to mention I can't seem to find the flat file in the Datastore any longer. Next time I'll create smaller VMDKs that I can offload from the host if I need to do this sort of thing again.

This whole thing is just really weird. This is not my first time messing with ESXi and I've never had these sorts of problems.

Also when you say it was treating the existing VMDK as a new disk, is that at boot up or when you added the VMDK?

When I added it to the new VM. Basically what would happen is I'd attach the drive and a message would pop up and say I can't create a new drive that is larger than my available free space. And the area where I could change the size of the disk is grayed out so I couldn't shrink it if I wanted to (which I wouldn't want to).

EDIT: Also what do you mean when you say:

On 6.0.0u2 I could only ever create a VM version 6, nothing higher. Consequently, I never had the option to attach a USB device (which begs the question why there are USB controllers). Unlike on 6.5 where the USB Device entry in the add device menu was grayed out, it was completely absent.

On 6.5 everything works fine. I can create version 13 VMs all day. It was just 6.0 that had problems.

It's good to hear your experience with the zwave.me controller. I might go that way if it proves that the S2 won't work even after reverting to the legacy USB drivers.

Thanks a ton for all your help. I've got my files back so the major crisis is averted. Now it is just a matter of time to build everything back up again. I should be back in business and then some by this weekend.

[smar]

Glad to hear you're back up and running. One other thing you may want to look at if you are worried about data loss in a virtual nas is to pass whole disks into your OMV as raw disks. That way, even if you are not able to get esxi back up and running, you can always access your data from a native OMV (or other linux) install. There are some downsides (e.g not being able to use vMotion etc) but they probably won't matter for home/free use. That is the way I have my disks. This also gives noticeably faster disk performance.

Good luck and don't hesitate to reach out if you come across any other stumbling blocks.

[rlkoshak]

That is really what I want to do but I could never figure out how to make it work. I will research it some more. I have only the one server so vmotion is irrelevant to me.

I looked into it some more and since I have my data backed up now I gave it the old college try and got raw device mapping to work. Woohoo! Second time through I'm able to make this look more like I originally wanted.

I think I'll still keep my backup drive as a USB so I can grab it and go in a pinch, though may change my mind later.

Thanks again for all the help!

[smar]

Can't go wrong with a backup USB drive! In fact, I use syncthings to automatically back up my data to a separate physical NAS.

[hyperbart]

Hi there

A lot of questions and experimenting with ESXi I see.

USB Passthrough should normally work, however you might want to check if you have passed on all devices, sometimes certain devices show up as a basic USB device Zwave is a serial protocol if I remember correctly so you might need to pass through a USB serial device too, post your devices here and I'll look in to it.

If the route above does not work: consider buying a well home-lab supported USB hub and pass this through completely and plug the Zwave usb stick in the hub.

And if your ESXi will serve some other things in the home too, consider upgrading to a VT-d capable setup. Doing so you can pass through a PCIe device (like a seperate USB controller card). Makes the investment a little less sour...

[rlkoshak]

Thanks for the offer. I'll take you up on that when I can. I've started over and am doing things "the right" way this time. Virtualization is on, I'm making 64-bit VMs, I'm passing my working drives in as RAW instead of vmdks, and I've reverted back to the older USB drivers.

But I've not yet had time to spin up a new VM for OH. I'm having some trouble getting my Gogs server running from the backup and have very little time. So far I have Plex (the 4-year-old has gotta have his movies), OpenMediaVault, and a desktop VM rebuilt. I still need to get Gogs working properly and restore my repos, a Calibre server, Alfresco, and all the home automation stuff like mosquitto, influxdb, grafana, and openHAB.

When I get back to rebuilding my home automation VM and if I still have trouble I'll come back and post. So far I've not had any more trouble with ESXi, though I haven't tried to pass through the S2 dongle yet.

Thanks again!

Rich

[hyperbart]

Would be nice if you could write up something of combining OpenHAB and InfluxDB with Grafana.

I have a friend who implemented Grafana + InfluxDB here to log all my temperature data from the KNX bus but since OpenHAB can do the same I would like to centralise this stuff.

InfluxDB + Grafana + KNX was a little bit too much for me. AMA about vSphere (ESXi) or Hyper-V but as always the road to getting something to work under Linux is steep as ****

You don't have any experience with Sonos and OpenHAB do you?
https://community.openhab.org/t/sonos-spotify-playlist-as-favorite-not-starting/28468

[rlkoshak]

Would be nice if you could write up something of combining OpenHAB and InfluxDB with Grafana.

Something like this write-up from Thom? That is what I used and I put my experiences deploying it in Docker in a posting further down. Being able to deploy all these servers as Docker containers is the primary reason I need 64-bit VMs.

If you already have them both installed, it will be a simple matter of creating the users and DB in InfluxDB and setting up Persistence properly.

You don't have any experience with Sonos and OpenHAB do you?

Unfortunately, I have no experience with Sonos. They are too pricey for me and I don't have audio integrated with my automation. Everything runs from our phones so I've had no need.

Once I do get this all set up and running I do plan on writing up a tutorial showing how I set up my environment, lessons learned, and posting my Ansible playbooks I used to configure everything. It might give someone somewhere some good ideas perhaps.