vSAN witness host patching using images

[Niyot1]

I am trying to patch the vSAN witness host but get an error -  Host not compatible with image.    All the other esxi hosts in the cluster remediate successfully.   Seeing as I can no longer go back to baselines how do I get to update the witness.  any ideas?   PS already opened a ticket with VMWare, that was 2 weeks ago. still waiting  

[Douglas Bowen]

The vSAN witness is not usually a real host but a virtual appliance in a remote site. It is only reported as a host by vCenter (by design). This could be your issue?

[Duncan Epping]

Which version are you running? As we don't support images for 7.0 U2 and lower, only U3 and up.

[Niyot1]

Hi 

 

Running version 7.0 U3j.    for to say that all the criteria has been met. 

• vCenter Server must be version 7.0 Update 3 and later.
• The witness host must be ESXi version 7.0 Update 2 and later.> this was verified already
• The witness host must be a virtual server and not a physical server. yes 
• The witness host must be a dedicated witness host and not a shared witness host. yes

[Duncan Epping]

Can you create a screenshot of the error, and does it give any specifics below that?

[Niyot1]

 

Got this KB from support - https://kb.vmware.com/s/article/90527.

[Duncan Epping]

The witness has some strange drivers listed, did you apply an add-on or something to the witness in the past?

I am deploying a virtual lab myself just to see if I can reproduce it...

[Niyot1]

Nope.   Always used baselines and only just switched to using images. 

[Duncan Epping]

Just testing it here in my lab and it just works fine... No idea why you have that issue. Maybe roll out a new Witness and replace the current one?

[Niyot1]

Thanks for looking and your quick responses.  I got VMWare support to check it out.  Turns out those drivers were the culprit and had to be removed and VMW vib drivers installed instead.   Must have been installed during the baseline days.   Interestingly,   the baseline patch didn't complain. 

[ezbg]

Nice one congrats!

[mike-p]

I have the same issue at one customer. I think this is the situation after the change of baselines to images. Perhaps waiting until a more actual ESXi version is available in the image list will fix it.

[Tibmeister]

Not following what you are meaning there.

[mike-p]

It is easy to reproduce. Take an actual version of a witness appliance 7.0U3o and create an image with the same esxi version.

The image versions of some components are lower than the installed ones.

Now there are 2 options. Either uninstalling all drivers with esxcli software via remove -n . . . . , or waiting until the image version contains higher versions.

[Tibmeister]

Did this occur on an existing witness that was previously patched using a vLCM image?

If the witness was ever patched with a OEM Custom ESXi ISO, then there will be these issues.  vLCM shouldn't apply the Vendor Addons or the Firmware packages as it knows the witness shouldn't get this.

[mike-p]

No, actually this are witness appliance is only connected to the VMware predefined baselines. It is possible that in the past someone used as wrong iso. I will check this.

 

[Tibmeister]

I had this happen as well.  I essentially deployed a new witness, updated it to the matching ESXi version of the cluster using an image I exported from vLCM, then moved the cluster to it and everything was happy.

Not sure why it had the other drivers, but doing the re-deploy seemed to resolved the issue this time anyway.  Will need to see if next time I have the issue, which then I will have to conclude vLCM is doing the deed and I will need to re-evaluate the whole image based remediation of my two-node clusters.

[mike-p]

Finally I found vib's from an old 7U2 DELL offline bundle. I deleted all and the update went smoothly.

Tip for all in a similar situation: You can remove multiple vib's in one command like this :  

esxcli software vib remove -n nenic -n nmlx4-core -n nmlx4-en -n nmlx4-rdma -n smartpqi -n ionic-en -n iavmd -n lpfc -n nfnic

[Tibmeister]

Little confused, I thought that you can't use vLCM to patch a Witness Appliance for a 2-node cluster and the 2-node cluster?  Are you saying if we have a Witness Appliance with only a single 2-Node cluster attached then we could use vLVM to patch with an image for both the cluster and witness?

[Niyot1]

I have a 6 node stretched cluster.    So what you mentioned here does not really apply to this environment. 

[Tibmeister]

Sorry, was replying to , but correct, it shouldn't apply unless you are using a shared witness.

[Duncan Epping]

I am not sure I am following you? We are talking about a 6-node cluster here? I didn't mention 2-node or shared witness either?