VMware vSphere

 View Only
Expand all | Collapse all

Ubuntu VMs freezing since vSphere ESXi 7u3c update

basicmonkey

basicmonkeyFeb 25, 2022 03:36 PM

basicmonkey

basicmonkeyJun 21, 2022 10:55 AM

  • 1.  Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Feb 23, 2022 08:11 PM

    Since upgrading our two main hosts to v7 update 3c, I'm getting VM freezes on a couple of VMs. One is happening every few hours.

    vSphere reports "The CPU has been disabled by the guest operating system. Power off or reset the virtual" at the time of the lockup. VM needs reset to continue.

    One VM on shared storage has had this happen once since the upgrade. Other one on local SSD is repeatedly suffering. I've move the repeat offender to shared storage to see if it happens again.

    Both VMs are hardware 13. I've tried upgrading the repeated one to 15 and made no difference.

    All my VMs are using NVME storage mode. Only these two have shown this issue, and never before u3c upgrade.

    Both VMs had this error in syslog about 15 mins before lockups:

     

    Feb 23 00:46:07 mon kernel: [36834.916147] nvme nvme0: I/O 213 QID 1 timeout, aborting
    Feb 23 00:46:07 mon kernel: [36834.916287] nvme nvme0: Abort status: 0x0
    Feb 23 00:46:37 mon kernel: [36865.123128] nvme nvme0: I/O 213 QID 1 timeout, reset controller
    Feb 23 00:46:37 mon kernel: [36865.165409] nvme nvme0: 15/0/0 default/read/poll queues
    Feb 23 02:34:22 mon kernel: [43329.673158] nvme nvme0: I/O 100 QID 4 timeout, aborting
    Feb 23 02:34:22 mon kernel: [43329.673393] nvme nvme0: Abort status: 0x0
    Feb 23 02:34:52 mon kernel: [43359.880150] nvme nvme0: I/O 100 QID 4 timeout, reset controller
    Feb 23 02:34:52 mon kernel: [43359.926041] nvme nvme0: 15/0/0 default/read/poll queues

     

    Versions:

    • Hypervisor:VMware ESXi, 7.0.3, 19193900
    • Model:PowerEdge R640
    • Processor Type:Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz
    • Ubuntu 20.04 LTS
    • Linux Kernel 5.4.0-100-generic x86_64

    Many thanks in advance!



  • 2.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Feb 24, 2022 12:54 PM

    Update: VMs dropping like flies, every day. Both on shared storage and local SSD. Going to have to re-image one of my hosts back to 7u2 as they were upgraded using patches. Ugh.



  • 3.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Feb 24, 2022 03:21 PM

    Update:

    Also NVME errors on my pfSense VM (FreeBSD 12).

     

    I've re-imaged a host to 7u2c and soaking for 24 hours to verify it's u3 related. Any issues will revert to u2a which is where the hosts were before I started the upgrade process.



  • 4.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Feb 25, 2022 03:36 PM

    No issues on same host back on U2e.



  • 5.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Mar 07, 2022 03:24 PM

    Is nobody else having these issues with U3? Seems to be unusable for me at the moment...



  • 6.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Mar 14, 2022 05:48 AM

    I am seeing this same issue. I have added a "SATA controller" and then changed the drive to "SATA controller 0" and my issue has went away. I have not dug into the issue further. 



  • 7.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Mar 18, 2022 07:46 PM

    This still exists even with the latest patches (including one NVME driver patch).

    Seems to only happen with a few VMs running on the host. Huge amount of headroom, only 15% CPU use and 50GB  RAM left.

    Would be great to know that VMWare are aware!



  • 8.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Mar 21, 2022 12:12 PM

    i have also faced same issue and now i am trying to rollback the update.

    wendy's lunch time surveyzop.com


  • 9.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Mar 30, 2022 08:28 AM

    We're experiencing the same.

    Very similar setup to you.

    We only see on selected Ubuntu VMs.

    I'm runnig VCF4.4, so rollback to an earlier esxi version is not an option.



  • 10.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Mar 30, 2022 10:54 AM

    Glad it's not just me. It's also happening on my pfSense VM which is FreeBSD 12 so not just a Ubuntu thing.

    Would really like to hear that VMware are aware and looking into it.



  • 11.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Mar 30, 2022 12:35 PM

    for the sake of argument, can you try running pv controllers?



  • 12.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Mar 30, 2022 12:45 PM

    This issue doesn't affect the few VMs that are running on SCSI controllers. We moved most over to NVME a while ago to improve performance and it did significantly.



  • 13.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Mar 30, 2022 12:54 PM

    So then would you say the issue is with NVME in this patch? I only ask because I'm running over 700 ubuntu vms...and really do not want those kinds of problems. But I want to get these security patches applied all the same. 



  • 14.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Mar 30, 2022 01:17 PM

    Yes, 98%.

    Since the VMs all seem to report an issue with NVME drive just before they die, it would make sense.

    It seems to affect VMs rather randomly, and some more susceptible than others (can't pin down what it is).

    The commonality is:

    • NVME storage controller used for boot drive
    • *nix OS
    • ESXi 7 u3c
    • Affects both local SSD and NFS storage (not tried on VSAN)

    I've migrated all VMs to 7 u2e hosts and 0 problems. Within 24 hours of migrating to a u3c host I'll lose at least 2 VMs.



  • 15.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Mar 31, 2022 08:30 AM

    I am completely on VSAN.

    I have an application which is clustered between 3 VMs (At least one of these VMs were freezing daily). I have changed all the Boot drives of these VMs to SCSI Controllers (I have left the application drives on NVME). So far 24hrs stable for the cluster.

    Out of interest, which version of VMTools are you running on the problematic VMs? (I am currently on OPEN-VM-TOOLS 11.3.0 - I know 11.3.5, and 12.0.0 is available, but I can't upgrade those at this point.)

     



  • 16.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Mar 31, 2022 09:14 AM

    Do you have any pending update for the sata storage firmware (on iDRAC) ?



  • 17.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Mar 31, 2022 10:00 AM

    No Firmware upgrades pending for me.

    As I am running full VCF, I am fully upgraded to everything within that. I also have no SATA backed disks other than the ESXi hypervisor itself. (Also VSAN datastore disks are SAS backed).



  • 18.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Broadcom Employee
    Posted Mar 31, 2022 10:54 AM

    Please file a VMare support ticket with VMware global services.
    There is a similar issue being tracked internally that matches some of symptoms quoted here.



  • 19.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted May 01, 2022 07:24 PM

    i'm having the same exact Issue with Oracle Linux 8.4 VM.

    Currently, the technician of our case doesn't work on the case saying it's a VM related issue. SR# 22322639904



  • 20.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted May 03, 2022 01:18 PM

    Are there any updates with the investigations into this?

    Many thanks.



  • 21.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted May 17, 2022 03:36 PM

    I posted too soon and the VM crashed. 



  • 22.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Jun 21, 2022 10:55 AM

    Any news on this? Not seen any patches to fix.



  • 23.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Broadcom Employee
    Posted Jun 21, 2022 11:27 AM

    From what I understood this is NVME driver related within the VM, not sure what the solution is right now if you hit this issue.



  • 24.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Jun 21, 2022 11:32 AM

    Thanks for coming back to me. Seems strange it's only in 7u3. 7u2 is absolutely solid and no issues with this. The only reason I'm trying to get onto 7u3 is because our Synology backup system doesn't seem to like our 7u2 hosts since upgrading some.



  • 25.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Jun 21, 2022 10:50 PM

    Just a datapoint: seeing the same problem with ubuntu and debian guests in Vmware Fusion preview for Apple Silicon (mac/arm64). I don't have an x86 mac laying around to try it on there, but since it's the same error that you're getting in VSphere, I suspect this is a bug in a shared code-base for both products.

     

    As others have said, switching to sata resolves it at the cost of some performance.



  • 26.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Jul 29, 2022 02:15 PM


  • 27.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Jul 29, 2022 02:46 PM

    Great news!

    Can see the fix in the article but no mention in the release notes for 7 U3F (big list).

    Will try after the weekend!



  • 28.  RE: Ubuntu VMs freezing since vSphere ESXi 7u3c update

    Posted Sep 26, 2022 10:15 PM

    I can confirm that this hasn't appeared since the update.

    Thank you!