Bluescreen booting Windows VM with latest CU on AMD with VBS enabled

[GabGo]

Hello Community!

We are currently migrating our virtual machines from our old Intel hosts to AMD hosts and ran into an issue with the latest Windows 10 21H2 Update.

We are using ESXi 7.0.3 Build 19898904 on HPE ProLiant Servers in a vCenter managed environment. Our old hosts running Intel Xeon E5-2660 v3 CPUs and our new hosts running AMD EPYC 74F3 CPUs.

Our Windows 10 21H2 (x64) VMs are on VM version 19 and have VBS enabled (in the VM configuration and Windows via group policy). These VMs are running fine on the Intel hosts. But if we start these VMs on the AMD hosts we get an bluescreen immediately with the following error message: “PNP DETECTED FATAL ERROR”.

We already did further investigations. The issue starts with the June 22 Preview Windows Update (KB5014023, Windows Build 19044.1741) and is also present in the latest July 22 Windows Update (KB5015807, Windows Build 19044.1826). It does not matter if VMware tools are installed or not.

To reproduce the problem,

• create a new Windows 10 (x64) VM with VBS enabled in the VM configuration,
• install Windows and patch to an affected build,
• enable VBS via group policy (cmd > gpedit.msc >> Computer Configuration > Administrative Templates > System > Device Guard > Turn On Virtualization Based Security >> just enable and leave all other options untouched).

The VM starts if we disable VBS in the VM options and disable I/O MMU in the VM CPU configuration (just leave EFI, Secure Boot and HW virtualization support for the guest OS enabled).

Is there anyone with similar hardware (Zen 3 architecture) who can also reproduce the problem? We are trying to figure out if this is an AMD, HPE, VMware issue or maybe just a Microsoft introduced bug.

Thank you!

EDIT 1: Just patched an affected host to ESXi 7.0.3 Build 20036589 but the Problem persists.

[GabGo]

Hi,

I just applied the latest July 22 Preview Windows Update (KB5015878, Windows Build 19044.1865) but the problem persists.

For reference, I also found someone with a similar problem back in 2020: https://communities.vmware.com/t5/VMware-vSphere-Discussions/7-0U1-Windows-boot-issues-after-upgrading-hw-version-to-18/m-p/2292668

Is there anyone with the same issue?

[Manu38]

Hi GabGo 

Same issue here. tested it with 20h2 / 21h2, as soon as the CU July or August is installed and needs a reboot, it runs into to the PNP Error. 

I tested as well with Vmware Tools (newer an older versions) and without... the issue is still the same.
ESXI 7.0 U2 (VM Ware Verson 19)

SecureBoot ---> Enabled 

Virtualization Based Security --> Enabled. 

We have it only on one VMware host in one location, i try to figure out the differences. Because im responsible for Client Engineering and SCCM, not for the Vmware Host or Hypervisor.
I do believe its a VMware issue... 

Do you have more News? 

 

[GabGo]

Hello Manu38,

currently we are working with Microsoft to find the root cause of the issue but don't have any findings yet.

In the meantime I updated the host to the current build (ESXi 7.0.3 20328353) and applied the Windows 10 21H2 August Preview CU (KB5016688) but the problem persists.

I'll keep this discussion updated if there are any news.

[GabGo]

Small update on the issue:

 

After Microsoft analyzed the memory dump there were traces that the problem might be VMware related so the case was transferred to them.

 

In the meantime we installed an ESXi 8.0 trial version on one of our affected hosts and transferred an problematic VM to that host. The problem persists as long as we stay on VM compatibility 19 (ESXi 7U2 ). After upgrading the VM compatibility to 20 (ESXi 8 ) the VM boots fine with VBS running. Unfortunately we cannot use vSphere 8 in production yet.

[agraupner]

Any Update? Or can you give me the Case Number? We ran into the same issue and i opened a ticket at vmware, but up to now the didn´t provide me the solution.

[GabGo]

Hello!

we also don't have an solution yet. As far as I understand the responses from VMware correct, they can reproduce the issue and are working on it.

Unfortunately I cannot give you a case number because we opened our ticket through HPE. We bought our VMware licenses there to (hopefully) have an combined hard and software support.

Sadly we have an new and related issue which also affects VMs running under ESXi 8 (this was our "workaround" if the problem is not fixed for ESXi 7 anymore): We have random daily BSODs on about 2% of our server VMs with VBS enabled. No pattern, any Windows Server version, always different VMs. On ESXi 7 these VMs just reboot so we didn't notice the issue first. On ESXi 8 there is an BSOD with "SECURE KERNEL ERROR" (BugcheckParameter1: 0x18c, BugcheckParameter2: 0x100b, BugcheckParameter3: 0x0, BugcheckParameter4: 0x209a). We opened another ticket with HPE/VMware which is also under investigation now.

[agraupner]

Hello GebGo,

is there an update that solves the issue?

Thx in advanced

[GabGo]

Hello!

VMware told us that both problems (Windows starting with BSOD and the random BSODs) will be fixed with the next ESXi patch.

Currently we are eagerly waiting for this release.

[agraupner]

Hello,

thx for your fast reply. Finally i also got a response from VMWare. They told me:

According to engineering the fix should be in the last preview of Windows (Not VMware)
"It should be available on the latest insider build already, starting from 22512.1000."

Confusing why we get different answers to the same issue. 

[GabGo]

Hello!

Interesting, I cannot find anything about the insider build number 22512.1000. But it sounds like a Windows 11 insider build.

We already tested a Windows 11 installation on ESXi 7 with VBS enabled. These VMs are starting fine like all Windows Server versions. Also Windows 10 VMs with VBS are starting on ESXi 8 but only with VM compatibility 20.

The immediate BSOD after starting the VM only occurs on Windows 10 with VBS enabled an VM compatibility 19 (ESXi 7 as well as ESXi 8).

Beside this issue we have random BSODs on all running Windows Server VMs with VBS enabled. This developed to the main issue fast since this potentially leads to data corruption.

[PhSLU]

Hello, we are experiencing the same here.

My case at VMware is closed already as mentioned in https://kb.vmware.com/s/article/89880 ... "If the issue still persists, please engage Microsoft Support to troubleshoot the issue."

So now I have a case logged with Microsoft investigating this.

On the same VM (Hardware version 19), I installed 4 versions of the OS ( Windows Server 2019 / Windows Server 2022 / Windows 11 and Windows 10 )  (not multiboot)

Only Windows 10 got BSOD 

[bentschrader]

Hi there,

exactly the same problems in our envorinment: 

• VMware ESXi, 7.0.3, 20842708
  
• Modell:ProLiant DL325 Gen10 Plus v2
  
• Prozessortyp:AMD EPYC 7513 32-Core Processor

Windows 10 22H2 19045.2604 with the same BSOD - until turned off VBS and using HW version 17. Each HW version above leads to the same BSOD, even with VBS disabled.

[GabGo]

Hi,

short update: The patch first promised for end of February is now delayed until early March.

In the meantime we tried every new Microsoft CU (even the preview releases) and the ESXi patches 7.0U3j (Build 21053776 from 2023-01-31) and 8.0b (Build 21203435 from 2023-02-14) with no luck.

As the BSOD at boot was kind of introduced by Microsoft with Windows 10 Build 19044.1741 (and therefore should be fixable), our biggest concerns now are the random VM crashes with the "SECURE KERNEL ERROR" BSOD which also affects all Windows Server versions with VBS and Credential Guard enabled.

Do you have also such random BSODs or reboots on VMs (Windows Server) which are running with VBS and CG enabled?

---

 wrote:
...

Each HW version above leads to the same BSOD, even with VBS disabled.

---

Interesting. Did you disable VBS on the VM options or inside Windows (Credential Guard). Is this also happening on a fresh VM on HW version 19 with VBS disabled in the VM options?

If we leave the VBS option off we can run any Windows version on HW version 19 (and of course cannot use Credential Guard then). With the VBS option enabled these VMs are also running fine as long as we do not enable Credential Guard.

It is amazing how less you can read about this problem. Still I don't know if this is only happening on ProLiant Gen10 Plus v2 Servers (we are running DL385 Gen10 Plus v2, I didn't mentioned this particularly yet). But I guess there are many of these servers out there running Windows 10 VDIs on VMware (Horizon). If you try to harden these VMs you probably read this blogpost (https://blogs.vmware.com/vsphere/2018/05/introducing-support-virtualization-based-security-credential-guard-vsphere-6-7.html) or this article (https://learn.microsoft.com/en-us/windows/security/identity-protection/credential-guard/credential-guard-manage) and you should end up with an BSOD after boot.

[GeoPerkins]

Are you using UEFI or BIOS for the guest boot option?

Maybe see: 

https://kb.vmware.com/s/article/90947

[PhSLU]

 Hello  

Your question is naive ... we all use UEFI (mandatory for what we are trying to achieve)

The KB you point us to is a new bug affecting Windows Server 2022 ... we are talking about Windows10 (and the issue is not new) 

[Maks Roshchyna]

Do you have only one VM with Windows 10? 

All your VMs having BSOD or only that one?

I would be digging into Guest OS problems

[PhSLU]

Hello  ,

as described multiple time this is easily reproducible.

Indeed, as I said, I have a case open with Microsoft investigating this.

 

[GabGo]

Hello,

I know this is a frustrating problem but lets try to stay calm anyway.

As PhSLU mentioned we use UEFI because this is a requirement for VBS and we have this problem on all Windows 10 systems.

We also started investigating this problem with Microsoft and handed it over to VMware since there were corresponding indications in the memory dump.

As this topic got some attention now and the problem is relatively easy to reproduce, are there other users with AMD CPUs (and maybe non HPE servers) who can reproduce these BSODs?

Are there users running Windows Server with VBS and Credential Guard enabled on such CPUs and encounter random reboots or BSODs? The latter issue is or was even present on ESXi 8 (patched our test host to 8.0b a few days ago still waiting for the first random crash of a VM).

[raphael2]

Hi all

I'm monitoring this thread since early december as we have the exact same random bsod on Windows Server 2019 with enabled VBS and thought I write my findings here:

We have 8 Lenovo SR635 (EPYC 7543P), 12 Dell PowerEdge R6525 (EPYC 7543) and 8 Dell PowerEdge R7515 (EPYC 7543P). On all of these servers newest firmwares were applied and have the newest ESXi patches 7.0.3 21053776. Some of the W2019 vms are on cu 17763.3887 and some already on 17763.4010.

We already had mulitple open cases with Microsoft, VMware and Dell without a solution or a commitment to further investigate. Troubleshooting is really annoying because we need to wait minimum one week after applying a possible fix.

This is the log output after a bsod:

2022-12-27T16:14:28.313Z Wa(03) vcpu-0 - WinBSOD: Synthetic MSR[0x40000100] 0x18b
2022-12-27T16:14:28.313Z Wa(03)+ vcpu-0 -
2022-12-27T16:14:28.313Z Wa(03) vcpu-0 - WinBSOD: Synthetic MSR[0x40000101] 0x18c
2022-12-27T16:14:28.313Z Wa(03)+ vcpu-0 -
2022-12-27T16:14:28.313Z Wa(03) vcpu-0 - WinBSOD: Synthetic MSR[0x40000102] 0x100b
2022-12-27T16:14:28.313Z Wa(03)+ vcpu-0 -
2022-12-27T16:14:28.313Z Wa(03) vcpu-0 - WinBSOD: Synthetic MSR[0x40000103] 0x0
2022-12-27T16:14:28.313Z Wa(03)+ vcpu-0 -
2022-12-27T16:14:28.313Z Wa(03) vcpu-0 - WinBSOD: Synthetic MSR[0x40000104] 0x209a
2022-12-27T16:14:28.313Z Wa(03)+ vcpu-0 -

Our last hope is the patch mentioned above which should arrive in early march.

 

 

[GabGo]

Hello,

these are exactly the WinBSOD entries we are getting in the vmware.log files of our VMs. Only a small percentage of our VMs are crashing daily. At some days none, some days 5 or more but always another VM. Absolutely random with no pattern.

The problem in troubleshooting the issue was, that there were no memory dumps written (even we configured this in Windows). We created a large amount of test VMs and configured them to not automatically reboot after a BSOD. Eventually one of these VM crashed after some time and stayed in the BSOD (with error "SECURE KERNEL ERROR"). We took an ESXi snapshot including memory and used the tool "vmss2core" to extract a memory dump from the memory part of the snapshot: https://kb.vmware.com/s/article/2003941

Microsoft was able to analyze this dump and detected an hardware issue so we could ask VMware for investigation. They then could reproduce and work on the problem. This took us about 3 month... annoying troubleshooting as you said.

At least we have the feeling that every party kind of committed to the problem so feel free to piggyback on us.

[GabGo]

Hello,

just tried patch 7.0U3k (Build 21313628) out of desperation knowing this is the fix for the Server 2022 issue. Of course with no luck.

I guess this is the kind of problem you need to get a fix in a week. As long as the VM starts with default settings (VBS disabled) it's probably not important enough if you want to gain some security on AMD CPUs and you have to wait about 6 month.  

March is coming, hope there are no more delays *fingers crossed*

[GabGo]

Hello,

I just want to inform everyone who also patiently waits for a fix that the promised patch is (of course) delayed and SHOULD be available by the end of march. The reason for this are pending product fixes and security.

Security, this is what we are talking about when we want to enable VBS. Early January we were told that a fix is ready and will be released in the next patch. Update 3j and 3k were released and still no fix.

Remember this also effects Windows Server with VBS enabled running on ESXi 8 (Update 8.0b). These VMs are crashing randomly and we already had a data loss because of this.

Sorry for ranting, I know that we are a minority with this problem. Still having in mind how long it took to fix the Server 2022 bug two weeks ago.

[GabGo]

Hello,

since VMware released build 21424296 for ESXi 7 and build 21493926 for ESXi 8 end of march I want to give a short update.

• the BSODs when booting a Windows 10 VM on ESXi 7 still persists
• the random BSODs on ESXi 7 seems to be gone
• the random BSODs on ESXi 8 still persists

It's not clear why VMware didn't fix these random crashes for ESXi 8 so there is no upgrade path for now. At least we don't have to worry more data corruption from this issue on ESXi 7.

For the BSODs at boot I heard rumors that maybe Microsoft is working on a fix (which would be reasonable since they "introduced" the problem last year with a CU).

[MartinRZ]

Hi all,

I found this thread looking up (again) my random reboot/crashes of our Windows Server 2019 DC VMs. We have three Lenovo SR655 with AMD EPYC 7443 CPUs. Since beginning we had some ramdom reboots but we ware not able to find any root cause. VBS enabled or not it just happens ony almost any VM. We have some Windows 10 32 bit VMs but there is no such VBS option - is only available with 64 bit?

Any way, we are on 7.0U3g and reading this I thin we will go to newest 3m. According to last entry by GabGo this SECURE KERNEL ERROR is gone going with 3l or later?

[MartinRZ]

Update.

I applied 7.0U3m (aka Build 21686933) and today a Windows Server 2019 VM crashed. This VM have VBS enabled. I am not sure if the VMs should be restarted after the ESXi patch have been applied or not?

Any other affected user have feedback on the 'm' and/or 'l' patch?

[GabGo]

Hello MartinRZ,

sorry for the late reply. First an easy answer: VBS can't be enabled on 32 bit VMs.

Now what happened since my last post:

Our Windows 10 VMs still don't boot with VBS enabled even with ESXi 7 Update 3m (Build 21686933). The potential Microsoft fix was announced for the June CU preview but with this update the VMs still boot into BSOD.

Our Windows Server VMs seemed to run stable but then we still saw very rare random reboots (one BSOD a week). The bug check codes are now different (mainly 0x20001: HYPERVISOR_ERROR).

In the meantime VMware released ESXi 8 Update 1 and Update 1a. With these releases the initial random reboots - which were still present on ESXi 8 Update c (Build 21493926) - are also gone. We think they incorporated the ESXi 7 fixes in these builds.

Since we can boot Windows 10 VMs with VBS on ESXi 8 (VM Version 20) we decided to upgrade to vSphere 8. If there are still BSOD we have to start over again and let Microsoft analyze the dumps.

[PhSLU]

Hello, 

I can add to that : I received and validate a private Fix from Microsoft for the Windows 10 ... it fixes the issue, but I don't know yet when it will be implemented.

 

[gpk]

Can you please provide me this fix? Because currently we have the same problems in our environment.

[PhSLU]

The fix should be KB5028244 ... but I don't thing you can download it as an individual patch yet ... a bit more patience is needed 

[gpk]

The patch is available, now (preview update) and solves the problem, thanks!

[markwpumphrey]

As a workaround you can set your "Guest OS Version" to "Microsoft Windows Server 2008 R2 (64-bit)"

you can even enable VBS during a new install while choosing this OS.

"Edit Settings"
in the "VM Options" tab expand "General Options"
"Guest OS Family" should not change "Windows"
"Guest OS Version" should be set to "Microsoft Windows Server 2008 R2 (64-bit)"

[Stellarier333]

Do anyone knows, if there is a Microsoft Patch also for Windows Server 2019. Patch KB5028244 is only for Windows 10 systems.

[GabGo]

Hello,

the Microsoft patch fixes the boot BSODs which we saw only on Windows 10 VMs with VBS on ESXi 7. Server 2019 VMs with VBS boot fine and "only" had this random reboots.

Here is a short recap since we have a stable ESXi environment now:

• KB5028244 (which was the preview) respectively KB5029244 fixed the boot BSODs of Windows 10 VMs with VBS on ESXi 7.
• Since we upgraded to ESXi 8 early in July we worked around the boot issue but still had random BSODs (now with different error codes) on VBS enabled Server VMs.
• These random BSODs where finally fixed with ESXi 8 Update 1c (build 22088125)

So this took almost a year to fix all issues where you had to decide if you wanted to have less security or potential data loss. I wonder if there is ever an explanation why this happened and what can be done to avoid such situations in the future.

[malloy993]

Have a customer that is just seeing this issue now on a windows server 2019 box. Anyone find a fix for windows server 2019 yet? 

Thank you, 

Malloy993

[PhSLU]

KB5030214 (September Update for Windows 2019) seems indeed to cause problem (AMD,ESXi-7.0U3n-21930508) if VBS is enable on the VM

[PhSLU]

Secure launch seems to be the responsible 

[GabGo]

I can confirm that Server 2019 VMs with secure launch and KB5030214 are crashing at boot.

There is no BSOD and the VM logs show only (guest initiated) CPU resets.

I reproduced the issue with fresh VMs on ESXi 7 and ESXi 8 (both with latest patches).

Our last used workarounds - first disabling VBS and then I/O MMU or the entire hardware virtualization in the VM configuration - did not recover the VMs. The only quick way for recovery without a restore from backup is to remove the update (RollupFix package) via DISM on the recovery cmd. Maybe there are other solutions.

[mkaetm]

We have the same issues with HPE ProLiant DL385 Gen11 and AMD EPYC 9374F CPUs.

The only way for us to get Windows Server 2019 VMs booting up again was to migrate them to Intel CPU-based ESXis to disable VBS completely according to these instructions from Microsoft. Then we had to boot them into the Windows Recovery environment to clear persistent UEFI settings. 

Afterwards they were bootable again on the AMD hardware. 

We were able to find the problematic setting: Secure Launch Configuration (Registry value name: HKLM\SOFTWARE\Policies\Microsoft\Windows\DeviceGuard\ConfigureSystemGuardLaunch) was set to "Enabled". Our workaround now is to re-enable VBS, but to set Secure Launch Configuration to "Not configured". 

Who can be blamed for this? Microsoft or VMware?

 

[GabGo]

Hard to tell who can be blamed. Hopefully this get fixed faster. The last VBS boot issue - which also occurred only on AMD CPUs - was introduced by Microsoft in July 22 and was fixed by them in August 23.

It's annoying that there is no QA for their security technologies on such a common hypervisor and CPU. I could understand the high expense if this where an uncommon setting but Microsoft recommends enabling Secure Launch in their own Security Compliance Toolkit Baselines.

Are you using vTPMs in these VMs? This is - if I remember correctly - a requirement for using Secure Launch. You can check this with msinfo >> VBS services configured vs. running. We don't use vTPMs yet so we (luckily) already disabled Secure Launch a while ago.

[PhSLU]

---

Are you using vTPMs in these VMs? This is - if I remember correctly - a requirement for using Secure Launch. You can check this with msinfo >> VBS services configured vs. running. We don't use vTPMs yet so we (luckily) already disabled Secure Launch a while ago.

---

You can enable Secure Launch without vTPM (wich we are - unluckily) ...

I opened a case at Microsoft ... will keep you posted. 

[Stellarier333]

We can confirm that Server 2019 VMs are crashing after installing KB5030214.

We are running 

• Hypervisor:VMware ESXi, 7.0.3, 20036589
  
• Model:ProLiant DL385 Gen10 Plus
  
• Processor Type:AMD EPYC 7402 24-Core Processor

Our VMs are using SCSI Controller " VMware Paravirtual".

As soon as we are running into the hanging Windows Logo screen, we are able to revert the MS update with these steps:

- Turn off VM
- Change SCSI Controller from "VMware Paravirtual" to "LSI Logic SAS"
- Deactivate VBS, I/O MMU and secure boot on VM virtual layer
- Boot into CMD (Windows Recovery environment)
- Sign in with local Administrator
- Mkdir C:\scratch
- dism /english /image:C:\ /Get-Packages /Format:Table
- dism /image:C:\ /scratchdir:C:\scratch /cleanup-image /revertpendingactions
- Power off VM
- Change SCSI Controller from "LSI Logic SAS" to " VMware Paravirtual"
- Activate VBS, I/O MMU and secure boot on VM virtual layer
- PowerOn VM
- VM is boot and screen appears "We couldn't complete the updates. Undoing changes. Don't turn off your computer"
- VM is running again

As soon as the VM is booting in OS again without installed KB5030214 we have performed this:

Deactivated VBS, I/O MMU and secure boot on VM layer, deactivating VBS via GPO. Deleted the Credential Guard EFI variables by using bcdedit like Microsoft has described it here: Disable Credential Guard with UEFI lock 

We did set this regkey additionally:

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\DeviceGuard]

"ConfigureSystemGuardLaunch"=dword:00000000

After installing KB5030214, we are still facing the same result: "VM is hanging in Windows Logo screen".

Why did you migrate the VMs from AMD to Intel Host? Was it because you needed the get Windows Recovery environment. In our case we have got it, like though changing the SCSI Controller like desrcibed above.

After you have re-enable VBS and set Secure Launch Configuration to "Not configured", where you able to patch the VM with KB5030214 successfully?

[mkaetm]

"Why did you migrate the VMs from AMD to Intel Host? Was it because you needed the get Windows Recovery environment. In our case we have got it, like though changing the SCSI Controller like desrcibed above."

I migrated the VMs to Intel, because I had some issues with disabling the VBS while Windows was running in safe mode. And on Intel they were able to boot and I could disable VBS then. 

"After you have re-enable VBS and set Secure Launch Configuration to "Not configured", where you able to patch the VM with KB5030214 successfully?"

I did not test this scenario. In our case the patch was installed before. 

 

[GabGo]

After you have re-enable VBS and set Secure Launch Configuration to "Not configured", where you able to patch the VM with KB5030214 successfully?

---

I can confirm that it is not enough to set Secure Launch to "Not configured" when it was enabled before. In this case it stays enabled (configured). You can check the configured VBS services through the "System Information" application (msinfo). So you have to set Secure Launch to disabled explicitly and reboot at least once.

In our environment I uninstalled KB5030214, set Secure Launch to disabled in the corresponding GPO, rebooted the VM and verified through msinfo that secure launch was not configured and installed KB5030214 again after that.

[Stellarier333]

In our last testing we had missunderstood the correct GPO configuration to get the VMs running after installed KB5030214. 

In the meanwhile Microsoft has provided an official workaround to us for Windows Server 2019 VMs, running on ESXi with AMD CPUs.

Official workaround for now

1. Workarounds:
   
   Workaround 1 of 2: Disable Secure Launch Policy under HKLM\SOFTWARE\Policies\Microsoft\Windows\DeviceGuard\ConfigureSystemGuardLaunch

2. OR
Workaround 2 of 2: Use VMWARE ESXI Hardware 17 or below

We have tested it with two VMs, running on ESXi7.0.3m. The first workaround has worked for us for the affected VMs!

We Activated VBS, I/O MMU and Secure boot on VM layer. Activated VBS via GPO with "Secure Launch Configuration = Disabled"

Result: After patching VM with MS September Updates, the VM is booting!!!

We hope that Microsoft will find the bug to fix it also with enabled Secure Launch Configuration on ESXi and AMD CPUs.

[PhSLU]

another patch Tuesday nightmare for those of us running ESXi on AMD and who care about VBS 

This time Windows Server 2022 with VBS enabled breaks after installation of  KB5031364.

Disabling Secure Launch is not enough. This time it seems to be deeper.

[GabGo]

I can confirm the issue with KB5031364 when running on ESXi 7. On ESXi 8 with VM version 20 the VM boots fine.

This time it seems the BSOD occurs even without enabling VBS via Group Policy. It's enough to enable VBS at the VM configuration. 

EDIT: You have to enable VBS in Windows to get the BSOD. It's the same behavior like the Windows 10 boot BSOD. You can work around by only disabling I/O MMU at the VM configuration (you have to uncheck VBS first) and at least still get (some) VBS features in Windows (check with msinfo).

[steven_vanpraet]

We have the same issue after applying the October MS security patches for Windows Server 2022 with a Dell AMD Epyc 3 servers. Disabling IO MMU is a workaround however VBS is not running anymore. Moving the VM to Intel based HW is working fine. Running ESX 7 u3 - O (22348816) with the lastest Dell updates. 

[GabGo]

I forgot to mention that if you disable I/O MMU in the VM configuration you have to set the Windows VBS settings (GPO) from "Secure Boot and DMA Protection" to "Secure Boot" only.

We do not have DMA protection enabled in our environment because it was not enabled in the MS Security Compliance Toolkit Baseline we use.

[TVolke]

Setting the DVD/CDROM drive to ide and remove the SATA controller, solves the problem for us. You have to change to IDE and save the settings, before removing the SATA controller. It can not be done in one operation.

[GabGo]

I can confirm this works. This is definitely the better workaround until a fix is available.

[RuneD]

I've had issues with Server 2019 since August 2023, Server 2016 since September 2023, and now Server 2022 since October 2023.
Most affected were 2 Server 2019's that were setup in January 2020 on some i5 CPU, before running on Intel Xeon Gold 5215's until this summer when they were migrated to AMD EPYC 7313.

 

Anything installed on the i5' the workarounds listed does not work for:
* Remove VBS
* Remove Secure Boot
* Remove IOMMU
* Remove CD/DVD + SATA Controller
* Upgrade ESXi to 8, it has caused new issues related to VSAN Stretched licensing since its a 2 node cluster with a witness (which no longer is allowed on a standard license it seems...)

 

Anything installed on the Xeon's directly the workaround related to IOMMU is the fix. Anything earlier refuses to budge...

[GabGo]

I assume you are also using VBS which causes your problems.

If you have installed KB5030214 for Server 2019 you have to disable Secure Launch in the group policy either by starting the VM on a unaffected host or by uninstalling the update first (which is a bit inconvenient if you are using paravirtualized hardware): https://communities.vmware.com/t5/VMware-vSphere-Discussions/Bluescreen-booting-Windows-VM-with-latest-CU-on-AMD-with-VBS/m-p/2988792/highlight/true#M46190

Another reason I could imagine is the VM hardware version of these old Server 2019 VMs. I guess these were created on ESXi 6.7, did you upgrade the VM hardware after migrating to ESXi 7? Maybe there are some VM settings not set correctly.

We have a similar issue with our VMs migrated to ESXi 8 and VM hardware version 20 regarding the VM (VMX) setting "chipset.motherboardLayout" which has to be set to "acpi" to solve the boot BSODs.

[RuneD]

Due to issues with production, I've been forced to reinstall the servers back to ESXi 7.0u3 due to lack of response from Dell which has its direct support in the licenses.

I tried a new restore from a (sadly) patched VM that has the updates installed, prior to backing it up I made sure to disable the GPO's that enforce VBS/Secure Launch/Credential Guard, rebooting, and then taking a new backup - to no avail. I disabled everything I wrote above + stopped applying the MS Security Baseline hardening for the server that enabled Secure Launch amongst other settings.

The VM is still unbootable. It was VM hardware for 6.7, now its 7.0u2. No setting in the VMX that refers to "chipset.motherboardLayout", adding it in with "acpi" didnt help, still BSOD'ing.

[GabGo]

I assume you are referring to your Server 2019 VMs.

Can you boot the VM on an unaffected host? If yes you have to apply a group policy which explicitly disables Secure Launch. Disabling the group policy itself is not enough.

If you are not able to boot the VM anymore you have to remove the patch first as described in the link in my previous post.

The motherboard layout switch is only relevant on ESXi 8 hosts.

[RuneD]

Oh, I didnt read that correctly the first time around.

Applied a policy which actively disables Secure Launch, that did the trick. I can keep VBS/Secure Boot enabled on the VM as a big bonus - less reconfiguration.

So:
* Deploy GPO that disables Secure Launch on the working VM's running on Intel processors.
* Made sure to gpupdate /force since I was not going to wait <90 minutes for a refresh
* Backed up 
* Restored to AMD processor, now it starts and doesnt BSOD. No other steps in regards to VBS/IOMMU/Secure Boot were needed.

Massive thanks and if you ever come by Oslo I owe you a drink & pizza.

 

[K1ngb0rA]

it seem's MS at least confirmed the bug

https://learn.microsoft.com/en-us/windows/release-health/status-windows-server-2022#windows-server-2022-vms-running-on-vmware-esxi-hosts-might-fail-to-start

 

[GabGo]

I can confirm that KB5032198 fixes the Server 2022 VBS issue.

It also seems that KB5032196 fixes the Server 2019 Secure Launch issue.

[DJMcKinnon]

Any 2022 server (VM) that received the October CU will not install the November CU: KB5032198 - simply fails with 0x8024200B - event id 20.

Ran DISM health check and SFC, no corruption - has anyone else experienced this?

 

[GabGo]

We skipped the October CU in production.

On a test VM with installed October CU (KB5031364) the November CU (KB5032198) installed fine. Are all your Server 2022 VMs affected?

You could also try to clear the Windows Update Software Distribution folder which solved some update issues for us in the past: https://www.thewindowsclub.com/software-distribution-folder-in-windows

 

[mike_tp]

Sadly KB5032196 does not fix it for us, nor does setting the DVD to IDE and removing the SATA controller.

There is a quick fix for a 2019 machine that won't boot (ours reboot up to three times then go to recovery screen) in this situation.  Boot into safe mode from the recovery screen, then use regedit (or reg or powershell) to set the value HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\SystemGuard\Enabled to 0, then reboot.  Once booted ensure you have a GPO that sets Secure Launch to Disabled before you reboot it again.  No need to move off AMD host.

[bcp11]

Is this still an issue for you? Curious as I just ran into it myself with an AMD host. Server 2019 VM will not boot without Secure Launch disabled. All available updates installed including KB5032196.

[mike_tp]

Yes, still an issue.  Been going backwards and forwards with MS support, they claim Secure Launch is not supported on 2019 but I've pointed out their Security Baseline documentation for 2019 says to enable it...  

[Przemysław Kuczyński]

• Disable Virtualization Based Security i I/O MMU on VM level
• Disable Virtualization Based Security  OS level ( Computer Configuration\Policies\Administrative Templates\System\Device Guard\)
• Enable  Virtualization Based Security i I/O MMU on VM level
• Enable Virtualization Based Security OS Level (DC without credential guard )
• Weryfikacja w msinfo32