Great to hear that it worked with the powered off VM! Note that the PCI device is not located within the host, but rather in the VM settings.
If the other VM snapshot works on another host, it's possible that the host may have some faulted services. You can test by vMotioning each VMs to the opposite host and see if it works there.
Please also note that snapshots should not be used as long-term backups and should be deleted within 72 hours to avoid performance issues, especially with the large storage size of the VMs. For backups that need to remain more than 72 hours, a proper backup solution should be used, such as Rubrik, Veeam etc.
https://kb.vmware.com/s/article/1025279