The suggestion to use CUDA samples or the Nvidia demo programs inside the VM was to verify that the Telsa P4 works inside the VM. From all the screenshots, it looks like it is working but the only thing lacking was proof an application being able to use it and evidenced by the GPU Activity.
Doing a search of XProtect Smart Client, I would guess it depends on the version of the software that you use. From what I can see XProtect H264 encoding/decoding is using Intel QuickSync (thus requires an Intel GPU not Nvidia).
SmartClient 2017 Documentation
https://www.milestonesys.com/files/General/MilestoneGuides/MilestoneXProtectSmartClient_HardwareAccelerationGuide_en-US.pdf#page=4&zoom=auto,-274,767
SmartClient 2016 Demonstration Video
https://youtu.be/_D5Rrw_mtVU?t=3m47s
The thing I found involving Nvidia is a press release from Milestone. The press release is vague. It doesn't say whether it is fully supported and availability. It doesn't say either exactly what the Nvidia GPUs will be used for. Is it for more than H264/H265 such as facial recognition?
https://www.milestonesys.com/press-releases/20170403-nvidia/
Intel QuickSync requires an application like XProtect Smart Client to use Intel's Media SDK for H264/H265 processing. The same thing with Nvidia as well requiring the application to use the Nvidia Video Codec SDK. Unless the version of the XProtect Smart Client you have explicitly supports and configurable to use Nvidia GPUs for H264/265 encoding/decoding, it would ignore the Tesla P4 altogether (regardless if it is inside a VM or not).
So you have to check with Milestone as to availability/support of Nvidia GPU with the version of XProtect that you have.
As to 2 cards with 2 different drivers, it works. There are many laptops with both Intel and a discrete graphics such as AMD or Nvidia. The laptop that I am typing this reply from has both Intel HD530 and Nvidia GTX 960M enabled. For desktops, it also works. I used to have a desktop that had both Intel HD4600 and Nvidia GTX 950 active at the same time. Even with the VM that you have created has a VMware SVGA 3D adapter (albeit virtual) and the Nvidia Tesla P4 active at the same time. DirectX 12 even allows any application that uses DX12 to be able to have both Nvidia and AMD GPUs in the same system.
The issue really is whether the OS and/or application(s) are written to explicitly take advantage of specific features of either or both cards. For example, a facial recognition algorithm could take advantage of the GPU compute capability of an Nvidia GPU if it was written using CUDA API while having another GPU for display purposes.