Thanks for the reply.
It's the underlying esx-i performance that I'm considering. The host will be presented with whatever ESX can provide through it's abstracted drivers.
My suspicion is as you say the guest OS won't utilise any video memory itself but use system memory but potentially you have a capable GPU and optimised memory to help the VM OS provide video to the hosts as part of it's VM SVGA driver.
For example Server A has the intel basic graphics with zero video memory whilst server B has an ATI PCI-E card with 512 MB (unlikely I know)
I was simply wondering if in Vsphere 4 they may be being that bit cleverer and able to use the extra resources of server B.