Hello Vinay,
For performance test against virtual services, you need to look at multiple things. Having required amount of memory allocation for VSE component is definitely important, and this can be tracked to see how much memory is being used by VSE process during the performance test. However, if the virtual service does not have the right capacity allocated, you can see performance issues.
We need more details about environment etc. to understand the issue here. We can connect offline to discuss more on this.
Thanks,
Surya