Citrix XenApp slowness on ESX 4

[rjaudon]

All,

I need some help with a some CPU constraint issues( i think) diagnosis.  I have a client with a Citrix Xenapp farm (5 desktop servers/9 Application servers) spread accross 12 ESX hosts running ESX4.0.  All Citrix servers, desktop and application servers have 2 cCPU's allocated and 54% of the VM's running on the ESX hosts are allocated 2 vCPU's.  The problem we are running into is that the Citrix Application servers are running(crawling) when trying to access the published applications.  I personally think it is a multiple VCPU issue but I would like to get an opinion on if my findings are true.

Below are some screen shots from the VSphere client and esxtop readings.

My questions are what are the best practices for Citrix on ESX? 

Is it typical to have multiple VCPU's per server in a Citrix environment?

For ESXTOP readings should I be looking at the %RDY and overall performance of the host? 

[M0rph77]

quick wins;

1) check for ballooning.. set reservations on the citrix vm's

2) if ready times are high (> 10-20%), try going for more 1 vcpu vm's and less users per server.. and talk to boss for more esx hosts

3) do you user xeon 5500 or higher? if so; force 2nd generation hardware assist, if not; think about upgrading to sandy bridge architecture..

couple of best practice sites (think/plan about best practices of others);

http://www.vmware.com/files/pdf/solutions/vmware-citrix-xenapp-best-practices-EN.pdf

http://geekswithblogs.net/eknowlogy/archive/2007/03/05/107950.aspx

http://virtrix.blogspot.com/2007/03/vmware-best-practices-for-deploying.html

http://ict-freak.nl/2008/07/02/vmware-citrix-on-vmware-recommendations/

[rjaudon]

@M0rph77,

Thank you for your help and time.

Thanks for the links as well. I am looking at them now.

To start off ...

1) check for ballooning.. set reservations on the citrix vm's 

***** See below image....Is this evidence of ballooning?  I'm sorry....still trying to wrap my head around the environment.

2) if ready times are high (> 10-20%), try going for more 1 vcpu vm's and less users per server.. and talk to boss for more esx hosts

I have checked on for the %RDY metrics and just need some clarification.  Please look at the screen shot below and for the circles should I concentrate on the ID 1 Idle when looking at %RDY?  When looking at the screen shot below is the CPU load OK?  For the VM CWSAPP# is the %RDY something to worry about?

3) do you user xeon 5500 or higher? if so; force 2nd generation hardware assist, if not; think about upgrading to sandy bridge architecture..

I am still researching this but here is the summary from one of the hosts...

[MKguy]

Just a few quick notes from what I can see in your screenshots:

- press Shift-v to exclude host processes from the esxtop list so you only see VM-metrics and post some screenshots again

- check (advanced) performance graphs from the vSphere Client for a historical view on metrics like swapping, ballooning, CPU %RDY or disk latency

- from the esxtop CPU screenshot, ready-times don't seem to be your issue

- from the esxtop memory screenshot, only one VM appears to be ballooning (as seen with MCTLSZ) and there is slight swapping going on (probably only one VM)

- check Duncan's compilation of important esxtop metrics here: http://www.yellow-bricks.com/esxtop/

From what I saw, I actually don't think CPU or memory is the problem. What's missing in the big picture are important storage-related metrics. Check latency metrics on your SAN or from esxtop/GUI graphs.

Also:

- consider upgrading to ESX(i) 4.1 (or ESXi5)

- Like already suggested, consider using newer hosts with Intel EPT/AMD RVI CPUs (e.g. Xeon 55xx). These provide memory assisted virtualization enhancements which can improve performance significantly.