Help understanding Performance files from VC.

[sergio1228]

We have 3 ESX 3.5 hosts and we seem to keep having performance issues. We had the network created for us so I'm not sure exactly what I'm looking at. I have looked at the best ways to monitor our esx servers but the only method that I was able to understand was the option in Virtual Center > Right click host > Report Performance. Unfortunately I do not understand any of the numbers and why we would be having performance issues.

We have 3 Dell poweredge/ 2 CPU 2.5 Ghz quad core/16 GB RAM/ FC San

They all seem to have issues, but currently I have one host that is showing memory concerns and we have no users on the servers.

Current VM's on system:

Domain Controller/Terminal Server/Exchange with 100 email accounts.

I have attached the files, please let me know if any more information is needed. Again, I'm a true beginner with vmware so I would appreciate any thorough explanations.

[tfskelly]

Hi Sergio,

The performance graphs provided only show an hour of time but it looks like you may have a CPU or memory problem. We need to isolate the problem better than those graphs can show us. Please answer these questions:

• What exactly makes you think you have performance problems?

• Is the performance problem happening with multiple guest systems, or just one?

• How are your VM's spread across your hosts?

• Is DRS turned on?(Right click the cluster, click "Edit Settings" to find out")

I need you to look a the individual virtual machine's performance. An easy way to do this is select your primary cluster in the "Hosts and Clusters" view, and then click on the virtual machine tab in the right pane of the VI client Look at the "Host CPU" column. You will easily be able to see what virtual machines are using a lot of CPU/memory at that specific time. Whenever you start to notice performance problems on a virtual machine, open the virtual machines tab. From the list of virtual machines in that view you should be able to answer these questions:

• are any VMs using 100% of their allocated memory?

• Are any VM's using significantly more CPU than others?

• What VM's have more than 2-3GB memory configured?

Let me know what you find out!

Kelly Burton

[sergio1228]

Hello Kelly,

Thank you for taking the time to

help me with our network.

• What exactly makes you think you have performance

problems?

Throughout the day, we seem to have stability issues. It's not like there is a specific time this happens,

there have been times when we only had a few users and no other jobs running

and everything was slow. I look at the Host usage and the memory usage

and cpu are high and there are only a few users on the network. At other

times, we can have 20 users on the host and the network is running fine, just

does not make sense.

• Is the performance problem happening with multiple

guest systems, or just one?

We only really run Terminal Servers that users access directly. We have Exchange/Sql server/App server/File

Server, but we have 4 TS servers that our users connect to do their jobs.+

• How are your VM's spread across your hosts?

We have 3 host, We have SQL on one, Exchange on another and then with DRS we allow VMWare to decide where to put the rest.

• Is DRS turned on?(Right click the cluster, click

"Edit Settings" to find out")

YES

*I will work on the below exercise

tomorrow during business hours to get the best data for your questions.+*

I need you to look a the individual virtual machine's performance. An easy way

to do this is select your primary cluster in the "Hosts and Clusters"

view, and then click on the virtual machine tab in the right pane of the VI

client Look at the "Host CPU" column. You will easily be able to see

what virtual machines are using a lot of CPU/memory at that specific time.

Whenever you start to notice performance problems on a virtual machine, open

the virtual machines tab. From the list of virtual machines in that view you

should be able to answer these questions:

• are any VMs using 100% of their allocated memory?

• Are any VM's using significantly more CPU than others?

• What VM's have more than 2-3GB memory configured? - I

can answer this one now: 4 Terminal Servers with 4 GB allocated, Exchange

and SQL have 8 GB allocated.

Thank you again for helping me out, as soon as I have some good data, I'll post

the answers to the other remaining questions.

Cheers,

Sergio

[Texiwill]

Hello,

Moved to the vCenter Server Forum.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast

[tfskelly]

Hi Sergio,

This problem sounds very similary to a problem I had a while back. Check out this KB article and see if your environment qualifies:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003638

If that isn't the problem, then post some more detailed performance reports. Maybe one for each esx host looking at 1 week's worth of time? Do this by right clicking on the host and clicking "Report Performance". Change the start date to 1 week ago. Gather data for the CPU, Network, and Memory counters. Post those and I'll take a look and see if I can't identify a pattern.

Later!

Kelly Burton

[sergio1228]

Thank you for the link. I will see what I can't figure out and also create the weeks performance records.

I have attached a screen shot of our VM's.

Thanks again,

Sergio

[tfskelly]

Hey Sergio,

The pictures really help! Could you one more picture with these columns visible:

• Name

• CPU Count

• Host

• Host CPU - MHz

• Host Mem - MB

• Guest Mem - %

• Memory Size - MB

To change the visible columns, right-click and place or remove checkmarks next to the column names.

Thanks!

Kelly Burton

[sergio1228]

Here is the screen shot you requested. I am still working on some other possiblities, but one thing that seems to be helping, is moving users to Thin Clients. I still think there are some configuration issues, but goiing to work through the KB and hopefully figure things out.

Thanks for all your help,

Sergio

[tfskelly]

Sergio,

How many CPUs/Cores do each of your ESX hosts have?

Right now there are 10-14 vCPUs on each host. If you only have 8 cores, and your VM's are busy(and they are), then the virtual machines are constantly competing to use 4 cores at one time. This can increase the cpu-wait time and your virtual machines will act slow while they are waiting for CPUs to become available. Rember - the virtual machine thinks it has 4 real CPUs. Therefore, ESX either schedules 4 real cores, or none, for every thread that VM executes. Virtual Machines in ESX often perform best when using only 1vCPU.

Reduce the vCPUs on all of your virtual machines to 2. The only machine on your infrastructure that MIGHT need 4 vCPUs is the SQL server, and even that is doubtful. Regardless, unless you have 16 cores per ESX server, you are currently having CPU contention problems. That is why the machines seem slow.

Reducing the domain controllers and exchange server to1vCPU is even better, but that is a somewhat dangerous process and you'll want to take a snapshot before you do. Read this discussion for more information about changing from a multiprocessor HAL to a uniprocessor HAL:

• http://communities.vmware.com/thread/84611

• http://communities.vmware.com/thread/84611

Hope that helps!

Kelly Burton

[sergio1228]

Thanks Kelly,

Each server is dual processor with quad core processors. I have been busy trying to get some of our physical servers ready to make the migration to a virtual state at one of our offices. I'll be back working on the ESX servers next week, so hopefully I'll be able to report back better news.

Thanks again,

Sergio

[sergio1228]

Well, we have been busy trying to make sure all aspects of the network work more reliable. After a lot of processor checking, we removed our End-Point A/v(Symantec) and Yahoo messenger from our servers and performance shot up immediately. We took off End-point first and that made a world of a difference and then yahoo and that helped even more. I'm goinng to see about switching some of the TS servers to 2 vCPU's and see how that goes just for testingn purposes.

Thanks again for everything Kelly.