Introduction

This document is a living, up-to-date version of the performance analysis methods whitepaper.

Host memory utilization represents the entirety of memory usage due to  the VM and all tasks required by ESX Server to manage and provide  control of the VMs.  Using ESX Server's monitoring capabilities there is  no visibility into improper usage of configuration of memory within the  guest.  Continue to use traditional monitoring tools in the guest to  identify memory-hungry applications or shortages that lead to in-guest  swapping.

Navigating esxtop

As before, bring up esxtop to inspect system specifics.  Hitting the ‘m' key will display the memory counters.



Once running, the following can be observed from the esxtop report:

• The header data contains host data that impacts all VMs running on  the host.  The physical memory row (PMEM) contains the total RAM  installed on the system, the amount used by the console operating system  (COS), the memory used by the kernel (VMK), and other statistics.
• The next few rows contains host-level memory statistics for various ESX subsystems: 
  • VMKMEM: shows memory statistics for the ESX Server VMkernel
  • COSMEM: displays the memory statistics as reported by the ESX Server service console.
  • PSHARE: displays the ESX Server page-sharing statistics.
  • SWAP: displays the ESX Server swap usage statistics.
  • MEMCTL: displays the memory balloon driver statistics.

Relevant Counters

Evaluate the Data

Memory analysis on an ESX Server means not just investigation of  server-side statistics but also a solid understanding of the application  that is running in the VM.  When memory is short on the host,  ballooning and swapping may be visible in esxtop, with swapping having a  great impact on performance.  When memory is short within the VM the  guest will swap.

• How much memory are the VMs actually using?  While they may have  been allocated large amounts of memory, its likely that the OS and  applications are only using a small percentage of what the VM was  assigned.  Check the active and touched memory counters for accurate  numbers on guest memory usage.
• Is memory short in the host?  Swapping (SWW/s and SWR/s) is a  certain sign of this problem.  Heavy use of the balloon driver may also  suggest this but ballooning has a very slight impact to guest  performance.
• Can memory deficiencies be addressed through VM resizing?  Checking  memory usage of critical apps within the VMs can help inform decisions  to decrease the amount of RAM provided to those VMs.  Some operating  systems will expand to utilize all available memory at little or no  value to the application.  Reducing the memory space and correcting  over-sized caches frees up memory for other VMs.
• Is the collection of all VMs' active memory (TCHD or %ACTV)  sustaining at an amount that exceeds the total available memory?  If so,  then either more memory must be added to the host or VMs must be  migrated to another DRS cluster.
• Are the guests swapping?  If the VM has been sized with too little  memory then the guest OS will swap inside the VM.  This will appear to  ESX Server as any other disk activity but should be investigated and  solved with traditional OS analysis tools.
• Can NUMA migrations (NMIG) be seen on the system?  NMIG reports  total migrations since the VM has been powered on.  If this number  continues to climb then the VM is being migrated from node to node which  most certainly degrades performance.
• Does the amount of memory located on a remote NUMA node (NRMEM)  remain at a non-zero number?  This may be a sign that the VM has been  sized to exceed the memory of a single NUMA node.  If the VM is using  more memory than fits on a single node, some of its memory is certain to  be located on a remote node.  Remote memory access is quite slow  relative to local memory access.

Correct the System

The prescriptive advice for memory shortages is fairly simple: use less  memory or buy more.  The following recommendations are variations on  this theme:

1. Verify that VMware Tools has been installed on every VM on the  system and that the memory balloon driver has not been disabled.  (The  balloon driver is always on by default and disabled manually through  text-based advanced configuration in extremely rare cases.)  When  provide the ability to balloon memory within the guests, ESX Server is  able to take memory from VMs that are not using it and make it available  to those that do need it.
2. Provide more memory to the DRS cluster.  As total resources go up,  VirtualCenter will balance VMs across the cluster so VMs that need the  memory are able to get it.
3. Set memory reservations to minimally provide the amount of memory  required of the OS and critical applications.  This will allow for  sustained, fast access for critical code and provide hints to  VirtualCenter for optimal VM positioning across the DRS cluster.
4. Make sure the amount of memory used by the VMkernel to maintain the  VMs is acceptable.  This value, reported for each VM with the overhead  counter (OVHD), is dependent on the memory size of the VM, the number of  vCPUs provided to it, and whether or not it is executing a 64-bit OS.   Fewer VMs on the host, fewer aggregate vCPUs, and lower precision OSes  (32-bit as opposed to 64-bit) will lower this number.  Reducing any of  these in the cluster will free up resources for every VM in the cluster.
5. Size VMs on NUMA systems to guarantee that each VM's memory will fit  on a single node.  This means either decreasing the memory allocated to  a VM or increasing the node memory size.
6. Size guests appropriately according to their needs.  For example:
   1. Depending on the access pattern of the data, databases may not  benefit from the last doubling of cache size.  Experiment with smaller  cache sizes and see if performance drops.  If not, decrease the VM's  available memory so it can be used by other VMs.
   2. Check the guest OS's statistics for in-guest swapping.  Provide  memory as its needed and pay attention to esxtop statistics to see if  the additional memory provided generates a new bottleneck in the host.

Understanding Page Sharing

One cannot fully optimize an ESX Server's memory without understanding  the performance implications of page sharing.  VMware's page sharing  algorithm was presented at EMC World 2008 as resulting in a 2% increase  in CPU load.  But the benefits of page sharing have been demonstrated to  provide overcommitment of memory safely to 2X and beyond.

The value of page sharing can be seen int the following counters:

Note that missing counters can be calculated using the other two.   Shared memory minus shared common memory equals shared savings.

References

The top-level Performance Monitoring and Analysis paper.

The esxtop Performance Counters index.

The Understanding VirtualCenter Performance Statistics page.