Problem with few logical processors

[Sebastian Ulatowski]

Hi, 

I have a strange problem. I noticed latelly, that my infrastructure is working somehow not as I assume should work. ESXTOP doesnt show any issues, %RDY about 2,3, %CSTP of coz 0, latency also ok, around 1, sometimes peak abount 12. But I looked at esxi monitor, on CPU Usage in % and there is something strange, 4 (of 32) logical processors have utilization 0. 

is this a correct ? I think not. 

What do you think guys ? 

Regards, 

Sebastian  

[StephenMoll]

I think people will need to know more about your setup to be able to help.

My first thought would be that it is something to do with the NUMA architecture of your system, in combination with the specifications of your VMs in terms of vCPU counts and vNUMA settings if any.

What server is it?

What pCPUs are you using?

How many sockets?

How much memory?

Are all the memory channels populated?

How many VMs?

How many vCPUs do these VMs have?

and so on...

[Sebastian Ulatowski]

Hi, 

Thanks for response. 

ad1. ProLiant DL380 Gen10

ad2. now I look at this and maybe there is no problem :) I have a ratio VMs per core 0,63. 

ad3. 2 sockets, 8 processors per socket, logical 32 

ad4. 194 GB RAM

ad5.  I dont think so

ad6. about 12 VMs per host 

ad7. summary about 31 vCPU. 

Regards, 

Sebastian

[Casey]

Hi Sebastian,

It sounds like you're encountering an unusual issue with your ESXi host's CPU usage. Based on the screenshot and your description, it appears that 4 out of your 32 logical processors consistently show 0% utilization. This behavior is not typical and could indicate several underlying issues. Here are a few steps and considerations that might help you troubleshoot the problem:

1. Check CPU Affinity Settings: Ensure that no virtual machines (VMs) or processes have been set with specific CPU affinity settings that might limit them to certain CPUs, leaving others underutilized.

2. Validate Power Management Settings: Sometimes power management settings can cause CPUs to be in a low-power state, appearing as if they have 0% usage. Check the BIOS/UEFI settings for power management options and ESXi power management policies.

3. Inspect Hyperthreading: Verify if hyperthreading is enabled. Sometimes logical CPUs (hyper-threads) might not be utilized effectively if hyperthreading is disabled or if the workload does not efficiently use hyper-threaded cores.

4. Review ESXi Scheduling: The ESXi CPU scheduler might sometimes appear to leave some logical CPUs idle if the overall workload is low or if other CPUs are handling the tasks more efficiently. Check if the VM distribution across the CPUs is balanced.

5. Monitor for Hardware Issues: While unlikely, hardware issues can sometimes cause certain CPUs to be underutilized. Use hardware diagnostics tools to rule out any underlying hardware problems.

6. Examine Resource Allocation: Make sure that resource allocation settings (such as reservations, limits, and shares) for the VMs are not causing the observed behavior.

7. Update ESXi: Ensure that your ESXi host is running the latest updates and patches. Sometimes bugs can lead to abnormal behavior in CPU utilization.

8. Check for CPU Scheduling Anomalies: Investigate if there are any CPU scheduling anomalies or bugs known for your ESXi version. Look into VMware's knowledge base and community forums for similar issues.

Please respond back and let us know if that worked out for you. Also include as much details as possible :D 

------------------------------
Love for Virtual Infrastructure!

Please mark as Recommended if this solved your problem. This will help the community find solutions to their problems as well.

More Logs, More Info, More Screenshots = Easier To Help You
------------------------------------------------------------
VCP-DCV 8.x | VCP-DCV 6.x | CompTIA A+, Net+, Sec+, Proj+, Server+ | MCSE | MCSA | MCITP
------------------------------

[Sebastian Ulatowski]

Hi, 

Thanks for answer :) 

I will answer your questions: 

ad1. no affinity is set

ad2. power is maximum performance

ad3. Hyperthreading is enabled. 

ad4. And I think it could be this. As I inspected this more deeply it looks like, there is no so much workload ;) overall ratio vms per core is less then 1 :) 

ad5. We will try to inspect this additionally using HPE Diagnostics tools. 

ad6. I have only reservations on DataCore vms :) 

ad7. Its not possible :D 

ad8. thx ;) 

Thank you for posts :) 

Regards, 

Sebastian

[Casey]

Based on your observations and the provided performance chart, it seems unusual for some of your logical processors to show 0% CPU usage consistently. This could potentially indicate an issue with CPU scheduling or resource allocation within your ESXi environment. Here are a few steps to help diagnose and address the problem:

Troubleshooting Steps

1. Check CPU Affinity: Ensure that no VMs have specific CPU affinities set that might be restricting their execution to certain CPUs only, causing other CPUs to be underutilized.

   • In vSphere Client, go to the VM settings, select the Options tab, and check the CPU Affinity settings.

2. Review DRS Settings: If you are using Distributed Resource Scheduler (DRS), ensure that it is set to automatically balance the load across CPUs.

   • Navigate to your cluster settings in vSphere Client and review the DRS settings.

3. Examine Resource Pools: Check if resource pools are configured and if they have any reservations or limits that might be affecting CPU allocation.

   • In vSphere Client, go to Resource Pools and review the configurations.

4. Check Power Management Settings: Ensure that your host's power management settings are not causing some CPUs to be in a low power state.

   • Access the host settings in vSphere Client and navigate to the Power Management settings.

5. Monitor VM Behavior: Use esxtop to monitor the behavior of VMs on the host. Pay attention to %RDY (CPU Ready), %CSTP (Co-Stop), and %USED (CPU Usage).

   • Connect to the ESXi host via SSH and run esxtop.
   • Press c to switch to the CPU screen and observe the values for each VM.

6. Review ESXi Logs: Check the ESXi logs for any warnings or errors that might indicate hardware issues or other anomalies.

   • Use vSphere Client to access the host logs or connect via SSH and check the /var/log directory.

Addressing Potential Issues

• Adjust CPU Affinity: If any VMs have specific CPU affinities set, consider removing them to allow the ESXi scheduler to balance the load automatically.

• Optimize DRS Settings: If DRS is set to manual or partially automated, change it to fully automated to ensure optimal load balancing.

• Reconfigure Resource Pools: Ensure that resource pools are not overly restrictive and allow sufficient CPU resources to be allocated as needed.

• Modify Power Management: Set the power management policy to "High Performance" to prevent CPUs from entering low power states.

By following these steps, you should be able to identify and rectify any issues causing the uneven CPU utilization. If the problem persists, providing more detailed logs and configurations might be necessary for further analysis.

Let us know how it works out! Be sure to include logs, screenshots, and any relevant details. 😊

------------------------------
Love for Virtual Infrastructure!

Please mark as Recommended if this solved your problem. This will help the community find solutions to their problems as well.

More Logs, More Info, More Screenshots = Easier To Help You
------------------------------------------------------------
VCP-DCV 8.x | VCP-DCV 6.x | CompTIA A+, Net+, Sec+, Proj+, Server+ | MCSE | MCSA | MCITP
------------------------------