vcpu and numa

[Bab]

Dear all

HI

i have a question about NUMA and Vcpu for create virtual machine i have read many articles about NUMA i saw this article:

How to check if NUMA is enabled on ESX hardware?

but when we see esxtop NHN parameter  for some virtual machines we see for example 0,1,2  and some 1   now i want to know when NHN for one vm is 0,1,2   is that mean this vm use from remote memory ?

when NHN for a vm is hust 1 or 0 or 2 is that mean this vm ust use local memory ?

Best regards

Babak

[Techie01]

You should be looking into the following fields to see if the memory access is local or remote

NRMEM (MB)	Current amount of remote memory being accessed by VM
NLMEM (MB)	Current amount of local memory being accessed by VM
N%L	Current percentage memory being accessed by VM that is local

NHN represents the node at which the VM running. If it is a single number, the vm is placed in a single numa node. If there are more multiple numbers, the vm configuration is such that, it cannot be fitted into single numa node

[Bab]

‌so thanks

but i have 2 question

1- how do i have to select sockets and core per socket if i want use just local memory?

2- if one of my virtual machine's NHN is more than one object is not that mean it use also from remote memory?

3- i read in many documents about local memory performance that if our virtual machine just use from local memory the latency in vms that have programmes which use multi vcpu will be so decrease and performance will be so increase is that correct? If yes how can select socket and core per socket for best performance

for example i have attached a pic that show 2G memory assigned to remote memory and 22G assigned to local memory cpu config for this machine is such as below :

Number of virtual sockets = 2

Number of core per sockets = 1

and my esxi cpu host  is such as this :

Processor Sockets =2

core per socket = 8

logical processor = 32

now how can change my vm cpu that it use just local memory ?

best regards

babak

[Bab]

Please help me

[Bab]

now i am realy confused

my server model is HP Proliant DL 380P G8

but now i don't know how much numa i have ?

i have readen many articles but now confused and don't know if i want create a new virtual machine what is best practice for choose cpu and core per cpu ?

Is best practice to choose 1 core per cpu for all vms ?

Best regards

Babak

[engineer4kailas]

Host memory ?

Memory Allocated to VM?

Run command to know NUMA on Host  " esxcli hardware memory get"   ?

this info is required to comment further

[Bab]

no i have read many article for example this article:

https://support.rackspace.com/how-to/numa-vnuma-and-cpu-scheduling/

in a part of this article said :

set the sockets to the number of vCPUs needed, and set the cores per socket to 1. That will let the underlying hardware determine the vNUMA nodes properly.

1 - is that correct ? set socket to number of vcpu and core = 1 ?

2- how can enable vnuma on my virtual machine?

3 - how can increase or decrease numa on vm ?

4 - what is better ? has a vm with more vnuma or have a vm with one vnuma?

Best regards

[hussainbte]

Find the answers to your questions.

1)It is recommended to keep 1 cores per socket and increase the number of sockets to allocate more CPUs to VMs.

2)If a VM has more than eight vCPUs, vNUMA is automatically enabled.

3) Explain the question

4)from the basic understanding that NHN reports the home node where the VM first booted up. this can change if Vm migrates.

A VM running within one NUMA node is better than a VM Spanned across more NUMA nodes.

[Bab]

so thanks

1 - can you say clearly to me what is mean when my vm numa is 2 or 3 ? NHN (0,1,2,3)

2 - how can understand my esxi host how many numa has?

3 - if my vms has less that 8 vcpu  thta vm will automatically just used from local memory ?

[hussainbte]

1- Could you elaborate or share a screenshot of exact setting/option

what do you mean when you say vm numa is 2 or 3

2) to answer that please share output of below command

esxcli hardware memory get

[Bab]

1 - if my vms has less that 8 vcpu  thta vm will automatically just used from local memory ?

2 - please see attach pic

i have 8 virtual machine on my esxi host attached my esxi host info and get esxtop my test 2 vm has number of virtual socket =11 and

number of core per socket = 1

and virtual machine test has number of virtual socket =21

and

number of core per socket = 1

now which of them better what does from NHN here for test2 is 0,1 and for test is 0,1,2,3,4,5,6,7,8,9,10,11,12,13

what does that means ?

[hussainbte]

NHN for test2 0,1 means it is running across 2 NUMA nodes.

NHN for test1????

could you share the output of below command

esxcli hardware memory get

[Bab]

[root@localhost:~] esxcli hardware memory get

   Physical Memory: 103042732032 Bytes

   Reliable Memory: 0 Bytes

   NUMA Node Count: 2

[root@localhost:~]

please see previous post test virtual machine and test2 virtual machine

i want to know when you say :  NHN for test2 0,1 means it is running across 2 NUMA nodes.

what does means ? is that means my virtual machine use from both physical cpu on my esxi host ?

best regards

[hussainbte]

yes it means that the vm test2 is spanning across 2 numa nodes.

you have 2 numa nodes in the ESXi host and based on the CPU configuration it seems that each NUMA node has 10 cores.

so when a VM is assigned more than 10 vCPUs it will span across 2 numa nodes.

I still don't understand why the test vm shows 0,1,2..13.

anything different configured on the test vm.?

[Bab]

thanks

realy don't  specific config on test virtual machine just create this virtual machine without os and question is :

1 - when you say : yes it means that the vm test2 is spanning across 2 numa nodes.

is that your means test2 virtual machine use from both of physical cpu esxi host ?

[hussainbte]

Yes it does.

[Bab]

but previously you said better use one numa instead of 2 numa is that true?

1- why 2 numa is not better than 1 numa ? ( 2 numa means our virtual machine use 2 physical esxi host )

[hussainbte]

in your case if you have to allocate more than 10 vCPUs to the VM(if you actually need it) there is no way you can make sure the VM will run in single NUMA node because each NUMA node only has 10 CPUs.

of course it is good for the VM to run in a single numa node because it can benefit from the shared L3 cache.

but your requirement of the test2 vm is 11 vCPU so a single NUMA node cannot accommodate it.

Try reducing the number of vCPU to 10 and you will see that the NHN will change.

when we talk about NUMA we are talking about make sure the vCPUs of the VM fall under single NUMA node.

ESXi is NUMA aware and that it what it will do by default.

but if the vm is big like test2(more than the NUMA node capacity) the ESXi host will have no option but to schedule the VM across 2 sockets(2 NUMA nodes)

[Bab]

1-means from numa is the number of  physical cpu my virtual machine used ?

2 - when say i have 4 numa on esxi host means my esxi host has 4 physical cpu ?

[hussainbte]

If you could explain it me your understanding of NUMA. it will be easier for me to answer the question.

What is your understanding of NUMA..?

[hussainbte]

please read page 22 on the linked guide.

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/vmware-perfbest-practices-vsphere6-0-white-p…

[Bab]

numa is an architecture for cpu means we have a virtual machine with 4vcpu and my esxi host  hast 4 physical cpu my virtual machine assign to one of physical cpu and use from local memory not use remote memory