Hi,
I have esx server i, 3.5.0 hypervisor on which i have installed ubuntu lts 10.04.1 (32bit).
My server specs are:
ibm eserver 336, cpu 3.2 ghz, Intel xeon with hyperthreading, 2gb memory.
I am having a problem where a program which is just calling nanosleep to sleep for 1msec has very high cpu usage in kernel space, upto 75% cpu is used.
I have 50 threads in the program, which are each calling nanosleep of 1 msec and looping infinitely. MY user space cpu usage is coming out as 0%. I am using nmon to display this information. There is no problem when i am not using vmware based system. I have same ubuntu installed on a pc as well.
My system has tsc based tickless high resolution timer.
I think the LOC interrupts are the reason for 75% cpu usage which is in kernel space but don't know how can i not have those interrupts? Also the timer on demand tsc timer behaviour should be same, unless nanosleep modifies the aperodic timer interval.
Can somebody from vmware support or community help me in this matter, this happens only on vmware?
Here is the interrupt situation for 10 sec interval, this is before my nanosleep based sample code.
---------------------------------------------------
Ubuntu 10.04.1 LTS \n \l
Linux ubuntu02 2.6.32-24-generic-pae #39-Ubuntu SMP Wed Jul 28 07:39:26 UTC 2010 i686 GNU/Linux
Fri Jan 28 11:51:46 GMT 2011
Fri 28 Jan 2011 11:51:51 GMT -0.751050 seconds
Fri Jan 28 11:51:47 GMT 2011
CPU0
0: 179 IO-APIC-edge timer
1: 94 IO-APIC-edge i8042
6: 3 IO-APIC-edge floppy
7: 0 IO-APIC-edge parport0
8: 5 IO-APIC-edge rtc0
9: 0 IO-APIC-fasteoi acpi
12: 376 IO-APIC-edge i8042
14: 57 IO-APIC-edge ata_piix
15: 0 IO-APIC-edge ata_piix
17: 64728 IO-APIC-fasteoi ioc0
18: 601489 IO-APIC-fasteoi eth0
NMI: 0 Non-maskable interrupts
LOC: 495881209 Local timer interrupts
SPU: 0 Spurious interrupts
PMI: 0 Performance monitoring interrupts
PND: 0 Performance pending work
RES: 0 Rescheduling interrupts
CAL: 0 Function call interrupts
TLB: 0 TLB shootdowns
TRM: 0 Thermal event interrupts
THR: 0 Threshold APIC interrupts
MCE: 0 Machine check exceptions
MCP: 518 Machine check polls
ERR: 0
MIS: 0
CPU0
0: 179 IO-APIC-edge timer
1: 94 IO-APIC-edge i8042
6: 3 IO-APIC-edge floppy
7: 0 IO-APIC-edge parport0
8: 5 IO-APIC-edge rtc0
9: 0 IO-APIC-fasteoi acpi
12: 376 IO-APIC-edge i8042
14: 57 IO-APIC-edge ata_piix
15: 0 IO-APIC-edge ata_piix
17: 64735 IO-APIC-fasteoi ioc0
18: 601507 IO-APIC-fasteoi eth0
NMI: 0 Non-maskable interrupts
LOC: 495881461 Local timer interrupts
SPU: 0 Spurious interrupts
PMI: 0 Performance monitoring interrupts
PND: 0 Performance pending work
RES: 0 Rescheduling interrupts
CAL: 0 Function call interrupts
TLB: 0 TLB shootdowns
TRM: 0 Thermal event interrupts
THR: 0 Threshold APIC interrupts
MCE: 0 Machine check exceptions
MCP: 518 Machine check polls
ERR: 0
MIS: 0
Fri Jan 28 11:51:57 GMT 2011
Fri 28 Jan 2011 11:52:02 GMT -0.940125 seconds
Fri Jan 28 11:51:58 GMT 2011
-------------------
252 interrupts in 10 second or 25 interrupts per second.
Now when i run it after with my nanosleep based program.
Ubuntu 10.04.1 LTS \n \l
-------------------------------------------
Linux m4mubuntu02 2.6.32-24-generic-pae #39-Ubuntu SMP Wed Jul 28 07:39:26 UTC 2010 i686 GNU/Linux
Fri Jan 28 12:01:00 GMT 2011
Fri 28 Jan 2011 12:01:05 GMT -0.670739 seconds
Fri Jan 28 12:01:01 GMT 2011
CPU0
0: 179 IO-APIC-edge timer
1: 94 IO-APIC-edge i8042
6: 3 IO-APIC-edge floppy
7: 0 IO-APIC-edge parport0
8: 9 IO-APIC-edge rtc0
9: 0 IO-APIC-fasteoi acpi
12: 376 IO-APIC-edge i8042
14: 57 IO-APIC-edge ata_piix
15: 0 IO-APIC-edge ata_piix
17: 64814 IO-APIC-fasteoi ioc0
18: 603351 IO-APIC-fasteoi eth0
NMI: 0 Non-maskable interrupts
LOC: 495916276 Local timer interrupts
SPU: 0 Spurious interrupts
PMI: 0 Performance monitoring interrupts
PND: 0 Performance pending work
RES: 0 Rescheduling interrupts
CAL: 0 Function call interrupts
TLB: 0 TLB shootdowns
TRM: 0 Thermal event interrupts
THR: 0 Threshold APIC interrupts
MCE: 0 Machine check exceptions
MCP: 520 Machine check polls
ERR: 0
MIS: 0
CPU0
0: 179 IO-APIC-edge timer
1: 94 IO-APIC-edge i8042
6: 3 IO-APIC-edge floppy
7: 0 IO-APIC-edge parport0
8: 9 IO-APIC-edge rtc0
9: 0 IO-APIC-fasteoi acpi
12: 376 IO-APIC-edge i8042
14: 57 IO-APIC-edge ata_piix
15: 0 IO-APIC-edge ata_piix
17: 64817 IO-APIC-fasteoi ioc0
18: 603365 IO-APIC-fasteoi eth0
NMI: 0 Non-maskable interrupts
LOC: 496015690 Local timer interrupts
SPU: 0 Spurious interrupts
PMI: 0 Performance monitoring interrupts
PND: 0 Performance pending work
RES: 0 Rescheduling interrupts
CAL: 0 Function call interrupts
TLB: 0 TLB shootdowns
TRM: 0 Thermal event interrupts
THR: 0 Threshold APIC interrupts
MCE: 0 Machine check exceptions
MCP: 520 Machine check polls
ERR: 0
MIS: 0
Fri Jan 28 12:01:11 GMT 2011
Fri 28 Jan 2011 12:01:16 GMT -0.627075 seconds
Fri Jan 28 12:01:12 GMT 2011
-----------------------------------------
99414 interrupts in 10 seconds, almost 9941 interrups per second.
Regards,
rui