High cpu usage on ESXi host

[Salli]

Just wondering, if it's normal to have ~25% cpu load (1cpu, quad core) on host when all guests are powered off :smileyshocked:

vSphere client on the host shows vmkapimod consuming most of the cpu-time (90% of the 25%)

Platform consists of components that are all fully supported by vmware.

[JimKnopf99]

Hi,

what happend when you restart the Host?

Maybe you can try this first.

1. Go to the ESXi console and press alt+F1

2. Type: unsupported

3. Enter the root password(No prompt, typing is blindly)

4. Restart the management service “/sbin/services.sh restart”

Frank

If you find this information useful, please award points for "correct" or "helpful".

[Salli]

As I didn't have local access to server, I rebooted it to maintenance mode ran all patches to U1 (3) and load looks much better for now; vmkapimod isn't taking any resources. I'll keep monitoring and see if it comes back (server was on for just ~8hours when the load issue hit last time) - hypervisor reboot is hardly a solution to anything :smileylaugh:

[tinipop]

Hi!

My system Tyan S7012GM4NR / 2 x Xeon 5630 / 32Gb DDR3ECC / 6x73Gb (Raid5) / AXX6DRV3GEXP / Adaptec3405 / BBU / 400W

On this server install ESXi 4Update1 (on USB stick)

After one day of good work begins high CPU usage on vmkapimod.

After reboot the server in Bios show the message BMC startap... (and this function iKVM IPMI 2.0 don't started, in BIOS options I can see that BMC status - "don't work"), after Adaptec bios initialization show NVRAM error. ESXi server start loaded BUT can't load... on message like "loading ipmi". Only clear the Bios help (NVRAM clear-YES don't help). (I dont configure iKVM function, all in default value, all bios settings in default value)

On ESXi host run one virtual mashine Server2003x64 TermService.

1. Log from dir /scratch/log

2,3. Log from /var/log/

Any help for this trable?

In documantation for Tyan iKVM: "Note that TYAN motherboard BIOS encodes IPMI Base IO address at 0xCA2 in its

DMI table IPMI entry, any generic OS IPMI drivers should have no problem to

support it."

In log ESX see: "ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 0"

Maybe in adress problem?

I also find in advance setting in ESXi VMKernel.Boot.ipmiEnabled=1. IF communities agree i can disable this value and test ESX!? I'am also find jumper on mother board for set the BMC functional to Disable...

I have problem in first installation of ESXi. Host cant see a datastore after reboot. Kernel speak "device buse" After I install option VMKernel.noIOMMU=0 datastore after reboot - ok! Sorry for my bad english!

[tinipop]

Hello!

I'am set option VMKernel.Boot.ipmiEnabled=0.\

4 days - cpu load ok! But with this option disabled ESXi don't read temparature parameters from motherboard.

Thaks

[Salli]

All seems to be fine in terms of CPU usage.. just need to figure out what's going on with disk latency (300-600ms on local disk)

Perhaps a broken disk in Raid5EE array, though I've replaced 2 out of 4drives to test it. Or perhaps ESXi doesn't like having scratch on USB. Would be odd if moving it on SSD array had any effect (there's plenty of memory to spare), but I'll try that tomorrow.

[Salli]

Update: I didn't find time to move ESXi installation, but it looks like vmkapimod is bugging again ?:| I'm getting bit concerned about purchasing the 5000$+ license while I'm running into all sort of troubles with certified hardware. Latency is as bad as ever even after installing battery module, replacing disks and enabling write cache on controller. Guess I'll try swapping remaining vmxnet3 adapters to E1000, moving the installation away from USB drive and changing raid level from 5EE to 10. If that doesn't solve problems, I guess I could give ESX a chance and start thining about alternatives, if all that fails. I'm just a bit puzzled, as I can't find any information about what vmkapimod is related to.

[J1mbo]

>Latency is as bad as ever even after installing battery module, replacing disks and enabling write cache on controller.

Could you provide some info on this problem, I can't see any reference to it in your other posts. I'd be interested to know the server hardware (CPU, RAM, disk) and the configuration of the 10 VMs (vCPU and RAM) and how they're split between the three hosts.

http://blog.peacon.co.uk

Please award points to any useful answer.

[Salli]

Hardware on the server I'm currently evaluating ESXi on (we have another identical server, but It's running Hyper-V for now):

• Intel S3200SHV

• 1x Xeon E5440 (Tried with Q9550 to rule out defected CPU)

• 8GB ram

• 1x Adaptec 3805 with Battery Unit

• 4x Seagate Barracuda ES 7200.2 1TB (Raid 5EE)

• 1x 16GB USB Flash

In terms of cpu usage everything looks good again ?:| ; I had left VSphere Client monitoring the ESXi host from terminal server (located on Hyper-V host). Closing it after use rather than leaving it on seems to "solve" the issue. It seems like that after <insert random time here> something on ESXi host fails when being monitored by VSphere Client and starts consuming those precious cpu cycles.

[J1mbo]

As you have summised the storage will performance very much better in RAID-10 than RAID-5EE with no change in capacity.

We need to understand the allocation of vCPU and RAM to guests in order to comment on the "latency" issue.

http://blog.peacon.co.uk

Please award points to any useful answer.

[Salli]

All guests (1 Domain controller, 1 Terminal server, 1 CentOS) have been allocated one vCPU. Domain controller has 2Gb, terminal server 4Gb and CentOS 384Mbs (CentOS is used for backup purposes; rsync).

Write rate peaks at around 8Mbytes/second :| when transfering files from local fileserver. During the peak, latency climbs to 2000+ms. Active memory for guests is hovering between 1 - 3Gbs.

[Salli]

Latency and performance issues were solved by switching to RAID10. For some reason ESXi didn't perform too well with 5EE.

CPU usage problem is still unaswered; some ESXi service seems to crash randomly, if I leave vSphere Client active for days in terminal server session.

[Frantisek Ferencik]

Hi,

I got exactly the same problem with vmkapimod consuming 100% CPU resources from one core on my ESXi 4.0.0, 208167. Please have someone found way how to fix it?