Problems with high cpu load when using network!

[Daniel844]

I have big problems with my network. Ehen transferring files to and from the ESX 3i server the CPU load goes up like crazy using allmost 50% of a Quad Core.

Currently i have 3 NIC's:

1 Intel PRO 1000 PT

2 Intel PRO 100

And i have moved them around without success and removed them and so on. The ESX 3i installation is fresch and updated.

[Dave_Mishchenko]

Welcome to the VMware Community forums. Your post has been moved to the Performance forum.

Dave Mishchenko

VMware Communities User Moderator

Could you tell us a bit more about your environment? How are you transfering data to the host?

[Daniel844]

Thanks for moving the post.

I have tested from/to an FTP-server that i run in Windows 2003 R2 Server (SP2)

And i have allso tested to download/upload vmdk files with WinSCP from/to my workstation and everytime i transfer something it jumps up in cpu usage.

[wila]

It is strange for the ftp server, that shouldn't really happen.

On the winscp download it actually makes sense that you see an increased CPU load as it encrypts everything before pushing it over the pipe.

In addition you are talking about moving vmdk files, so I am suspecting that you are moving those files over an opened ssh connection in your ESX3i server which might not be as well balanced as when doing this in a real guest.

--

Wil

[Daniel844]

I have some share on the 2003 server that i have mapped to my workstations and i still g et high cpu load when i move files in that way to.

About ~3000Mhz acording to the CPU-usage graph.

Allso running ubuntu server and i have the same problem when i transfer files to/from the shares on that VM.

[wila]

That shouldn't happen like that. It seems like that there's some problem with your host's NIC of some kind.

Have you checked the logfiles on the host ?

--

Wil

[Daniel844]

What logs do i have the look at?

Can there be a problem with the motherboard?, im using Intel DP35DP i know it is desktop but ESX 3i seems to run without problem execpt the CPU-usage...

[wila]

Hi,

Using the Virtual Infrastructure Client you can have a look at the /var/log/messages file and the /var/log/vmkernel (or at least that's how the files are called on a full blown ESX 3.5 server)

As most network traffic is actually handled by the CPU (Yes I did some additional reading on the subject) some upswing in CPU usage is expected, but I've not noticed it to go up in the way you describe.

For kicks I just had a look here and I do notice an approximately 1000MHz upswing when copying a larger file over the network out of the ESX server, hmmm :smileywink:

Here is a link to an article that goes in on details on how this works:

http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf

--

Wil

[Daniel844]

I cant find anything in the log that has anything to do with the network..

Seems to me that my problem is not that easy to solve :smileysad:

My friend that are using the same chipset and same NIC,s (execpt he has PCI cards instead of PCI-E)

and he dosnt get that high cpu-usage.. he get up to around ~1600mhz, but mine goes up to ~3000 :smileysad:

[wila]

No suspicious messages in the logs OK...

Well I still think that the CPU usage is pretty much out of the ordinary.

Are you by any chance using VLAN's?

--

Wil

[Daniel844]

Nope no VLAN's

well i coulnt find any suspicious messages in the logs,don´t know what i have to look for exaclty..

[wila]

Well what you could do is to test this and make a note of the start and endtime and then check your logs.

I also don't know how big the files are that you are testing with, if you are testing with VM's internally only, if they are local to the ESX server, it will be different when the VMs are connected to the same vSwitch as when they are connected to different switches. Copying data over the uplink adapter out of your ESX server or towards the ESX server, it all might get different results.

If your files are really big and you are using local storage on the server, then it is also possible that you are getting higher CPU usage due to the CPU also having to get the data from disk.. in such a case it is important to have a battery backed Write cache on your scsi controller. There are quite a few parameters that can make a difference when measuring performance..

--

Wil

[Daniel844]

wila: thanks for all your help!

I think i have found the problem!!

sitting here with the server and testing one thing after another, and what i did is i took out my older intel pro 100 adapter and running only with the new Intel PRO 1000 PT adapter and now when i copy an ISO file (about 4gigs)

the CPU-load is only around 550Mhz!! thats an huge drop from ~3000Mhz

But i dont understand how the other NIC can do something like that, i only use the pro 100 for management and the pro 1000 for the VM,s

i will try to install only the pro 100 adapter and see what happens, if i get low load with only that card i guess ESX cant handle PCI and PCI-E NIC,s at the same time!

i would be glad is anyone else can test this in their ESX 3i machine.

Or it can be my motherboard that dosnt handle PCI and PCI-E cards good.

[Daniel844]

Btw i forgott to say that i got ~550Mhz when using 100Mbit and now that i have tested with 1000Mbit the load was around 1500-1700Mhz.

used 2 uplinks on my Switch for gigabit between the 2 machines.