I have a handful of Linux NFS servers running on Virtual Machines. Sometimes NFS performance will take an absolute nosedive and it will take about 15x longer than normal to copy files to the NFS file system. This is not a simple resource issue. If I forgo NFS and copy the file using scp of ftp, it is nice and fast. Nor does this seem to be an NFS tuning issue on Linux -- all the NFS metrics look fine. Same for the physical servers that are the NFS clients -- if I mount up an NFS file system from another NFS Server VM, I can write to that with good performance.
So the problem seems to be centered on the NFS Server VM, and I suspect it is at the VMware layer rather than the OS layer. Until recently we were able to resolve the problem by doing a vMotion to another hosts and then back again, or we'd put the NIC in promiscuous mode using ifconfig in Linux on the guest OS. Strange, huh? Anyway, these two tricks are currently not working, and I have some VMs that are acting as nice and speedy NFS servers, and a few others that are dogs, and the dogs are sometimes running on the same ESXi hosts as the speedy servers.
I don't suppose anyone else out there is using a Linux VM as an NFS server...? Any suggestions?