VMWare workstation in a fistfight with Linux Memory Compactor kcompact

[linutic]

I run VMWare Workstation Pro 16.2.9 on a AMD 3900X with 64 GB memory, and an SSD running Ubuntu Linux 5.11.0-38-generic.

Starting about 1 year ago VMWare has been fighting with the Linux memory compactor kcompct.  When hostilities begin (Windows 10 with 4 cores at 16 GB configured) VMWare freezes with its 4 cores at 100% while kcopact runs 100% CPU.  After a few seconds, the combatants restrain themselves for about 30 seconds, then begin again.  This doesn't happen every day, but when it does happen the two combatants never seem to tire of this annoying behavior.  A Linux reboot clears up the problem, for a time.

This behavior has been mentioned on a number of forums, but I have not seen a discussion on VMWare.  (which is why I am here)   Initially the problem could be resolved by turning off Transparent Hugepages.   Then after a time, this was no longer necessary, and the two behaved themselves.  About 3 months ago, hostilities resumed, but now turning off Transparent Hugepages has little or no effect.

I'm guessing that VMWare has it's own algorithms to collect 4K pages into hugepages, which is otherwise the job of kcompact.   Possibly the two are squabbling over a futex or other exclusive resource, or maybe a pair of them.

Any chance VMWare guys could look at this problem and break up the fight?

[simmonsjw]

I've got the same problem on a Fedora 34 system.  All kernels starting with 5.12 show the problem, going back to 5.11.21-300 and the issue pretty much disappears.  When this happens, kcompactd0 goes to 100% cpu use and the virtual is unresponsive (you can move the cursor but not get any other response).  At the same time, the vmware process for the virtual goes to basicallly 100% * cpu cores (i.e. if you have 2 cpu cores in the virtual it will sit at 200% or so).  When kcompactd0 goes back below 100%, which can sometime take minutes, the virtual comes back but it often happens again fairly quickly.  Strangely, at times with the newer kernels you can run for quite a while without the issue.

Updating to 16.1.2, then 16.2.0, and 16.2.1 didn't help, along with using patches like https://github.com/mkubecek/vmware-host-modules/archive/workstation-16.2.1.tar.gz.

I'd like to update to Fedora 35 but suspect I won't be able to use the older kernel.  I'm considering converting the virtual to kvm, but I'd prefer to stay with VMware Workstation.

As mentioned, turning off hugepages did not help, either.

Anyone have any ideas?

Thanks, Jim

[wila]

Hi,

Please see the following for a workaround:
https://gist.github.com/2E0PGS/2560d054819843d1e6da76ae57378989

This was found by the who posted this thread:
https://communities.vmware.com/t5/VMware-Workstation-Player/Over-CPU-usage-in-idle/m-p/2878569

 

Hope this helps,
--
Wil

[simmonsjw]

Thanks, but I've tried it and it doesn't help with the latest kernels in Fedora 34 (I've tried it through all of them up to the current 5.15.4-101.fc34.x86_64).  The 5.15.4-101.fc34.x86_64 is the last kernel vmware has worked reliably with on this system.  I don't believe it is actual memory - the system has 6G of memory with 2G (and later 3G) allocated to the windows 10 virtual.  I've also tried changing the vmware preference for reserved memory to 4096MB but that didn't help.

I'm going to try converting the virtual to a kvm virtual and see if it fixes it.  This is an old system with a "Intel(R) Core(TM) i7-2630QM CPU @ 2.00GHz" CPU, which is currently one of the oldest supported by vmware.  It is possible there is something with it that is causing the issue.

Jim

[toesterdahl]

Hello,

I have this problem too, and the problem seem to be hitting systems running Ubuntu as host somewhat reliably. I find it would be in the intrest of vmware to finda permanent solution with Ubuntu, alternative upstream with kernel developers. For every user that find the correct tweak there will be plenty that just give up.

[linutic]

I received a solution on another thread, and it's working for me.  All credit to neves-0 on Github. See:

https://gist.github.com/2E0PGS/2560d054819843d1e6da76ae57378989

As root, execute the following command:

echo 0 > /proc/sys/vm/compaction_proactiveness

So far, this fix seems to completely disable kcompactd0.  There may be side effects, but so far I haven't noticed any.  After 12 hours, on Ubuntu 20.04 with VMware Workstation 16.2.1 build-18811642 running a Windows 10 vm, kcompactd0 has accumulated zero cpu time:

# ps -ef | grep kcompactd0
root         165       2  0 Dec04 ?        00:00:00 [kcompactd0]
gene       24364   20919  0 10:05 pts/0    00:00:00 grep kcompactd0

If you would like to apply the fix every time you boot, add the following line to /etc/sysctl.conf:

# Disable kcompactd0 to work around a conflict with VMWare Workstation.
vm.compaction_proactiveness=0

 

[kmahyyg]

# Disable kcompactd0 to work around a conflict with VMWare Workstation.
vm.compaction_proactiveness=0

 

This solution offered by another guy works fine here. Transparent hugepage is no need to be modified for this problem.

This problem became much more obvious after upgrade Linux kernel to 5.16+ as I'm running Arch Linux.

VMWare official, plz, fix ASAP.

[ijbgreen]

This works for me!!. Ubuntu 20.0.4 kernel 5.13.0-37-generic at this moment without side effects. Tested on Windows 10 and Windows 11 guests.!!! I only apply:

# Disable kcompactd0 to work around a conflict with VMWare Workstation.
vm.compaction_proactiveness=0

Using:

sudo sysctl vm.compaction_proactiveness=0

 Only when i will go to start a Windows VM. Because for me only happens with Windows Guests.

[reddavid]

Excellent info. Been dealing with this for a couple years on multiple kernels/distros. Work around was adding a cron job that fired every 10 min to drop caches which worked "fine" until this AM when i was compacting a large VM on a USB attached disk. Long process multiple kcompatd runs turned my running VM into a useless mess.

 

 

[linutic]

I've tried disabling transparent hugepages.  It used to work, but no longer does.  See the link below for an alternative fix:

https://gist.github.com/2E0PGS/2560d054819843d1e6da76ae57378989

I have been strugging with this problem for *years*.   No more.

[jiaxinslee]

Comfirm that win10 guest freezes and kcompactd0 kicks in with compaction_proactiveness=0.

Tested with:
Ubuntu 22.04 kernel 5.15.35 / 5.16.20
VMWare 16.2.3 / 16.2.0 / 16.1.2
Win10 Guest, 4GB / 6GB memory allocated

[jvasa]

I'm on Fedora 36, and it's good on the 5.17 kernel, but on 5.18 the kcompactd0 constantly freezes the Win10 VM.

My settings:
echo never | tee /sys/kernel/mm/transparent_hugepage/defrag
echo 0 | tee /sys/kernel/mm/transparent_hugepage/khugepaged/defrag
echo 0 | tee /proc/sys/vm/compaction_proactiveness

Specs:
Fedora 36
VMWare Workstatoin 16.2.3 with mkubecek vmware-host-modules
Windows 10 VM: 16GB / 32GB allocated
CPU : Intel i7-10700

Working:
Kernel: 5.17.14 (fedora rpm)

Not Working
Kernel: 5.18.5 (fedora rpm)

I am reverted back to 5.17 kernel, as atleast for me it gives me issues running 5.18.

[nayr1]

OS: Pop!_OS 22.04 LTS x86_64
Host: 20QVS0FP00 ThinkPad X1 Extreme 2nd
Kernel: 5.17.15-76051715-generic
CPU: Intel i9-9880H (16) @ 4.800GHz
GPU: NVIDIA GeForce GTX 1650 Mobile / Max-Q
Memory: 23582MiB / 31783MiB

Disabling kcompactd0 seems to have solved this problem for me also!

[FZ2]

It seems

vm.compaction_proactiveness = 0

works properly till Kernel 5.13. It doesn't work anymore from Kernel 5.15.

- Ubuntu 22.04 LTS use v5.15 /5.17 as default

- Ubuntu 20.04.4 LTS use v5.13
- Ubuntu 20.04.5 LTS use v5.15 <-- Released 1/9/202

[robep00]

I concur: 

 It doesn't work anymore from Kernel 5.15.

and none of the other tricks mentioned work for extended period of times (as in more than a day or two). It seems to always creep back in  after a day or two and just won't go away afterwards. I also tried compiling my own Linux Kernel without the features I suspected could conflict, but some are deeply rooted in the kernel now.

I have tried many variations of the following without any success. 64GB of ram, can't use it. The more ram VM guests use, the worse it gets.

/////////////// vmware and other configs \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
# add to /etc/sysctl.conf
# to appy right away: sysctl -p
vm.swappiness = 4
vm.vfs_cache_pressure=100
vm.compaction_proactiveness = 0
vm.compact_unevictable_allowed = 0

echo never | tee /sys/kernel/mm/transparent_hugepage/defrag
echo 0 | tee /sys/kernel/mm/transparent_hugepage/khugepaged/defrag
echo 0 | tee /proc/sys/vm/compaction_proactiveness
echo never | tee /sys/kernel/mm/transparent_hugepage/defrag
echo never | tee /sys/kernel/mm/transparent_hugepage/enabledsudo sync

sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
sudo sh -c 'echo 1 > /proc/sys/vm/compact_memory'
https://communities.vmware.com/t5/VMware-Workstation-Pro/kcompacd0-using-100-CPU-with-VMware-Workstation-16/td-p/2896972

https://unix.stackexchange.com/questions/679706/kcompacd0-using-100-cpu-with-vmware-workstation-16

 

[dittothat]

I have encountered this same issue. I recently upgraded to host Ubuntu 22.04 and my Windows 10 guest VMs run in Workstation Pro 17 are no longer functional. I am about to downgrade to 20.04 to get back to a pre 5.15 kernel because I must have working VMs for occasional work tasks. Posting here as a last hope before I do. Has anyone made any discoveries about how to work around this issue?

[sth-peter]

Having the same issue with Workstation 16.2.4 on Xubuntu 22.04 all of a sudden this week. Rebooted for the first time since September which was running 5.15.0-47-generic and now 5.15.0-56-generic since the reboot Jan 4. I'll try a reboot an see if that help otherwise will revert back to the older kernel.

 

 

[morebikethanman]

As with the others, I have tried everything suggested in the various forums including everything suggested by  (even the intel_iommuand VMware continues to lock up with kcompactd0 seeming to be the culprit.  I am running 

• VMware Workstation 17 Pro17.0.0 build-20800274
• Ubuntu 22.04.1 LTS
• Linux 5.15.0-57-generic #63-Ubuntu SMP

[sth-peter]

The first answer from this link has been working for me for a few hours now;

https://unix.stackexchange.com/questions/679706/kcompacd0-using-100-cpu-with-vmware-workstation-16

No more kcompacd0 process using 100% CPU after running:

sudo sh -c 'echo 0 > /proc/sys/vm/compaction_proactiveness'

This is on Xubuntu 22.04.1 LTS, 5.15.0-56-generic and VMware Workstation 16.2.4.

[goldeneye_007]

I have not seen this issue for awhile but today, Win11 guest on 17.0.1, crippling and I mean absolutely crippling slowdown of VM, kcompacd0 running at max CPU. I have implemented the /etc/sysctl.conf fix and also echo. No more kcompacd0.

Specs:

Operating System: openSUSE Leap 15.4
KDE Plasma Version: 5.24.4
KDE Frameworks Version: 5.90.0
Qt Version: 5.15.2
Kernel Version: 5.14.21-150400.24.41-default (64-bit)
Graphics Platform: X11
Processors: 16 × 11th Gen Intel® Core™ i9-11900 @ 2.50GHz
Memory: 31.2 GiB of RAM
Graphics Processor: NVIDIA GeForce GT 710/PCIe/SSE2

[Rondo85]

This does not work for me

[Rondo85]

This does not work for me. 4k Monitor, Guest with 4GB

It freeze....

[morebikethanman]

I switched from Ubuntu to Arch hoping this problem would go away but it did not.  As I mentioned in an earlier post, I have tried all the suggestions and inevitably after a few days kcompactd0 has the system pegged and I have to reboot in order to use my virtual machine.  My system info is:

• Host:
  • 17.0.1 build-21139696
  • Arch Linux 6.2.12-arch1-1
  • 16 GB RAM
  • Intel Core i7-10700 CPU @ 2.90GHz
• Guest:
  • Windows 11 Pro 22H2
  • 4 GB Ram

Do we know if VMWare is looking into this at all?

[philadev]

Same issue, hours of lost time fighting this until I came across the trick to disable kcompact. I was ready to give up on VMWare on Linux.

It seems that this could be better advertised, ideally have an advanced setting to let you fix it from VM config UI (with the appropriate warnings).

[goldeneye_007]

As this toipic was begun in 2021, the answer is apparently, 'No'.