VMware Workstation

 View Only

 Problem with 17.6 Update on Ubuntu 22.04

rainer_xxl's profile image
rainer_xxl posted Sep 04, 2024 06:40 AM

The virtual machines will run, but it will kill the host after startup and network connections will no longer be available. All processes that require a network connection are no longer executable and can no longer be started. Existing processes such as Firefox can no longer even be terminated.

It does not help to close vmware again, you have to reboot the computer completely, which leads to further errors. The only thing that helped me was to switch off the computer!

Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.009633] /dev/vmmon[10682]: PTSC: initialized at 3399999000 Hz using TSC, TSCs are synchronized.
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.073955] /dev/vmmon[10682]: Monitor IPI vector: 0
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.073958] /dev/vmmon[10682]: HV      IPI vector: 0
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.074995] ------------[ cut here ]------------
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.074997] WARNING: CPU: 0 PID: 10682 at kernel/rcu/tree_plugin.h:734 rcu_sched_clock_irq+0x3ad/0x3c0
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075003] Modules linked in: rfcomm veth xt_nat nf_conntrack_netlink xfrm_user xfrm_algo xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_addrtype br_netfilter xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink bridge stp llc vmnet(OE) vmw_vsock_vmci_transport vsock vmw_vmci nls_utf8 vmmon(OE) cifs cifs_arc4 nls_ucs2_utils cifs_md4 netfs cmac algif_hash algif_skcipher af_alg bnep overlay intel_rapl_msr intel_rapl_common binfmt_misc snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_intel edac_mce_amd snd_intel_dspcfg snd_intel_sdw_acpi snd_usb_audio snd_hda_codec snd_usbmidi_lib kvm_amd snd_ump snd_hda_core nls_iso8859_1 mc snd_hwdep snd_pcm kvm snd_seq_midi snd_seq_midi_event irqbypass snd_rawmidi btusb btrtl btintel snd_seq btbcm btmtk joydev input_leds rapl snd_seq_device bluetooth snd_timer ecdh_generic wmi_bmof ecc snd k10temp ccp soundcore mac_hid sch_fq_codel nct6775 nct6775_core hwmon_vid msr parport_pc ppdev lp
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075076]  parport efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 hid_generic amdgpu amdxcp drm_exec gpu_sched drm_buddy drm_suballoc_helper drm_ttm_helper ttm drm_display_helper cec usbhid hid usb_storage crct10dif_pclmul rc_core nvme crc32_pclmul polyval_clmulni polyval_generic mxm_wmi ghash_clmulni_intel sha256_ssse3 sha1_ssse3 igb i2c_piix4 video ahci nvme_core dca xhci_pci i2c_algo_bit libahci xhci_pci_renesas nvme_auth wmi gpio_amdpt aesni_intel crypto_simd cryptd
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075121] CPU: 0 PID: 10682 Comm: vmware-vmx Tainted: G           OE      6.8.0-40-generic #40~22.04.3-Ubuntu
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075124] Hardware name: Micro-Star International Co., Ltd. MS-7A32/X370 GAMING PRO CARBON (MS-7A32), BIOS 1.Q1 05/19/2023
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075125] RIP: 0010:rcu_sched_clock_irq+0x3ad/0x3c0
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075127] Code: 04 25 40 43 03 00 f0 80 08 08 65 81 25 f0 3d 65 69 ff ff ff 7f e9 a2 fe ff ff 8b 87 38 08 00 00 85 c0 0f 84 23 fd ff ff eb b2 <0f> 0b e9 ed fc ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075129] RSP: 0018:ffffa0ac00003e68 EFLAGS: 00010082
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075131] RAX: ffff8f319761a900 RBX: 0000000000000000 RCX: 0000000000000000
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075132] RDX: 00000000ffffff8a RSI: ffffffff983b4945 RDI: ffff8f319761a900
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075134] RBP: ffffa0ac00003e80 R08: 0000000000000000 R09: 0000000000000000
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075135] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075136] R13: ffff8f3efe225000 R14: ffffa0ac10b176b8 R15: ffff8f3efe224ac0
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075137] FS:  00007ca8d5238c00(0000) GS:ffff8f3efe200000(0000) knlGS:0000000000000000
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075139] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075140] CR2: 00007ca8d2031000 CR3: 0000000267fea000 CR4: 0000000000f50ef0
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075141] PKRU: 55555554
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075142] Call Trace:
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075144]  <IRQ>
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075147]  ? show_regs+0x6d/0x80
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075150]  ? __warn+0x89/0x160
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075154]  ? rcu_sched_clock_irq+0x3ad/0x3c0
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075156]  ? report_bug+0x17e/0x1b0
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075160]  ? handle_bug+0x46/0x90
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075164]  ? exc_invalid_op+0x18/0x80
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075166]  ? asm_exc_invalid_op+0x1b/0x20
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075172]  ? rcu_sched_clock_irq+0x3ad/0x3c0
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075174]  ? srso_alias_return_thunk+0x5/0xfbef5
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075177]  update_process_times+0x76/0xb0
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075180]  tick_sched_handle+0x28/0x70
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075184]  tick_nohz_highres_handler+0x78/0xa0
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075186]  ? __pfx_tick_nohz_highres_handler+0x10/0x10
Sep  4 09:35:52 rainer-MS-7A32 kernel: [ 1356.075189]  __hrtimer_run_queues+0x112/0x2a0

lnxres's profile image
lnxres

Hello!

Same problem here.

I installed the update, rebooted the machine, and started experiencing issues.

The system works fine until I start a Virtual Machine. When I start my Windows Guest, after a few minutes, the host system becomes unstable. The network connection stops working, some processes cannot be terminated, and the sudo command gets stuck. Power Off and Reboot do not work, and I have to perform a hard shutdown.

I checked dmesg and syslogand noticed messages like the following:

----

[  963.639739] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 0-...D } 62867 jiffies s: 953 root: 0x1/.
[  963.639753] rcu: blocking rcu_node structures (internal RCU debug):
[  963.639756] Sending NMI from CPU 1 to CPUs 0:
[  963.639766] NMI backtrace for cpu 0 skipped: idling at intel_idle+0x72/0xe0
[ 1107.002981] INFO: task kworker/1:3:4852 blocked for more than 122 seconds.
[ 1107.002991]       Tainted: G        W  OE      6.8.0-40-generic #40~22.04.3-Ubuntu
[ 1107.002994] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[ 1107.003167] INFO: task vmx-vmem:6673 blocked for more than 122 seconds.
[ 1107.003169]       Tainted: G        W  OE      6.8.0-40-generic #40~22.04.3-Ubuntu
[ 1107.003171] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[ 1143.865777] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 0-...D } 243091 jiffies s: 953 root: 0x1/.
[ 1143.865790] rcu: blocking rcu_node structures (internal RCU debug):

---

From my side I had to uninstall version 17.6.0 and installed again the previous one 17.5.2.

I hope VMware Broadcom will fix it soon.

ambroz09's profile image
ambroz09

I can confirm the same problem on Ubuntu 24.04.1.

Symptoms:

(1) Soon after I start (any) VM the host OS starts "losing integrity".

(2) Other programs which I want to start suddenly don't run anymore (nothing happens, they just don't open).

(3) I f I wait long enough the whole host Ubuntu 24.04.1 OS just freezes.

(4) Usually I just close the VM and want to reboot the machine. The reboot screen gets filled with a plethora of blue errors and waits for the processes to finish. They never do and I have power down the machine the hard way (with a button) and restart it.

----

17.6.0 is the culprit. 17.5.2 was working fine.

The problems started immediately after 17.6.0 installation.

So Workstation has become unusable as it hangs the host OS.

Never encounted anything similar with Linux OSes.

jon zuilkowski's profile image
jon zuilkowski

I'm having the exact same behavior right now, it's making my system unusable:

Sep  5 10:41:32 Desktop kernel: [ 2827.230521] INFO: task vmx-vmem:11946 blocked for more than 737 seconds.
Sep  5 10:41:32 Desktop kernel: [ 2827.230533]       Tainted: P        W  OE      6.8.0-40-generic #40~22.04.3-Ubuntu
Sep  5 10:41:32 Desktop kernel: [ 2827.230537] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  5 10:41:32 Desktop kernel: [ 2827.230540] task:vmx-vmem        state:D stack:0     pid:11946 tgid:11921 ppid:4206   flags:0x00000002
Sep  5 10:41:32 Desktop kernel: [ 2827.230546] Call Trace:
Sep  5 10:41:32 Desktop kernel: [ 2827.230550]  <TASK>
Sep  5 10:41:32 Desktop kernel: [ 2827.230555]  __schedule+0x27c/0x6a0
Sep  5 10:41:32 Desktop kernel: [ 2827.230564]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230574]  schedule+0x33/0x110
Sep  5 10:41:32 Desktop kernel: [ 2827.230578]  synchronize_rcu_expedited+0x20c/0x290
Sep  5 10:41:32 Desktop kernel: [ 2827.230585]  ? __pfx_wait_rcu_exp_gp+0x10/0x10
Sep  5 10:41:32 Desktop kernel: [ 2827.230589]  ? __pfx_autoremove_wake_function+0x10/0x10
Sep  5 10:41:32 Desktop kernel: [ 2827.230595]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230600]  ? __put_mountpoint.part.0+0x7b/0xa0
Sep  5 10:41:32 Desktop kernel: [ 2827.230607]  namespace_unlock+0xdc/0x1b0
Sep  5 10:41:32 Desktop kernel: [ 2827.230612]  put_mnt_ns+0x76/0xb0
Sep  5 10:41:32 Desktop kernel: [ 2827.230617]  free_nsproxy+0x1b/0x1b0
Sep  5 10:41:32 Desktop kernel: [ 2827.230623]  exit_task_namespaces+0x71/0xa0
Sep  5 10:41:32 Desktop kernel: [ 2827.230630]  do_exit+0x2ae/0x530
Sep  5 10:41:32 Desktop kernel: [ 2827.230635]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230641]  __x64_sys_exit+0x1b/0x20
Sep  5 10:41:32 Desktop kernel: [ 2827.230645]  x64_sys_call+0x1a02/0x24b0
Sep  5 10:41:32 Desktop kernel: [ 2827.230650]  do_syscall_64+0x81/0x170
Sep  5 10:41:32 Desktop kernel: [ 2827.230656]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230661]  ? zap_pmd_range.isra.0+0x133/0x280
Sep  5 10:41:32 Desktop kernel: [ 2827.230668]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230672]  ? unmap_page_range+0x28f/0x4f0
Sep  5 10:41:32 Desktop kernel: [ 2827.230679]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230682]  ? tlb_finish_mmu+0x73/0x180
Sep  5 10:41:32 Desktop kernel: [ 2827.230687]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230691]  ? zap_page_range_single+0x162/0x210
Sep  5 10:41:32 Desktop kernel: [ 2827.230701]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230706]  ? madvise_vma_behavior+0x5c0/0x780
Sep  5 10:41:32 Desktop kernel: [ 2827.230711]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230715]  ? __blk_flush_plug+0xdf/0x130
Sep  5 10:41:32 Desktop kernel: [ 2827.230721]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230725]  ? madvise_walk_vmas+0xc4/0x140
Sep  5 10:41:32 Desktop kernel: [ 2827.230730]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230733]  ? blk_finish_plug+0x31/0x50
Sep  5 10:41:32 Desktop kernel: [ 2827.230737]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230741]  ? do_madvise+0x1d1/0x390
Sep  5 10:41:32 Desktop kernel: [ 2827.230747]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230751]  ? syscall_exit_to_user_mode+0x89/0x260
Sep  5 10:41:32 Desktop kernel: [ 2827.230756]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230760]  ? do_syscall_64+0x8d/0x170
Sep  5 10:41:32 Desktop kernel: [ 2827.230764]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230768]  ? irqentry_exit_to_user_mode+0x7e/0x260
Sep  5 10:41:32 Desktop kernel: [ 2827.230773]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230777]  ? irqentry_exit+0x43/0x50
Sep  5 10:41:32 Desktop kernel: [ 2827.230781]  ? srso_return_thunk+0x5/0x5f
Sep  5 10:41:32 Desktop kernel: [ 2827.230785]  ? exc_page_fault+0x94/0x1b0
Sep  5 10:41:32 Desktop kernel: [ 2827.230790]  entry_SYSCALL_64_after_hwframe+0x78/0x80
Sep  5 10:41:32 Desktop kernel: [ 2827.230796] RIP: 0033:0x72416da94a66
Sep  5 10:41:32 Desktop kernel: [ 2827.230832] RSP: 002b:00007241691ff9a0 EFLAGS: 00000246 ORIG_RAX: 000000000000003c
Sep  5 10:41:32 Desktop kernel: [ 2827.230836] RAX: ffffffffffffffda RBX: 0000724169200640 RCX: 000072416da94a66
Sep  5 10:41:32 Desktop kernel: [ 2827.230839] RDX: 000000000000003c RSI: 00000000007fb000 RDI: 0000000000000000
Sep  5 10:41:32 Desktop kernel: [ 2827.230842] RBP: 0000724168a00000 R08: 0000000724158001 R09: 0000000000000000
Sep  5 10:41:32 Desktop kernel: [ 2827.230844] R10: 0000000000000008 R11: 0000000000000246 R12: 0000724169200640
Sep  5 10:41:32 Desktop kernel: [ 2827.230847] R13: 0000000000000002 R14: 000072416da947d0 R15: 00007ffd5ce8ffc0
Sep  5 10:41:32 Desktop kernel: [ 2827.230855]  </TASK>

johncenix's profile image
johncenix

The problem starts happening when Ubuntu 22.04 is updated to utilize Linux kernel 6.8.0 - or if you have Ubuntu 24.04, which comes with 6.8.0 by default (despite VMware claiming that 17.6.0 has been updated to have Ubuntu 24.04 host support).

This commit on GitHub fixes the problem. The same repository also has fixes for later kernels (UBSAN warning messages will need x86msr.h and x86cpuid.h to have [0] changed to [] and vmnetInt.h will need read_lock(& and read_unlock(& replaced with rcu_read_lock() and rcu_read_unlock()), but if all you need is for VMware Workstation 17.6.0 to start working with latest patched Ubuntu 22.04 or 24.04, do this to surgically inject only the required changes for 6.8.0: (make sure your system is not already in a state of degradation - so reboot first!)

tar xvf /usr/lib/vmware/modules/source/vmmon.tar
curl -o vmmon-only/include/pgtbl.h https://raw.githubusercontent.com/mkubecek/vmware-host-modules/4c2a103fd2d71f2084f1fe7ceacb816b9832ffa2/vmmon-only/include/pgtbl.h
tar cvf vmmonnew.tar vmmon-only/

sudo cp /usr/lib/vmware/modules/source/vmmon.tar /usr/lib/vmware/modules/source/vmmonold.tar
sudo cp vmmonnew.tar /usr/lib/vmware/modules/source/vmmon.tar

sudo vmware-modconfig --console --install-all

If you ever want to revert the changes, copy vmmonold.tar back to vmmon.tar and re-run the vmware-modconfig command.

I've tested the change myself - I am now happily on kernel 6.8.0 and running VMs in 17.6.0 no longer causes the immediate rcu_sched_clock_irq kernel message and other instabilities and lockups. And yes, you're downloading code from the Internet to be compiled straight into your kernel modules - this is how Linux works... (in other words, at your own risk, though of course you can audit the changes yourself.)

rainer_xxl's profile image
rainer_xxl

Before I sent this question, I had tried several times, hours before, to send a warning via the forum!

This warning was not published by the team, which I find very confusing when the error is known and more customers are left out in the cold.

Robert Peters's profile image
Robert Peters

I also had this problem after updating the Ubuntu 24.04.1 LTS, then I saw the update for VMWare, and updated to 17.6.


I'm currently trying the fix from johncenix.

One of the interesting side effects of the problem is that you can start vmware 17.6, then close it down, and it will still cause my system to lockup. So, I think that's supporting evidence that it's a problem with a kernel module that's still loaded.

Ali Reza Vojdanitalab's profile image
Ali Reza Vojdanitalab

I tried it in Linux Mint 22 and the outcome was pretty bad.

When i started VMWare workstation player 17.6.0 not much happened. Then i  loaded Firefox and closed Firefox and after that no other program worked right anymore. I could not even kill Firefox anymore.

Only solution is a hard reset.

Had to go back to 17.5.2 with hacked vmmon / vmnet files.

One problem has been solved 17.6.0 though. One does not need to to use the hacked vmmon / vmnet files for VMWare player to start anymore.

gary fargusson's profile image
gary fargusson

Same issue here. Fortunately, I experienced this in my sandbox system. I was able to revert back to 17.5.2 and will test for stability.

It seems that the kernel cannot spawn new processes. I played around with ulimits but it didn't help. Something is clogging up the kernel and when it happened, I couldn't fork to an additional process to probe for diagnostics. 

I will build a small Ubuntu 22.04 LTS build to reproduce the problem on a clean install. 

bertramaerts's profile image
bertramaerts

Fix from johncenix is working perfectly.
I now run VMware Workstation Pro 17.6 on host Ubuntu 24.04.1 (kernel 6.8.0-44) with guests Windows 11 and Windows XP.

Thanks a lot johncenix !!!

Dejan Rodiger's profile image
Dejan Rodiger

I had available kernel 6.5.0-45 on 24.04 and if I boot with this kernel, then Workstation 17.6 works and doesn't stop network or whole kernel.

But I have microphone issues with my Jabra headset on the host and in the VM itself (Windows 10 VM).

So, what is the best solution for 6.8.0 kernel?

6.8.0 with patch for 17.6 or downgrade to 17.5.2?

LucMorizur's profile image
LucMorizur

I had this issue too (Ubuntu 22.04.5 LTS, Linux 6.8.0-44-lowlatency #44.1~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC, X.Org 1.21.1.4, KDE 5.92.0, plasmashell 5.24.7, Qt 5.15.3): around 10 minutes after reboot, Ubuntu freezes except mouse cursor. A console can still be reached (CTRL+ALT+F2), allowing to show that not process is zombie.

solution seems to have fixed it, many thanks. No issue with microphone. VMWare 17.6.0 works fine too.

 
Linux is an amazing world...