How to install on Linux

[RaSystemlord]

I have VMware 17.5.2 running on one Kubuntu workstation. Now I tried to install it to a newer one, having latest Kubuntu LTS 24.04.2, but failed. I read several instruction sets and the one including downloading 17.5.1, editing initialization script and compiling again (with a conditional statement "if you have higher than 6.0.9 kernel then do something else"), got VMware 17.5.2 to start. However, no VM starts, but complains that vmmon module is not loaded.

Then I tried 17.6.3, which complains the same about vmmon not loaded. Installation didn't have any other problem.

How to get this to work? Is this perhaps about vmware uninstaller not removing everything OR is there something else that needs to be done?

I need this for making performance comparisons in several cases. The system has Asus ROG Strix Z590 motherboard and Intel i7 11th gen.

[louyo]

It is because the modules are not signed and you are booting with UEFI. You have to manually sign with MOK or turn off UEFI boot.

There is a FAQ on using MOK but I do it like:

sudo vmware-modconfig --console --install-all
sudo su
openssl req -new -x509 -newkey rsa:2048 -keyout MOK.priv -outform DER -out MOK.der -nodes -days 36500 -subj "/CN=VMware/"
sudo /usr/src/linux-headers-`uname -r`/scripts/sign-file sha256 ./MOK.priv ./MOK.der $(modinfo -n vmnet)
sudo /usr/src/linux-headers-`uname -r`/scripts/sign-file sha256 ./MOK.priv ./MOK.der $(modinfo -n vmmon)
mokutil --import MOK.der

You will need the password you give it when it asks for it during reboot.

When you reboot, it will ask if you want to do the MOK process. 

Best to read the FAQ, I don't remember the link. sorry

You will have to do this when the kernel is updated. I use Debian so the updates aren't frequent .

I choose to do this rather than turn off UEFI boot.

Lou

We the unwilling are lead by the unknowing

[Technogeezer]

Have you droppped into a root shell and checked the status of the VMware service with systemctl status vmware? Or checked the ststem log for errors upon start of the VMware services? That would give you a clue to what was happening - as@louyo notes, it could be the module signing if you're running with UEFI and have Secure Boot enabled in the firmware. 

I believe the following KB article is the one that's referenced in a prior reply https://knowledge.broadcom.com/external/article/315309/cannot-open-devvmmon-no-such-file-or-dir.html

------------------------------
- Paul (technogeezer)
vExpert 2025
------------------------------

[RaSystemlord]

Paul: Don't know what the below means, other than those modules are not loaded into kernel. I believe this is a standard case with 17.6.3 and Kubuntu 24.04.2, the latest LTS. I'm NOT using a Secure Boot (referring to my previous answer about "no UEFI boot", "not possible to be without it with Win 11 VM). (The same computer works, with same BIOS settings, with Win 11 Pro, however, no tweeks to support hardware virtualization has been done.)

Perhaps the MOK-thingy is now always required?

~/deli/vmware-host-modules-workstation-17.6.3$ systemctl status vmware
× vmware.service - LSB: This service starts and stops VMware services
     Loaded: loaded (/etc/init.d/vmware; generated)
     Active: failed (Result: exit-code) since Sun 2025-06-08 19:15:34 EEST; 11min ago
       Docs: man:systemd-sysv-generator(8)
    Process: 1759 ExecStart=/etc/init.d/vmware start (code=exited, status=1/FAILURE)
      Tasks: 1 (limit: 37998)
     Memory: 11.4M (peak: 14.7M)
        CPU: 134ms
     CGroup: /system.slice/vmware.service
             └─1866 /usr/sbin/vmware-authdlauncher

rogkubu vmware[1759]:    Virtual machine monitor - failed
rogkubu vmware[1759]:    Virtual machine communication interface - done
rogkubu vmware[1759]:    Virtual ethernet - failed
rogkubu vmware[1759]:    VMware Authentication Daemon - done
rogkubu systemd[1]: vmware.service: Control process exited, code=exited, status=1/FAILURE
rogkubu systemd[1]: vmware.service: Failed with result 'exit-code'.
rogkubu systemd[1]: vmware.service: Unit process 1812 (vmware-authdlau) remains running after unit stopped.
rogkubu systemd[1]: vmware.service: Unit process 1813 (logger) remains running after unit stopped.
rogkubu systemd[1]: Failed to start vmware.service - LSB: This service starts and stops VMware services.
~/deli/vmware-host-modules-workstation-17.6.3$

[RaSystemlord]

I'm still studying this, but as an answer to MOK/UEFI-thing:

a)

My Kubuntu VM has choices in Options-Advanced: BIOS/UEFI.

It was NOT UEFI

b)

My Win11 Pro VM, says that UEFI cannot be changed when TPM device is present.

So, that is a no-go to my understanding.

[Technogeezer]

If you're not running with UEFI secure boot enabled, anything to do with kernel signing and MOK shouldn't be necessary. Yes, Windows 11 pretty much requires UEFI Secure Boot and a TPM to run. Ubuntu does not. 

I built a a Ubuntu Desktop 24.04.2 VM (with Workstation 17.6.3 on Fedora 42 on a 2014 Mac mini) from scratch (UEFI firmware in the VM, no secure boot) and was able to get Workstation 17.6.3 installed in it. The kernel modules compiled (although I did have to install the build-essential package first to get the C compiler and development tools needed to build the modules). The Workstaiton GUI came up with no problems -- although I did not try to build a virtual machine. I'll enable nested virtualization and make sure that a small VM works. 

I'll give Kubuntu 24.04 LTS a try to see if I can reproduce. 

------------------------------
- Paul (technogeezer)
vExpert 2025
------------------------------

[RaSystemlord]

Paul, thanks a lot for your answers and confirmations.

I have built "the essentials" and gcc's according to previous instruction sets. And I was able to compile with 17.5.1 according to one instructions set - but vmmon still gave this error message.
With 17.5.2 I got the comments during install that vmnet and vmmon did not load to kernel - 17.6.3 is silent. However, in my previous post it shows that they are not with kernel also in 17.6.3.

I have Kubuntu 24.04.2 - in the other Kubuntu computer - which worked - I might have it without .2 (not certain, it might have been 22.04).

But yes, 17.6.3 installs and it does start. The problem is booting a VM (I tried Kubuntu 24.04 and Win 11 Pro). Those VMs are in regular use with my Win 10 Pro.

I do have NTFS-filesystem and running VMs from an external M.2 SSD with NTFS - some years ago, on Linux platform, running a VM from NTFS did not work at all (it loaded for ages and probably would never have finished). It seems too far-fetched to think that this is the problem, since the boot-up of a VM seemed to work and error message is pointing to something else entirely.

As a perhaps separate issue, the physical computer probably boots with UEFI - since I have a switchable disks for Win 11 Pro in it. I haven't changed BIOS settings between these separate OS boots from a different disk.

[RaSystemlord]

Paul, I tried with my old system:
-  Kubuntu 24.04 (probably upgraded from old versions), but not sure what Upgrades have been run afterwards

- i7 (probably 1st Gent), Asus Motherboard
- Kernel is 6.8 (not 6.11 like with .2 version)

It does open VMs, including the same as previously mentioned Kubuntu 24.04. (providing that I said "I copied it", which means that it's fit for this older infrastructure and not the same as with newer Intel&else). It was very slow, the interface was USB 2.0, which explains.

Is there something very wrong how the instructions deal with a newer Kernel?

[Technogeezer]

17.6 seems to deal better with newer kernels. From what I've seen, it hasn't needed to use the patched module sources available on the 'net to work with kernels up to and including 6.14. 17.6 has compiled out of the box for me - no patched sources needed. 

I can't remember, though, if 17.5 needs the patched module sources to run on newer kernels - there are some indications that it does. I can give that a try on my VM.

 I do know that 17.0 needs patched sources to run on newer kernels. 

------------------------------
- Paul (technogeezer)
vExpert 2025
------------------------------

[Technogeezer]

From latest testing, 17.5.2 definitely does not compile the kernel modules out-of-the-box on newer kernels (appears to be 6.9 and later). It needs patched kernel module source. Unfortunately, the mkubecek versions that are the usual go-to source for patched kernel modules are not up to date with latest kernels and don't work. I've found that there are at least two forked versions of the mkubecek sources on GitHub that advertise as updated for 17.5.2 and newer kernels:

https://github.com/nan0desu/vmware-host-modules/tree/workstation-17.5.2-k6.9%2B

https://github.com/bytium/vm-host-modules/tree/17.5.2

I used the bytium version and it compiled and installed on the 6.11 kernel I have in my Kubuntu 24.04.2 VM using the instructions posted with those modules. No editing necessary. Workstation starts and I can fire up a small VM without issue. 

------------------------------
- Paul (technogeezer)
vExpert 2025
------------------------------

[RaSystemlord]

Ok, thanks a lot Paul :) 

I will test this later in my system and definitely having 17.5.2 is OK for me, even preferred for performance comparison.

[RaSystemlord]

No, the same thing goes on. Perhaps something left from previous attempts that uninstall doesn't take away?

This obviously isn't the entire story of trying this out, but is perhaps enough.

 
/include/linux/minmax.h:330: note: this is the location of the previous definition
  330 | #define MAX(a,b) __cmp(max,a,b)
      |
  LD [M]  /home/jorma/vm-host-modules/vmnet-only/vmnet.o
  MODPOST /home/jorma/vm-host-modules/vmnet-only/Module.symvers
  CC [M]  /home/jorma/vm-host-modules/vmnet-only/vmnet.mod.o
  LD [M]  /home/jorma/vm-host-modules/vmnet-only/vmnet.ko
  BTF [M] /home/jorma/vm-host-modules/vmnet-only/vmnet.ko
Skipping BTF generation for /home/jorma/vm-host-modules/vmnet-only/vmnet.ko due to unavailability of vmlinux
make[2]: Leaving directory '/usr/src/linux-headers-6.11.0-26-generic'
make -C $PWD SRCROOT=$PWD/. \
  MODULEBUILDDIR= postbuild
make[2]: Entering directory '/home/jorma/vm-host-modules/vmnet-only'
make[2]: 'postbuild' is up to date.
make[2]: Leaving directory '/home/jorma/vm-host-modules/vmnet-only'
cp -f vmnet.ko ./../vmnet.o
make[1]: Leaving directory '/home/jorma/vm-host-modules/vmnet-only'

~/vm-host-modules$
~/vm-host-modules$
~/vm-host-modules$ sudo make install
install -D -t /lib/modules/6.11.0-26-generic/misc vmmon-only/vmmon.ko vmnet-only/vmnet.ko
strip --strip-debug /lib/modules/6.11.0-26-generic/misc/vmmon.ko /lib/modules/6.11.0-26-generic/misc/vmnet.ko
if test -z ""; then /sbin/depmod -a 6.11.0-26-generic; fi
~/vm-host-modules$ sudo modprobe vmmon
modprobe: ERROR: could not insert 'vmmon': Key was rejected by service
~/vm-host-modules$
~/vm-host-modules$
~/vm-host-modules$ sudo modprobe vmnet
modprobe: ERROR: could not insert 'vmnet': Key was rejected by service
~/vm-host-modules$
~/vm-host-modules$
~/vm-host-modules$
~/vm-host-modules$

[Technogeezer]

I would expect these messages if you are running Workstation on a Linux system that has EFI secure boot enabled. In that case you need to go through the process to sign the newly built kernel modules and register the key (as noted by @louyo's post above). 

------------------------------
- Paul (technogeezer)
vExpert 2025
------------------------------

[RaSystemlord]

Paul, you are probably right - I will test later. The workstation itself has surely that kind of boot system since it has Win 11 Pro (some latest) on one disk and Linux try-out comes from a different disk ... and no changes to BIOS between using them.

I will let you know how it goes.

[RaSystemlord]

Thanks Paul. I thought the discussion was about VM and not the Host system. It works now.

Thanks for the help. I got some perhaps interesting performance comparisons done.