It is either something very deep and complicated or too simple to notice.
The setup right now is:

==================
Managed Switch
Port 1 - Trunk to Gateway - a pfSense box
Port 2 - Trunk to Host - ESXi 7 ----> dvSwitch
Port 3 - Tagged (VLAN5) to IPMI of Host - a separate interface on a Supermicro motherboard
Port 4 - Tagged (VLAN5) to Laptop
vDSwitch
PortGroup 1 - Management - VLAN5
PortGroup 2 - System - VLAN5
VMs
VM1 - FreeNAS - PortGroup 1
VM2 - Windows 10 - PortGroup 2
==================
--- IPMI and Laptop are setup to NOT tag their frames with Vlan-IDs - the frames get tagged at the ingress at Port 3 and Port 4, respectively
--- VM1, VM2, pfSense, Laptop - can ping each other just fine
--- pfSense, Laptop are able to ping IPMI
--- VM1, VM2 cannot ping IPMI
For some reason either the pings from the VMs do not get to IPMI or the replies get dropped.
It might have something to do with this:
A while back I needed to free up Port 3, so I combined the IPMI to the "Trunk to Host" traffic on the same cable (by setting up IMPI as "shared", and to send its frames as tagged VLAN5)
I do not remember if VM access to IPMI worked before this change, but it did not work during the change and now too - after I put things back the way they were.
Cannot figure it out. Any ideas?
Thanks!