If I understand the situation correctly, it should be easy to change the network setup.
From what you write I assume that you are using Cisco switches (Access Ports, Trunk Ports). I further assume that the native/default VLAN is VLAN 1 (default).
The host is currently connected to an Access Port on VLAN 5, so the port groups for the Management Network, and the VM network do not have a VLAN ID assigned.
If you change the physical switch port to a Trunk Port you will either have to change the VLAN ID on the ESXi host's exiting port groups from 0 to 5, or define VLAN 5 as the default VLAN on the physical port.
Personally I prefer the first method, i.e. not touching the native/default VLAN on the physical side, but setting the VLAN ID on the ESXi host's port groups. To do this with only minimal network interruption, either set the VLAN ID on the Management Network first (you will lose connection after doing this), followed by changing the physical port, or change the physical setup first, and then change the VLAN ID for the Management Network from the ESXi host's DCUI.
André