Hi, thanks for the reply. What I'm trying to do is to test the feasibility of using a software firewall. I understand the ESXi has its built in firewall but I prefer to use a proper firewall with its associated set of management features and other features such as one time password. So in the end it'd look like the following:
Internet -> vmnic0 -> vSwitch0 -> Firewall VM -> vSwitch Private -> VMkernal Port vmk0
I'm trying to do this in a two phase approach:
Phase 1 - Setup an additional VMkerel Port via vmnic1
This is done so in case I mis-configure the firewall I'd still somehow be able to control ESXi from vSphere via the vmnic1 connection. I just tried configuring the second connection for vmk1 and it looks like this. Port vmk0 is still connected directly to the Internet via vmnic0 at the moment. Note I haven't setup the Firewall VM here yet as you can see vmk0 is still connected directly to vmnic0 via vSwitch0.

However I'm not able to ping the second IP for vmk1. Not sure if it's a routing issue or ESXi just don't support having two management port both accessible via 2 NICs. Any ideas?
Phase 2 - Assign vmk0 to vmnic1 via DCUI
This is the end goal with vmk0 be the only management port (ie vmk1 will be removed by this time) with the following connection, which is what I stated above.
Internet -> vmnic0 -> vSwitch0 -> Firewall VM -> vSwitch Private -> VMkernal Port vmk0
Then if the Firewall VM screws up due to firmware upgrade or configuration issues I and VPN in to manage the host via IPMI and use console redirect to access DCUI. From within the DCUI, I'll then assign the management port vmk0 to vmnic1 resulting in this, bypassing the firewall.
Internet -> vmnic1 -> vSwitch1 -> VMkernal Port vmk0
I have yet to test this but do you see any issues with this?