VSphere Routing Concern

[caster]

We have two sites, our main site, which Ill refer to as SiteA, and a remote site, which Ill refer to as SiteB.

SiteA has a VCenter appliance running with the IP 10.20.4.20. SiteA and SiteB are connected via a MetroE connection. The route from SiteA to SiteB is reachable via IP 10.20.4.250. In addition to this there is also a VPN connection on their firewall for redundancy, which we manually control at this point.

SiteB's VSphere server IP is 10.30.4.10. SiteB’s to SiteA’s MetroE connection routes via IP 10.30.4.250 .

During normal operation SiteB 10.30.4.10 gets to 10.20.4.20 via the MetroE connection on 10.30.4.250 . When we need to schedule maintenance on our MetroE connection we failover to VPN. The VPN route is reachable via SiteB default gateway which is on the firewall, 10.30.4.1. The VSphere server at SiteB is also configured with a default gateway of 10.30.4.1 . My problems begin with VSphere when we failover to VPN. Vsphere is hardcoding a manual route of 10.20.4.20 255.255.255.255 10.30.4.250 vmk1 Manual which can be seen running the command esxcli network ip route ipv4 list.

The MetroE route might still be reachable but is not the way we want the traffic to take. All other machines when we failover to VPN take the VPN route.

I reached out to VmWare who says the route is always added to their route table and does not disappear until it is unreachable for an HR. As no other devices do this, I am wondering what is going on with VSphere. How can we get VSphere to behave like all other devices that always go to their default gateway to get the route? I can simulate this issue with other sites as well.

[NirbhayRathore]

Can you please get the output for the following two commands on ESXi host.

# esxcfg-vmknic -l

# esxcli network ip route ipv4 list

[caster]

FROM SITEB

Interface  Port Group/DVPort/Opaque Network        IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type                NetStack

vmk0       Management Network                      IPv4      10.30.4.10                       255.255.255.0   10.30.4.255           00:1f:fe:ab:50:ef 1500    65535     true    STATIC              defaultTcpipStack

Network          Netmask          Gateway          Interface  Source

---------------  ---------------  ---------------  ---------  ------

default          0.0.0.0          10.30.4.1   vmk0       MANUAL

10.20.4.20  255.255.255.255  10.30.4.250  vmk0       MANUAL

10.30.4.0    255.255.255.0    0.0.0.0          vmk0       MANUAL

[NirbhayRathore]

The following route is coming from ICMP redirects from the firewall / router. You can disable ICMP redirects on the router and check.

10.20.4.20  255.255.255.255  10.30.4.250  vmk0       MANUAL

=> Reference KB:

VMware Knowledge Base

VMware Knowledge Base

[caster]

I agree as we see that in the wireshark.

Why is VSphere adding these as a manual route and keeping it there?  Other network devices are switching their routes based on the route change on the firewall.

[NirbhayRathore]

• Dynamic host route entries have a timeout of one hour by default, during this timeout period if the ESXi communicates with the destination host the timeout will be reset and restarts its countdown.
• When more number of dynamic routes created then the timeout period will be shorter than 1 hour. When timeout expires, dynamic route entry will be removed.

To manually trigger the clearing of the ICMP Redirect generated routes, restart the network using these commands:

# esxcli network ip interface set -e false -i vmk0

# esxcli network ip interface set -e true -i vmk0

[caster]

is there a way to deny icmp redirects to vsphere hosts?  Found the below

Deny IPv4 ICMP Redirect Messages

[NirbhayRathore]

These commands would work on vCenter Appliance.

But in this case the routes are added on ESXi host. So either disable / enable vmk0 from DCUI or disable ICMP redirect on router / firewall.

[caster]

appreciate your help!