vmkernel removing an IPv4 Route

[NickDaGeekUK]

Hi everyone,
I have a weird one, I have a couple of custom VM stacks duplicated on three physical hosts.

I am now seeing an entry in the IPv4 Routing Table in vSphere Web Client for one of those custom stacks. It is an entry I do not recall ever having set as a static route.

Not only is it very specific (a /32 prefix length) but the Gateway is outside the Network Address range i.e. network address is in the 192.168.x.x class but the gateway is in the 10.x.x.x class. This is not a configuration I have ever used.

I have tried to remove it using esxcli

esxcli network ip route ipv4 remove -N "netstack" -n 192.168.x.x/32 -g 10.x.x.x (I am using the full IP address but have obscured the actual IP for security reasons)

and even if I specify the -N "netstack instance name" option it says unable to find route 192.168.x.x/32 with gateway 10.x.x.x

If I attempt to list the IP Route Table of the specific vmk using esxcli network ip route ipv4 list -N "netstack name" it does not show the new route I am seeing in vSphere only the specific routes I configured manually when I created them.

What worries me is that it is a route to the VPN interface of our firewall and I do not ever recall seeing it before or entering it as an static route. I have read that ICMP redirects can add static routes that cannot be removed.

Internet Control Management Protocol Redirects (broadcom.com)

Any advice on this matter would be greatly appreciated.

@Casey

can't seem to reply to you or use the @ mention as you don't appear on the list. Hence editing this post.
Really appreciate you coming back to me on this with such helpful and specific recommendations. I will check again to be sure but the first command you are suggesting I believe does not return the suspicious route, it only shows in the vSphere GUI. Your step number 4 "If the route doesn't appear in your manual checks but is visible in the vSphere Client, try to remove it by specifying the netstack again"

I have tried and it still can't find it. I am going to try again this afternoon as the logs you mention might help me track this down. :)

[Casey]

Hi there,

It sounds like you're experiencing an unusual routing issue, potentially caused by ICMP redirects or an inadvertent configuration change. Let's work through the troubleshooting steps to identify and resolve this problem.

Steps to Diagnose and Remove the Route

1. Verify Current Routes:

   • Use esxcli to list all current routes, ensuring you're not missing any details.

     
     esxcli network ip route ipv4 list
     
     
     

2. Check Specific Netstack Routes:

   • List routes for the specific netstack you're concerned with.

     
     esxcli network ip route ipv4 list -N "netstack_name"
     
     
     

3. Check for ICMP Redirects:

   • As you mentioned, ICMP redirects can add routes. To see if this is happening, review the logs.

     
     grep ICMP /var/log/syslog.log
     
     
     

   • If ICMP redirects are the cause, consider disabling them temporarily to see if the issue persists.

     
     esxcli network ip set --ipv4-gateway-disable=true
     
     
     

4. Remove Specific Routes:

   • If the route doesn't appear in your manual checks but is visible in the vSphere Client, try to remove it by specifying the netstack again.

     
     esxcli network ip route ipv4 remove -N "netstack_name" -n 192.168.x.x/32 -g 10.x.x.x
     
     
     

5. Restart Network Service:

   • Sometimes, changes don't take effect immediately. Restart the management network to force a refresh.

     
     /etc/init.d/network restart
     
     
     

6. Check VMkernel Logs:

   • Look for any additional clues in the VMkernel logs that might indicate how the route was added.

     
     tail -f /var/log/vmkernel.log
     
     
     

7. Check Host Network Configuration:

   • Verify there are no misconfigurations in the network settings of the host that could lead to unexpected routes being added.

Disabling ICMP Redirects Permanently

If ICMP redirects are indeed the cause, and you determine they need to be permanently disabled, you can add the following configuration:

1. Disable ICMP Redirects:

   • Create or edit the esx.conf file to include the following:

     
     /net/tcpip/ipv4/conf/all/accept_redirects=0
     /net/tcpip/ipv4/conf/default/accept_redirects=0
     /net/tcpip/ipv4/conf/eth0/accept_redirects=0
     
     
     

2. Restart the Host:

   • A restart may be necessary to apply these changes effectively.

Let us know how it works out! Be sure to include logs, screenshots, and any relevant details. 😊