ADX-1000 Why does a VIP see the real server ports as Failed. How to debug why failed state?

[ops7]

Fairly simply setup. Load Balancing a TCP port 11972.

The ADX has several virtual ethernet interfaces (for various network/firewall issues)

Its connected in leaf mode with only one ethernet port active. Hence real servers use source-nat

ver      12.5.01cT403

interface management 1  ip address 172.25.86.181 255.255.255.0

interface ethernet 6  ip address 192.168.100.1 255.255.255.252  no spanning-tree

interface ve 1  ip address 172.25.66.217 255.255.255.0

interface ve 2  ip address 172.25.68.88 255.255.255.0

interface ve 3  ip address 172.25.67.178 255.255.255.0

interface ve 4  ip address 172.25.65.58 255.255.255.

When I look at the real server its status is active. I can telnet to the real servers on port 11972 from the ADX-1000

But when I create a virtual server and bind to the remote-server port they show up as failed??

telnet@DC1-ADX1000(config)#show server real mvstmcdr11                  
Real Servers Info
========================
Remote Name: mvstmcdr11      State: Active       Cost: 1  IP:172.25.68.198:   1
Mac: 0050.5697.5803          Weight: 1/1              MaxConn: n/a
SrcNAT: cfg, op              DstNAT: not-cfg, not-op    Serv-Rsts: 0
Rx throughput: 0 Kbps    Tx throughput: 0 Kbps
tcp conn rate:udp conn rate = 0:0, max tcp conn rate:max udp conn rate = 0:0
BP max local conn configured No: 0 0 0 0 0 0
BP max conn percentage configured No: 0 0 0 0 0 0
Use local conn : No
SIP current TCP connections = 0

Port    St  Ms CurConn TotConn    Rx-pkts   Tx-pkts   Rx-octet   Tx-octet   Reas
----    --  -- ------- -------    -------   -------   --------   --------   ----
default DIS 0  0       0          0         0         0          0          0
11972   FAL 0  0       0          0         0         0          0          0

telnet@DC1-ADX1000(config)#show server virtual mvstmcdr 11972

Name: mvstmcdr               State: Enabled             IP:172.25.65.153:   1
Pred: round-robin            ACL-Id: 0                  TotalConn: 0
Sym: group =  1 state =  5 priority = 100 keep =  0
     dyn priority/factor = 100/  0
 Activates =    1, Inactive= 0 sym-active = 1
     Best-standby-mac: 0000.0000.0000

Total weight for virtual port = 2

Bind count for virtual port = 2
Active count for virtual port = 0
SLB state for vport = Not healthy

Rx PPS =        0                        Tx PPS =        0              
Rx Throughput = 0          Kbps          Tx Throughput = 0          Kbps
Note: The above statistics lag by 1 second

Port    State     Sticky  Concur  Proxy  DSR   CurConn  TotConn  PeakConn  
----    -----     ------  ------  -----  ---   -------  -------  --------  
11972   enabled   NO      NO      NO     NO    0        0        0         

Port    Rx-pkts    Tx-pkts    Rx-octet             Tx-octet             
----    -------    -------    --------             --------             
11972   0          0          0                    0                    

Binding Information:
=====================
       11972 -------> mvstmcdr11: 172.25.68.198,  11972 (remote) (Failed)
                      mvstmcdr12: 172.25.68.199,  11972 (remote) (Failed)

Bound Port Information:
========================
State(St) - ACT:active, ENB:enabled, FAL:failed, TST:test, DIS:disabled,
            UNK:unknown, UNB:unbind, AWU:await-unbind, AWD:await-delete
            HLD:held-down

Port    St  Ms CurConn TotConn    Rx-pkts   Tx-pkts   Rx-octet   Tx-octet   Reas
----    --  -- ------- -------    -------   -------   --------   --------   ----
mvstmcdr11: 172.25.68.198
11972   FAL 0  0       0          0         0         0          0          0

mvstmcdr12: 172.25.68.199
11972   FAL 0  0       0          0         0         0          0          0

server virtual mvstmcdr 172.25.65.153
 sym-priority 100
 sym-active
 port default disable
 predictor round-robin
 port 11972
 port 11972 tcp-only
 bind 11972 mvstmcdr11 11972 mvstmcdr12 11972


server remote-name mvstmcdr11 172.25.68.198
 port default disable
 no-l3-check
 source-nat
 port 11972
 port 11972 tcp-only
 port 11972 keepalive
 hc-track-port 11972 11972


server remote-name mvstmcdr12 172.25.68.199
 port default disable
 no-l3-check
 source-nat
 port 11972
 port 11972 tcp-only
 port 11972 keepalive
 hc-track-port 11972 11972

[ops7]

Further I have set a trace on one of the real servers and found the ADX is sending  UDP.? when I said tcp-only for port 11972

2    0.000044    172.25.68.198    172.25.68.88    ICMP    326    Destination unreachable (Port unreachable)

3    1.899623    172.25.68.88    172.25.68.198    UDP    298    1795 → 11972  Len=256

4    1.899665    172.25.68.198    172.25.68.88    ICMP    326    Destination unreachable (Port unreachable)

5    3.899541    172.25.68.88    172.25.68.198    UDP    298    1795 → 11972  Len=256

6    3.899584    172.25.68.198    172.25.68.88    ICMP    326    Destination unreachable (Port unreachable)

7    5.799324    172.25.68.88    172.25.68.198    UDP    298    1795 → 11972  Len=256

8    5.799367    172.25.68.198    172.25.68.88    ICMP    326    Destination unreachable (Port unreachable)

9    7.799096    172.25.68.88    172.25.68.198    UDP    298    1795 → 11972  Len=256

10    7.799139    172.25.68.198    172.25.68.88    ICMP    326    Destination unreachable (Port unreachable)

[Mick.Day · Accepted solution]

Hi,

You need to define the port type, here is extract from manual

Adding a port and specifying its type
By adding a port, you also automatically enable periodic Layer 4 (and Layer 7, if applicable) keepalive health checks for the port. If you do not specify the port type (TCP or UDP), the ServerIron ADX assumes the port type is UDP.
To add a port and specify that it is a TCP port, enter commands such as the following.
ServerIronADX(config)# server port 8080
ServerIronADX(config-port-8080)# tcp

Hopefully this will solve your problem, obviously in your case using port 11972

Regards

Mick

[ops7]

Thank you Issue resolved.
I was confused by the setting of the tcp-only config against the real server port.
I guess this means that global setting of the port then applies to the virtual server which in turns polls the real server. SO having the virtual server (global setting) defined as a tcp only port then checks the real server only with tcp.
Cheers

[Mick.Day]

Hi,

Good news your problem is resoved.  Yes the global setting for the port designates how all L4-7 healthcheck will treat a non-well know ports. Default is always UDP unless specified.

Regards

Mick