VCF Automation is sometimes pingable and sometimes not. [VCFA 9.0.0.0]

[aenagy]

After deploying VCF Automation I was able to ping and login to the web console. Then I wasn't able to ping or login, then I was able to. Now I can't ping or login.

Config from VCF Operations > Fleet Management > Lifecycle. Note that all IP addresses are on the same /24 subnet.
Primary VIP: 10.1.95.157
IP Pool: 10.1.95.236,10.1.95.237
Additional VIPs: 10.1.95.234,10.1.95.235

IP addresses of virtual appliance:
10.1.95.236 (IP Pool) and 10.1.95.234 (Additional VIPs)

When VCFA was pingable and web console worked:

PowerShell> tracert vcf-lab02-m01-aria-auto.convergelabs.ca

Tracing route to vcf-lab02-m01-aria-auto.myOrg,myTLD [10.1.95.157]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  10.50.0.3
  2    21 ms    <1 ms    <1 ms  vcf-lab02-m01-aria-auto.myOrg.myTLD [10.1.95.157]

Trace complete.

Then this happened:

ping -t vcf-lab02-m01-aria-auto.myOrg.myTLD

Pinging vcf-lab02-m01-aria-auto.myORG.myTLD [10.1.95.157] with 32 bytes of data:
Reply from 10.1.95.157: bytes=32 time=3ms TTL=63
Reply from 10.1.95.157: bytes=32 time=1ms TTL=63
Reply from 10.1.95.157: bytes=32 time<1ms TTL=63
Reply from 10.1.95.157: bytes=32 time<1ms TTL=63
Reply from 10.1.95.236: Destination host unreachable.
Reply from 10.1.95.236: Destination host unreachable.
Reply from 10.1.95.236: Destination host unreachable.
.... and so on ....

When it didn't work:

PowerShell> tracert vcf-lab02-m01-aria-auto.myOrg.myTLD

Tracing route to vcf-lab02-m01-aria-auto.myOrg.myTLD [10.1.95.157]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  10.50.0.3
  2     1 ms    <1 ms    <1 ms  10.1.95.236
  3  10.1.95.236  reports: Destination host unreachable.

Trace complete.

I suspect that 10.50.0.3 is a VRP router. Both IP addresses are pingable. I am able to ssh to the VCFA appliance.

Q1: What is broken and/or how do I fix this? This happened on a previous deployment with exactly the same configuration and after attempting to reboot via VCF Operations the virtual appliance just seemed to die. After 2+ hours VCF Operations wasn't able to see the virtual appliance. I eventually re-deployed and got a repeat of the above symptom.

Q2: What logs on the VCFA appliance should I be looking at?

Q3: Why can't I find the load balancer in NSX Manager or vCenter?

-------------------------------------------

[Ron Perron]

I too am experiencing the same thing!! Perhaps this issue is related to the "native" Load Balancer..??

Only speculation at this moment. Problem I am having is there appears to be little to no info on where or what this "native" LB is??

https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/design/design-library/vcf-automation-deployment-models(1)/load-balancing.html

"The native load balancer is the default automatically configured load balancer for both the simple and highly available deployment models."

Can anyone provide some assistance or feedback as to where to troubleshoot this thing further?

Thanks. RonP

-------------------------------------------

[Ron Perron]

I discovered and drafted a few pointers below that will provide an opportunity to assess whether the native (internal) load balancing feature is functioning correctly. 

Native Load Balancer Services in VCFA 9 

 

The VCF 9 Automation UI services are behind the Istio Service mesh. External traffic must go through the istio-ingressgateway service, which then routes to the various internal services. 

 

Below are a few simple commands to assess both the configuration and operational status of the internal load balancer:

 

• To determine which IPs the istio-ingressgateway and Ports the Service is actually listening on, run the following command as root: 

 

• • kubectl get svc istio-ingressgateway -n istio-ingress  

  

• The command to Describe the Services of the Load Balancer is as follows: 

• • kubectl describe svc istio-ingressgateway -n istio-ingress 

 

• To test the VIP properly, run the following command (Note: The "Host" and "https" values below are based on output from above commands): 

 

• • curl -k -v -H "Host: vcf-m1-auto1.example.com" https://x.x.x.x

 

• To see the actual HTML content, follow the redirect by executing the following command: 

 

• • curl -k -L -H "Host: vcf-m1-auto1.example.com" https://x.x.x.x/automation 

Hope this helps.

-------------------------------------------

[aenagy]

Ron:

The next time I encounter this symptom I'll try your troubleshooting and report back the results here.

-------------------------------------------