the IP and DNS required for the TKGI deployment?

[niceguy001]

Hi guys,

trying to deploy a brand new VMware TKGI 1.9 with NSX-T 3.0 environment in a POC infra.

okay,

at first I believe that this question shall be categorized into the "enterprise PKS" forum page, however, the question involves complicated overlay network settings therefore I decided to post here.

my target is to deploy a minimum tanzu kubernetes grid integration topology (just management nodes and a basic k8s cluster) with "Automated NSX-T deployment" networking option, and the topology is generally the same as figure below.

but things got tricky when I was planning the network (and DNS)in the management console VM, as shown in figure below.

actually, there is official description about the meaning of each section, as described here: Network Planning for Installing Tanzu Kubernetes Grid Integrated Edition with NSX-T | VMware Tanzu Docs

after several tires, my deployment hanging at configuring routing for ops manager which failed all the time and it seemed that the deployment assigns "floating IP" with the ops manager's IP address for NAT.

(the NSX-T network links such as T0, T1 and logical switches were deployed successfully.)

My questions are:

1. what is "floating IP" trying to achieve actually??? is it destined to do NAT on T0 router for management nodes and k8s cluster??

2. do the "floating ip" and "deployment CIDR" exist only on the NSX-T network not on the physical VLAN network?

3. I had a DNS server VM for the overall environment which is locating on physical VLAN network(VDS portgroup) , is it correct to utilize it for "Node DNS" and "Deployment DNS"?or probably there should be a specific DNS server for "Node DNS" and "Deployment DNS"?

hope my questions are on the topic cause the TKGI involved complicated network settings...

thanks in advance no matter the answers help or not.:smileyconfused:

[daphnissov]

1. what is "floating IP" trying to achieve actually??? is it destined to do NAT on T0 router for management nodes and k8s cluster??

The floating IP pool must be a routable segment. IPs from this pool will be used to create load balancer VIPs and SNAT rules (if applicable).

2. do the "floating ip" and "deployment CIDR" exist only on the NSX-T network not on the physical VLAN network?

See above. The floating IP pool is always routable and must be known to the physical world. The deployment CIDR is the CIDR for the management components.

3. I had a DNS server VM for the overall environment which is locating on physical VLAN network(VDS portgroup) , is it correct to utilize it for "Node DNS" and "Deployment DNS"?or probably there should be a specific DNS server for "Node DNS" and "Deployment DNS"?

You can use whatever you want as a global DNS server. The management components as well as the K8s clusters will use this DNS server as their main lookup, so you need to ensure it is accessible by all.

[niceguy001]

Hi daphnissov​ thanks for the explaination!

 

after another careful try i experienced a failed opsman deployment again, as shown in figure below.

my configurations for the TKGI deployment are:

1. 172.16.5.0/24(it's VLAN) for edge uplink network IPs/VIP

2. the "deployment CIDR" was set to 10.2.0.0/24(this should be the overlay network on logical switch)

3. the Pod and Node CIDP remained the same by default(which are 10.10.0.0/16 & 10.20.0.0/16 by default)

4. i configured "floating IP" range from 172.16.55.20 to 172.16.55.35 and this is another VLAN network different from edge uplinks'.

 

obviously, the vSphere showed that the opsman VM has an IP address 10.2.0.2 and it's located on LS that NSX-T created for TKGI.

however, the TKGI management console  showed the error of the routing to ops manager, so i started to guess whether the "deployment CIDR" should be the same as edge uplink network or not?

 

I can ping from the TKGI management console to the edge uplinks' IP but couldn't reach ops manager's ip 172.16.55.20(as shown in figure above).

note that the floating IPs are located on a VLAN which is routable to other VLANs, such as edge uplink's.

 

one of the weird part is the SNAT and DNAT rules that NSX-T created on edge indeed map the 172.16.55.X IP to the 10.2.0.x IP but routing problem remained the same.

 

***update

can someone tell me which load-balancer topology does TKGI utilize? one-arm or inline? or probably TKGI deploys load-balancer with random topology?

[RaymundoEC]

it uses one arm for each segment you create for TKGI