DFW Policy for Antrea Tanzu Cluster not working as intended

[Stephan_Zur]

Dear NSX Community,

I have some questions regarding NSX DFW security policy with regards to Tanzu/Antrea cluster integration. 

Versions used:
VMware ESXi, 8.0.3
NSX 4.2.0
TKG v1.29.4+vmware.3-fips.1 Photon OS
Antrea v1.13.3

I have created an NSX Group to match the  External-IP of a Kubernetes Service of type LoadBalancer within a namespace. This works and proper IP gets dynamically allocated to the NSX Group.
Next I configure an NSX DFW Policy and Rule using this group as the destination to block the traffic from any source.
- When testing to access the service from a VM living on another T1 of same NSX infra the traffic is blocked as intended.
- When testing to access the service from NSX external sources the traffic is still allowed. I can only block access to the Kubernetes service from external when replicating same policy on Gateway FW T0. Is this intended behavior? Does not make sense IMHO.
- When setting the NSX Group in the applied to field, the rule does not get intantiated on vnic leading to Tanzu Node holding the workload and rule is not enforced. Verified with vsipioctl getrules. It only works when applied to DFW.

Any comment that help me better understand the innerworkings of Antrea/Tanzu/NSX is much appreciated. 

Thanks ahead

Stephan

[Yves Hertoghs]

Hi,

Unfortunately using a Group based on IP addresses/IP-sets in the Apply-To field, will not work.  The Apply-To field will be evaluated by the NSX Management Plane , from which it will create a list of vNICs.  The NSX Management Plane has no way to find out what vNic points to which IP address.  It knows what vNICs owns what IP address, but not the other way around.  You could create another Group , which is not IP based, and configure that under the Applied-To field.

As to the other problem you are referring, note that GFW will only happen on the uplink of the T1, so GFW will not work between segments on the same T1 (unless you are using a Service Interface on the T1).  But I am a bit confused why you are referring to DFW in your question, as it seems more related to GFW?

[Stephan_Zur]

Hi Yves,

Many thanks for your reply.

I have not configured the group based on IP address. I have used dynamic membership criteria Kubernetes Cluster AND Kubernetes Namespace AND Kubernetes Service.
This is working fine and NSX is picking up the correct IP and list them in the Effective member section of that group. But when adding same group in the applied-to field it is not instantiated at the vnic and rule is not enforced. This only works when rule is applied to DFW.

I am interested in the DFW rule. This is what i would like to get working. When the rule is applied to DFW following scenario works:

source is VM on another T1 than Tanzu --> destination is Kubernetes service as defined in Group --> DFW rule is blocking as intended 

However following scenario does not work:
source is outside NSX --> destination is Kubernetes service as defined in Group --> DFW rule is not dropping this flow as it should.

I created/replicated same rule on T0 GW FW for test purpose. --> this rule blocks traffic coming from NSX external sources as intended. I don't understand this behavior. 

Regards
Stephan

[DanielKrieger]

However following scenario does not work:
source is outside NSX --> destination is Kubernetes service as defined in Group --> DFW rule is not dropping this flow as it should.

The problem here is that the dynamic group does not know the ingress IP, but only the cluster network IPs. You have to write a DFW rule based on IP and set it as Applied To on DFW. Deny any to Ingress IP

------------------------------
-----------
If you found this post useful, please click recommend. Thank you
-----------
My Blog: https://evoila.com/blog/author/danielkrieger/
------------------------------

[Stephan_Zur]

Hi All,

Thanks for your replies. 

The solution is to apply the policy on GW FW T1 going to TKGS Cluster. Default policy can be set to drop. And with additional policies above the needed ports can be allowed.

Have nide day.

Stephan