NCP nsx_node_agent crashing on startup

[50_ZZZZZ]

pod/nsx-node-agent-2p8g4       1/2     CrashLoopBackOff   160        16h   17.100.48.85   k8sn2.crhc.cn   <none>           <none>

pod/nsx-node-agent-8dlhv       2/2     Running            148        16h   17.100.48.81   k8sm1.crhc.cn   <none>           <none>

pod/nsx-node-agent-ff4mj       1/2     Error              110        16h   17.100.48.84   k8sn1.crhc.cn   <none>           <none>

pod/nsx-node-agent-wbshz       1/2     CrashLoopBackOff   142        16h   17.100.48.86   k8sn3.crhc.cn   <none>           <none>

kubectl logs  -f pod/nsx-node-agent-ff4mj -c nsx-node-agent -n nsx-system

1 2019-12-25T02:31:27.735Z k8sn1.crhc.cn NSX 7 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO" security="True"] nsx_ujo.common.nsx_log_adaptor Initialized log configuration

1 2019-12-25T02:31:28.233Z k8sn1.crhc.cn NSX 7 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="WARNING"] nsx_ujo.common.privilege Privsep daemon check failed for context nsx_ujo.common.privilege.node_agent_pri: 'NoneType' object has no attribute 'exchange_ping'

1 2019-12-25T02:31:28.234Z k8sn1.crhc.cn NSX 7 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] oslo.privsep.daemon Running privsep helper: ['sudo', '-E', 'privsep-helper', '--config-file', '/etc/nsx-ujo/ncp.ini', '--privsep_context', 'nsx_ujo.common.privilege.node_agent_pri', '--privsep_sock_path', '/tmp/tmpniXfQf/privsep.sock']

1 2019-12-25T02:31:28.751Z k8sn1.crhc.cn NSX 7 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] oslo.privsep.daemon Spawned new privsep daemon via rootwrap

1 2019-12-25T02:31:28.752Z k8sn1.crhc.cn NSX 7 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="DEBUG"] oslo.privsep.daemon Accepted privsep connection to /tmp/tmpniXfQf/privsep.sock

1 2019-12-25T02:31:28.710Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] oslo.privsep.daemon privsep daemon starting

1 2019-12-25T02:31:28.718Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] oslo.privsep.daemon privsep process running with uid/gid: 0/0

1 2019-12-25T02:31:28.722Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] oslo.privsep.daemon privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/none

1 2019-12-25T02:31:28.722Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] oslo.privsep.daemon privsep daemon running as pid 31

1 2019-12-25T02:31:28.985Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO" security="True"] nsx_ujo.agent.agent Starting nsx_node_agent

1 2019-12-25T02:31:28.986Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="WARNING"] nsx_ujo.common.privilege Privsep daemon check failed for context nsx_ujo.common.privilege.ovslib_pri: 'NoneType' object has no attribute 'exchange_ping'

1 2019-12-25T02:31:28.987Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] oslo.privsep.daemon Running privsep helper: ['sudo', '-E', 'privsep-helper', '--config-file', '/etc/nsx-ujo/ncp.ini', '--privsep_context', 'nsx_ujo.common.privilege.ovslib_pri', '--privsep_sock_path', '/tmp/tmpn_oIl8/privsep.sock']

1 2019-12-25T02:31:29.461Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] oslo.privsep.daemon Spawned new privsep daemon via rootwrap

1 2019-12-25T02:31:29.424Z k8sn1.crhc.cn NSX 47 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] oslo.privsep.daemon privsep daemon starting

1 2019-12-25T02:31:29.432Z k8sn1.crhc.cn NSX 47 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] oslo.privsep.daemon privsep process running with uid/gid: 0/0

1 2019-12-25T02:31:29.435Z k8sn1.crhc.cn NSX 47 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] oslo.privsep.daemon privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/none

1 2019-12-25T02:31:29.435Z k8sn1.crhc.cn NSX 47 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] oslo.privsep.daemon privsep daemon running as pid 47

1 2019-12-25T02:31:29.603Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] nsx_ujo.agent.nsxrpc_client Listening NSX RPC connection...

1 2019-12-25T02:31:29.607Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] cli.server.container_cli_server Starting node_agent CLI server

1 2019-12-25T02:31:29.635Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] nsx_ujo.agent.cni_watcher_lin CNI socket is listening...

1 2019-12-25T02:37:29.205Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] cli.server.container_cli_server Received request "{u'cmd': u'get_hyperbus_status', u'id': u'fbf3e16a-64f9-4ea9-92df-1e9188b203e2', u'args': {}}" from node_agent CLI client

1 2019-12-25T02:37:29.239Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] cli.server.container_cli_server Executed client request "node_agent" and sending response on {u'cmd': u'get_hyperbus_status', u'id': u'fbf3e16a-64f9-4ea9-92df-1e9188b203e2', u'args': {}} CLI server

1 2019-12-25T02:37:39.183Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] cli.server.container_cli_server Received request "{u'cmd': u'get_hyperbus_status', u'id': u'e4f3aa22-4db0-4f69-9044-621fd07eb177', u'args': {}}" from node_agent CLI client

1 2019-12-25T02:37:39.186Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] cli.server.container_cli_server Executed client request "node_agent" and sending response on {u'cmd': u'get_hyperbus_status', u'id': u'e4f3aa22-4db0-4f69-9044-621fd07eb177', u'args': {}} CLI server

1 2019-12-25T02:37:49.191Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] cli.server.container_cli_server Received request "{u'cmd': u'get_hyperbus_status', u'id': u'0dc93608-e568-4494-98c7-5b2c0b7343ea', u'args': {}}" from node_agent CLI client

1 2019-12-25T02:37:49.194Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] cli.server.container_cli_server Executed client request "node_agent" and sending response on {u'cmd': u'get_hyperbus_status', u'id': u'0dc93608-e568-4494-98c7-5b2c0b7343ea', u'args': {}} CLI server

1 2019-12-25T02:37:59.211Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] cli.server.container_cli_server Received request "{u'cmd': u'get_hyperbus_status', u'id': u'e2b44a29-9219-4d38-9574-3c7c9b926789', u'args': {}}" from node_agent CLI client

1 2019-12-25T02:37:59.214Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] cli.server.container_cli_server Executed client request "node_agent" and sending response on {u'cmd': u'get_hyperbus_status', u'id': u'e2b44a29-9219-4d38-9574-3c7c9b926789', u'args': {}} CLI server

1 2019-12-25T02:38:09.174Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] cli.server.container_cli_server Received request "{u'cmd': u'get_hyperbus_status', u'id': u'a97056be-10e3-4076-8621-dd0de554c301', u'args': {}}" from node_agent CLI client

1 2019-12-25T02:38:09.177Z k8sn1.crhc.cn NSX 31 - [nsx@6876 comp="nsx-container-node" subcomp="nsx_node_agent" level="INFO"] cli.server.container_cli_server Executed client request "node_agent" and sending response on {u'cmd': u'get_hyperbus_status', u'id': u'a97056be-10e3-4076-8621-dd0de554c301', u'args': {}} CLI server

I am trying to deploy ncp to join my kubernetes cluster to my 2.5 nsx-t deployment. All the nsx_node_agent containers are returning these logs. @

[50_ZZZZZ]

It's there anybody who knows?

[Mauricio Queiroz Santos Amorim]

Couldn't find the errors in the log. Could you send more logs?

Have you also tagged all the needed interfaces with NCP info?

[50_ZZZZZ]

I have only configured tags on the transport nodes. Does NCP info refer to yaml?

[Mauricio Queiroz Santos Amorim]

The tags I asked about are the ones regarding ncp/node_name and ncp/cluster, as mentioned here: Configure NSX-T Data Center Networking for Kubernetes Nodes

If this is not done correctly nsx_node_agent keeps crashing.