Error with management cluster deployment "control plane is not available yet, retrying"

[Kenny]

Dear all,

I am facing the above issue when building TKG management cluster

The control plane node was created so when I logged in it I can found that there is error

"Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized". There is no "10-antrea.conflist" file in the path /etc/cni/net.d and there is no image for antrea. And the node is not ready due to this error.

However, I could manually pull images and there is no issue with flows.

When it's timeout. If I redo the command "tanzu mc create -f mc-cluster.yaml", sometimes it's successful, sometimes it's failed again.

I tried to check but there is no useful info in cloud-init log.

Please help to provide your advice how to fix this issue.

Thanks and Regards,

[Shubham Sharma]

Hi, could you share the following info

• Is it a prod plan or dev plan? How many control plane nodes were created if it's a prod plan?
• Are you using AVI or kube-vip?
• Output of ping command from
  • jumpbox to vcenter
  • jumpbox to control plane node created
• Are you using private image repo?
• Is this environment using proxy or internet restricted?
• crictl images output from the node that was created?
• If the kind cluster is still around, check the logs for tanzu-addons-controller-manager and kapp-controller to see if you get any more info.

[Kenny]

Hello,

Thank you for your response. I would like to update below info

• Is it a prod plan or dev plan? How many control plane nodes were created if it's a prod plan?
  • It happened for both. Now I'm using dev plan to save time (1 control plane node and 1 worker node)
• Are you using AVI or kube-vip?
  • I'm using AVI. 
• Output of ping command from
  • jumpbox to vcenter
  • jumpbox to control plane node created
    • The flows are opened and If I rerun it completed successfully sometimes (randomly)
• Are you using private image repo?
  • I'm using public repo but at the moment the error happened, I tried manually form the control plane node and could pull the image successfully
• Is this environment using proxy or internet restricted?
  • It's restricted but the prerequisites flows are opened that's why the process can be completed sometimes
• crictl images output from the node that was created?
  • at that moment, it didn't pull the image for antrea but I could pull it successfully from the error control plane node (manually)
• If the kind cluster is still around, check the logs for tanzu-addons-controller-manager and kapp-controller to see if you get any more info.
  • The kind cluster is still there, could you please help to explain how to check the logs

Thanks and Regards,

[Kenny]

Hello,

There is nothing for kapp-controller. I checked log for tanzu-addons-controller-manager and found that there was error message "PackageInstallStatusController "msg"="failed to reconcile PackageInstallStatus for package 'tkg-system/antrea.tanzu.vmware.com.1.11.3+vmware.2-tkg.1-advanced'" "error"="unable to get PackageInstall 'tkg-system/mc-tshoot-ams2-12-antrea': PackageInstall.packaging.carvel.dev \"mc-tshoot-ams2-12-antrea\" not found" "cluster-name"="mc-tshoot-ams2-12" "cluster-ns"="tkg-system""

I'm still investigating to find the root cause.

Please give me your advice if any.

Thanks and Regards,