vSphere with Tanzu TKC Creation Error

[abdurrahman-12]

Hi everyone

I am deploying a TKC cluster using v1alpha2 API on vSphere 7U3.

The control plane and worker VMs are deploying, but the cluster pending on Creating phase.

When I describe the cluster using:

kubectl describe tkc tkgs-prod-cluster -n ns-prod

I am getting: 
......
Conditions:
Last Transition Time: 2023-02-16T05:54:40Z
Message: node pools [] are unknown, [] are failed, [worker-nodepool-a1] are updating
Reason: NodePoolsUpdating
Severity: Info
Status: False
Type: Ready
..............

I have created a cluster with 1 control plane, 3 worker nodes. but when I run command kubectl get nodes inside the cluster it lists as fllowing:

tkgs-prod-cluster-control-plane-9w8zz Ready control-plane,master 32m v1.23.8+vmware.3
tkgs-prod-cluster-worker-nodepool-a1-7bq9v-65b6485f56-9k72d Ready <none> 28m v1.23.8+vmware.3
tkgs-prod-cluster-worker-nodepool-a1-7bq9v-65b6485f56-bq89s Ready <none> 28m v1.23.8+vmware.3

there is only 2 worker nodes! but in vSphere inventory I see 3 worker VMs

when I run kubectl get po -A all the pods are running and fine.

 

Anyone knows what can be the issue?

[codedoings]

Late reply but you can get some more information regarding the provisioning of the virtual machines for the kubernetes cluster by doing a kubectl get machines -n <your vsphere namespace> while being on the supervisor context.

 

There is a pretty nice troubleshooting guide available here: https://core.vmware.com/blog/tanzu-kubernetes-grid-service-troubleshooting-deep-dive-part-3

[Tian Peng Yap]

 I have a similar problem. The control plane coming up, but the worker nodes are not.

if you do a "kubectl get machinedeployments"

NAME                             CLUSTER   REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE       AGE   VERSION
tkc2-worker-nodepool-a11-hhv8c   tkc2      1                  1         1             ScalingUp   12h   v1.27.10+vmware.1-fips.1
tkc3-worker-nodepool-a11-8tl97   tkc3      1                  1         1             ScalingUp   12h   v1.27.10+vmware.1-fips.1
tkc4-worker-nodepool-a11-8wxxj   tkc4      2                  1         2             ScalingUp   9h    v1.26.12+vmware.2-fips.1

you will see that the worker nodes have been scaling up for many hours. In vCenter you will see that the machine is powered off.

This is on vcf 4.5.2 with it async upgraded to 3p ..esxi and vcenter.

a describe will show up as "

 node pools [] are unknown, [] are failed, [worker-nodepool-a11] are updating"

[Tian Peng Yap]

It seems it might to be a convention issue. I created 3 workers and 3
control nodes. Still showed workers not ready but they are powered on this
time. I scaled down to 1 control and 1 worker.. then it shows ready. Maybe
because I also kept recreating the iSCSI backing to make sure its not
hanging. Using targetcli on Ubuntu 20.04 backed by vsan

On Mon, 7 Oct 2024, 11:06 Tian Peng Yap via Broadcom, <[e-mail]>
wrote:

> I have a similar problem. The control plane coming up, but the worker
> nodes are not. if you do a "kubectl get machinedeployments" NAME ...
> [image: Broadcom] <https: community.broadcom.com>
> VMware vSphere
> <https: community.broadcom.com vmware-cloud-foundation communities community-home digestviewer?communitykey=c24ac095-2065-4261-a4b5-6de9dade8734>
> Post New Message <[e-mail]>
> Re: vSphere with Tanzu TKC Creation Error
> <https: community.broadcom.com vmware-cloud-foundation discussion vsphere-with-tanzu-tkc-creation-error#bm692691a6-453a-496c-8ac2-019264e70459>
> Reply to Group
> <[e-mail]?subject=re:+vsphere+with+tanzu+tkc+creation+error> Reply
> to Sender
> <https: community.broadcom.com vmware-cloud-foundation communities all-discussions postreply?messagekey=692691a6-453a-496c-8ac2-019264e70459&ListKey=eabedc9c-d055-471e-9665-018ed42bdca6&SenderKey=8e2e60a3-26bc-4997-9ddc-018b2e90082d>
> [image: Tian Peng Yap]
> <https: community.broadcom.com network members profile?userkey=8e2e60a3-26bc-4997-9ddc-018b2e90082d>
> Oct 6, 2024 11:06 PM
> Tian Peng Yap
> <https: community.broadcom.com network members profile?userkey=8e2e60a3-26bc-4997-9ddc-018b2e90082d>
>
> I have a similar problem. The control plane coming up, but the worker
> nodes are not.
>
> if you do a "kubectl get machinedeployments"
>
>
> NAME CLUSTER REPLICAS READY UPDATED
> UNAVAILABLE PHASE AGE VERSION
> tkc2-worker-nodepool-a11-hhv8c tkc2 1 1 1
> ScalingUp 12h v1.27.10+vmware.1-fips.1
> tkc3-worker-nodepool-a11-8tl97 tkc3 1 1 1
> ScalingUp 12h v1.27.10+vmware.1-fips.1
> tkc4-worker-nodepool-a11-8wxxj tkc4 2 1 2
> ScalingUp 9h v1.26.12+vmware.2-fips.1
>
> you will see that the worker nodes have been scaling up for many hours. In
> vCenter you will see that the machine is powered off.
>
> This is on vcf 4.5.2 with it async upgraded to 3p ..esxi and vcenter.
>
> a describe will show up as "
>
> node pools [] are unknown, [] are failed, [worker-nodepool-a11] are
> updating"
> *Reply to Group Online
> <https: community.broadcom.com vmware-cloud-foundation communities all-discussions postreply?messagekey=692691a6-453a-496c-8ac2-019264e70459&ListKey=eabedc9c-d055-471e-9665-018ed42bdca6>*
> *Reply to Group via Email
> <[e-mail]?subject=re:+vsphere+with+tanzu+tkc+creation+error>*
> *View Thread
> <https: community.broadcom.com vmware-cloud-foundation discussion vsphere-with-tanzu-tkc-creation-error#bm692691a6-453a-496c-8ac2-019264e70459>*
> *Recommend
> <https: community.broadcom.com:443 vmware-cloud-foundation discussion vsphere-with-tanzu-tkc-creation-error?messagekey=692691a6-453a-496c-8ac2-019264e70459&cmd=rate&cmdarg=add#bm692691a6-453a-496c-8ac2-019264e70459>*
> *Forward
> <https: community.broadcom.com vmware-cloud-foundation communities all-discussions forwardmessages?messagekey=692691a6-453a-496c-8ac2-019264e70459&ListKey=eabedc9c-d055-471e-9665-018ed42bdca6>*
> *Flag as Inappropriate
> <https: community.broadcom.com vmware-cloud-foundation discussion vsphere-with-tanzu-tkc-creation-error?markappropriate=692691a6-453a-496c-8ac2-019264e70459#bm692691a6-453a-496c-8ac2-019264e70459>*
>
> -------------------------------------------
> Original Message:
> Sent: Aug 31, 2023 11:54 AM
> From: codedoings
> Subject: vSphere with Tanzu TKC Creation Error
>
> Late reply but you can get some more information regarding the
> provisioning of the virtual machines for the kubernetes cluster by doing a
> kubectl get machines -n <your vsphere namespace> while being on the
> supervisor context.
>
>
>
> There is a pretty nice troubleshooting guide available here:
> core.vmware.com/blog/...
> <https: core.vmware.com blog tanzu-kubernetes-grid-service-troubleshooting-deep-dive-part-3>
>
>
>
>
> You are subscribed to "VMware vSphere" as [e-mail]. To
> change your subscriptions, go to My Subscriptions
> <http: community.broadcom.com preferences?section=Subscriptions>. To
> unsubscribe from this community discussion, go to Unsubscribe
> <http: community.broadcom.com higherlogic egroups unsubscribe.aspx?userkey=8e2e60a3-26bc-4997-9ddc-018b2e90082d&sKey=KeyRemoved&GroupKey=eabedc9c-d055-471e-9665-018ed42bdca6>.
>
>
> Copyright © 2005-2023 Broadcom. All Rights Reserved. The term "Broadcom"
> refers to Broadcom Inc. and/or its subsidiaries.
>
> Hosted by Higher Logic, LLC on the behalf of Broadcom - Privacy Policy
> <https: www.broadcom.com company legal privacy-policy> | Cookie Policy
> <https: www.higherlogic.com legal privacy> | Supply Chain Transparency
> <https: www.broadcom.com company citizenship governance-and-ethics#supply>
> | Terms of Use <http: termsandconditions>
>

--
This electronic communication and the information and any files transmitted
with it, or attached to it, are confidential and are intended solely for
the use of the individual or entity to whom it is addressed and may contain
information that is confidential, legally privileged, protected by privacy
laws, or otherwise restricted from disclosure to anyone else. If you are
not the intended recipient or the person responsible for delivering the
e-mail to the intended recipient, you are hereby notified that any use,
copying, distributing, dissemination, forwarding, printing, or copying of
this e-mail is strictly prohibited. If you received this e-mail in error,
please return the e-mail to the sender, delete it from your computer, and
destroy any printed copy of it.