va:
$ kubectl get volumeattachments.storage.k8s.io/csi-518e826b201e83eda5acdfeba091c92e8a1532e1b434e602588091dce4a008c3 -o yaml
apiVersion: storage.k8s.io/v1
kind: VolumeAttachment
metadata:
annotations:
csi.alpha.kubernetes.io/node-id: tkg-dev-workers-cdxpm-7dc9746d87-z9mcg
creationTimestamp: "2023-11-30T11:24:11Z"
finalizers:
- external-attacher/csi-vsphere-vmware-com
name: csi-518e826b201e83eda5acdfeba091c92e8a1532e1b434e602588091dce4a008c3
resourceVersion: "101682996"
uid: 8b8b6000-d48d-4d2d-ab1e-3150d81bb1ef
spec:
attacher: csi.vsphere.vmware.com
nodeName: tkg-dev-workers-cdxpm-7dc9746d87-z9mcg
source:
persistentVolumeName: pvc-8576028f-96de-4a2a-8d8d-788025fcc53f
status:
attachError:
message: 'rpc error: code = Internal desc = Watch on virtualmachine "tkg-dev-workers-cdxpm-7dc9746d87-z9mcg"
timed out'
time: "2023-11-30T11:32:13Z"
attached: falsepod events:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 45m default-scheduler Successfully assigned dev/task-pv-pod-data to tkg-dev-workers-cdxpm-7dc9746d87-z9mcg
Normal SuccessfulAttachVolume 45m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-2adb07cb-f1d9-49a7-b53d-a826ea664eb7"
Normal SuccessfulAttachVolume 45m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-c2815e06-984b-4169-8374-d64e760a8667"
Warning FailedAttachVolume 41m (x9 over 43m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-8576028f-96de-4a2a-8d8d-788025fcc53f" : rpc error: code = Internal desc = observed Error: "failed to attach cns volume" is set on the volume "9fb8fabc-e1e8-4191-a06b-f71c3b7a9ba2-8576028f-96de-4a2a-8d8d-788025fcc53f" on virtualmachine "tkg-dev-workers-cdxpm-7dc9746d87-z9mcg"
Warning FailedMount 27m (x2 over 29m) kubelet Unable to attach or mount volumes: unmounted volumes=[opt-data], unattached volumes=[edge-home-dil kube-api-access-xbmhg opt-data edge-tmp]: timed out waiting for the condition
Warning FailedMount 11m (x2 over 18m) kubelet Unable to attach or mount volumes: unmounted volumes=[opt-data], unattached volumes=[kube-api-access-xbmhg opt-data edge-tmp edge-home-dil]: timed out waiting for the condition
Warning FailedMount 4m37s (x3 over 33m) kubelet Unable to attach or mount volumes: unmounted volumes=[opt-data], unattached volumes=[edge-tmp edge-home-dil kube-api-access-xbmhg opt-data]: timed out waiting for the condition
Warning FailedAttachVolume 28s (x20 over 39m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-8576028f-96de-4a2a-8d8d-788025fcc53f" : rpc error: code = Internal desc = Watch on virtualmachine "tkg-dev-workers-cdxpm-7dc9746d87-z9mcg" timed out
Warning FailedMount 5s (x13 over 43m) kubelet Unable to attach or mount volumes: unmounted volumes=[opt-data], unattached volumes=[opt-data edge-tmp edge-home-dil kube-api-access-xbmhg]: timed out waiting for the conditionvsphere-csi-controller log
$ k logs -n vmware-system-csi pod/vsphere-csi-controller-ffcbbf45b-59928 vsphere-csi-controller
...
{"level":"error","time":"2023-11-30T12:08:29.732105962Z","caller":"wcpguest/controller.go:647","msg":"Watch on virtualmachine \"tkg-dev-workers-cdxpm-7dc9746d87-z9mcg\" timed out","TraceId":"ddee367e-041a-4ba6-a1a3-d9808315ba5d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/wcpguest.controllerPublishForBlockVolume\n\t/build/mts/release/bora-22031571/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/wcpguest/controller.go:647\nsigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/wcpguest.(*controller).ControllerPublishVolume.func1\n\t/build/mts/release/bora-22031571/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/wcpguest/controller.go:539\nsigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/wcpguest.(*controller).ControllerPublishVolume\n\t/build/mts/release/bora-22031571/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/wcpguest/controller.go:542\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerPublishVolume_Handler\n\t/build/mts/release/bora-22031571/cayman_vsphere_csi_driver/vsphere_csi_driver/go/pkg/mod/github.com/container-storage-interface/spec@v1.7.0/lib/go/csi/csi.pb.go:5707\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/build/mts/release/bora-22031571/cayman_vsphere_csi_driver/vsphere_csi_driver/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:1283\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/build/mts/release/bora-22031571/cayman_vsphere_csi_driver/vsphere_csi_driver/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:1620\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/build/mts/release/bora-22031571/cayman_vsphere_csi_driver/vsphere_csi_driver/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:922"}
{"level":"error","time":"2023-11-30T12:08:29.732260375Z","caller":"wcpguest/controller.go:545","msg":"Operation failed, reporting failure status to Prometheus. Operation Type: \"attach-volume\", Volume Type: \"block\", Fault Type: \"csi.fault.Internal\"","TraceId":"ddee367e-041a-4ba6-a1a3-d9808315ba5d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/wcpguest.(*controller).ControllerPublishVolume\n\t/build/mts/release/bora-22031571/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/wcpguest/controller.go:545\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerPublishVolume_Handler\n\t/build/mts/release/bora-22031571/cayman_vsphere_csi_driver/vsphere_csi_driver/go/pkg/mod/github.com/container-storage-interface/spec@v1.7.0/lib/go/csi/csi.pb.go:5707\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/build/mts/release/bora-22031571/cayman_vsphere_csi_driver/vsphere_csi_driver/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:1283\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/build/mts/release/bora-22031571/cayman_vsphere_csi_driver/vsphere_csi_driver/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:1620\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/build/mts/release/bora-22031571/cayman_vsphere_csi_driver/vsphere_csi_driver/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:922"}
{"level":"info","time":"2023-11-30T12:09:29.723776959Z","caller":"wcpguest/controller.go:508","msg":"ControllerPublishVolume: called with args {VolumeId:9fb8fabc-e1e8-4191-a06b-f71c3b7a9ba2-8576028f-96de-4a2a-8d8d-788025fcc53f NodeId:tkg-dev-workers-cdxpm-7dc9746d87-z9mcg VolumeCapability:mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > Readonly:false Secrets:map[] VolumeContext:map[storage.kubernetes.io/csiProvisionerIdentity:1674815255199-8081-csi.vsphere.vmware.com type:vSphere CNS Block Volume] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"06248116-f687-492c-a055-368073897fa6"}
{"level":"info","time":"2023-11-30T12:09:29.723837904Z","caller":"wcpguest/controller_helper.go:384","msg":"attacherTimeout is set to 4 minutes","TraceId":"06248116-f687-492c-a055-368073897fa6"}
{"level":"info","time":"2023-11-30T12:09:29.730092803Z","caller":"wcpguest/controller.go:583","msg":"Volume \"9fb8fabc-e1e8-4191-a06b-f71c3b7a9ba2-8576028f-96de-4a2a-8d8d-788025fcc53f\" is already present in the virtualMachine.Spec.Volumes","TraceId":"06248116-f687-492c-a055-368073897fa6"}We have several PV's and all can be attached except this one.
Please advise!
thanks