VMmark

View Only

vmmark4 run a quick start test failed

jiang jianfeng posted Feb 11, 2025 06:53 AM

run commad like this : vmmark4service --mode quick_start --vcenter_ip 198.168.55.79 --vcenter_password Superuser9! --datacenter vmmark --cluster SUTCluster --client_cluster ClientCluster --tile_number 1 --provisioning_source VMmark-4.0.2-209 --datastore sutlun --client_datastore clientlun --network_label vmmark-workload --time_zone UTC --infra_datastore vmotion --primeclient_ip 177.177.0.1 --deploy_vminfo DeployVM0:177.177.0.2 --static_ip_startaddress 177.177.0.3 --provisioning_ip_gateway 177.177.0.1

but failed,it looks like some issues with k8s vms

Tue Feb 11 07:07:33 2025: Configuring and starting data services for appInstance 1 of workload 2.

Tue Feb 11 07:07:33 2025: Configuring and starting data services for appInstance 1 of workload 1.

Tue Feb 11 07:18:34 2025: Couldn't bring to running all data services for appInstance 1 of workload 1.

Tue Feb 11 07:18:35 2025: Couldn't start data services for appInstance 1 of workload 1.

See the Troubleshooting section of the User's Guide for assistance.

If this problem recurs, you can enable auto-remediation by setting "reloadOnFailure": true, in your configuration file.

Tue Feb 11 07:19:47 2025: Could not properly start the data services for run 0-0. Exiting.

Attachment View in library

log.tar 2.35 MB

Broadcom Employee Benjamin Hoflich posted Feb 11, 2025 12:06 PM

A file to help debug this issue is postprovision-auction<ip>.out.
If this file is not in the Provision-<id> directory, then it might be in the directory on the primeclient where the vmmark4service command was invoked from.

Some possible issues might be network connectivity between the Client0 VM and the vCenter, or conflicts with the CNS-DATASTORE role.
Is this out file available to debug with?

There also might be a file named like weathervane-provision-fail-*.zip in ~/VMmark4/tmp/ on the primeclient that has useful information for debugging any kubernetes cluster issue.

Broadcom Employee Benjamin Hoflich posted Feb 12, 2025 09:48 AM

removing duplicate post

jiang jianfeng posted Feb 12, 2025 08:01 PM

the network connectivity between the Client0 VM and the vCenter is good,i can ping or SSH vCenter from Client0

jiang jianfeng posted Feb 12, 2025 08:19 PM

Thank you for replay, I use default ip address run the test again, and also the same fail reason.

it seems like some error happens during creating pvc.

{"level":"error","time":"2025-02-12T09:54:42.08582399Z","caller":"vanilla/controller.go:1853","msg":"Operation failed, reporting failure status to Prometheus. Operation Type: \"create-volume\", Volume Type: \"block\", Fault Type: \"csi.fault.Internal\"","TraceId":"a153f3d9-e426-468c-bb0a-6b2b11c5322d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:1853\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.7.0/lib/go/csi/csi.pb.go:5671\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1301\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1642\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:938"}
{"level":"error","time":"2025-02-12T09:55:14.521921466Z","caller":"vsphere/datacenter.go:214","msg":"failed to get all the datastores in the Datacenter Datacenter:datacenter-3 @ /vmmark with error: datastore '*' not found","TraceId":"aed703cb-f28f-4f1c-9daa-c78797613f84","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.(*Datacenter).GetAllDatastores\n\t/build/pkg/common/cns-lib/vsphere/datacenter.go:214\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/common.GenerateDatastoreMapForBlockVolumes\n\t/build/pkg/csi/service/common/authmanager.go:232\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/common.(*AuthManager).refreshDatastoreMapForBlockVolumes\n\t/build/pkg/csi/service/common/authmanager.go:158\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/common.ComputeDatastoreMapForBlockVolumes\n\t/build/pkg/csi/service/common/authmanager.go:199"}
{"level":"error","time":"2025-02-12T09:55:14.522023927Z","caller":"common/authmanager.go:235","msg":"failed to get dsURLTodsInfoMap for vCenter \"198.168.55.79\". err: datastore '*' not found","TraceId":"aed703cb-f28f-4f1c-9daa-c78797613f84","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/common.GenerateDatastoreMapForBlockVolumes\n\t/build/pkg/csi/service/common/authmanager.go:235\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/common.(*AuthManager).refreshDatastoreMapForBlockVolumes\n\t/build/pkg/csi/service/common/authmanager.go:158\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/common.ComputeDatastoreMapForBlockVolumes\n\t/build/pkg/csi/service/common/authmanager.go:199"}

Attachments View in library

Provision-2025021205.zip 176 KB

weathervane-provision-fail-tile0-20250212-0155.zip 189 KB

postprovision-auction-198.18.4.22.out.zip 5 KB

Broadcom Employee Benjamin Hoflich posted Feb 13, 2025 09:26 AM

I see this
govc: role "CNS-DATASTORE" not found
in postprovision-auction-198.18.4.22.out, which results from a CNS role conflict.

Following the steps in the "CNS Naming in vCenter 8.0 Update 3" text in the Troubleshooting section of the VMmark Users Guide will likely resolve this issue.
(after deleting the tiles, delete the "CNS-Datastore" or "CNS Datastore" role from vCenter prior to the first tile provisioning).

jiang jianfeng posted Feb 14, 2025 12:52 AM

it works! thank you very much