compiling fails at DS3DB0

[Riehmulus]

2024-07-24T09:56:59.935 [pool-3-thread-9] INFO  ST219 : Starting Provision DS3DB0 onto Host phx-e0-sddc-r2-s05.lab.aexp.com

 2024-07-24T10:01:01.077 [pool-3-thread-9] ERROR ST219 : DS3DB0, Unable to Create VM : Msg An error occurred during host configuration.

2024-07-24T10:01:01.077 [pool-3-thread-9] ERROR ST219 : Terminating Early Due to Provision Thread Errors (1)

2024-07-24T10:01:01.082 [pool-3-thread-9] ERROR MAIN : Exception Caught: Terminating Early Due to Provision ThreadErrorsErrorsFound: 1 : exitSetting true

Tried 10 times using different storage/timeouts/etc, always the same problem

[Fred Abounader]

For one it looks like you are trying to deploy VMmark3. Is that correct?  The reason I mention it is VMmark3 is at end-of-life and other than you using it for internal testing it is no longer excepted for results publications.  We have released VMmark4 and can be dowloaded from out website.

https://www.vmware.com/products/vmmark

If you do want to pursue with debugging this DS3DB issue on VMmark3 I would need you to upload the compressed /root/VMmark3/provision-output/ directory from the failed provision attempt.

Fred

[Riehmulus]

I'm testing an environment using esxi/vcsa v7 and vmmark requires v8 right?

[Fred Abounader]

You are correct that we do require 8.0.u2 or newer for VMmark4 but that is what we require for publications. VMmark4 should install on 7.X (it was initially tested on it) but you can't publish to VMmark Results but neither can you for VMmark3 (eol). 

I looked at your properties file and noticed that your datastores have a space in them (e.i. DS3DB:ROW4-5 VSAN,\).  It does look like it is provisioning some of the VMs but wondering if this could be an issue with the DS3DB0 provision.  Is it possible for you to try provisioning DS3DB0 on a datastore that doesn't have a space in it's name.  

Did it create an entire directory under /root/VMmark3/provisioning-output/* directory with more files?

[Riehmulus]

I added a dash to the storage to remove the xtra space.  Still stops at same place.

[Riehmulus]

disregard this comment

[Riehmulus]

What is the cli command in version4 to start provisioning?  no jar file like v3

[Fred Abounader]

Have you looked at the VMmark4 users guide on provisioning?

ovftool --diskMode=thin --noSSLVerify --acceptAllEulas --datastore=[datastore_name] /path/to/VMmark-4.*.*-nnn.ova vi://[e-mail]:[VC password]@[VC IP]/[Datacenter name]/host/[SUT cluster name]/

Fred

[Riehmulus]

what you posted above is for deploying the template, not for provisioning the vmmark4.properties file.  I setup vmmark4 the same way I did vmwark3 by putting the template in the client cluster and creating the primeclient.  The one difference is VMmark4 doesn't have this command anymore to run the test "java -jar tools/VMmark3Service.jar -c VMmark3.properties".  I did run VMmark4 from STAX via the console and also cli by using "VMmark4-STAX-console.py -c /root/VMmark4/VMmark4.properties -j VMmark4.properties".  Both ran through what looked like provisioning, but it didn't create any VM's.

[Fred Abounader]

Please upload a compressed copy of the /root/VMmark4/provision-output/ directory that was created from the failed provisioning attempt.

[Fred Abounader]

So did you follow page 48 of Users Guide and try Quick Start mode?  I see in the VMark4.properties file you sent the ProvisioningDatastores are all commented out.. 

#ProvisioningDatastores = \

#DS35DB:ROW4-5-VSAN,\

#DS35WebA:ROW4-5-VSAN,\

#DS35WebB:ROW4-5-VSAN,\

#DS35WebC:ROW4-5-VSAN,\

#AuctionWebA:ROW4-5-VSAN,\

#AuctionWebB:ROW4-5-VSAN,\

#AuctionWebC:ROW4-5-VSAN,\

#AuctionWebDNosql:ROW4-5-VSAN,\

#AuctionWebEDB:ROW4-5-VSAN,\

#AuctionWebF:ROW4-5-VSAN,\

#AuctionAppA:ROW4-5-VSAN,\

#AuctionAppB:ROW4-5-VSAN,\

#AuctionAppC:ROW4-5-VSAN,\

#AuctionKA:ROW4-5-VSAN,\

#AuctionKB:ROW4-5-VSAN,\

#AuctionKC:ROW4-5-VSAN,\

#AuctionKD:ROW4-5-VSAN,\

#NoSQLBenchA:ROW4-5-VSAN,\

#NoSQLBenchB:ROW4-5-VSAN,\

#NoSQLBenchC:ROW4-5-VSAN,\

#SocialNetwork:ROW4-5-VSAN,\

#Standby:ROW4-5-VSAN

[Riehmulus]

I've tried it both ways and result is the same.

[Riehmulus]

provision folders are empty but these are the results

[Fred Abounader]

I just got on a setup and did a provision. Try doing a provision without quickstart trying to do a run immediately afterwards. 

For my setup here is the exact command used:

/root/VMmark4/tools/vmmark4service --mode provision --vcenter_ip 10.210.19.124 \

--vcenter_password 'P0rm!' --datacenter 'vmmark' --cluster 'milsut' \

--client_cluster 'milclient' --tile_number 1 --provisioning_source 'VMmark-4.0.1-208' \

--datastore 'Unity-3' --client_datastore 'Unity-9' --infra_datastore 'Unity-3' \

--network_label 'vmmarke' >  1tile-provision.txt

Substitute your parameters and make sure it provisions by checking  /root/VMmark4/provision/Provision-202407XXXX  (XXXX date stamp info when executed).  

[Riehmulus]

i did try your way but couldnt get it to work but you got me on the right track:

/root/VMmark4/tools/vmmark4service -m provision -pci 10.61.26.98 -vc 10.61.24.132 -vcu [e-mail] -vcp '!Passw0rd' -dc 'SDDC' -cl Row4-5 -ccl Row2-3 -t 1 -ps VMmark-4.0.1-208 -ds ROW4-5-VSAN -cds ROW2-3-VSAN -ids ROW4-5-VSAN -si static -pig 10.61.26.97 -pim 255.255.255.224 -sis 10.61.26.99 -sie 10.61.26.126 -nl '04-10.61.26.96%2f27-LS' -nst dvswitch > test1.txt

[Riehmulus]

got it running but guide could definitely be clearer

[Riehmulus]

First provisioning missed 5 or so VM's so had to restart test until completely provisioned them all.  Test ran for 10 hours but errored right at the end of the log with weathervane?  Restarted test again and it started doing the review of weathervane again.  

[Rebecca Yo]

Hi, my name is Rebecca and I will be taking over support of this case from Fred. I also work on the VMmark 4 team.

Thank you for providing the vmmark4service-output.log.

From looking at the logs, I can see that you started provisioning several times with errors, probably due to the provisioning command needing a certain syntax, although I can't see those details from this log. This created some VMs but didn't get everything set up.

To fix this, please delete all VMmark VMs; from the Prime Client, run the command:

vmmark4service --mode delete_all_vmmark4 

You'll need to include parameters in the command to describe your environment which can be found in the vmmark4service help, like --vcenter_ip, --vcenter_username, --vcenter_password, --datacenter, --cluster, --client_cluster, --datastore, --csi_datastore and --tile_number.

Then rerun your correct provisioning command.

If you run into issues, please zip the most recent /root/VMmark4/provision-output/ directory and upload here. That will contain logs from the provisioning attempt. 

Thank you,

Rebecca

[Riehmulus]

Any chance you can just join a webex?

[Rebecca Yo]

Yes! Let's talk privately.

[Riehmulus]

attached

[Rebecca Yo]

Thank you for the files.

[Riehmulus]

got alot further today.  error at socialnetwork0

2024-08-08 11:16:43,842 [vmmark4service MainThread] [ERROR]: opID[3520.6] Connectivity Check Unable to SSH Connect to VM 'SocialNetwork0' at IP '198.18.4.21'
2024-08-08 11:16:43,843 [vmmark4service_SVC_p3520] [ERROR]: opID[3520.6]: ERROR : Provisioning Automation Cancelled due to lack of VM 'SocialNetwork0' connectivity

[Riehmulus]

provision folders are always empty?

[Rebecca Yo]

When you run vmmark4service -mode quick_start, logs will be saved at /root/VMmark4/provision. Also vmmark4service-output.log will be saved to the directory where the command was run. This contains the console output from Quick Start. 

[Riehmulus]

don't worry about the image.  got past that issue because of this line "Deploy/DeployVMinfo ="

seems i'm stuck on ssh issue with some hosts that originally were fine.  

[Rebecca Yo]

Thank you for the files, your VMmark 3 tile provisioning completed successfully:

/root/VMmark3/provisioning-output/test7/
Automation Ending:
Please add the contents of hosts-stub.txt to your PrimeClient's /etc/hosts file
STAX should be started before beginning new runs
DS3DB0 is still being created : Review VM for status::  :: Detail (if applicable): IP: 192.168.1.5
Creation Status: ssh DS3DB0 'tail -f /ds3/VMmark3-DS3installoutput.txt'
Error Output:: Found 0 Errors

Did you ensure DS3DB0 completed initialization before starting the run in STAX? It takes upwards of 12 hours to complete typically.

Did you add the contents of hosts-stub.txt to your Prime Client's /etc/hosts file?

You mentioned an SSH issue, did you configure passwordless SSH between your hosts and the Prime Client?

If so please reboot your Prime client and start a VMmark run via STAX. If the run has issues, please zip the most recent results folder and attach it to this thread. 

[Riehmulus]

here are the esx files for completed tile5