Management domain deployment keeps failing

[Giridhar Sakineni]

During management domain deployment , I am repeatedly running into this error message:

Can you help me how to troubleshoot this error message

[Giridhar Sakineni]

I am able to get the exact error message 

[Dhruv Tyagi]

I'd suggest using Holodeck 9.1: https://vmware.github.io/Holodeck/release_notes/#holodeck-91-major-release

This is fixed in the latest release.

[Giridhar Sakineni]

@Dhruv Tyagi Thanks for the response. I am now using the new holodeck 9.1 and after some hicccups could proceed to online depot part of the install. I see now I am getting the following error. Is there a new way to get the token for VCF 9.1.0 as my manager said we would be using tokens from support.broadcom.com and generating a token from there, however that token is not working anymore. It seems like we need to login to vcf.broadcom.com to get the token?

[Dhruv Tyagi]

https://knowledge.broadcom.com/external/article/443647/download-token-has-been-replaced-by-acti.html

[Giridhar Sakineni]

Can I still try to install VCF 9.0 while using holorouter 9.1?

[Dhruv Tyagi]

yes

[Giridhar Sakineni]

Where should I check the logs of why it keeps failing

In VCF Installer VM I dont see any sddc-manager log specifically

[Giridhar Sakineni]

I see this error at holodeck run time error logs. and I am not sure how to resolve this 

[Giridhar Sakineni]

During VCF 9.0.2.0 Management Domain deployment on Holodeck, the deployment consistently fails at the "Create vSAN Disk Groups" step with a JWT token expiry error: "JWT expired, difference of ~4000 milliseconds, allowed clock skew: 0 milliseconds" - caused by clock drift between the VCF Installer appliance and SDDC Manager accumulating over the ~45 minute deployment window before this step.

The workaround of manually running ntpdate -u 10.1.1.1 on the VCF Installer and SDDC Manager is blocked because SDDC Manager holds a system lock during the bring-up workflow, making the NTP reconfiguration API (PUT /v1/system/ntp-configuration) unavailable - is there a recommended way to either increase the allowed JWT clock skew tolerance or ensure NTP stays in sync during the bring-up process?

[Dhruv Tyagi]

Do you provide the same NTP source to VCF Installer and management domain? You need to fix the NTP to ensure times are synced between various appliances using your NTP server

[Giridhar Sakineni]

 tried to wipe out and started a fresh install but still I get this error consistently.

I am not sure how we control this NTP time sync. We tried to restart NTP service in vcenter and it sycned all hosts successfully but this error continues. And up checking Holorouter was not configured with an external NTP upstream and it is using local as its clokc source.

• Is there a supported way to configure NTP upstream on Holorouter before starting the Holodeck deployment, so nested VMs get an accurate time source from the start?
• Is the zero-millisecond JWT clock skew tolerance by design in VCF 9.x, or is there a configuration to relax it for lab/nested environments?
• IS there a way to sync SDDC Manager's clock mid-deployment ?

[Nikhil M Kulkarni]

Did you provide a NTP server when deploying Holorouter? 

You can update the upstream NTP on Holorouter by editing the /etc/ntp.conf file and adding a line - server <upstream_ntp_server>. Then run - 
- systemctl stop ntpd
- ntpq -gq
- systemctl start ntpd

After 3-4 mins, it should start syncing with the upstream NTP server. And then you can start the holodeck deployment.

[Giridhar Sakineni]

Yes I did sync from holo-router VM and then restarted the deployment but the same error keeps happening after every 45 mins.

I think SDDC manager is controlling the clock and there is no way to SSH into it and only way to control it is thru API call.