vra 7.6 deployments stuck at provisioning

[zatco20]

Hello,

Recently ran into an error in which our vRA 7.6 environment would not deploy VMs. 

These are only stuck at provisioning stage.  Some of them eventually will deploy after several hours, but not many.  

I went into Managed Machine and cleared out any set to SubmittingRequest status, which there was over 100.  After clearing them, deployment delays are still occurring.

One VM that deployed I am able to use the controls (power on, reboot, create snapshot etc) and are normal response as previous.

Any help to point out where the issue is would be appreciated or any ideas where to look for the issue.

One thought I have is the DB could be full of pending jobs delaying jobs.

[bdamian]

It seems that you have problems with the DEM Worker and/or Agent services.

• Is a distributed installation of vRA7.6?
• Are DEM Orchestrator, DEM Worker and Agent on the same server?

Maybe it is time to restart the hole plataform. And if you do, please check if the time between the appliances and the windows (all of them) are in sync.

 

[zatco20]

Yes, setup

1 Server with SQL
1 with Orchestrator, Demworker and Agent
2 x Appliance VMs

 

Everything was working fine up until a week ago.   System was restarted last Friday and that is when the issue appears to have occurred. 

VMs are currently set to sync time on startup and I just re-stared the whole cluster. 

Once restarted, I was able to deploy 2 VM's without issue and they updated with completed status.   I tried a 3rd and 4th and it attempts to deploy, but status is stuck at provisioning.  Allocate is successful, but Provision is stuck in progress.

Other behaviour is deploys the VM and powers it on, but stuck at in progression for provision.  

Once thing that was done a couple weeks ago was the Vcenter STS cert.  

Anything else you can suggest would be great.

 

[bdamian]

Go to the VAMI console (Appliance #1 on port 5480) and tell me if the "Cluster" Tab shows everything normal.

[zatco20]

Everything is showing as healthy from I can tell.

 

 

[zatco20]

Also to update, all my VMs did eventually deploy but took close to 45 min to do so unlike the first 2 I did.  

[bdamian]

Can you start a Data Collection? Takes too long?

[zatco20]

I have run several data collections.   Does not take long at all.

[zatco20]

Hi,

Looked at logs it seems deployments are timing out waiting for the Event Broker.   I read on another post that disabling the extensibility service is a workaround for this.

 

 

[Logan Skrzypczak]

This is an unprovision machine event that could be timing out due to another health issue in the environment.

This information is useful, but can we check the Infrastructure > Audit Log for behavior in the lifecycle states? This can be useful to determine which component is failing based upon which component is responsible for which workitem during said lifecycle.

We are wanting to filter on the machine from Requested onward.

V/R,

Logan