Configure-vcd script failed to complete

[waseemarif]

Hi Guys,

 

I am trying to set up a new vCD 10.3.1 and I am able to access the management and I am sure that my NFS is correctly configured. But When it tries to finish it fails during clean up.

I have attached the screenshot and logs from root@vcd [ /opt/vmware/vcloud-director/logs ]# more cell-management-tool.log and I could see the below error:

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) 

Could you please look into this and help  I have attached both screenshot and logs

 

Regards,

Waseem

[jrehman]

Inspect your logs to determine why the vcd setup did not complete (opt/vmware/var/log/firstboot.log and /opt/vmware/var/log/vcd/setupvcd.log). If you see the following error in your setupvcd.log, this issue occurred because the password for the private key, keystore, and root user account was not all the same.

[waseemarif]

I see these logs in (opt/vmware/var/log/firstboot.log

2021-11-11 17:48:27 | Updating /etc/hosts file ...
2021-11-11 17:48:27 | System hostname is: vcd.infobloxdemo.com
2021-11-11 17:48:27 | Completed Appliance OS Configuration Phase
2021-11-11 17:48:28 | Writing UI configuration file to enable the primary setup wizard in the appliance management UI and setting the systemSetupComplete pro
perty to false
2021-11-11 17:48:28 | Setting content in deployment.node.type file ...
2021-11-11 17:48:28 | Checking for all required OVF parameters, for VCD Configuration Phase
Please provide NFS mount
Please provide Administrator password
Please provide DB password
Please provide Administrator's email
Please provide Administrator's full name
Please provide Administrator's installation ID
Please provide system name
Not all required ovf parameters were provided, aborting Cloud Director configuration phase
2021-11-11 17:48:30 | [WARNING] Not all required parameters for VCD configuration were provided.
2021-11-11 17:48:30 | For more details, see: /opt/vmware/var/log/vcd/configure-vcd.log
2021-11-11 17:48:30 | Please complete the appliance configuration using the appliance management UI.

And if i do more on /opt/vmware/var/log/vcd/configure-vcd.log as I don't see setupvcd.log in there

Invoking firstrunwizard.sh now.
2021-11-17 17:35:00 | setupvcd.sh | Invoking firstrunwizard.sh now.
2021-11-17 17:35:01 | firstrunwizard.sh | Running the system-setup CMT command .
..
Creating admin user.
Setting system details.
Completing system setup.
System setup is complete.
2021-11-17 17:37:07 | firstrunwizard.sh | Clearing sensitive OVF properties...

2021-11-17 17:37:07 | setupvcd.sh | Copying local responses.properties to NFS share.
2021-11-17 17:37:08 | setupvcd.sh | Done setting up vCD.
Reconfiguring vCD to use a secure database connection.
2021-11-17 17:37:08 | setupvcd.sh | Reconfiguring vCD to use a secure database connection.
Invalid database configuration provided. Aborting configuration of local cell.
2021-11-17 18:07:42 | setupvcd.sh | [ERROR] Failed to run reconfigure-database. See /opt/vmware/vcloud-director/logs/cell-management-tool.log for details. Ab
orting.
2021-11-17 18:07:42 | configure-vcd.sh | [ERROR] setupvcd script failed to execute.
2021-11-17 18:07:42 | cleanup-vcd.sh | Starting cleanup script.
2021-11-17 18:07:42 | cleanup-vcd.sh | The following error code was provided as argument to script: 23
2021-11-17 18:07:42 | cleanup-vcd.sh | Attempting cleanup on primary appliance.
2021-11-17 18:07:42 | cleanup-vcd.sh | Executing configure_vcd_primary_cleanup.
2021-11-17 18:07:42 | cleanup-vcd.sh | Executing configure_vcd_base_cleanup.
2021-11-17 18:07:43 | cleanup-vcd.sh | Executing vcd_db_erase.
2021-11-17 18:07:43 | cleanup-vcd.sh | Deleting PostgreSQL database (vcloud) and user (vcloud).
2021-11-17 18:07:43 | cleanup-vcd.sh | Removing files from transfer share placed there by systemSetup, etc.
2021-11-17 18:07:43 | cleanup-vcd.sh | Unmounting NFS share and cleaning up /etc/fstab.
2021-11-17 18:07:43 | cleanup-vcd.sh | Executing clear_ovfenv_params.

 

Any Idea what am I missing

[naser_mx]

it seems i have same issue.
after i removed my first cell , i tried to re install it again , but :
1) setup stuck on "Invoking setupvcd script." ( attached photo )

2) "setupvcd.log" not exist .

3) "configure-vcd.log" show :
2022-01-14 15:02:24 | check-vcd-params.sh | Checking ovf parameters needed for proper Cloud Director configuration ...
2022-01-14 15:02:25 | check-vcd-params.sh | nfs_mount ovf parameter not specified
2022-01-14 15:02:25 | check-vcd-params.sh | Number of missing parameters: 1
2022-01-14 15:02:25 | check-vcd-params.sh | [ERROR] Not all required ovf parameters were provided, aborting Cloud Director configuration phase
2022-01-14 15:39:36 | configure-vcd.sh | Appliance OS Phase has been Completed.
2022-01-14 15:39:36 | configure-vcd.sh | Starting Appliance Cloud Director Configuration Phase.
2022-01-14 15:39:36 | configure-vcd.sh | Invoking check-vcd-params script again.
2022-01-14 15:39:36 | check-vcd-params.sh | Checking ovf parameters needed for proper Cloud Director configuration ...
2022-01-14 15:39:36 | check-vcd-params.sh | All the required ovf parameters provided for the Cloud Director configuration phase
2022-01-14 15:39:36 | configure-vcd.sh | Invoking nfs-setup script.
2022-01-14 15:39:36 | nfs-setup.sh | Mounting NFS file share ...
2022-01-14 15:39:37 | nfs-setup.sh | Listing contents of transfer share directory
total 8
drwxr-x--- 5 vcloud vcloud 0 Jan 14 15:39 .
drwxrwx--- 4 vcloud vcloud 4096 Jan 14 15:39 ..
drwxr-x--- 3 vcloud vcloud 0 Jan 13 12:37 appliance-nodes
drwxr-x--- 4 vcloud vcloud 0 Jan 13 10:22 cells
-rwxr-x--- 1 vcloud vcloud 663 Jan 13 10:20 responses.properties
2022-01-14 15:39:37 | configure-vcd.sh | Invoking setupvcd script.
2022-01-14 15:39:37 | setupvcd.sh | System ip0 is: 172.x.x.11

After several hours, nothing happened. i use vCD 10.3.1
(I had no problem deploying cells on the first day(clean install). But after a while, I will not be able to deploy any new cell.)

any idea ?

[SeanArmstrong]

Did you ever get to the bottom of this error? as I have the exact same scenario

Thanks

Sean

[AbbedSedkaoui]

Hi all,

we all have in the /opt/vmware/vcloud-director/logs/cell-management-tool.log the following error:

ERROR | main | ReconfigureDatabaseCommand | Invalid database configuration: SSL error: PKIX path buildi
ng failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target |
org.postgresql.util.PSQLException: SSL error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid cer
tification path to requested target

 

Please confirm that you have "Timed out" in your appliance-sync.log when adding the certificate node

cat /opt/vmware/var/log/vcd/appliance-sync.log | grep Timed

Timed out adding certificate for node node-35184bba-f181-469d-be92-ebb45ff6727d, continuing to other nodes.

 

i found a way to complete the setup successfully with cluster HEALTHY and services running.

the trick was the import from my NFS share (windows) into vcd appliance may take longer than 10seconds, i gave it 60.

sed -i s/10s/60s/ /opt/vmware/appliance/bin/appliance-sync.sh

resulted line should be:
cat /opt/vmware/appliance/bin/appliance-sync.sh | grep force
/usr/bin/timeout 60s $VCLOUD_HOME/bin/cell-management-tool import-trusted-certificates --source $APPLIANCE_SHARE/$i/vcd_ova.crt --force

then try again completing the setup, while watching the log i confirmed that the operation took more than 10s , here nearly 30s in my environment.

tail -f /opt/vmware/var/log/vcd/configure-vcd.log

"
Reconfiguring vCD to use a secure database connection.
2022-09-25 18:36:29 | setupvcd.sh | Reconfiguring vCD to use a secure database connection.
Database configuration in properties files complete for local cell.
Database configuration was successfully refreshed in running Cell.
2022-09-25 18:36:57 | configure-vcd.sh | Invoking cellstartup script.
"

the setup complete successfully,

then revert back the original setting

sed -i s/60s/10s/ /opt/vmware/appliance/bin/appliance-sync.sh

systemctl restart appliance-sync.timer

Shutdown/Power on the VM, do not use Restart

Increase vCPUs from 2 to 4. (Updated)

wait few minutes for the vcd appliance to fully load before opening http://{VCD_IP}/provider

Let me know if this hasn't worked for you.

cheers

Abidi

[Vijaya Bhaskar Mekala]

Thanks Abidi, this helped to fix my issue.

[Martin Benitez]

Hi , I have problems to start the cell. any idea? the cell was working find and them I Couldn't access any more.

2025-04-12 16:06:54,795 | DEBUG    | HeartBeat-1               | CellManagerImpl                | No change in the active cell list wast detected: [b6c3d684-af69-4225-b43d-816786c94876] |
2025-04-12 16:07:09,811 | DEBUG    | HeartBeat-1               | CellManagerImpl                | No change in the active cell list wast detected: [b6c3d684-af69-4225-b43d-816786c94876] |
2025-04-12 16:07:25,229 | DEBUG    | HeartBeat-1               | CellManagerImpl                | No change in the active cell list wast detected: [b6c3d684-af69-4225-b43d-816786c94876] |
2025-04-12 16:07:40,242 | DEBUG    | HeartBeat-1               | CellManagerImpl                | No change in the active cell list wast detected: [b6c3d684-af69-4225-b43d-816786c94876] |
2025-04-12 16:07:43,833 | DEBUG    | Message Broker Topology Snapshotter | BrokerContext                  | Updated the last effective date of the latest topology state from 2022-12-15T17:14:53.467043Z |
2025-04-12 16:07:47,283 | DEBUG    | Message Broker Topology Logger | BrokerContext                  |
defaultTopology=topology on Topology@6db41a7f[owner=ServerLocatorImpl [initialConnectors=[], discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='vcd-discovery-group', refreshTimeout=10000, discoveryInitialWaitTimeout=1000}]]:
        b6c3d684-af69-4225-b43d-816786c94876 => TopologyMember[id=b6c3d684-af69-4225-b43d-816786c94876, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=172-20-10-100, b=null], backupGroupName=null, scaleDownGroupName=null]
        nodes=1 members=1,
-----------------------------------
vcd-cluster=topology on Topology@515fa565[owner=ClusterConnectionImpl@881116585[nodeUUID=b6c3d684-af69-4225-b43d-816786c94876, connector=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=172-20-10-100, address=, server=ActiveMQServerImpl::name=b6c3d684-af69-4225-b43d-816786c94876]]:
        b6c3d684-af69-4225-b43d-816786c94876 => TopologyMember[id=b6c3d684-af69-4225-b43d-816786c94876, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=172-20-10-100, b=null], backupGroupName=null, scaleDownGroupName=null]
        nodes=1 members=1 |
2025-04-12 16:07:55,278 | DEBUG    | HeartBeat-1               | CellManagerImpl                | No change in the active cell list wast detected: [b6c3d684-af69-4225-b43d-816786c94876] |
2025-04-12 16:08:06,855 | WARN     | Module: HTML5 Cell Application | DirectlyConfigurableXmlApplicationContext | 300,007 milliseconds creating bean: filterUtils; SLOW BEAN |
2025-04-12 16:08:06,856 | WARN     | Module: HTML5 Cell Application | DirectlyConfigurableXmlApplicationContext | 300,012 milliseconds creating bean: sessionFilter; SLOW BEAN |
2025-04-12 16:08:06,856 | WARN     | Module: HTML5 Cell Application | DirectlyConfigurableXmlApplicationContext | 326,437 milliseconds creating bean: vcdSecuredWebappConfiguration; SLOW BEAN |
2025-04-12 16:08:06,857 | WARN     | Module: HTML5 Cell Application | DirectlyConfigurableXmlApplicationContext | Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'vcdSecuredWebappConfiguration': Unsatisfied dependency expressed through field 'sessionFilter'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sessionFilter': Unsatisfied dependency expressed through field 'filterUtils'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'filterUtils': Unsatisfied dependency expressed through field 'securityService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityService': FactoryBean threw exception on object creation; nested exception is java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)' |
2025-04-12 16:08:06,862 | FATAL    | Module: HTML5 Cell Application | ModuleRegistry                 | Error in module: "HTML5 Cell Application"; Bean: vcdSecuredWebappConfiguration, Message: Error creating bean with name 'vcdSecuredWebappConfiguration': Unsatisfied dependency expressed through field 'sessionFilter'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sessionFilter': Unsatisfied dependency expressed through field 'filterUtils'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'filterUtils': Unsatisfied dependency expressed through field 'securityService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityService': FactoryBean threw exception on object creation; nested exception is java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)' |
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'vcdSecuredWebappConfiguration': Unsatisfied dependency expressed through field 'sessionFilter'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sessionFilter': Unsatisfied dependency expressed through field 'filterUtils'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'filterUtils': Unsatisfied dependency expressed through field 'securityService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityService': FactoryBean threw exception on object creation; nested exception is java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)'
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:660)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
        at com.vmware.vcloud.common.main.bootstrap.DirectlyConfigurableXmlApplicationContext$LoggingBeanFactory.doCreateBean(DirectlyConfigurableXmlApplicationContext.java:54)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:920)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
        at com.vmware.vcloud.common.main.bootstrap.DirectlyConfigurableXmlApplicationContext.refresh(DirectlyConfigurableXmlApplicationContext.java:151)
        at com.vmware.vcloud.common.main.ModuleRegistry$3.call(ModuleRegistry.java:90)
        at com.vmware.vcloud.common.main.ModuleRegistry$3.call(ModuleRegistry.java:70)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sessionFilter': Unsatisfied dependency expressed through field 'filterUtils'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'filterUtils': Unsatisfied dependency expressed through field 'securityService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityService': FactoryBean threw exception on object creation; nested exception is java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)'
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:660)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
        at com.vmware.vcloud.common.main.bootstrap.DirectlyConfigurableXmlApplicationContext$LoggingBeanFactory.doCreateBean(DirectlyConfigurableXmlApplicationContext.java:54)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:657)
        ... 21 more
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'filterUtils': Unsatisfied dependency expressed through field 'securityService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityService': FactoryBean threw exception on object creation; nested exception is java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)'
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:660)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
        at com.vmware.vcloud.common.main.bootstrap.DirectlyConfigurableXmlApplicationContext$LoggingBeanFactory.doCreateBean(DirectlyConfigurableXmlApplicationContext.java:54)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:657)
        ... 36 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityService': FactoryBean threw exception on object creation; nested exception is java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)'
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:176)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:135)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1898)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getObjectForBeanInstance(AbstractAutowireCapableBeanFactory.java:1284)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:267)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1616)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1573)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1354)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:657)
        ... 51 more
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)'
        at com.vmware.vcloud.common.service.OsgiServiceReferenceFactoryBean.getObject(OsgiServiceReferenceFactoryBean.java:238)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:169)
        ... 62 more

[AbbedSedkaoui]

Hi, if VCD is deployed as small form factor, increasing the vCpus from 2 to 4 help the cell boot,

note the medium use 8 vCpus and that is the minimum for Production (i've seen it the 8 cores hardcoded on a file, but i don't remember where).

How to find out after setting 4 cores, SSH to VCD use "top" then "1" to show all cores
```
top -o +%CPU
```
the goal is validate 400% can be reached for the cell to boot

I guess the root cause maybe that message queueing is getting stuck at one point and then need extra CPU power cycles to empty the queues, this can take time..., certainly Engineering should know better the app, that just my guess.

[Martin Benitez]

Hello Abbed Sedkaoui,

Thank you very much for your response. Unfortunately, it is still not working. The issue is that the lab environment was functioning correctly until it suddenly stopped working. I have backups stored on the NFS, and I managed to recover them by deploying a new cell. This new cell is deployed as medium, not small with 8vcpu and 16Gb ram

However, I applied the following command:
sed -i s/10s/60s/ /opt/vmware/appliance/bin/appliance-sync.sh
since the storage is a V5000 and not very fast.

The data seems to restore correctly, but the cell never fully starts. It fails when it reaches the module "com.vmware.vcloud.backend-core", where it throws an exception and then restarts the process.

The first error in the log is:
Error in module: "HTML5 Cell Application"; Bean: vcdSecuredWebappConfiguration

I would really appreciate any guidance you could provide.

Thank you in advance.

Best regards,

Martín

efaultTopology=topology on Topology@2a5d498c[owner=ServerLocatorImpl [initialConnectors=[], discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='vcd-discovery-group', refreshTimeout=10000, discoveryInitialWaitTimeout=1000}]]:
        29076e42-85d4-416b-a4e4-d35af65e0800 => TopologyMember[id=29076e42-85d4-416b-a4e4-d35af65e0800, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=172-20-10-100, b=null], backupGroupName=null, scaleDownGroupName=null]
        nodes=1 members=1,
-----------------------------------
vcd-cluster=topology on Topology@10e19861[owner=ClusterConnectionImpl@1793309107[nodeUUID=29076e42-85d4-416b-a4e4-d35af65e0800, connector=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=172-20-10-100, address=, server=ActiveMQServerImpl::name=29076e42-85d4-416b-a4e4-d35af65e0800]]:
        29076e42-85d4-416b-a4e4-d35af65e0800 => TopologyMember[id=29076e42-85d4-416b-a4e4-d35af65e0800, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=172-20-10-100, b=null], backupGroupName=null, scaleDownGroupName=null]
        nodes=1 members=1 |
2025-04-15 23:24:12,919 | DEBUG    | HeartBeat-1               | CellManagerImpl                | No change in the active cell list wast detected: [b6c3d684-af69-4225-b43d-816786c94876, 29076e42-85d4-416b-a4e4-d35af65e0800] |
2025-04-15 23:24:25,158 | WARN     | Module: HTML5 Cell Application | DirectlyConfigurableXmlApplicationContext | 300,084 milliseconds creating bean: filterUtils; SLOW BEAN |
2025-04-15 23:24:25,159 | WARN     | Module: HTML5 Cell Application | DirectlyConfigurableXmlApplicationContext | 300,089 milliseconds creating bean: sessionFilter; SLOW BEAN |
2025-04-15 23:24:25,163 | WARN     | Module: HTML5 Cell Application | DirectlyConfigurableXmlApplicationContext | 324,504 milliseconds creating bean: vcdSecuredWebappConfiguration; SLOW BEAN |
2025-04-15 23:24:25,164 | WARN     | Module: HTML5 Cell Application | DirectlyConfigurableXmlApplicationContext | Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'vcdSecuredWebappConfiguration': Unsatisfied dependency expressed through field 'sessionFilter'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sessionFilter': Unsatisfied dependency expressed through field 'filterUtils'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'filterUtils': Unsatisfied dependency expressed through field 'securityService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityService': FactoryBean threw exception on object creation; nested exception is java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)' |
2025-04-15 23:24:25,171 | FATAL    | Module: HTML5 Cell Application | ModuleRegistry                 | Error in module: "HTML5 Cell Application"; Bean: vcdSecuredWebappConfiguration, Message: Error creating bean with name 'vcdSecuredWebappConfiguration': Unsatisfied dependency expressed through field 'sessionFilter'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sessionFilter': Unsatisfied dependency expressed through field 'filterUtils'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'filterUtils': Unsatisfied dependency expressed through field 'securityService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityService': FactoryBean threw exception on object creation; nested exception is java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)' |
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'vcdSecuredWebappConfiguration': Unsatisfied dependency expressed through field 'sessionFilter'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sessionFilter': Unsatisfied dependency expressed through field 'filterUtils'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'filterUtils': Unsatisfied dependency expressed through field 'securityService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityService': FactoryBean threw exception on object creation; nested exception is java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)'
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:660)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
        at com.vmware.vcloud.common.main.bootstrap.DirectlyConfigurableXmlApplicationContext$LoggingBeanFactory.doCreateBean(DirectlyConfigurableXmlApplicationContext.java:54)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:920)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
        at com.vmware.vcloud.common.main.bootstrap.DirectlyConfigurableXmlApplicationContext.refresh(DirectlyConfigurableXmlApplicationContext.java:151)
        at com.vmware.vcloud.common.main.ModuleRegistry$3.call(ModuleRegistry.java:90)
        at com.vmware.vcloud.common.main.ModuleRegistry$3.call(ModuleRegistry.java:70)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sessionFilter': Unsatisfied dependency expressed through field 'filterUtils'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'filterUtils': Unsatisfied dependency expressed through field 'securityService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityService': FactoryBean threw exception on object creation; nested exception is java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)'
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:660)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
        at com.vmware.vcloud.common.main.bootstrap.DirectlyConfigurableXmlApplicationContext$LoggingBeanFactory.doCreateBean(DirectlyConfigurableXmlApplicationContext.java:54)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:657)
        ... 21 more
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'filterUtils': Unsatisfied dependency expressed through field 'securityService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityService': FactoryBean threw exception on object creation; nested exception is java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)'
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:660)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
        at com.vmware.vcloud.common.main.bootstrap.DirectlyConfigurableXmlApplicationContext$LoggingBeanFactory.doCreateBean(DirectlyConfigurableXmlApplicationContext.java:54)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:657)
        ... 36 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityService': FactoryBean threw exception on object creation; nested exception is java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)'
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:176)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:135)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1898)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getObjectForBeanInstance(AbstractAutowireCapableBeanFactory.java:1284)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:267)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1616)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1573)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1354)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:657)
        ... 51 more
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)'
        at com.vmware.vcloud.common.service.OsgiServiceReferenceFactoryBean.getObject(OsgiServiceReferenceFactoryBean.java:238)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:169)
        ... 62 more