vCenter

 View Only
  • 1.  Failed upgrade to vCenter 8u2

    Posted Sep 26, 2023 10:59 AM

    Running on VMUG licensing, so no support entitlement here, but I figured it was worth asking if anyone else has hit on this.

    The update process (have tried both the online update as well as the full patch ISO) seems to run through pretty smoothly, and gets to the "converting data" stage, before bombing out with

    "Exception occured in postInstallHook for sps:Patch"

    Checking PatchRunner.log I see the below, which seems to be the breaking point

    "2023-09-25T11:38:55.465Z [main] INFO opId= com.vmware.sps.upgrade.b2b.patch.action.RemoveUnusedSpbmKvProviders - Deleting spbmVpUrlToIdMapProvider provider
    2023-09-25T11:38:55.465Z [main] INFO opId= com.vmware.vim.storage.common.kv.KvManager - Deleting the provider 'spbmVpUrlToIdMapProvider'.
    2023-09-25T11:38:55.473Z [main] DEBUG opId= com.vmware.vim.storage.common.kv.KvManager - Deleted the provider 'spbmVpUrlToIdMapProvider'.
    2023-09-25T11:38:55.474Z [main] INFO opId= com.vmware.sps.upgrade.b2b.patch.action.RemoveUnusedSpbmKvProviders - Deleting spbmRollupComplianceResultProvider provider
    2023-09-25T11:38:55.474Z [main] INFO opId= com.vmware.vim.storage.common.kv.KvManager - Deleting the provider 'spbmRollupComplianceResultProvider'.
    2023-09-25T11:38:55.474Z [pool-3-thread-15] DEBUG opId= com.vmware.vim.storage.common.task.CustomThreadPoolExecutor - [VLSI-client] Request took 9 millis to execute.
    2023-09-25T11:38:55.483Z [main] DEBUG opId= com.vmware.vim.storage.common.kv.KvManager - Deleted the provider 'spbmRollupComplianceResultProvider'.
    2023-09-25T11:38:55.483Z [pool-3-thread-16] DEBUG opId= com.vmware.vim.storage.common.task.CustomThreadPoolExecutor - [VLSI-client] Request took 7 millis to execute.
    2023-09-25T11:38:55.483Z [main] INFO opId= com.vmware.sps.upgrade.b2b.patch.action.RemoveUnusedSpbmKvProviders - Deleting fcdProfileAssociationProvider provider
    2023-09-25T11:38:55.483Z [main] INFO opId= com.vmware.vim.storage.common.kv.KvManager - Deleting the provider 'fcdProfileAssociationProvider'.
    2023-09-25T11:38:55.491Z [main] DEBUG opId= com.vmware.vim.storage.common.kv.KvManager - Deleted the provider 'fcdProfileAssociationProvider'.
    2023-09-25T11:38:55.491Z [pool-3-thread-17] DEBUG opId= com.vmware.vim.storage.common.task.CustomThreadPoolExecutor - [VLSI-client] Request took 7 millis to execute.
    2023-09-25T11:38:55.491Z [main] INFO opId= com.vmware.sps.upgrade.b2b.patch.action.RemoveUnusedSpbmKvProviders - [printPatchInfo] Patch SPBM unused provider cleanup patch support executed successfully to patch to VERSION11
    2023-09-25T11:38:55.491Z [main] INFO opId= com.vmware.vim.storage.common.serviceclient.ls.LsServiceEndpointUpdater - Fetching the SPS service info from the lookup service.
    2023-09-25T11:38:55.496Z [main] DEBUG opId= com.vmware.vim.storage.common.serviceclient.vmomi.AbstractVmomiClient - Creating stub for com.vmware.vim.binding.lookup.ServiceRegistration
    2023-09-25T11:38:55.503Z [main] INFO opId= com.vmware.vim.storage.common.util.OperationIdUtil - New operationID: 27ba4ad7-f703-4a41-aa22-0e7a01cfc4b1 set in request context
    2023-09-25T11:38:55.527Z [main] ERROR opId= com.vmware.vim.storage.common.serviceclient.ls.impl.ExternalServiceRegistryImpl - LS returned null or empty result, for filter={product = com.vmware.vim.sms, type = sms, nodeId = d66e54e0-c05c-4981-a931-46d2567a9cb6}
    2023-09-25T11:38:55.528Z [main] ERROR opId= com.vmware.vim.storage.common.serviceclient.ls.LsServiceEndpointUpdater - Could not find the SPS service info.
    com.vmware.vim.storage.common.serviceclient.ls.LsException: Could not find SPS info"

     

    This led me to https://kb.vmware.com/s/article/78900, but unfortunately none of the VASA providers are showing a statusFault = null, so no dice there.


    Any thoughts before I just redeploy?



  • 2.  RE: Failed upgrade to vCenter 8u2

    Broadcom Employee
    Posted Sep 26, 2023 04:07 PM

    Hey Kev - Can you provide support bundle from the appliance?



  • 3.  RE: Failed upgrade to vCenter 8u2

    Posted Sep 26, 2023 07:38 PM

    I *could* share a support bundle  , but I've reverted it back to the pre-upgrade snapshot. I can happily rerun the upgrade and snag the upgrade bundle afterwards if that would help, else I have a copy of everything in /var/log/vmware/ after the upgrade. If any of that would be helpful, happy to share, just let me know what you need.



  • 4.  RE: Failed upgrade to vCenter 8u2

    Posted Sep 26, 2023 08:11 PM

    (As an aside, I opted to deploy a fresh vCenter in the meantime, so there's no all fired rush on this - I posted this more as a curiosity and in case anyone else got hit by it - please don't feel like you need to spend any time on this!)



  • 5.  RE: Failed upgrade to vCenter 8u2

    Broadcom Employee
    Posted Sep 26, 2023 04:26 PM

    By any chance sps service was down before upgrade ? I feel like you have sps registration missing. But logs could give more insight. 
    From what version are you upgrading ?

    /usr/lib/vmware-lookupsvc/tools/lstool.py list --url http://localhost:7090/lookupservice/sdk > /var/core/services.txt
    and share the services.txt file 



  • 6.  RE: Failed upgrade to vCenter 8u2

    Posted Sep 26, 2023 07:37 PM

    I don't believe the service was down, and there are references further up the PatchRunner.log file (I'll attach here so you can see what I mean) that imply that it was checked and fine, and then the update deleted some references. 

    The VCSA has been reverted to snapshot before the upgrade, but I've attached the services.txt from the pre-upgrade appliance in case this is helpful. I can happily rerun the upgrade and capture this after the upgrade too, just let me know.

     

    Attachment(s)

    log
    PatchRunner.log   2.76 MB 1 version
    txt
    services.txt   266 KB 1 version


  • 7.  RE: Failed upgrade to vCenter 8u2

    Broadcom Employee
    Posted Sep 27, 2023 05:52 AM

    Hi Kev,

    Going by the service registration text file, Service Type: sms is missing.

    Since you have reverted the setup, double check again if the above one is missing. Also make sure vmdird is NORMAL. 

    You can use lsdoc tool to try and recreate the missing SMS service registration. https://kb.vmware.com/s/article/80469

    Regards,

    Nirmal Nair



  • 8.  RE: Failed upgrade to vCenter 8u2

    Broadcom Employee
    Posted Sep 27, 2023 06:01 AM

    Perfect  .  Please follow the same to check and create the missing registrations before upgrade. 

    Below from a 8.x lab. 
    [ /var/core ]# grep "Service Type:" services.txt |sort | uniq -c
    1 Service Type: activedirectory
    1 Service Type: applmgmt
    1 Service Type: certificateauthority
    1 Service Type: certificatemanagement
    1 Service Type: cis.cls
    1 Service Type: cis.vlcm
    1 Service Type: cis.vmonapi
    1 Service Type: client
    1 Service Type: com.vmware.vsphere.client
    1 Service Type: cs.authorization
    1 Service Type: cs.componentmanager
    1 Service Type: cs.ds
    1 Service Type: cs.eam
    1 Service Type: cs.identity
    1 Service Type: cs.inventory
    1 Service Type: cs.keyvalue
    1 Service Type: cs.license
    1 Service Type: cs.perfcharts
    1 Service Type: cs.registry
    1 Service Type: cs.vapi
    1 Service Type: cs.vsm
    1 Service Type: cs.vstats
    1 Service Type: hvc
    1 Service Type: imagebuilder
    1 Service Type: infraprofile
    1 Service Type: observability
    1 Service Type: phservice
    1 Service Type: rbd
    1 Service Type: sca
    1 Service Type: sms
    1 Service Type: sso:admin
    1 Service Type: sso:groupcheck
    1 Service Type: sso:sts
    1 Service Type: svcaccountmgmt
    1 Service Type: tokenservice
    1 Service Type: topologysvc
    1 Service Type: trustmanagement
    1 Service Type: vcenter.apigw
    1 Service Type: vcenter.computepolicy
    1 Service Type: vcenterserver
    1 Service Type: vcha
    1 Service Type: vcIntegrity
    1 Service Type: vsan-dps
    1 Service Type: vsan-health
    1 Service Type: vsphereui
    1 Service Type: vsphereui-plugins
    1 Service Type: wcp