VUM stopped service , VCSA fail

[gabomedina83]

Hi , I have a VCSA 6.7 with Build number 11726888 , A few days ago, VUM service began to fail and shows the following error, the service is stopped and when I try to start it, the following error is observed

could someone please give me some idea what is causing the failure???, thanks

ERROR TEXT

root@vcenter [ ~ ]# service-control --start vmware-updatemgr

Operation not cancellable. Please wait for it to finish...

Performing start operation on service updatemgr...

Error executing start on service updatemgr. Details {

    "detail": [

        {

            "localized": "An error occurred while starting service 'updatemgr'",

            "translatable": "An error occurred while starting service '%(0)s'",

            "id": "install.ciscommon.service.failstart",

            "args": [

                "updatemgr"

            ]

        }

    ],

    "componentKey": null,

    "problemId": null,

    "resolution": null

}

Service-control failed. Error: {

    "detail": [

        {

            "localized": "An error occurred while starting service 'updatemgr'",

            "translatable": "An error occurred while starting service '%(0)s'",

            "id": "install.ciscommon.service.failstart",

            "args": [

                "updatemgr"

            ]

        }

    ],

    "componentKey": null,

    "problemId": null,

    "resolution": null

}

[vijayrana968]

Did you make any changes to certificates ! Or any recent activity performed !

[gabomedina83]

Hello, export the virtual machine and pass it to another esxi host, after making this migration I do not know if it started to fail

I have this error in  vmware-vum-server.log

root@vcenter [ ~ ]# vim /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

2019-03-05T11:09:06.052-05:00 info vmware-vum-server[34463] [Originator@6876 sub=ThreadPool] Thread enlisted^M

2019-03-05T11:09:06.052-05:00 info vmware-vum-server[34463] [Originator@6876 sub=ThreadPool] Entering worker thread loop^M

2019-03-05T11:09:40.853-05:00 error vmware-vum-server[33772] [Originator@6876 sub=Ufa.HTTPService] Failed to read request; stream: <SSL(<io_obj p:0x00007fe8c4002a30, h:-1, <TCP '127.0.0.1 : 8084'>, <TCP '127.0.0.1 : 49154'> FD Closed>)>, error: N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <SSL(<io_obj p:0x00007fe8c4002a30, h:-1, <TCP '127.0.0.1 : 8084'>, <TCP '127.0.0.1 : 49154'> FD Closed>)>, duration: 00:00:45.793412 (hh:mm:ss.us))^M

--> [context]zKq7AVECAQAAAOkXmwALdm13YXJlLXZ1bS1zZXJ2ZXIAAJR/NWxpYnZtYWNvcmUuc28AANQ9HACLShgAhGQbAJhCLADx8CgAYXEoAEV1KAAn8jUB1HMAbGlicHRocmVhZC5zby4wAAKdjA5saWJjLnNvLjYA[/context]^M

2019-03-05T11:17:46.101-05:00 error vmware-vum-server[34463] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

2019-03-05T11:18:06.100-05:00 error vmware-vum-server[33771] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

2019-03-05T11:18:26.109-05:00 error vmware-vum-server[33823] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

2019-03-05T11:18:46.098-05:00 error vmware-vum-server[33820] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

2019-03-05T11:19:06.100-05:00 error vmware-vum-server[33770] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

2019-03-05T11:19:26.103-05:00 error vmware-vum-server[34389] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

2019-03-05T11:19:46.095-05:00 error vmware-vum-server[33772] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

2019-03-05T11:20:00.257-05:00 info vmware-vum-server[39217] [Originator@6876 sub=Default] [shutdownMgr,92] Ufa shutdown manager stopping... first make sure this thread is enlisted.^M

2019-03-05T11:20:00.257-05:00 info vmware-vum-server[39217] [Originator@6876 sub=ThreadPool] Thread enlisted^M

2019-03-05T11:20:00.257-05:00 info vmware-vum-server[39217] [Originator@6876 sub=Default] [shutdownMgr,97] Ufa shutdown manager stopping... Total number of callbacks: 1^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[39217] [Originator@6876 sub=Default] [shutdownMgr,109] Ufa shutdown manager stopped^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33775] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33774] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33773] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33820] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33772] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33823] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33822] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33776] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33770] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[34463] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33819] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[34389] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33771] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:01.362-05:00 info vmware-vum-server[33591] [Originator@6876 sub=Ufa] Plugin stopped^M

2019-03-05T11:20:01.362-05:00 info vmware-vum-server[33591] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:01.362-05:00 info vmware-vum-server[33591] [Originator@6876 sub=Default] [main,416] VCI service stopped.^M

[mfirth]

Hi,

I'm having the same issue - the VUM service is refusing to start on my vCenter.

We are running with an official SSL certificate for the GUI, to avoid users having to click through SSL errors to access vCenter.

The SSL certificate was able to successfully install in January, so VUM must have started after the installation.

However, we now need to update the SSL certificate again, and can't because VUM is in a failed state (and so any attempt to install a new certificate results in the vCenter rolling back because VUM fails to start).

The problem appears to be that /usr/lib/vmware-updatemgr/bin/refreshCerts-utility.py is failing with a certificate error - from its log file:

----------------

Can't update VUM service info: (vmodl.fault.InvalidArgument) {

   dynamicType = <unset>,

   dynamicProperty = (vmodl.DynamicProperty) [],

   msg = '',

   faultCause = <unset>,

   faultMessage = (vmodl.LocalizableMessage) [],

   invalidProperty = 'Invalid certificate'

}

Traceback (most recent call last):

  File "/usr/lib/vmware-updatemgr/bin/vumUtils.py", line 370, in updateServiceInfo

    lsClient.ReRegister(vumServiceInfo.serviceId, setSpec)

  File "/usr/lib/vmware-updatemgr/bin/vumUtils.py", line 813, in addSecurityctxToRequests

    reqMethod(self, *args)

  File "/usr/lib/vmware-updatemgr/bin/vumUtils.py", line 823, in ReRegister

    sr.Set(svcId, newSpec)

  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 557, in <lambda>

    self.f(*(self.args + (obj,) + args), **kwargs)

  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 363, in _InvokeMethod

    return self._stub.InvokeMethod(self, info, args)

  File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1418, in InvokeMethod

    raise obj # pylint: disable-msg=E0702

pyVmomi.VmomiSupport.vmodl.fault.InvalidArgument: (vmodl.fault.InvalidArgument) {

   dynamicType = <unset>,

   dynamicProperty = (vmodl.DynamicProperty) [],

   msg = '',

   faultCause = <unset>,

   faultMessage = (vmodl.LocalizableMessage) [],

   invalidProperty = 'Invalid certificate'

}

Unable to update VUM service info

----------------

From what little I have been able to work out from the files involved, it looks like it is submitting a certificate from /usr/lib/vmware-updatemgr/bin/ssl/rui.crt to gain access to some service point (I haven't worked out the URL for what it is trying to access), and that is being rejected.

From what little I can see in the vCenter GUI, that certificate file seems to match the certificate for "vpxd-extension" shown in the vCenter administration section.

Is it possible to find logs from the "server" end of the connection, to see why it is having a problem?

Is this someone that is being seen more widely? - it seems to be the same issue as is being described in this thread, which shows that at least some other people are having the problem.

[mfirth]

Hi,

I did a bit more digging, and (for my case) it seems that the old certificate was still being retrieved from somewhere (using "LookupServiceClient().GetServiceInfoList()") by vumUtils.py as the "sslTrust" field of the objects it was trying to re-register, and whatever it was trying to register them with was then rejecting the request because that certificate had expired.

I hacked the script to replace that field with the certificate that should have been present (which was already being used by vumUtils.py), and I was then able to get the update service to start, and it seems to be working.

I haven't yet checked to see whether the "LookupServiceClient().GetServiceInfoList()" call is still returning the old certificate, or whether the "re-registration" updated it to the correct one.

Attached is a patch with what I changed, in case it is useful to anyone else.

[irvingpop_chef]

thank you mfirth​ this patch fixed my VUM!!

[icorbitt]

Thanks a ton mfirth, got me out of a pinch.

[Nacer_Bell]

Hi,

I'm having exactly the same issue in my VCSA 6.5, I did not find vumUtils.py so I'm not sure how can I apply this patch?

Appreciate your help !! 

[mfirth]

Hi - I'm not sure if the file is the same or in the same place on 6.5 - the original poster and me are both using 6.7

On 6.7, the file is in /usr/lib/vmware-updatemgr/bin

[psyko1337]

Thank you!

worked in VCSA 6.7.0.42000

[luckyloveu]

vumUtils.patch.zip how to use this patch

[mfirth]

Hi

If you are using VMware V6.7, then the process is:

Extract vumUtils.patch from the ZIP (on a PC, though it could be done on the VCSA if you really wanted)

Log in to the VCSA and get a root Bash shell (just "shell" if you log in as root, a little more complex if you log in as a non-root user - there must be instructions somewhere on here!)

Cd into /usr/lib/vmware-updatemgr/bin

Copy the patch file into that directory on the VCSA (I did it by using "sftp" out from the VCSA to a Linux machine - there are probably other ways)

Run "patch -p0 < vumUtils.patch" to patch the vumUtils.py file

You should now be able to start the VUM service, either with "service-control --start vmware-updatemgr" from the VMware shell, or by restarting the VCSA

[tranquilnet]

This fixed my issue, thanks! Thankfully I came across this post after already trying multiple "fixes" over several hours. Question: Do you think spinning up a new vCenter, and importing a backup created from the appliance GUI would help solve the issue more permanently? 

[AlbertoCallegar]

Hi mfirth​... How can I use your patch?

[mfirth]

Hi,

See the reply I posted to luckyloveu​ - hopefully that helps

[Krexis]

Many thanks and kudos  

Worked like a charm and put an end to the hours and hours of finding and trying fixes that didn't make the grade.

[Anton Somov]

Thank you! Fixed it for Version 6.7.0 Build 19832280

[rebru]

Thank you, fixed it for Version 6.7.0.53000

[Nodirbek13]

Had this issue on a VCSA which got all the expiring certificates renewed recently. The patch worked like a charm.

Thank's a lot !

I did it directly on the server

cd /usr/lib/vmware-updatemgr/bin
cp vumUtils.py vumUtils.py.backup
wget https://communities.vmware.com/wbsdv95928/attachments/wbsdv95928/2514/762/1/vumUtils.patch.zip
unzip vumUtils.patch
chown updatemgr:updatemgr ./vumUtils.patch
patch -p0 < vumUtils.patch

It ran fine and showed::

patching file vumUtils.py
Hunk #1 succeeded at 241 (offset 5 lines).
Hunk #2 succeeded at 299 (offset 5 lines).
Hunk #3 succeeded at 344 (offset 5 lines).
Hunk #4 succeeded at 369 (offset 5 lines).
Hunk #5 succeeded at 400 (offset 5 lines).
Hunk #6 succeeded at 409 (offset 5 lines).
Hunk #7 succeeded at 531 (offset 5 lines).

Once again, thank you.

[ChadMarkley]

I literally logged in to say THANK YOU!! Had all kinds of issues with the certs expiring and services failing. Got everything back up and this was the last piece. You all are rock stars!!!

[initsol]

thank you so much for providing this! I was searching forever for a solution, this patch was simply great.

Thanks so much

[Termont]

*send beer + Hi5*

[Danilor1]

Thank you so much your patch solved my issue.. 

[antspopov]

The patch did not work on version 7.0.0, but the fixes were made manually. This worked for me. Thank you.

[NathanosBlightc]

I have a similar problem and upgrading the VCSA solves the problem. Can you try this? (provide a full backup of your VCSA, if you decide to upgrade)

[mfirth]

It looks to me that the certificate is installed right the first time, but then VUM keeps retrieving the original certificate, rather than updating it when it changes.

So a re-install will fix the problem for however long your certificate is valid, but I don't think it is a permanent fix.

So far, I've just kept reapplying my patch whenever I update the VCSA if I have to - it seems to revert it on some updates but not others. I haven't had a problem doing that, so either the patch is still needed or it is at least doing no harm to the newer versions

For every version of VCSA through to 6.7.0.44100 the original vumUtils.py was binary the same as the version I created my patch against, so either VMware still haven't looked into this, or they have fixed it in a different way.

[Grzegorz Suproniuk]

Have anyone this patch?

vumUtils.patch.zip