vCenter

 View Only
Back to discussions
Expand all | Collapse all

vCenter not working after log4 mitigation

cwc-kavaa

cwc-kavaaDec 13, 2021 12:50 PM

We have the same issue...
ronancosta

ronancostaDec 13, 2021 12:59 PM

Same problem

  • 1.  vCenter not working after log4 mitigation

    Posted Dec 13, 2021 10:25 AM

    Version: vCenter 6.7 on Windows server

    I mitigated vCenter log4 vulnerability as described here: https://kb.vmware.com/s/article/87096 This means I have altered vsphere-ui.json, vsphere-client.json, log4j-core.jar and JndiLookup.class. But now my vCenter doesn't seem to be working anymore. Whenever I go the vCenter in my browser, it just times out. I have backups of the files but I'm wondering what's going wrong since I believe I followed the exact instructions from the KB. I've already tried a server reboot but no difference.

    Anyone idea what's going on?

     

     

     

     

    C:\Program Files\VMware\vCenter Server\bin>.\service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vmware-cis-config...
Successfully started service vmware-cis-config
Performing start operation on service VMWareAfdService...
Successfully started service VMWareAfdService
Performing start operation on service VMWareDirectoryService...
Successfully started service VMWareDirectoryService
Performing start operation on service VMWareCertificateService...
Successfully started service VMWareCertificateService
Performing start operation on service VMwareIdentityMgmtService...
Successfully started service VMwareIdentityMgmtService
Performing start operation on service VMwareSTS...
Successfully started service VMwareSTS
Performing start operation on service VMwareDNSService...
Successfully started service VMwareDNSService
Performing start operation on profile: ALL...
Service-control failed. Error: Failed to start vmon. Error: (1056, 'StartService
', 'An instance of the service is already running.')

C:\Program Files\VMware\vCenter Server\bin>.\service-control --status
Stopped:
 EsxAgentManager VMWareCAMService VMwareComponentManager VServiceManager content
-library mbcs rhttpproxy vPostgres vapiEndpoint vimPBSM vmonapi vmsyslogcollecto
r vmware-autodeploy-waiter vmware-imagebuilder vmware-license vmware-network-cor
edump vmware-perfcharts vmwareServiceControlAgent vpxd vpxd-svcs vsan-health vsp
here-ui vspherewebclientsvc
StartPending:
 vmon
Running:
 VMWareAfdService VMWareCertificateService VMWareDirectoryService VMwareDNSServi
ce VMwareIdentityMgmtService VMwareSTS vmware-cis-config

C:\Program Files\VMware\vCenter Server\bin>

C:\Program Files\VMware\vCenter Server\bin>.\service-control --stop
Operation not cancellable. Please wait for it to finish...
Performing stop operation on profile: ALL...
Successfully stopped profile: ALL.

C:\Program Files\VMware\vCenter Server\bin>.\service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vmware-cis-config...
Successfully started service vmware-cis-config
Performing start operation on service VMWareAfdService...
Successfully started service VMWareAfdService
Performing start operation on service VMWareDirectoryService...
Successfully started service VMWareDirectoryService
Performing start operation on service VMWareCertificateService...
Successfully started service VMWareCertificateService
Performing start operation on service VMwareIdentityMgmtService...
Successfully started service VMwareIdentityMgmtService
Performing start operation on service VMwareSTS...
Successfully started service VMwareSTS
Performing start operation on service VMwareDNSService...
Successfully started service VMwareDNSService
Performing start operation on profile: ALL...
Service-control failed. Error: Failed to start vmon. Error: (1056, 'StartService
', 'An instance of the service is already running.')

     

    edit: I had to have the server running again so I reverted log4j-core.jar and JndiLookup.class, same problem. After that I reverted vsphere-ui.json, vsphere-client.json, still same problem. The last step to completely revert everything is to undo the actions from vMON.py script. I'm currently trying to figure out what the script did.

     



  • 2.  RE: vCenter not working after log4 mitigation

    Posted Dec 13, 2021 12:50 PM

    We have the same issue...



  • 3.  RE: vCenter not working after log4 mitigation

    Posted Dec 13, 2021 12:59 PM

    Same problem



  • 4.  RE: vCenter not working after log4 mitigation

    Posted Dec 13, 2021 02:27 PM

    Did anyone of you already found a solution? 

    Still waiting on a Call from vmware...



  • 5.  RE: vCenter not working after log4 mitigation

    Posted Dec 13, 2021 02:35 PM

    No solution yet for me. If you got a call back with some more info, I'm eager to hear.

    I reverted log4j-core.jar and JndiLookup.class, same problem. After that I reverted vsphere-ui.json, vsphere-client.json, still same problem. The last step to completely revert everything is to undo the actions from the vMON.py script. I'm currently trying to figure out what the script did.

    edit: nvm, I didn't have to undo the script it seems. A reboot made my server available again but without mitigations so I'm still curious on how to solve that.



  • 6.  RE: vCenter not working after log4 mitigation

    Posted Dec 13, 2021 02:41 PM

    We have put back all the files as well and rebooted, but did not work.



  • 7.  RE: vCenter not working after log4 mitigation

    Posted Dec 14, 2021 08:50 AM

    Hi,

    we've had the same issue (VMON not starting).
    Since then VMware has updated their KB with the following sentence:
    "Make a backup of the below two .json files in a different location than the originals"
    In the first try we had the backups in the same folder, after re-doing everything and moving them to a different location the service started successfully.



  • 8.  RE: vCenter not working after log4 mitigation

    Posted Dec 14, 2021 08:54 AM

    If we only knew that!

    We made the same mistake i think, at least we had the files moved, but also left .bak files in original location.

    We ended up migrating to vCenter 7 in the end. We got the old vCenter server working somehow, don't ask me how... But it did eventually.

     



  • 9.  RE: vCenter not working after log4 mitigation

    Posted Dec 14, 2021 03:45 PM

    Having the same issues with the vCenter running on the Virtual Appliance.



  • 10.  RE: vCenter not working after log4 mitigation

    Posted Dec 14, 2021 03:49 PM

    KB87096 was fixed, after retry following the KB it worked correctly.

    https://kb.vmware.com/s/article/87096

    Note: Take a snapshot before the changes.



  • 11.  RE: vCenter not working after log4 mitigation

    Posted Dec 13, 2021 05:53 PM

    Exactly the same case but with vcenter 7.0.2 Version: 7.0.2.00400
    Only changed vi /usr/lib/vmware-vmon/java-wrapper-vmon file as a workaround from:

    Original
    exec $java_start_bin $jvm_dynargs $security_dynargs $original_args
     
    Updated
    log4j_arg="-Dlog4j2.formatMsgNoLookups=true"
    exec $java_start_bin $jvm_dynargs $log4j_arg $security_dynargs $original_args


    and rebooted services.. does not come up anymore

    Sent a ticket



  • 12.  RE: vCenter not working after log4 mitigation

    Posted Dec 13, 2021 07:41 PM

    resolved my case

    went to compare the original:
    /usr/lib/vmware-vmon/java-wrapper-vmon file
    and backuped /usr/lib/vmware-vmon/java-wrapper-vmon.bak file

    and even if I did not change that myself then there was a empty line in the beginning of the file.
    After removing that and restarting the vcenter. All is fine again.. 



  • 13.  RE: vCenter not working after log4 mitigation

    Posted Dec 15, 2021 06:56 PM

    Just throwing my hat in the ring, also on 6.7 on Windows. We are having the same issue, vmon is "Start Pending" and will not come back up. I followed the KB article AFTER the changes clarifying that the backup files should be moved to another directory.

    Support ticket is open, will update if any fix is found.



  • 14.  RE: vCenter not working after log4 mitigation

    Posted Dec 16, 2021 07:56 PM

    Update: TAC was able to get our server back up and running by just repeating the steps in KB 87096, then rebooting the server.



  • 15.  RE: vCenter not working after log4 mitigation

    Posted Dec 22, 2021 02:59 AM

    i have a similar issue, but vimPBSM wont start ( i think its part of sps )

    but i only got to step1 and stopped. firstly those 2 json files it askes to remove lines, well those lines of code dont exist...

    the vcenter is acessible and working,besides this not running.

     

    i have backups of the json files in the different location

    i noticed all the files in C:\ProgramData\VMware\vCenterServer\cfg\vmware-vmon\svcCfgfiles have been touched my the vmon py script. the touched files seem to have increased in size a little and some extra code added.

    i have all the files from another vcenter healthy server, should i stop all services

    copy all from healthy to to bad vcenter, and start services.

    PS on 6.5.0 BUILD 9451637

    my first psc seem to go ok.