CVE-2021-44228 -- Apache Log4j - Remote code execution vulnerability

[Owen Sheehy]

VMware has released a critical security advisory, VMSA-2021-0028. This advisory is for multiple VMware products that use the popular open source log4j Java logging component, which was discovered to have a critical vulnerability.

Due to the 0-day nature of this disclosure, this will be a developing situation. Please subscribe to our Security Advisories mailing list and ensure that you revisit the VMSA-2021-0028 advisory periodically to get the latest information. 

First, if you haven’t read the original advisory or are a returning visitor here are some links to the different resources available:

VMware Security Advisory VMSA-2021-0028 (descriptions of the issues and workarounds)
VMSA-2021-0028 Questions & Answers (common questions & answers hosted in VMware Tech Zone)
Tips for Patching VMware vSphere (practical advice for ensuring patching success)
VMware vSphere Security Configuration Guide (baseline security best practices for vSphere)
VMware Ransomware Resource Center (discussion around tactics to help prevent, deter, and recover from attacks)
VMware Ports & Protocols Firewalling Guidance (ports.vmware.com)

The current expectation is that we will be publishing a number of product specific KBs with a workaround which is designed to be a temporary solution, pending the release of an upgrade/patch for the impacted product.

As such, the current recommendation is that you review the advisory and implement any workarounds or product updates as soon as possible.

If your product is pending an update, the advisory will be updated with the workaround/upgrade details as they become available 

Please fell free to post any questions you may have here and we will be happy to help in any way we can.

[jwdg]

Hi

In the workaround section for 6.7 VCSA the following statement is made: 

NOTE:- The below workaround is applicable for vCenter Server Appliance 6.7 P05 and Older versions only.

I'm not clear where that version naming fits into the numbers I'm familiar with - In my case I have build 18485185 (6.7U3o) - is this before or after P05? 

Thanks

John

 

[Martin Meinhardt]

6.7 P05 is Build 17700523. That means your Build is after P05.

see the following KB:

https://kb.vmware.com/s/article/2143832

 

[MartinLucanskyU]

 we need vCenter Server Appliance 6.7 P05 build number not ESXi 6.7 P05.

[Martin Meinhardt]

Correct, P05 is a name for ESXi Patches (has been mistakenly used). If you look at the release date of this ESXi Patch P05 and then look for the same release date of the vCenter Patch on the according KB

https://kb.vmware.com/s/article/2143838

you can see that they actually meant vCSA 6.7 U3m (which has been corrected now)

[behrent]

I have to chime in with John (jwdg) here: the workarounds described in https://kb.vmware.com/s/article/87081?lang=en_US#vCenter6.7 are targeted towards vCenter Server Appliances - and there is no such thing as a "vCenter Server Appliance 6.7 P05": VCSA-naming-schemes are different (see: https://kb.vmware.com/s/article/2143838).

The suffix "6.7 P05" refers to the ESXi build 17700523, that's correct - but ESXi itself is not affected by the Log4j2-RCE-vulnerability.

So the note "The below workaround is applicable for vCenter Server Appliance 6.7 P05 and Older versions only." is confusing in different ways: what if you're running a vCenter Appliance 6.7 Update 3p (6.7.0.51000), VAMI-Build: 18831133? The Jndi-Lookup-Class exists in both jar-files (Analytics and CM, /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar, /usr/lib/vmware-cm/lib/log4j-core.jar), but it is totally unclear, what this note is referring to (Analytics only? both Analytics and CM service?)

What side effects are to be expected if you apply the workarounds to a "newer" VCSA (6.7.0.51000 U3p, 6.7.0.50000 U3o) or in other words: what's the reason for this warning note with no correct VCSA-Build-referral?

Please advice,
Nico

[Baiju Parambil]

  Thanks for posting the questions. vCenter Server Appliance 6.7 P05 is actually U3m (vCenter Appliance 6.7 Update 3m 6.7.0.47000). We agree to the concerns mentioned in this thread, just changed P05 to U3m in the VMSA workaround KB.

We will also look at the other questions mentioned in this thread and circle back with updates.

[MartinLucanskyU]

Hi, now it is more clear.

NOTE:- The below workaround (Analytics service) is applicable for vCenter Server Appliance 6.7 Update 3m and Older versions only.

[MartinLucanskyU]

The version of vCenter Server was changed. From 6.7 u3m to 6.7u3o.

NOTE:- The below workaround (Analytics service) is applicable for vCenter Server Appliance 6.7 Update 3o and Older versions only.  The JAR is already updated to 2.11 on the later versions.

[Stacy McClure]

Hi, Please go here https://www.vmware.com/security/advisories/VMSA-2021-0028.html  for the latest updates. 

[Stacy McClure]

Content is being update frequently  

For Tanzu you can find updated KBs and Answers here

For VMware Core you can find updates KBs and Answers here

[Donghoon Lee]

Hi, I have a question.

Currently, I'm using the NSX for vSphere.

Does this issue also occur in NSX for vSphere Versions 6.3.x and 6.4.x ?

I can't see the NSX for vSphere in the list on this page.

https://www.vmware.com/security/advisories/VMSA-2021-0028.html

[Owen Sheehy]

As per the above, this is a developing situation and the information in the VMSA will be updated with all the latest information. 

Please ensure that you revisit the advisory throughout the day. A number of products are still under investigation and updates will continue throughout the day

[takedahideyuki]

Hello, I have a question about workaround of vCenter and  vCenter server appliance.

Is there any disadvantage if we do workaround? (eg. the log level become down or certain log messages are not generated.)

[markus101]

Concerns VCSA 6.7:

In my opinion, the section "verify the changes - step 2" is not 100% precise, if VCSA newer than Update 3o.
grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
returns "1"

but:
ps auxww | grep analytics | grep formatMsgNoLookups
returns that "-Dlog4j2.formatMsgNoLookups=true" IS used

That's logical, because analytics service on Update 3p works with log4j version 2.11, not 2.8. So "verify the changes - step 2" is ONLY important, if VCSA is on Update 3o or older and you have done the mitigation described under "Analytics Service". Here should be a note too, to prevent confusion. At least I was a bit confused ...

[MrtnsPaul]

Hi, If today I do the workaround when release the fix, do I update normally or should I undo the workaround first? Thank you!

[Baiju Parambil]

 Q7 in official Q&A for this VMSA-2021-0028 would help - https://core.vmware.com/vmsa-2021-0028-questions-answers-faq#sec19130-sub7

[MrtnsPaul]

Thanks a lot!

[casaub]

Hello,

reg. vCSA 6.7.0.51000 U3p (release 18831049).

 

We noticed one difference in the workaround KB87081, under 'Verify the changes' it is saying:

 

    2. Verify the Analytics Service changes:

                grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l

                This should return 0 lines

 

But the output from our vCSA is:

                # grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l

                1

 

 

The output of the other query

 

    3. Verify the CM Service changes:

               grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l

                This should return 0 lines

 

is showing what it should:

                # grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l

                0

 

So one of these is not correct. Is there an error in KB87081 or is our vCSA having an issue?

(We did not run the workaround for 'Analytics Service' because it says: applicable for vCSA 6.7 Update 3o and older versions only.)

 

[behrent]

---

 wrote:

... 

So one of these is not correct. Is there an error in KB87081 or is our vCSA having an issue?

(We did not run the workaround for 'Analytics Service' because it says: applicable for vCSA 6.7 Update 3o and older versions only.)

---

Your result is -as far as I can tell- perfectly fine:

The workaround-part you did not apply (Analytics Service) is responsible for "zeroing" out the grep result (deleting the JndiLookup.class from the jar) - but this is not needed on U3p, since the jar-file /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar is no longer used in this release (Hello? Housekeeping anyone?)

Nevertheless, the "Verify the changes"-section does not reflect the fact, that there are different approaches for <= U3o and U3p.
The workaround KB-article is a living document and VMware started out without these insights, so step 2 in the verification process is only relevant for VCSA 6.7.50000 (U3o) or less and does not take the version-differentiation in account.

Step 2 of the verification instruction for 6.7.x should sound:

2. Verify the Analytics Service changes (ONLY if you are on U3o or older)

 

BTW: This is exactly what noted here, but the instruction was not clarified since.

[casaub]

For vCSA 6.7.0 U3p, the answer was given indirectly by VMware today by corrected version of KB87081,  see Note:

 

2. Verify the Analytics Service changes:

grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
 

Note: This should return 0 lines. This does not apply to vCenter 6.7 U3p.

[Ed3]

If a version is not listed, is it considered not vulnerable? 

Example: SRM 8.3, 8.4, 8.5 listed as vulnerable. What is the status of SRM 8.1?

[Baiju Parambil]

 Details mentioned in the Related Information section in the KB https://kb.vmware.com/s/article/87123 would help to address your query. You may search for keyword SRM 8.1.x in this KB.

[Dan_Johnson]

According to this https://kb.vmware.com/s/article/87123

"Note: Products older than SRM 8.3 are beyond End of General Support date, so patches will not be made available for these older versions. As there may be other non-validated exposure, VMware recommends updating EOGS versions to a supported release as a priority.
SRM 8.1.x and 8.2.x on Windows or Photon Appliances were shipped with log4j 1.x. Based on https://logging.apache.org/log4j/2.x/security.html we believe these versions are not exposed to CVE-2021-44228 because the product uses log4j 1.x and does not have JMSAppender configured in its log4j configuration. Customers are urged to monitor https://logging.apache.org/log4j/2.x/security.html as guidance from the Apache Logging team may change.  It is important to note that Log4j 1.x has reached end of life and is no longer supported by the Apache Logging team. As per https://logging.apache.org/log4j/2.x/security.html "Vulnerabilities reported after August 2015 against Log4j 1.x were not checked and will not be fixed". VMware strongly recommends upgrading to a supported version of SRM to minimize exposure."

[thomasicbcs]

Hi, my setup for vCenter 6.5 is that the PSC functionality is separated from the vCenter Appliance itself.

I assumed that the PSC appliance were affected by Log4j and thus need to be rectified like the vCenter appliance but my team disagree with me for some reason.

Am i right on this?

[Baiju Parambil]

 You are absolutely right, you need to apply the steps on PSC as well.

Execute both the scripts :
- remove_log4j_class.py attached to KB 87081 and
- sctipt vmsa-2021-0028-kb87081.py attached to KB87088