vCSA: log4j Vulnerability Mitigation Script Using Posh-Ssh

[faherne_CTI]

Kicking off a post to see if the community has shell scripters who can help out all the vCenter admins out there to draft a script to mitigate the log4j RCE vulnerability in vCSA.

The mitigation steps for each vCSA version are detailed in webpage: 

https://kb.vmware.com/s/article/87081

 

[LucD]

Although the KB clearly mentions that the instructions are "... a temporary solution only", I automated the instructions for my lab environment.

DISCLAIMER: The following code has only been tested in a limited environment. Use at your own risk!!!

Some caveats:

• Although the VCSA has the VMware Tools running, the code uses SSH. This is because a session ran via Invoke-VMScript does not have all the required environment variables set as needed.
  
• The code needs the Posh-Ssh module and at least version 3.0.0. Older versions of the module have issues connecting to a VCSA 7.*
  
• The snippet uses a ViCredentialStoreItem to retrieve the credentials for the SSH session to the VCSA. Replace this part with whatever code you use to provide such credentials
• The snippet only handles VCSA 7.*. This because I don't have older VCSA versions in my lab, and hence could not test the code against those versions. Based on the VCSA 7.* code it shouldn't be too hard to code the VCSA 6.* code in a similar way
  
• I didn't create an extended blog post because of the previous bullet
• The snippet where all VCSA services are restarted uses a Timeout of 5 mins. This can be too high or too low for your environment. The default Timeout was in any case too low for my test environment.

Again: Use at your own risk!!!

#Requires -Modules @{ ModuleName="Posh-Ssh"; ModuleVersion="3.0.0" }

$vcsaFQDN = 'your.vcsa'               # FQDN of the VCSA
$user = 'root'                        # User that can SSH into VCSA

# Provide PSCredential
# This sample uses a VICredentialStoreItem
$viCred = Get-VICredentialStoreItem -Host $vcsaFQDN -User $user
$cred = New-Object -TypeName PSCredential -ArgumentList $user, (ConvertTo-SecureString -String $viCRed.Password -Force -AsPlainText)

# Required for the stop/start of All Services
$vcsaTimeout = 300

#region Code snippets
$codeVersion = @'
vpxd -v
'@
$codeVMON0and1 = @'
sed -i.bak 's/exec $java_start_bin $jvm_dynargs "$@"/log4j_arg="-Dlog4j2.formatMsgNoLookups=true"\nexec $java_start_bin $jvm_dynargs $log4j_arg "$@" /usr/lib/vmware-vmon/java-wrapper-vmon
'@
$codeVMON2and3 = @'
sed -i.bak 's/exec $java_start_bin $jvm_dynargs $security_dynargs $original_args/log4j_arg="-Dlog4j2.formatMsgNoLookups=true"\nexec $java_start_bin $jvm_dynargs $log4j_arg $security_dynargs $original_args/g' /usr/lib/vmware-vmon/java-wrapper-vmon
'@
$codeVMON7Restart = @'
chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon;
chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon;
service-control --stop --all;
service-control --start --all;
'@
$codeUpdMgr = @'
sed -i.bak -e '$a\-Dlog4j2.formatMsgNoLookups=true' /usr/lib/vmware-updatemgr/bin/jetty/start.ini;
service-control --restart vmware-updatemgr;
'@
$codeAnaSrv = @'
cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak;
zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class;
service-control --restart vmware-analytics;
'@
$codeDBCC = @'
cp /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar.bak;
zip -q -d /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
'@
$codeVerVMON = @'
ps auxww
'@
$codeVerUpdMgr = @'
cd /usr/lib/vmware-updatemgr/bin/jetty/;
java -jar start.jar --list-config
'@
$codeVerAnaSrv = @'
grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
'@
$codeVerDBCC = @'
grep -i jndilookup /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar | wc -l
'@
#endregion

$session = New-SSHSession -ComputerName $vcsaFQDN -Credential $cred -AcceptKey

$result = Invoke-SSHCommand -SSHSession $session -Command $codeVersion -Verbose
$vcsaVersion = [Version]($result.Output.Split()[2])

switch ($vcsaVersion.Major) {
  # VCSA 6
  '6' {
    # To be completed
  }
  # VCSA 7
  '7' {
    #region vMON Service
    if ('2', '3' -contains $vcsaVersion.Build ) {
      $result = Invoke-SSHCommand -SSHSession $session -Command $codeVMON2and3
    } else {
      $result = Invoke-SSHCommand -SSHSession $session -Command $codeVMON0and1
    }
    $result = Invoke-SSHCommand -SSHSession $session -Command $codeVMON7Restart -TimeOut $vcsaTimeout
    #endregion

    #region Update Manager Service
    $result = Invoke-SSHCommand -SSHSession $session -Command $codeUpdMgr
    #endregion

    #region Analytics Service
    $result = Invoke-SSHCommand -SSHSession $session -Command $codeAnaSrv
    #endregion

    #region DBCC Utility
    $result = Invoke-SSHCommand -SSHSession $session -Command $codeDBCC
    #endregion

    #region Verify

    #region vMON Service
    $result = Invoke-SSHCommand -SSHSession $session -Command $codeVerVMON
    Write-Host 'vMON Service ' -NoNewline
    if ($result.Output | Where-Object { $_ -match '/usr/java/jre-vmware/bin' -and $_ -match '-Dlog4j2.formatMsgNoLookups=true' }){
      Write-Host -ForegroundColor green 'ok'
    }
    else{
      Write-Host -ForegroundColor red 'nok'
    }
    #endregion

    #region Update Manager
    $result = Invoke-SSHCommand -SSHSession $session -Command $codeVerUpdMgr
    Write-Host 'Update Manager Service ' -NoNewline
    if ($result.Output | Where-Object { $_ -match [RegEx]::Escape('log4j2.formatMsgNoLookups = true (/usr/lib/vmware-updatemgr/bin/jetty/start.ini)')}) {
      Write-Host -ForegroundColor green 'ok'
    } else {
      Write-Host -ForegroundColor red 'nok'
    }
    #endregion

    #region Analytics Service
    $result = Invoke-SSHCommand -SSHSession $session -Command $codeVerAnaSrv
    Write-Host 'Analytics Service ' -NoNewline
    if ($result.Output -eq 0) {
      Write-Host -ForegroundColor green 'ok'
    } else {
      Write-Host -ForegroundColor red 'nok'
    }
    #endregion

    #region  DBCC Utility
    $result = Invoke-SSHCommand -SSHSession $session -Command $codeVerDBCC
    Write-Host 'DBCC Utility ' -NoNewline
    if ($result.Output -eq 0) {
      Write-Host -ForegroundColor green 'ok'
    } else {
      Write-Host -ForegroundColor red 'nok'
    }
    #endregion
    #endregion
  }
  Default {
    Throw "Unhandled VCSA version $vcsaVersion"
  }
}

Remove-SSHSession -SSHSession $session | Out-Null

 

[faherne_CTI]

Thanks Luc!

I will be testing (in DEV) this evening and will give feedback on how it goes. 

[LucD]

There is an official Python script in KB87088

[faherne_CTI]

Hi Luc,

I'm encountering errors on the following line:

$result = Invoke-SSHCommand -SSHSession $session -Command $codeVersion -Verbose

 

Error: Unknown command: `vpxd'

[faherne_CTI]

I've encountered issues like this before when using Posh-Ssh to execute command against vCSA, but we never got to the bottom of it.

New-SSHSession executes successfully, but Invoke-SSHCommand fails. I'm unsure if the root cause is related to timing issues, or the default shell that the Invoke-SSHCommand commands execute against.

 

[faherne_CTI]

I am able to successfully execute "service-control --status" with Invoke-SSHCommand.

[LucD]

Which VCSA version are you running this against?
Did you check that the Posh-Ssh module is at least version 3.0.0?

[faherne_CTI]

PoshSsh Version: 3.0.0

PowerShell Version: 5.1.19041.1320

PowerCLI Version: 12.4.1

 

I'm running my tests against a single, standalone vCSA 7.0.2 appliance.

 

[LucD]

When you establish an intercative SSH session with the VCSA, does the vpxd command work?
Or do you get the same error?

[faherne_CTI]

No, the vpxd command fails with:

Host : vcsa-01.lab.local
Output : {Unknown command: `vpxd'}
ExitStatus : 0

[faherne_CTI]

I get the same error when I SSH to the vCSA via Putty and execute "vpxd -v" against the default appliance shell:

Connected to service

* List APIs: "help api list"
* List Plugins: "help pi list"
* Launch BASH: "shell"

Command> vpxd -v
Unknown command: `vpxd'
Command>

[faherne_CTI]

In Putty, once I type "shell" and access the Bash shell, i can successfully execute "vpxd -v".

It looks to me that the Invoke-SSHCommand command is executing against the appliance shell instead of the Bash shell.

I tried the following code:

__________________________________________________________________

$shellChange = @'
shell
'@

Invoke-SSHCommand -SSHSession $session -Command $shellChange -Verbose

__________________________________________________________________

But the command times out.

_______________________________________________________________________

Exception calling "EndExecute" with "1" argument(s): "Command 'shell' has timed out."
At C:\Program Files\WindowsPowerShell\Modules\Posh-SSH\3.0.0\Posh-SSH.psm1:266 char:25
+ $Output = $_.cmd.EndExecute($_.Async)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : SshOperationTimeoutException

[LucD]

I suspect I don't have this issue since I set the default shell for root to /bin/bash.
That is probably also the reason the KB with the Python script asks you to run it inside the VCSA.

You can change the default shell with the command (while in the bash shell)

chsh -s /bin/bash root

this will permanently change that for root.

To go back to the appliance, use (again inside the bash shell).

chsh -s /bin/appliancesh root

 
I'm not sure if there is an easier solution.

[LucD]

Seems I forgot my own solution for this VCSA - shell issue, see Solved: Re: Script to Create vCenter Accounts - VMware Technology Network VMTN
By using a Stream you can enter the 'shell' command.

[faherne_CTI]

Hi Luc,

For further feedback:

Once I manually SSH to the vCSA, enter the shell (by typing 'shell'), then reconfigure the default shell to bash (by typing 'chsh -s /bin/bash root'), the script runs like a dream. It took approximately 10 mins to execute against an empty vCSA 7.0.2:

vMON Service ok
Update Manager Service ok
Analytics Service ok
DBCC Utility ok

[faherne_CTI]

Hey Luc, I think I figured out a solution. I implemented a prep script to ensure that bash was configured as the default vCSA shell before I ran your script. If the prep script prints out the vCSA software version, your script will run without issue.

#Requires -Modules @{ ModuleName="Posh-Ssh"; ModuleVersion="3.0.0" }

$vcList = Import-Csv .\Log4j_vCSA_Inventory.csv | Select DNSName | Sort-Object DNSName
$cred = Get-Credential -Username "root" -Message "Please Enter Root Password:"

Foreach($vcsa in $vcList) {

# Create SSH Session
$session = New-SSHSession -ComputerName $vcsa.DNSName -Credential $cred –AcceptKey
$session

# Create SSH Shell Stream
$stream = New-SSHShellStream -SSHSession $session -TerminalName tty

# Switch to bash shell
$stream.WriteLine("shell")
sleep 3
while ($stream.Length -ne 0){
  $stream.Read()
}

# Configure Bash as default shell
$stream.WriteLine("chsh -s /bin/bash root")
sleep 3
while ($stream.Length -ne 0){
  $stream.Read()
}

# Execute Bash shell command as a test
$stream.WriteLine("vpxd -v")
sleep 3
while ($stream.Length -ne 0){
  $stream.Read()
}

# Disconnect stream
$stream.Close()
Remove-SSHSession -SSHSession $session | Out-Null

       Write-Host "Bash enabled as default shell..."
}

[LucD]

Great!

[faherne_CTI]

Thanks again Luc. I learnt so many new PowerShell skills in your solution. How to implement different types of "sed" commands (substitute and append), how to implement Posh-Ssh SSH shell stream and shell commands, and how to do a PowerShell regex! You're an absolute legend when it comes to tackling the unknowns in automated VMware PowerShell scripted solutions.