EXPECTED Behavior or Bug in 3.2.6 when managing windows remote accounts

[Sebastiano Alighieri]

I was reviewing a client's setup related to managing Windows Remote accounts with another 'Admin Account' when i observed a setup that boggled my mind.

FTR: My understanding is that PAM expects both the Management and Managed accounts to be of the same "TYPE" - ie linked to an application of the same type. So only a windows remote account can manage a windows remote account and an Active Directory Account can manage an Active Directory account... but you cannot have an Active Directory account (AD Application type) manage a windows remote account (Windows Remote Application Type); the accounts must be of the same type.

Here is the setup i observed.

The client has PAM 3.2.6, several domains and an Account (call it ADADMINACCT) in each domain that has been granted local administrative privileges on respective member servers by way of nested groups in each server's Local Administrators group.

Each of the AD (ADADMINACCT) Accounts has been on-boarded into PAM as follows

1. Once for each domain as an AD TYPE of Account (linked to respective Active Directory application for that domain)

1. These AD Type accounts are Synchronized and Verified - the password is known
2. Password rotation is not expected to ever happen on these accounts.
3. These AD accounts however, are not being used for any particular purpose in PAM.

 2. Once per Target Device as Windows Remote TYPE Account (linked to a windows remote application for that target device);

1. These WinRemoteAgent ADADMINACCT accounts are NEITHER synchronized nor verified
2. Password is stored in Password Authority only
3. To be 100% clear, these ADADMINACCT  accounts ONLY EXIST IN AD - they DO NOT EXIST on the member servers, the Password in PAM on each of these accounts has been manually set to the correct password / to match the AD Account Password (having fun yet?)

Now comes the part i can't wrap my mind around

They have, somehow, been able to on-board member server's local admin accounts (call them LOCALADMIN) using the Windows Remote (NON-SYNCH'd ADADMINACCT  as the Managing Account).

What's more they've been able to on-board several hundreds LOCALADMIN accounts (as Windows Remote accounts) and force-set the initial password on those accounts using the ADADMINACCT Windows Remote Account as the management account.

And it works - Initial on-boarding of the account works as does subsequent Password Rotation and verification; in the GUI the account is shown as verified and we receive success messages while verifying and updating the password; Although i can't explain why it would.

Although it does work, the catalina log shows symptoms of why it shouldn't. Here's an excerpt of from the client's catalina log:

Mar 25, 2020 2:51:50 PM com.cloakware.cspm.server.plugin.targetmanager.WindowsRemoteAgentTargetManager verifyWindowsCredentials

WARNING: Verifying credential for account LOCALADMIN on server cdp-4dgcsvc001.SUB.DOM.com with net rpc didn't succeed. Use rwin to do this operation again.

Mar 25, 2020 2:51:50 PM com.cloakware.cspm.server.plugin.targetmanager.windowsRemoteAgent.WindowsRemoteAgent a
INFO: invokeSMB exit code = 0

Mar 25, 2020 2:51:54 PM com.cloakware.cspm.server.plugin.targetmanager.windowsRemoteAgent.WindowsRemoteAgent execute
INFO: execute exit code = 0

Mar 25, 2020 2:51:54 PM com.cloakware.cspm.server.plugin.targetmanager.windowsRemoteAgent.WindowsRemoteAgent a
INFO: invokeSMB exit code = 0

Mar 25, 2020 2:51:54 PM com.cloakware.cspm.server.plugin.targetmanager.WindowsRemoteAgentTargetManager updateWindowsCredentials

WARNING: Updating credential for account LOCALADMIN on server cdp-4dgcsvc001.SUB.DOM.com by OTHER account with net rpc didn't succeed.Reason: [Failed to set password for 'LOCALADMIN' with error: Failed to connect to IPC$ share on cdp-4dgcsvc001.SUB.DOM.com.protocol negotiation failed: NT_STATUS_CONNECTION_RESET]. Use rwin to do this operation again.

Mar 25, 2020 2:51:54 PM com.cloakware.cspm.server.plugin.targetmanager.windowsRemoteAgent.WindowsRemoteAgent a
INFO: invokeSMB exit code = 0

Mar 25, 2020 2:51:58 PM com.cloakware.cspm.server.plugin.targetmanager.windowsRemoteAgent.WindowsRemoteAgent execute
INFO: execute exit code = 0

For testing purposes, in PAM, we changed the Password on one of the (ADADMINACCT) Windows Remote Accounts to something that we knew was NOT the correct password for that account; we again, attempted to change the password for the managed LOCALADMIN account that previously succeeded, but now it failed, with a different error:
PAM-CM-4050: Windows Remote logon failed because of bad username or password, with Administrator account ADADMINACCT on target server 10.14.134.137.

I've tested this setup in my 3.3 clustered environment and i cannot get it to work, i cannot get the password to force set upon initial on-board, i can't get it to spin while managed by an similarly configured admin account - i get this error - which is the behavior that i would expect.

Mar 25, 2020 8:31:26 PM com.ca.pam.rest.PAUtil generateExceptionFromAppCtx
SEVERE: UpdateTargetAccountCmd.invoke Failed to synchronize password with target

Mar 25, 2020 8:31:47 PM com.cloakware.cspm.server.app.impl.VerifyAccountPasswordCmd invoke
WARNING: VerifyAccountPasswordCmd.invoke, end: result=true, accounts=1, duration=106.38549ms

Mar 25, 2020 8:32:23 PM com.cloakware.cspm.server.plugin.targetmanager.WindowsRemoteAgentTargetManager verifyWindowsCredentials
WARNING: Verifying credential for account LOCALADMIN on server 192.168.2.102 with net rpc didn't succeed. Use rwin to do this operation again.

Mar 25, 2020 8:32:28 PM com.cloakware.cspm.server.app.impl.VerifyAccountPasswordCmd invoke
WARNING: VerifyAccountPasswordCmd.invoke, end: result=true, accounts=1, duration=4337.4937ms

Mar 25, 2020 8:32:42 PM com.cloakware.cspm.server.plugin.targetmanager.WindowsRemoteAgentTargetManager verifyWindowsCredentials
WARNING: Verifying credential for account LOCALADMIN on server 192.168.2.102 with net rpc didn't succeed. Use rwin to do this operation again.

Mar 25, 2020 8:32:46 PM com.cloakware.cspm.server.plugin.targetmanager.WindowsRemoteAgentTargetManager updateWindowsCredentials
WARNING: Updating credential for account LOCALADMIN on server 192.168.2.102 by OTHER account with net rpc didn't succeed.
Reason: [mkdir failed on directory /var/run/samba/msg.lock: Permission denied
Failed to set password for 'LOCALADMIN' with error: Failed to connect to IPC$ share on 192.168.2.102.
session setup failed: NT_STATUS_LOGON_FAILURE]. Use rwin to do this operation again.

Mar 25, 2020 8:32:49 PM com.cloakware.cspm.server.app.impl.UpdateTargetAccountCmd invoke
SEVERE: UpdateTargetAccountCmd.invoke 4664: mkdir failed on directory /var/run/samba/msg.lock: Permission denied
Failed to set password for 'LOCALADMIN' with error: Failed to connect to IPC$ share on 192.168.2.102.
session setup failed: NT_STATUS_LOGON_FAILURE

PAM-CM-1104: The specified network account name or password is not correct.
null

Of course the password is correct on both accounts, i've verified them individually.

Any thoughts?

------------------------------
Services Architect
HCL Technologies Ltd
------------------------------

[Ralf Prigl]

Hi Sebastiano, I don't see anything out of the ordinary here. You state that the customer has the ADADMINACCT account defined as a Windows Remote type of account for each target device, and that these accounts are used to rotate the LOCALADMIN accounts. That would satisfy the requirement that the accounts are of the same type. Why are you having a problem with that? As long as the ADADMINACCT (Win Remote) accounts are configured with the right password, that's fine.
The logs are consistent with that configuration, and the password update for LOCALADMIN works, because the ADADMINACCT is able to logon and update the local user password.
In you case there may be a permission issue. You do not have messages at log level INFO, which would help understand the problem. But unlike you I see nothing wrong with the customer's configuration, other than that it's not advisable to have a bunch of unsynchronized accounts that you have to update manually whenever the AD account password changes. On the other hand, it's possible to use API calls to keep those accounts in sync with the AD account, with small out-of-sync time windows. Maybe that's what they do?

[Sebastiano Alighieri]

You say "The logs are consistent with that configuration, and the password update for LOCALADMIN works, because the ADADMINACCT is able to logon and update the local user password."

however the logs state:

"WARNING: Updating credential for account LOCALADMIN on server cdp-4dgcsvc001.SUB.DOM.com by OTHER account with net rpc didn't succeed.Reason: [Failed to set password for 'LOCALADMIN' with error: Failed to connect to IPC$ share on cdp-4dgcsvc001.SUB.DOM.com.protocol negotiation failed: NT_STATUS_CONNECTION_RESET]. Use rwin to do this operation again."

and as i mentioned, the ADADMINACCT does not exist on the target device.

Question -
Does the ADADMINACCT need to exist on the target device?
Does the ADADMINACCT need to be verified in order to manage another account?




------------------------------
Services Architect
HCL Technologies Ltd
------------------------------

[Sebastiano Alighieri]

i increased the logging on my 3.3 cluster and attempted it again.

see attached.

------------------------------
Services Architect
HCL Technologies Ltd
------------------------------

Attachment(s)

catalina_log.txt   20 KB 1 version

[Ralf Prigl]

The Windows Remote target connector has two ways to verify or update an account. If the first doesn't work, it will try the second. That's what the log shows. The second way works in your customer environment. The log messages don't show details for the other account. Could it be that the customer defines the account name such that it is recognized by the Windows server as domain account, e.g. [e-mail], and you don't do that? Or maybe the customer has the target application for the admin account configured with account type "domain account", and you don't?

[Sebastiano Alighieri]

​Hi @Ralf Prigl - No, they define it as ADADMINACCT 

They do not define the accounts as  DOAMAIN\samAccountName  nor do they define the User Principal Name. ​

I had thought about the Windows Remote Application being configured as Domain Account; I did ask the client for screenshots they have yet to get back to me-but I actually tested this configuration in my environment - and it doesn't work either. Same or similar errors.

It's that "second way of verifying / updating" that i'm curious about... could the answer lie therein?

Where can I read about what RWIN is and what it actually does - I'm wondering if RWIN isn't using a protocol that causes the OS to "ALSO" attempt domain authentication against its domain with the credentials USERNAME/PWD provided even though the account is a windows remote account.

Thanks in advance.

------------------------------
Services Architect
HCL Technologies Ltd
------------------------------

[Ralf Prigl]

RWIN in the pam log refers to using "net rpc ..." commands going to port 135 on the Windows server. That's not working for you or the customer. If that fails, we use smbclient, which goes to port 445. This is done by a python script based on the wmiexec script you find in a Google search.

[Sebastiano Alighieri]

Thanks for clarifying..

so i found this https://github.com/SecureAuthCorp/impacket/blob/master/examples/wmiexec.py

Admittedly i haven't spent much time going through the code and of course i don't know how PAM implements the wmiexec script, if it's a word-for-word copy of the script i found, but...

the wmiexec.py script imports this class: https://github.com/SecureAuthCorp/impacket/blob/master/impacket/smbconnection.py

which defines this method:
def kerberosLogin(self, user, password, domain='', lmhash='', nthash='', aesKey='', kdcHost=None, TGT=None,
TGS=None, useCache=True):
"""
...
:param string domain: domain where the account is valid for (required)

which contains this bit of code:
if domain == '':
domain = ccache.principal.realm['data'].decode('utf-8')
LOG.debug('Domain retrieved from CCache: %s' % domain)

principal = 'cifs/%s@%s' % (self.getRemoteName().upper(), domain.upper())
creds = ccache.getCredential(principal)

Which leads me to believe that i may be right - the script is attempting domain authentication on Windows Remote Accounts - even though, that was never the conventional understanding or design for Windows Remote type of accounts that were meant to manage 'LOCAL' accounts.

That said, it may not work in my environment because the SMB connection may only work on older SMB versions / OS versions (ie windwos 2016 disables SMBv1 by default)

a lot of speculation, or possible hypothesis?

I guess i have a little more work to do.

------------------------------
Services Architect
HCL Technologies Ltd
------------------------------

[Sebastiano Alighieri]

And here is the full excerpt from the client's catalina logs

------------------------------
Services Architect
HCL Technologies Ltd
------------------------------

Attachment(s)

cert2_PIMADM.txt   96 KB 1 version