Automic Workload Automation

Expand all | Collapse all

Timestamp password for Service Manager

  • 1.  Timestamp password for Service Manager

    Posted 17 days ago
    Edited by Michael A. Lowry 10 days ago
    Knowledge base article KB224476 documents a bug (AE-26648) fixed in 12.3.6 HF2. The problem is that the Automation Engine uses an undocumented password based on the UTC timestamp to control its own processes via the Service Manager. Someone with knowledge of the format of this password could also control the AE Server processes.

    The format of the the timestamp-based password is described with a Windows PowerShell example:
    $((Get-Date).ToUniversalTime().ToString("yyyMMdd$([char]0x01)HHmmss"))

    I was able to reproduce the described behavior. Below is a summary of my tests. All tests were conducted using UCYBSMCL.EXE on a Windows VM.

    Service Manager version

    TLS 1.2 secured

    Hosted service & version

    Timestamp-based password accepted

    12.3.6

    Automation Engine 12.3.6

    12.3.6

    Automation Engine 12.3.6

    12.3.6

    Automation Engine 12.3.6 HF3

    12.3.6

    Automation Engine 12.3.6 HF3

    12.3.6 HF2

    Automation Engine 12.3.6

    12.3.6 HF2

    Automation Engine 12.3.6

    12.3.6 HF2

    Automation Engine 12.3.6 HF3

    12.3.6 HF2

    Automation Engine 12.3.6 HF3

    12.3.6

    UNIX Agent 12.3.6

    12.3.6

    UNIX Agent 12.3.6

    12.3.6

    UNIX Agent 12.3.6 HF3

    12.3.6

    UNIX Agent 12.3.6 HF3

    12.3.6 HF2

    UNIX Agent 12.3.6

    12.3.6 HF2

    UNIX Agent 12.3.6

    12.3.6 HF2

    UNIX Agent 12.3.6 HF3

    12.3.6 HF2

    UNIX Agent 12.3.6 HF3


    The fix for AE-26648 in 12.3.6 HF2 definitely changes the behavior. Under the fix description, the only changed component listed is the Automation Engine. It seems obvious from the above results however that the only thing that matters is whether the Service Manager has been updated to 12.36 HF2.

    Whether TLS is enabled, and whether the hosted service (AE or Agent) has also been updated, appear to be irrelevant to whether the timestamp-based password is accepted. (In an earlier revision to this post, I reported that in one scenario even the updated 12.3.6 HF2 Service Manager accepted the timestamp-based password. This was incorrect.)

    It also seems clear that both the AE Server and the Service Manager would have to be updated to change the behavior without breaking things. I have not tested yet in what configurations it is possible to use the Automation Engine sheet of the Administration perspective in the Automic Web Interface to stop & start AE Server processes.

    Perhaps the updated Automation Engine uses a different undocumented internal password that is harder to guess. Perhaps the Automation Engine uses its back-channel to the Service Manager (specified via the -svc command line option for the CP & WP). In any event, I get the impression that 12.3.6 HF2 represents only the first part of what will eventually be a more thorough fix.

    One can also reproduce the behavior using the ucybsmcl binary on Linux:
    "$(date -u +%Y%m%d)$(printf '\x01')$(date -u +%H%M%S)"

    In ASCII, x01 is the SOH (Start of Header) character. Thanks to @Ivan Barumov, who discovered how to get it working on Linux.

    Any additional information would be appreciated.​​


  • 2.  RE: Timestamp password for Service Manager

    Posted 15 days ago
    Edited by Michael A. Lowry 15 days ago
    With additional testing, I confirmed my suspicion.

    Once you have updated the Service Manager that hosts the Automation Engine to 12.3.6 HF2, you will no longer be able to start these server processes using the Automation Engine sheet of the Administration perspective in the Automic Web Interface. Attempts to start AE server processes via the updated Service Manager 12.3.6 HF2 will result in the error message:
    Service Manager reports unknown error, error code '112'.

    This is essentially the same error that appears when trying to start the process using UCYBSMCL.EXE and specifying the timestamp-based password.
    *ERROR 3* Protocol violation (112)

    ERROR 3 is what one gets if one specifies an invalid password for an operation that requires a password.

    I imagine that the capability to start processes will be restored in a future update to the SMgr/AE/AWI.


  • 3.  RE: Timestamp password for Service Manager

    Posted 11 days ago
    Thank you Michael,

    I can confirm that this works the same from unix ucybsmcl

    Changing the '\x0001' to '\x01'  did the trick not sure why as the printed in file characters are the same SOH but it does not matter in this case.

    So in theory to get this particular vulnerability fixed for the moment only the service manager can be updated as in any case braking the functionality  from the AWI.









  • 4.  RE: Timestamp password for Service Manager

    Posted 10 days ago
    Edited by Michael A. Lowry 10 days ago
    I just discovered a quick way to test Service Manager passwords that is non-destructive. I found that ucybsmgr checks any provided password even if the requested operation does not require a password. This means you can test a password by submitting a command like GET_PROCESS_LIST that normally does not require a password.

    Details
    In ucsybsmgr.ini, there are three different passwords:
    • password_l1: allows read-only operations like GET_PROCESS_LIST
    • password_l2: allows L1 operations plus change operations like START_PROCESS and STOP_PROCESS
    • password_l3: allows L1 & L2 operations plus admin operations like SET_DATA
    Typically, no L1 password will be set, which means that you do not need to provide a password just to run the GET_PROCESS_LIST command. I found however, that in this case if you use ucybsmcl to send the GET_PROCESS_LIST command, but specify a password anyway, the Service Manager will still check the password even though a password is not required.

    Passwords that are accepted:
    • The L2 or L3 password.
    • The timestamp-based password (prior to SMgr 12.3.6 HF2)


  • 5.  RE: Timestamp password for Service Manager

    Posted 10 days ago
    Edited by Michael A. Lowry 5 days ago
    I received an update from Broadcom on this question. The support specialist helpfully pointed me to a part of the documentation I had not seen before. At the end of the page describing the Service Manager - Dialog Program, there is a section describing how to tell the Automation Engine & Automic Web Interface what passwords to use when contacting Service Managers.

    The documentation is characteristically unclear, so I did some tests to see how it works. Below are my instructions.


    UC_SMGR_LOGINS

    Service Manager passwords can be stored a special LOGIN object called UC_SMGR_LOGINS in client 0.

    To store the password to use when communicating with an Agent Service Manager, select the name of the agent in the Agent / Name field.
    To store the password to use when communicating with the Automation Engine Service Manager, insert the placeholder * into the Agent / Name field and the Type field.

    In both cases, the Username / ID is irrelevant. You must put a value in this field, but the value does not matter.
    The password should be the password (L2 or L3) of the corresponding Service Manager.

    Here's an example.


    In this example, two Service Manager passwords are stored:
    • The password for controlling AGENT123 via its Service Manager;
    • The password for controlling the Automation Engine server processes via the AE Service Manager.
    There is no way to specify a wildcard to indicate a group of agents. If you have 1000 agents and they all use the same Service Manager password, you must unfortunately add 1000 agent entries to to login object.


    Additional details

    The UC_SMGR_LOGINS object (ID 50048) is included in the AE 12.3.6 HF2 initial data. It appears in client 0 under the folder DIV_VARIABLES (folder ID 822).
    F001+00000
    F002CLOGIN
    F003CUC_SMGR_LOGINS
    F004+0000050048
    F005CStores the SMGR login
    F006+0000000005
    F00720200726092618
    F00920210914102740
    F010+0000000001
    F063C00000000000000000000000000000000
    F072C11.2
    R

    This object is not present in the AE 12.3.6 initial data. I gather the documentation was updated for 12.3.6 HF2 as well.

    We can therefore conclude that the complete fix for AE-26648 in 12.3.6 HF2 encompasses at least these four changes:
    • Service Manager: close timestamp password backdoor;
    • Automation Engine: add ability to use Service Manager passwords stored in UC_SMGR_LOGINS;
    • Initial Data: add UC_SMGR_LOGINS object;
    • Documentation: Add description of UC_SMGR_LOGINS object and how to use it.
    I hope this helps clarify the matter.

    I trust I am not the only one who believes it would be preferable if Broadcom made a effort to proactively publish complete and timely descriptions of vulnerabilities and fixes.


  • 6.  RE: Timestamp password for Service Manager

    Posted 12 days ago
    Hi @Michael A. Lowry,

    It's always interesting to follow your posts in the communities. Keep going !​

    /Keld.


  • 7.  RE: Timestamp password for Service Manager

    Posted 3 days ago
    Edited by Carlos Cardano 3 days ago
    Hello Michael,

    Thanks for posting this topic as we are also in the middle of this topic.

    Just want to ask you if or how you are managing the password for each service manager since changing password via command line is not yet available and we are talking 15000+ Agents.

    Scenario is you have a third party application that monitors and starts your Windows Agent when tagged as "Automatic Start" and for this, you need password.

    Now, we want to provide only the password_l2 to the third party and the password_l3 for us admin.

    But you can only change password via Service Manager Dialog.

    Have you also seen this scenario and best practice on handling this password?

    Thanks for the reading this comment.





    ------------------------------
    Regards,
    Carlos
    ------------------------------



  • 8.  RE: Timestamp password for Service Manager

    Posted 2 days ago
    Edited by Marcin Uracz 2 days ago
    Hi Carlos,
    I will chip in.

    The passwords in the ini file are, based on my research/tests, SHA-512 hashes with no salt. So you can generate one with simple linux command:

    echo -en test| shasum -a 512​

    And then put it in the right place using sed for example. No restart of the SMGR is necessary.

    ------------------------------
    Cheers,
    Marcin
    ------------------------------



  • 9.  RE: Timestamp password for Service Manager

    Posted 2 days ago
    Hi Marcin,
    Thanks for your input.
    Can you share also how it is done via Windows command prompt?


    ------------------------------
    Regards,
    Carlos
    ------------------------------



  • 10.  RE: Timestamp password for Service Manager

    Posted 2 days ago
    Edited by Michael A. Lowry 2 days ago
    @Marcin Uracz:  Good find. I knew it was different from the algorithm used by UCYBCRP.EXE but I didn't realize it was just a hash.

    This means it would be straightforward to automatically generate ucybsmgr.ini files as well as insert the corresponding passwords into the UC_SMGR_LOGINS object, e.g., with the LoginDefinition Java API class.


  • 11.  RE: Timestamp password for Service Manager

    Posted 2 days ago
    Edited by Ivan Barumov 2 days ago
    I would recommend a docu object  to store the ini file/s  and run this true the system via simple job on OS level.
    Whenever password change is needed create new one and replace the hashed string in the docu object and run the job again
    Loop true the Docu with PREP_PROCESS_DOCU and write it to file on the agent WRITE_PROCESS
    This runs no matter of the OS or version of it. Runs like a charm and it can be customized and automated as much as anyone likes