Symantec Access Management

 View Only
Expand all | Collapse all

Smart Debugging: Update JBOSS/Wildfly Log4j without restart for Identity Suite (and vApp)

  • 1.  Smart Debugging: Update JBOSS/Wildfly Log4j without restart for Identity Suite (and vApp)

    Posted Oct 13, 2017 01:18 AM

    Team,

     

    While assisting a customer with their documentation efforts of their current production IM business logic, we typically use the ims.policyxpress=DEBUG logger, using the dynamic logger tool:  logging.jsp.

     

     

    However, there are times when you would like to leave these loggers in place. for test/dev/staging environments; or to use in production without restarting the solution.

     

    Fortunately, if you are using JBOSS/Wildfly, this functionality does exist, within the JBOSS_HOME/bin/jboss-cli.sh (.bat)

     

    2018-08-25 - Update.  Added text download versions to add/remove the new loggers/appender file.

    2018-09-10 - Update.  Added new versions that include IP/nodename as part of the appender file, to assist with RCA for cluster logging.

    2018-10-01 - Update.  One caution note for the vApp, ensure you are using the "batch" mode during removal of objects to prevent a possible error.  Otherwise, if you remove a dependency before another object, you may impact startup of Wildfly.  On standalone deployments, this is not an issue, but the vApp's secured deployments permissions would not allow the 'config' userID to update the standalone-full-ha-***.xml configuration file, that is owned by "wildfly" userID.

     

     

     

     Using select loggers will only add add < 10% delay to processing time.

     

    Below is an example to add the select loggers, with a NEW file appender/handler to allow business logic to be separated from the default J2EE server.log

     

    Step 1:  Create a batch file to add the loggers:  jbosslog.cli

     

    connect

    batch

    /subsystem=logging/periodic-rotating-file-handler=im_business_logic:add(file={"path"=>"im_business_logic_debug.log", "relative-to"=>"jboss.server.log.dir"}, suffix=".yyyy.MM.dd", level=DEBUG, append=true, autoflush=false, suffix=".yyyy-MM-dd-HH", formatter="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n")

    /subsystem=logging/logger=ims.tasktrack.custom:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)

    /subsystem=logging/logger=im.feeder:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)

    /subsystem=logging/logger=ims.policyxpress:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)

    /subsystem=logging/logger=ims.llsdk.role.azengine:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)

    /subsystem=logging/logger=ims.llsdk.role.azcache.ridiculouslydetailed:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)

    /subsystem=logging/logger=ims.tmt.submit.validation.blth:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)

    /subsystem=logging/logger=ims.tmt.submit.execute.blth:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)

    run-batch

     

    Step 2:  Execute this batch file by navigating to the JBOSS_HOME\bin\ folder; and execute  .

    ./jboss-cli.sh --file=jbosslog.cli

     

    The logger will now be active, and you will notice a new FILE handler under:

    JBOSS_HOME/standalone/log/im_business_logic_debug.log

     

     

    If you wish to remove these loggers and FILE handler

     

     

    Step 3:   Create a 2nd batch file:   jbosslogremove.cli

     

    connect

    batch

    /subsystem=logging/periodic-rotating-file-handler=im_business_logic:remove

    /subsystem=logging/logger=ims.tasktrack.custom:remove

    /subsystem=logging/logger=im.feeder:remove

    /subsystem=logging/logger=ims.policyxpress:remove

    /subsystem=logging/logger=ims.llsdk.role.azengine:remove

    /subsystem=logging/logger=ims.llsdk.role.azcache.ridiculouslydetailed:remove

    /subsystem=logging/logger=ims.tmt.submit.validation.blth:remove

    /subsystem=logging/logger=ims.tmt.submit.execute.blth:remove

    run-batch

     

     

    Step 4:  Execute this batch file by navigating to the JBOSS_HOME\bin\ folder; and execute  .

    ./jboss-cli.sh --file=jbosslogremove.cli

     

     

     

     

     

    Optional Notes:   To execute these steps on the secure Identity Suite Virtual Appliance, it is necessary to create a local JBOSS/Wildfly account.   Fortunately, this batch "add-user.sh" is part of the sudoer list for the "config" userID.

     

    Ref: Administrating CA Identity Suite Virtual Appliance (vApp) - CA Identity Suite - 14.0 - CA Technologies Documentation 

     

     

     

    Step A:   Execute the following command:

    sudo /opt/CA/wildfly-idm/bin/add-user.sh

     

    Add an "administrator" user, with a good password.

     

    Step B:  Execute the IDM Wildfly command, jboss-cli.sh, with a redirect to the batch file

    /opt/CA/wildfly-idm/bin/jboss-cli.sh  --file=/tmp/jbosslog.cli

     

    This batch information will now be added to JBOSS_HOME/standalone/configuration/ca-standalone-full-ha.xml & be available immediately for use (and upon reboot of the system or restart of Wildfly).

     

     

     

    Let me know if you find this of value for your own debugging or reverse-engineering documentation purposes.

     

     

    Cheers,

     

    A.

     

     

    Attaching a PDF note that explains the value of the loggers that I have selected.

     

     

    Edit:  9/10/2018     Two (2) items:

     

     1)  Update logger/appender's formatter pattern to have IP address/NodeName within the log file, so when consolidation is utilized, the resource will be able to identify where the cluster logs originated from.

    2)   Added space for the %s%E pattern &  %l to assist with RCA .

     

    im_business_logic.cli.txt

     

    connect
    batch
    /subsystem=logging/periodic-rotating-file-handler=im_business_logic:add(file={"path"=>"im_business_logic_debug.log", "relative-to"=>"jboss.server.log.dir"}, suffix=".yyyy.MM.dd", level=DEBUG, append=true, autoflush=false, suffix=".yyyy-MM-dd-HH", formatter="%d{HH:mm:ss,SSS} [${jboss.bind.address}] [${jboss.node.name}] %-5p [%c] (%t) %-120s%E {%l}%n")
    /subsystem=logging/logger=ims.tasktrack.custom:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)
    /subsystem=logging/logger=im.feeder:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)
    /subsystem=logging/logger=ims.policyxpress:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)
    /subsystem=logging/logger=ims.llsdk.role.azengine:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)
    /subsystem=logging/logger=ims.llsdk.role.azcache.ridiculouslydetailed:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)
    /subsystem=logging/logger=ims.tmt.submit.validation.blth:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)
    /subsystem=logging/logger=ims.tmt.submit.execute.blth:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)
    run-batch

     

     

     

    im_business_logic_remove.cli.txt

     

    connect
    batch
    /subsystem=logging/periodic-rotating-file-handler=im_business_logic:remove
    /subsystem=logging/logger=ims.tasktrack.custom:remove
    /subsystem=logging/logger=im.feeder:remove
    /subsystem=logging/logger=ims.policyxpress:remove
    /subsystem=logging/logger=ims.llsdk.role.azengine:remove
    /subsystem=logging/logger=ims.llsdk.role.azcache.ridiculouslydetailed:remove
    /subsystem=logging/logger=ims.tmt.submit.validation.blth:remove
    /subsystem=logging/logger=ims.tmt.submit.execute.blth:remove
    run-batch

     

     

    im_full_business_logic.cli.txt

     

    connect
    batch
    /subsystem=logging/periodic-rotating-file-handler=im_all_business_logic:add(file={"path"=>"im_all_business_logic_debug.log", "relative-to"=>"jboss.server.log.dir"}, suffix=".yyyy.MM.dd", level=DEBUG, append=true, autoflush=false, suffix=".yyyy-MM-dd-HH", formatter="%d{HH:mm:ss,SSS} [${jboss.bind.address}] [${jboss.node.name}] %-5p [%c] (%t) %-120s%E {%l}%n")
    /subsystem=logging/logger=im:add(handlers=["im_all_business_logic"], level=DEBUG, use-parent-handlers=false)
    /subsystem=logging/logger=ims:add(handlers=["im_all_business_logic"], level=DEBUG, use-parent-handlers=false)
    run-batch

     

     

    im_full_business_logic_remove.cli.txt

     

    connect
    batch
    /subsystem=logging/periodic-rotating-file-handler=im_all_business_logic:remove
    /subsystem=logging/logger=im:remove
    /subsystem=logging/logger=ims:remove
    run-batch

     

     

     

    Log4J Formatter Syntax Reference:    

    https://access.redhat.com/documentation/en-us/jboss_enterprise_application_platform/6/html/administration_and_configuration_guide/chap-the_logging_subsystem#Log_Formatter_Syntax1

     

     

     

     

    Output example:

     

    15:51:07,804 [192.168.242.146] [iamnode1]  WARN  [im.plugins.IMEnvironmentInitializerPlugin] (MSC service thread 1-1) http://192.168.242.146:8080/iam/im/ETACALLBACK/?env=identityEnv is already registered as a inbound URI on the Provisioning Server. Just updating the shared secret. {com.ca.identitymanager.plugins.IMEnvironmentInitializerPlugin.startEnvironment(IMEnvironmentInitializerPlugin.java:209)}

    15:51:09,901 [192.168.242.146] [iamnode1]  INFO  [ims.Main] (MSC service thread 1-1) * Started environment: identityEnv                                                                                       {com.netegrity.llsdk6.imsimpl.utility.log.Log4JLogImpl.logInfo(Log4JLogImpl.java:66)}

    15:51:10,088 [192.168.242.146] [iamnode1]  INFO  [ims.Main] (MSC service thread 1-1) ** FIPS mode enabled : false                                                                                             {com.netegrity.llsdk6.imsimpl.utility.log.Log4JLogImpl.logInfo(Log4JLogImpl.java:66)}

    15:51:10,338 [192.168.242.146] [iamnode1]  INFO  [ims.Main] (MSC service thread 1-1) * Startup Step 30 : Attempting to start ApplicationContextInitializer plug-ins                                           {com.netegrity.llsdk6.imsimpl.utility.log.Log4JLogImpl.logInfo(Log4JLogImpl.java:66)}

    15:51:10,388 [192.168.242.146] [iamnode1]  INFO  [ims.Main] (MSC service thread 1-1) ---- CA IAM FW Startup Sequence Complete. ----                                                                           {com.netegrity.llsdk6.imsimpl.utility.log.Log4JLogImpl.logInfo(Log4JLogImpl.java:66)}



  • 2.  Re: Update JBOSS/Wildfly Log4j without restart for Identity Suite (and vApp)

    Posted Oct 13, 2017 08:48 PM

    Alan,

     

    Great post on setting up a new logger to isolate the classes included in the "business logic" logs. This process can be applied to additional classes as needed, as in authentication events forwarded to Splunk. Hopefully this will trigger more submissions from the Community.

     

    Enrique



  • 3.  Re: Update JBOSS/Wildfly Log4j without restart for Identity Suite (and vApp)

    Posted Jan 10, 2018 01:11 PM

    Thanks for sharing Alan. The process to isolate log entries in a different file than the main server.log is very helpful for troubleshooting issues.



  • 4.  Re: Update JBOSS/Wildfly Log4j without restart for Identity Suite (and vApp)

    Broadcom Employee
    Posted Jan 10, 2018 01:20 PM

    Thanks, this is very helpful. 

    SumeetM



  • 5.  Re: Update JBOSS/Wildfly Log4j without restart for Identity Suite (and vApp)

    Posted Jul 12, 2018 08:12 PM

    This is great info Alan.



  • 6.  Re: Smart Debugging: Update JBOSS/Wildfly Log4j without restart for Identity Suite (and vApp)

    Broadcom Employee
    Posted Sep 12, 2018 03:19 AM

    Great stuff Alan, thanks a lot!