Symantec IGA

Expand all | Collapse all

IDM Upgrade from 14.3 CP2 to 14.4

  • 1.  IDM Upgrade from 14.3 CP2 to 14.4

    Posted 12 days ago
    Hi!

    We recently upgraded our stage environment from 14.3 CP2 to IDM 14.4 (along with upgrade of Wildfly to 15.x, and migration to AdoptOpenJDK).

    Since the upgrade, we got an issue, where the IDM tasks would get stuck in "in-progress" state randomly. Out of 10 submitted tasks on 4-6 would succeed, and rest would fail (the type of task did not matter). our of 2 exactly same tasks, 1 would succeed, and another would fail.

    Opened a Broadcom ticket, and engineering team advised some performance tuning options, since the applications of changes from engineering we have been getting the SQLExceptions.

    So far Broadcom has been unable to help on this issue, any ideas from community would help.

    2021-10-03 00:04:10,038 ERROR [ims.tmt.persistence] (default task-1) PersistenceProvider: default task-1: getEvent: No Event found for id=5bac4341-0ae6156f-3636d2b6-dd180cb3 java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/iam/im/jdbc/jdbc/idm
    at com.netegrity.idmutils.DBConnectionUtil.getConnection(DBConnectionUtil.java:51)
    at com.netegrity.ims.persistence.PersistenceProvider.getConnection(PersistenceProvider.java:4634)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:417)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.persistence.PersistenceProvider.getEvent(PersistenceProvider.java:423)
    at com.netegrity.ims.etacallback.ETACallbackEventBase.submitTask(ETACallbackEventBase.java:317)
    at com.netegrity.ims.etacallback.ETACallbackEventBase.submitTask(ETACallbackEventBase.java:223)
    at com.netegrity.ims.etacallback.ETACallbackEventBase.submitTask(ETACallbackEventBase.java:213)
    at com.netegrity.ims.etacallback.ProvisioningActivityNotificationEvent.processNonExistingCorporateManagedObjectCommand(ProvisioningActivityNotificationEvent.java:70)
    at com.netegrity.ims.etacallback.ProvisioningActivityNotificationEvent.processImpl(ProvisioningActivityNotificationEvent.java:43)
    at com.netegrity.ims.etacallback.ETACallbackEventBase.process(ETACallbackEventBase.java:137)
    at com.netegrity.ims.etacallback.ETACallbackServiceServlet.doPost(ETACallbackServiceServlet.java:180)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
    at com.netegrity.webapp.filter.ConsolePageFilter.doFilter(ConsolePageFilter.java:226)
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at com.netegrity.webapp.authentication.FrameworkLoginFilter.doFilter(FrameworkLoginFilter.java:242)
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at com.netegrity.webapp.filter.LocaleFilter.doFilter(LocaleFilter.java:101)
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at com.netegrity.webapp.filter.ClientExtractFilter.doFilter(ClientExtractFilter.java:52)
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at io.opentracing.contrib.jaxrs2.server.SpanFinishingFilter.doFilter(SpanFinishingFilter.java:55)
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
    at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
    at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
    at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
    at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
    at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
    at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
    at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
    at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
    at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
    at java.lang.Thread.run(Thread.java:748)

    Thanks
    -G


  • 2.  RE: IDM Upgrade from 14.3 CP2 to 14.4

    Posted 11 days ago
    We also upgraded to 14.4 from 14.3 CP2 and have run into tasks getting stuck In Progress, meaning an IM restart was required.

    We too have an open support case about this, and have received Performance Tuning tips but these have not helped.

    We are running the Virtual Appliance.  This has been seen in our AT and Prod environments (which are HA) but not in our ST enviornment (which is a single leg system).  So we suspect clustering to play a part, but nothing has been confirmed from Support yet.  We are currently running our AT env as a single node solution to see if we see the error occur there or not (running as a HA setup it regularly appeared).

    The errors we are seeing in the wildfly logs are timeout messages relating to ActiveMQ.

    As yet no solution has been provided - we just check the logs for In Progress tasks, and when we see then from our nightly scheduled jobs we have to restart IM for the system to function properly.

    "javax.jms.JMSException: AMQ119014: Timed out after waiting 600,000 ms for response when sending packet 71" is a typical error.

    Support provided the following link (we do not have the relevant Red Hat subscription): "https://access.redhat.com/solutions/6190581 but u need a subscription to view the information."

    They are currently liaising with SE about the exception.

    Is anybody else having In Progress issues since going from 14.3 CP2 to 14.4?

    Adrian


  • 3.  RE: IDM Upgrade from 14.3 CP2 to 14.4

    Posted 11 days ago
    Hi Adrian,

    We are facing this issue in the non-clustered environment. We are still waiting to hear from engineering.

    Did you get a recommendation from engineering to change the DB for snapshot isolation mode? if yes, have you made that change?

    ALTER DATABASE IMStore SET allow_snapshot_isolation ON
    ALTER DATABASE IMStore SET read_committed_snapshot ON

    Regards
    Garima


  • 4.  RE: IDM Upgrade from 14.3 CP2 to 14.4

    Posted 2 days ago
    Hi Adrian,


    We are facing same issue too. Have you received any update on your support case?

    thank you

    kamala



  • 5.  RE: IDM Upgrade from 14.3 CP2 to 14.4

    Posted 11 days ago
    No, we haven't been asked to make that change.


  • 6.  RE: IDM Upgrade from 14.3 CP2 to 14.4

    Posted 2 days ago
    No, we haven't heard from Support for a while concerning this issue unfortunately.