Hello EveryOne,
We have two nodes in an gateway cluster.And we have observed that the primary gateway goes down with clients getting timeout response.
We have disabled the logs(as its production environment),still able to get the below audits.
Before the gateway gone down, we did found below observations.
1.Whenever Client has requested for a token(invoking the auth/oauth/v2/token API).
In audits we found below logs
com.l7tech.server.transport.http.TimeoutInputStream$TimeoutIOException: Stream timeoutcom.l7tech.server.transport.http.TimeoutInputStream$TimeoutIOException: Stream timeout at com.l7tech.server.transport.http.TimeoutInputStream.d(Unknown Source) at com.l7tech.server.transport.http.TimeoutInputStream.exitBlocking(Unknown Source) at com.l7tech.server.transport.http.TimeoutInputStream.read(Unknown Source) at java.io.FilterInputStream.read(FilterInputStream.java:133) at java.io.PushbackInputStream.read(PushbackInputStream.java:186) at com.l7tech.common.io.ByteLimitInputStream.read(Unknown Source) at java.io.FilterInputStream.read(FilterInputStream.java:107) at com.l7tech.common.mime.HybridStashManager.stash(Unknown Source) at com.l7tech.common.mime.i.a(Unknown Source) at com.l7tech.common.mime.i.stashAndRecall(Unknown Source) at com.l7tech.common.mime.k.getInputStream(Unknown Source) at com.l7tech.common.mime.MimeBody.readAndStashEntireMessage(Unknown Source) at com.l7tech.common.mime.MimeBody.getEntireMessageBodyLength(Unknown Source) at com.l7tech.message.j.getContentLength(Unknown Source) at com.l7tech.server.policy.assertion.ServerRequestSizeLimit.doCheckRequest(Unknown Source) at com.l7tech.server.policy.assertion.AbstractMessageTargetableServerAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.ServerPolicy.checkRequest(Unknown Source) at com.l7tech.server.policy.w.call(Unknown Source) at com.l7tech.server.policy.w.call(Unknown Source) at com.l7tech.common.log.HybridDiagnosticContext.doInContext(Unknown Source) at com.l7tech.server.policy.ServerPolicyHandle.checkRequest(Unknown Source) at com.l7tech.server.ap.a(Unknown Source) at com.l7tech.server.ap.processPreServicePolicies(Unknown Source) at com.l7tech.server.ap.h(Unknown Source) at com.l7tech.server.ap.a(Unknown Source) at com.l7tech.server.ap.access$700(Unknown Source) at com.l7tech.server.MessageProcessor.a(Unknown Source) at com.l7tech.server.MessageProcessor.processMessageNoAudit(Unknown Source) at com.l7tech.server.SoapMessageProcessingServlet.serviceNoAudit(Unknown Source) at com.l7tech.server.SoapMessageProcessingServlet.access$000(Unknown Source) at com.l7tech.server.a2.call(Unknown Source) at com.l7tech.server.audit.AuditContextFactory.doWithNewAuditContext(Unknown Source) at com.l7tech.server.SoapMessageProcessingServlet.service(Unknown Source) at javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:342) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302) at com.l7tech.server.transport.http.HttpNamespaceFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.l7tech.server.WsdlFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.l7tech.server.transport.http.ConnectionIdFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.l7tech.server.transport.http.InputTimeoutFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.l7tech.server.log.HybridDiagnosticContextServletFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:234) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:181) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at com.l7tech.server.tomcat.ResponseKillerValve.invoke(Unknown Source) at com.l7tech.server.tomcat.ConnectionIdValve.invoke(Unknown Source) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:295) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:610) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:410) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
2.Also when Client is invoking the Token API.We found below audits
"Perform JDBC Query" assertion failed due to: Could not get JDBC Connection: Could not get JDBC Connection; nested exception is java.sql.SQLException: Connections could not be acquired from the underlying database! A ResourcePool could not acquire a resource from its primary factory or source.
Please suggest on below if anyone as faced the same issues.
Any links,guides would be appreciated .
Thanks in Advance...!!!
Shabaz