Primary Node going down

[Shabaz Shaikh]

Hello EveryOne,

We have two nodes in an gateway cluster.And we have observed that the primary gateway goes down with clients getting timeout response.
We have disabled the logs(as its production environment),still able to get the below audits.

Before the gateway gone down, we did found below observations.
1.Whenever Client has requested for a token(invoking the auth/oauth/v2/token API).

In audits we found below logs
com.l7tech.server.transport.http.TimeoutInputStream$TimeoutIOException: Stream timeoutcom.l7tech.server.transport.http.TimeoutInputStream$TimeoutIOException: Stream timeout at com.l7tech.server.transport.http.TimeoutInputStream.d(Unknown Source) at com.l7tech.server.transport.http.TimeoutInputStream.exitBlocking(Unknown Source) at com.l7tech.server.transport.http.TimeoutInputStream.read(Unknown Source) at java.io.FilterInputStream.read(FilterInputStream.java:133) at java.io.PushbackInputStream.read(PushbackInputStream.java:186) at com.l7tech.common.io.ByteLimitInputStream.read(Unknown Source) at java.io.FilterInputStream.read(FilterInputStream.java:107) at com.l7tech.common.mime.HybridStashManager.stash(Unknown Source) at com.l7tech.common.mime.i.a(Unknown Source) at com.l7tech.common.mime.i.stashAndRecall(Unknown Source) at com.l7tech.common.mime.k.getInputStream(Unknown Source) at com.l7tech.common.mime.MimeBody.readAndStashEntireMessage(Unknown Source) at com.l7tech.common.mime.MimeBody.getEntireMessageBodyLength(Unknown Source) at com.l7tech.message.j.getContentLength(Unknown Source) at com.l7tech.server.policy.assertion.ServerRequestSizeLimit.doCheckRequest(Unknown Source) at com.l7tech.server.policy.assertion.AbstractMessageTargetableServerAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.ServerPolicy.checkRequest(Unknown Source) at com.l7tech.server.policy.w.call(Unknown Source) at com.l7tech.server.policy.w.call(Unknown Source) at com.l7tech.common.log.HybridDiagnosticContext.doInContext(Unknown Source) at com.l7tech.server.policy.ServerPolicyHandle.checkRequest(Unknown Source) at com.l7tech.server.ap.a(Unknown Source) at com.l7tech.server.ap.processPreServicePolicies(Unknown Source) at com.l7tech.server.ap.h(Unknown Source) at com.l7tech.server.ap.a(Unknown Source) at com.l7tech.server.ap.access$700(Unknown Source) at com.l7tech.server.MessageProcessor.a(Unknown Source) at com.l7tech.server.MessageProcessor.processMessageNoAudit(Unknown Source) at com.l7tech.server.SoapMessageProcessingServlet.serviceNoAudit(Unknown Source) at com.l7tech.server.SoapMessageProcessingServlet.access$000(Unknown Source) at com.l7tech.server.a2.call(Unknown Source) at com.l7tech.server.audit.AuditContextFactory.doWithNewAuditContext(Unknown Source) at com.l7tech.server.SoapMessageProcessingServlet.service(Unknown Source) at javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:342) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302) at com.l7tech.server.transport.http.HttpNamespaceFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.l7tech.server.WsdlFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.l7tech.server.transport.http.ConnectionIdFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.l7tech.server.transport.http.InputTimeoutFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.l7tech.server.log.HybridDiagnosticContextServletFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:234) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:181) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at com.l7tech.server.tomcat.ResponseKillerValve.invoke(Unknown Source) at com.l7tech.server.tomcat.ConnectionIdValve.invoke(Unknown Source) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:295) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:610) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:410) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

2.Also when Client is invoking the Token API.We found below audits

"Perform JDBC Query" assertion failed due to: Could not get JDBC Connection: Could not get JDBC Connection; nested exception is java.sql.SQLException: Connections could not be acquired from the underlying database! A ResourcePool could not acquire a resource from its primary factory or source.

Please suggest on below if anyone as faced the same issues.
Any links,guides would be appreciated .

Thanks in Advance...!!!
Shabaz

[Shabaz Shaikh]

Hi EveryOne,

We are using Gateway 9.2 and Ouath Toolkit 3.6.

Thanks,
Shabaz.

[Joe Dascole]

Hi Shabaz,

What type of database are you using for OTK? 
Are you able to successfully test the connection to the database in policy manager (via the JDBC connection parameters)?

Regards,
Joe

[Vince Baker]

Hiya,

What do your SSPC logs say? I guess that the Process Controller is restarting the gateway process due to lack of probe liveness response.

This timeout can be extended from the default if needed, but there is obviously an underlying problem which needs fixing.

Regards

Vince

------------------------------
Senior Architect
Apiida AG
Https://www.apiida.com
------------------------------

[Daren Chernoff]

Hi Shabaz,

Were you able to respond to prior questions inquiring as to the type of database that OTK is running on?
Do you notice any kind of performance problems with the gateway processing requests prior to the gateway node going down?  If so, you may want to run our DCT tool to gather thread dumps and config files etc then kick off an issue with our support team to take a further look into things.

Daren

[Shabaz Shaikh]

HI Darren,

We are using MYSQL database for OTK.
We have raised the same concern with the support team and they suggested to reboot the primary server once.

Thanks,
Shabaz.

[Zhijun He]

"A ResourcePool could not acquire a resource from its primary factory or source" is more likely a jdbc driver problem.
Was it working before? any change had been done before the problem occurred?
Do you use the default mysql driver for the otk jdbc connection?
Do you test the OTK jdbc connection on "Manage jdbc connections" window?

Regards,
Mark