Layer 7 API Management

Expand all | Collapse all

Primary Node going down

Jump to Best Answer
  • 1.  Primary Node going down

    Posted 11-19-2019 03:25 AM
    Hello EveryOne,

    We have two nodes in an gateway cluster.And we have observed that the primary gateway goes down with clients getting timeout response.
    We have disabled the logs(as its production environment),still able to get the below audits.

    Before the gateway gone down, we did found below observations.
    1.Whenever Client has requested for a token(invoking the auth/oauth/v2/token API).

    In audits we found below logs
    com.l7tech.server.transport.http.TimeoutInputStream$TimeoutIOException: Stream timeoutcom.l7tech.server.transport.http.TimeoutInputStream$TimeoutIOException: Stream timeout at com.l7tech.server.transport.http.TimeoutInputStream.d(Unknown Source) at com.l7tech.server.transport.http.TimeoutInputStream.exitBlocking(Unknown Source) at com.l7tech.server.transport.http.TimeoutInputStream.read(Unknown Source) at java.io.FilterInputStream.read(FilterInputStream.java:133) at java.io.PushbackInputStream.read(PushbackInputStream.java:186) at com.l7tech.common.io.ByteLimitInputStream.read(Unknown Source) at java.io.FilterInputStream.read(FilterInputStream.java:107) at com.l7tech.common.mime.HybridStashManager.stash(Unknown Source) at com.l7tech.common.mime.i.a(Unknown Source) at com.l7tech.common.mime.i.stashAndRecall(Unknown Source) at com.l7tech.common.mime.k.getInputStream(Unknown Source) at com.l7tech.common.mime.MimeBody.readAndStashEntireMessage(Unknown Source) at com.l7tech.common.mime.MimeBody.getEntireMessageBodyLength(Unknown Source) at com.l7tech.message.j.getContentLength(Unknown Source) at com.l7tech.server.policy.assertion.ServerRequestSizeLimit.doCheckRequest(Unknown Source) at com.l7tech.server.policy.assertion.AbstractMessageTargetableServerAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source) at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source) at com.l7tech.server.policy.ServerPolicy.checkRequest(Unknown Source) at com.l7tech.server.policy.w.call(Unknown Source) at com.l7tech.server.policy.w.call(Unknown Source) at com.l7tech.common.log.HybridDiagnosticContext.doInContext(Unknown Source) at com.l7tech.server.policy.ServerPolicyHandle.checkRequest(Unknown Source) at com.l7tech.server.ap.a(Unknown Source) at com.l7tech.server.ap.processPreServicePolicies(Unknown Source) at com.l7tech.server.ap.h(Unknown Source) at com.l7tech.server.ap.a(Unknown Source) at com.l7tech.server.ap.access$700(Unknown Source) at com.l7tech.server.MessageProcessor.a(Unknown Source) at com.l7tech.server.MessageProcessor.processMessageNoAudit(Unknown Source) at com.l7tech.server.SoapMessageProcessingServlet.serviceNoAudit(Unknown Source) at com.l7tech.server.SoapMessageProcessingServlet.access$000(Unknown Source) at com.l7tech.server.a2.call(Unknown Source) at com.l7tech.server.audit.AuditContextFactory.doWithNewAuditContext(Unknown Source) at com.l7tech.server.SoapMessageProcessingServlet.service(Unknown Source) at javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:342) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302) at com.l7tech.server.transport.http.HttpNamespaceFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.l7tech.server.WsdlFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.l7tech.server.transport.http.ConnectionIdFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.l7tech.server.transport.http.InputTimeoutFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.l7tech.server.log.HybridDiagnosticContextServletFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:234) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:181) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at com.l7tech.server.tomcat.ResponseKillerValve.invoke(Unknown Source) at com.l7tech.server.tomcat.ConnectionIdValve.invoke(Unknown Source) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:295) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:610) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:410) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

    2.Also when Client is invoking the Token API.We found below audits

    "Perform JDBC Query" assertion failed due to: Could not get JDBC Connection: Could not get JDBC Connection; nested exception is java.sql.SQLException: Connections could not be acquired from the underlying database! A ResourcePool could not acquire a resource from its primary factory or source.

    Please suggest on below if anyone as faced the same issues.
    Any links,guides would be appreciated .

    Thanks in Advance...!!!
    Shabaz


  • 2.  RE: Primary Node going down

    Posted 11-19-2019 03:25 AM
    Hi EveryOne,

    We are using Gateway 9.2 and Ouath Toolkit 3.6.

    Thanks,
    Shabaz.


  • 3.  RE: Primary Node going down

    Posted 11-26-2019 12:11 PM
    Hi Shabaz,

    What type of database are you using for OTK?
    Are you able to successfully test the connection to the database in policy manager (via the JDBC connection parameters)?

    Regards,
    Joe


  • 4.  RE: Primary Node going down

    Posted 11-27-2019 02:45 AM

    Hiya,

    What do your SSPC logs say? I guess that the Process Controller is restarting the gateway process due to lack of probe liveness response.

    This timeout can be extended from the default if needed, but there is obviously an underlying problem which needs fixing.

    Regards

    Vince



    ------------------------------
    Senior Architect
    Apiida AG
    Https://www.apiida.com
    ------------------------------



  • 5.  RE: Primary Node going down
    Best Answer

    Posted 11-27-2019 02:08 PM
    Hi Shabaz,

    Were you able to respond to prior questions inquiring as to the type of database that OTK is running on?
    Do you notice any kind of performance problems with the gateway processing requests prior to the gateway node going down?  If so, you may want to run our DCT tool to gather thread dumps and config files etc then kick off an issue with our support team to take a further look into things.

    Daren


  • 6.  RE: Primary Node going down

    Posted 12-09-2019 01:25 AM
    HI Darren,

    We are using MYSQL database for OTK.
    We have raised the same concern with the support team and they suggested to reboot the primary server once.

    Thanks,
    Shabaz.


  • 7.  RE: Primary Node going down

    Posted 12-09-2019 04:35 PM
    "A ResourcePool could not acquire a resource from its primary factory or source" is more likely a jdbc driver problem.
    Was it working before? any change had been done before the problem occurred?
    Do you use the default mysql driver for the otk jdbc connection?
    Do you test the OTK jdbc connection on "Manage jdbc connections" window?

    Regards,
    Mark