Layer7 API Management

  • 1.  ssg process restarting frequently

    Posted Aug 04, 2017 12:25 AM

    Hi All,

    In few nodes the ssg process getting restarted suddenly and it's happening frequently, before and after ssg process getting restart the node load is less and https connections ( ESTABLISHED,CLOSE_WAIT,TIME_WAIT) are less than 100 and memory and disk utilization also less but in logs we can able to see IO errors ( please find below detailed error), is it causing ssg restart ? or what might be the valid reasons behind ssg process restart.?

     

    error  logs:

    ------------------

    2017-08-03T21:47:35.725-0400 WARNING 1585 com.l7tech.server.policy.variable.ServerVariables: Unsupported variable: size (Stream timeout)
    com.l7tech.server.transport.http.TimeoutInputStream$TimeoutIOException: Stream timeout
    at com.l7tech.server.transport.http.TimeoutInputStream.d(Unknown Source)
    at com.l7tech.server.transport.http.TimeoutInputStream.exitBlocking(Unknown Source)
    at com.l7tech.server.transport.http.TimeoutInputStream.read(Unknown Source)
    at java.io.FilterInputStream.read(FilterInputStream.java:133)
    at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
    at com.l7tech.common.io.ByteLimitInputStream.read(Unknown Source)
    at java.io.FilterInputStream.read(FilterInputStream.java:107)
    at com.l7tech.common.mime.HybridStashManager.stash(Unknown Source)
    at com.l7tech.common.mime.h.a(Unknown Source)
    at com.l7tech.common.mime.h.stashAndRecall(Unknown Source)
    at com.l7tech.common.mime.i.getInputStream(Unknown Source)
    at com.l7tech.common.mime.MimeBody.readAndStashEntireMessage(Unknown Source)
    at com.l7tech.common.mime.MimeBody.getEntireMessageBodyLength(Unknown Source)
    at com.l7tech.message.j.getContentLength(Unknown Source)
    at com.l7tech.server.policy.variable.bg.select(Unknown Source)
    at com.l7tech.server.policy.variable.MessageSelector.select(Unknown Source)
    at com.l7tech.server.policy.variable.MessageSelector.select(Unknown Source)
    at com.l7tech.server.policy.variable.ExpandVariables.a(Unknown Source)
    at com.l7tech.server.policy.variable.ExpandVariables.a(Unknown Source)
    at com.l7tech.server.policy.variable.ExpandVariables.processSingleVariableAsObject(Unknown Source)
    at com.l7tech.server.policy.variable.ExpandVariables.processSingleVariableAsObject(Unknown Source)
    at com.l7tech.server.policy.variable.we.get(Unknown Source)
    at com.l7tech.server.policy.variable.og.a(Unknown Source)
    at com.l7tech.server.policy.variable.ServerVariables.get(Unknown Source)
    at com.l7tech.server.message.b.getBuiltinVariable(Unknown Source)
    at com.l7tech.server.message.b.a(Unknown Source)
    at com.l7tech.server.message.b.getVariableMap(Unknown Source)
    at com.l7tech.external.assertions.comparison.server.ServerComparisonAssertion.checkRequest(Unknown Source)
    at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)
    at com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source)
    at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)
    at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source)
    at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)
    at com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source)
    at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)
    at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source)
    at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)
    at com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source)
    at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)
    at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source)
    at com.l7tech.server.policy.ServerPolicy.checkRequest(Unknown Source)
    at com.l7tech.server.policy.z.call(Unknown Source)
    at com.l7tech.server.policy.z.call(Unknown Source)
    at com.l7tech.common.log.HybridDiagnosticContext.doInContext(Unknown Source)
    at com.l7tech.server.policy.ServerPolicyHandle.checkRequest(Unknown Source)
    at com.l7tech.server.mb.b(Unknown Source)
    at com.l7tech.server.mb.a(Unknown Source)
    at com.l7tech.server.mb.access$700(Unknown Source)
    at com.l7tech.server.MessageProcessor.a(Unknown Source)
    at com.l7tech.server.MessageProcessor.access$200(Unknown Source)
    at com.l7tech.server.kb.call(Unknown Source)
    at com.l7tech.server.kb.call(Unknown Source)
    at com.l7tech.server.audit.AuditContextFactory.doWithNewAuditContext(Unknown Source)
    at com.l7tech.server.MessageProcessor.processMessage(Unknown Source)
    at com.l7tech.server.SoapMessageProcessingServlet.service(Unknown Source)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:342)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
    at com.l7tech.server.transport.http.HttpNamespaceFilter.doFilter(Unknown Source)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.l7tech.server.WsdlFilter.doFilter(Unknown Source)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.l7tech.server.transport.http.ConnectionIdFilter.doFilter(Unknown Source)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.l7tech.server.transport.http.InputTimeoutFilter.doFilter(Unknown Source)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.l7tech.server.log.HybridDiagnosticContextServletFilter.doFilter(Unknown Source)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
    at com.l7tech.server.tomcat.ResponseKillerValve.invoke(Unknown Source)
    at com.l7tech.server.tomcat.ConnectionIdValve.invoke(Unknown Source)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:295)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:396)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
    2017-08-03T21:47:35.726-0400 WARNING 1585 com.l7tech.server.policy.variable.ServerVariables: Unsupported variable: size on com.l7tech.message.Message
    2017-08-03T21:47:35.726-0400 WARNING 1585 com.l7tech.server.policy.variable.ServerVariables: Unsupported variable: request.size
    2017-08-03T21:47:35.727-0400 WARNING 1585 com.l7tech.server.SoapMessageProcessingServlet: I/O error while processing message: Stream timeout
    2017-08-03T21:47:35.821-0400 WARNING 1702 com.l7tech.message.JsonFacet: Did not need to create json data
    2017-08-03T21:47:35.821-0400 WARNING 1702 com.l7tech.message.JsonFacet: Did not need to create json data
    2017-08-03T21:47:35.822-0400 WARNING 1702 com.l7tech.message.JsonFacet: Did not need to create json data
    2017-08-03T21:47:35.822-0400 WARNING 1702 com.l7tech.message.JsonFacet: Did not need to create json data
    2017-08-03T21:47:35.824-0400 WARNING 1702 com.l7tech.server.policy.variable.ServerVariables: Unsupported variable: now for date time. Illegal pattern character 'n'
    2017-08-03T21:47:35.824-0400 WARNING 1702 com.l7tech.server.policy.variable.ServerVariables: Unsupported variable: gateway.time.now
    2017-08-03T21:47:35.824-0400 WARNING 1702 com.l7tech.server.policy.variable.ServerVariables: Unsupported variable: now for date time. Illegal pattern character 'n'
    2017-08-03T21:47:35.824-0400 WARNING 1702 com.l7tech.server.policy.variable.ServerVariables: Unsupported variable: gateway.time.now

     

    gateway version is 8.1.0

     

    Thanks in advance for your inputs and help.

     

    --Uday



  • 2.  Re: ssg process restarting frequently

    Posted Aug 04, 2017 05:04 AM

    Hi Uday,

     

    What form factor of gateway is this? Virtual, harware, software?

    We will need the full ssg log

     

    I suggest a ca support case be raised with the full log as it will need review.

     

    Regards

    Seenu Mathew



  • 3.  Re: ssg process restarting frequently

    Posted Aug 04, 2017 05:10 AM

    it's Virtual



  • 4.  Re: ssg process restarting frequently
    Best Answer

    Broadcom Employee
    Posted Aug 04, 2017 12:49 PM

    Uday,

    This indeed is best for a support case.

    And it seems you covered many of the typical scenarios but there may be something lacking. 

    So the stacks you outline are really just timeouts to a back-end server and would be considered somewhat normal and wouldnt in itself cause a gateway restart.

     

    Usually if the Gateway is restarting its because the Parent Process Process Controler (sspc_***.log) has tried to make a socket or connection to the gateway and it has not gotten response in certain time frame. And if so it will force a restart. So the first step is ensuring it is sspc (in its log) that has a killing default starting default. And is restarting the gateway. Similar to what you might do by just hitting a ssg/ping page.

     

    The next portion is determining the why. Usually the gateway will go unresponsive if it runs out of threads (to many connections) the netstat you mention. Or runs out of memory (as you mention). Now by default the gateway only takes up to 1/2 or 2/3 of available memory. So you dont need to see the system at 100 percent memory for the JVM not to be able to allocate more memory. So how much memory is available on this vm is very important. Hopefully its much more then the 2Gb and depending on load it would definitely need to be more than 2gb if its production and getting a lot of traffic. A lot of times which may be hard to do on a unexpected restart its best to collect the following tech doc just prior to restart for your support case

    Troubleshooting Gateway application failures, performance concerns, and service outages 

     

    As well gather a tar of the gateway/process controller logs. 
    If you havent opened an issue yet please do and if I can provide any further information please let me know.

    Thanks....