Re: java.lang.ArrayIndexOutOfBoundsException error

[Prem]

Hello All,

I'm testing a Gateway service from POSTMAN and when i'm making a first call to the endpoint I'm seeing some java error as below in ssg logs, buy when I trigger same request again seeing the excepted response. Could any one please let me what is the reason for this behaviour?

2019-10-16T11:17:33.296-0500 SEVERE 592 org.apache.catalina.core.ContainerBase.[ssg].[/].[SoapMessageProcessingServlet]: Servlet.service() for servlet SoapMessageProcessingServlet threw exception
java.lang.ArrayIndexOutOfBoundsException
at org.apache.coyote.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:706)
at org.apache.coyote.http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:443)
at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:1641)
at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:956)
at org.apache.coyote.Response.action(Response.java:190)
at org.apache.coyote.Response.sendHeaders(Response.java:393)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:318)
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:278)
at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104)
at com.l7tech.server.SoapMessageProcessingServlet.sendExceptionFault(Unknown Source)
at com.l7tech.server.SoapMessageProcessingServlet.sendExceptionFault(Unknown Source)
at com.l7tech.server.SoapMessageProcessingServlet.serviceNoAudit(Unknown Source)
at com.l7tech.server.SoapMessageProcessingServlet.access$000(Unknown Source)
at com.l7tech.server.a1.call(Unknown Source)
at com.l7tech.server.audit.AuditContextFactory.doWithNewAuditContext(Unknown Source)
at com.l7tech.server.SoapMessageProcessingServlet.service(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:342)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
at com.l7tech.server.transport.http.HttpNamespaceFilter.doFilter(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.l7tech.server.WsdlFilter.doFilter(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.l7tech.server.transport.http.ConnectionIdFilter.doFilter(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.l7tech.server.transport.http.InputTimeoutFilter.doFilter(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.l7tech.server.log.HybridDiagnosticContextServletFilter.doFilter(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:234)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:181)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at com.l7tech.server.tomcat.ResponseKillerValve.invoke(Unknown Source)
at com.l7tech.server.tomcat.ConnectionIdValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:295)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:610)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:410)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


Regards,
Prem.

[Joe Dascole]

Prem,

You may want to open a support case if you are still seeing this issue. Ideally, we would like to see the complete logs along with your policy and request body.

Regards,
Joe

[Shashi Ranjan]

Hi Prem, have you found any solution for this?
We are also noticing the same behavior for few policies.

[Charles Lilienkamp]

What gateway version is this for? We had some problems similar to this in some of our older versions. But this was corrected.

[Shashi Ranjan]

Hi Charles, 

We have Gateway v9.2.00 & Java Runtime Environment (build 1.8.0_181-b13). Do you know if this is resolved in v9.2 or higher?

Thanks,
Shashi

[Charles Lilienkamp]

It would have been fixed in 9.2 based on recollection but I apologize I realize i was thinking of something slightly different which was a class cast exception.
 
But based on recollection typically the index out of bounds means that something in the request was larger then the allotted size allowed. 
* Maybe large headers or large body or something that exceeded a defined size. Could it be that the first connection the load balancer is adding stuff then due to tcp re-use (keep-alive)  the second connection is slightly smaller? 

1 Example. The gateway has a 2gb message size limit in general and a number of assertions make 2 copies of the message restricting it to 1gb. So trying to audit or act on a message passed to the gateway that is larger then around 900mb would cause a exceptionoutofbounds.
(this is just an example meaning the size/data type was exceeded causing it to out of bounds. The example was not meant to mean that the message size (body) is what you ran into. 

But I would suspect from the first request to second even if not evident to you the message (headers/body/or other) has changed. 
Charlie.

[Shashi Ranjan]

For us it is random behavior meaning I see results sometime for exact same get request.
And sometime it fails with following exception at policy assertion "Route via HTTP".

Dec 9 16:03:52 SSG[339]: SEVERE com.l7tech.server.SoapMessageProcessingServlet: java.lang.ArrayIndexOutOfBoundsExceptionjava.lang.ArrayIndexOutOfBoundsException
Dec 9 16:03:52 SSG[339]: WARNING com.l7tech.server.util.SoapFaultManager: Ignoring invalid http status code for fault ''.
Dec 9 16:03:52 SSG[339]: SEVERE org.apache.catalina.core.ContainerBase.[ssg].[/].[SoapMessageProcessingServlet]: Servlet.service() for servlet SoapMessageProcessingServlet threw exceptionjava.lang.ArrayIndexOutOfBoundsException#012#011at org.apache.coyote.http11.InternalOutputBuffer.write(InternalOutputBuffer.java:706)#012#011at org.apache.coyote.http11.InternalOutputBuffer.sendStatus(InternalOutputBuffer.java:443)#012#011at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:1643)#012#011at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:958)#012#011at org.apache.coyote.Response.action(Response.java:190)#012#011at org.apache.coyote.Response.sendHeaders(Response.java:393)#012#011at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:314)#012#011at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:274)#012#011at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104)#012#011at com.l7tech.server.SoapMessageP

[Prem]

Hi Shashi,

Let me know if you are using return template response assertion used in the policy?

Regards,
Prem.

[Shashi Ranjan]

Hi Prem, thanks for replying.
Policy Assertions 'Return Template Response to Requestor' is not used. I have used assertion 'Route via HTTP to downstream application' and send the same response back to the browser/consumer of the service.

What was the actual issue in your case and how you have addressed that?