DX Application Performance Management

Expand all | Collapse all

What does this error mean?

Jump to Best Answer
  • 1.  What does this error mean?

    Posted 09-19-2017 08:54 AM

    Hi,

    What does the below error mean and what are the steps to correct it? This error is in the MOM's IntroscopeEnterpriseManager.log.

     

    Thanks!

    Sam

     

     

    2017-09-19 08:10:25,715 [ERROR] [btpool0-82] [Manager.AppMap] Internal Server Error happend during controller invocation!
    org.mortbay.jetty.EofException
            at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:789)
            at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:568)
            at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1010)
            at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:650)
            at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:579)
            at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2003)
            at com.fasterxml.jackson.core.json.UTF8JsonGenerator._verifyValueWrite(UTF8JsonGenerator.java:973)
            at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeNumber(UTF8JsonGenerator.java:804)
            at com.fasterxml.jackson.databind.ser.std.NumberSerializers$LongSerializer.serialize(NumberSerializers.java:212)
            at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:693)
            at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:675)
            at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:157)
            at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
            at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)

            at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
            at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:693)
            at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:675)
            at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:157)
            at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:693)
            at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:675)
            at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:157)
            at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:149)
            at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serialize(CollectionSerializer.java:111)
            at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serialize(CollectionSerializer.java:24)
            at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:693)
            at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:675)
            at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:157)
            at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
            at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2383)
            at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:231)
            at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:208)
            at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:205)
            at org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:146)
            at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:71)
            at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:126)
            at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:775)
            at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:705)
            at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
            at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
            at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
            at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:965)
            at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:867)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
            at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:841)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
            at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
            at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
            at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
            at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
            at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
            at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
            at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
            at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
            at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:440)
            at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
            at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
            at org.mortbay.jetty.Server.handle(Server.java:326)
            at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
            at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)
            at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
            at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
            at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
            at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
            at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
    Caused by: java.net.SocketException: Broken pipe
            at java.net.SocketOutputStream.socketWrite0(Native Method)
            at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
            at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
            at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368)
            at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:122)
            at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:723)
            ... 64 more



  • 2.  Re: What does this error mean?

    Posted 09-19-2017 09:05 AM

    Some of the same error but not exactly the same 

     

    MOM not registering to ETC, getting "Configuration test failed. Security token is invalid". 

     

    Do you have any idea what was happening at the time? Has other APM Admins, Partners seen those errors? CA  Staff may respond if no answers are given after the usual period of waiting

     

    Thanks

    Hal German



  • 3.  Re: What does this error mean?
    Best Answer

    Posted 09-19-2017 09:27 AM

    Hello Samantha,

    I have seen the same error/ stack trace when running long queries and seems to be related to the load

    Could you verify the below?

     

    1) The exception itself is about broken TCP connection, it could be caused in case the client refuses to get more data from server in case of large data portion or the network error.

    Check if the problem occurs when running specific queries : Introcsope webservices calls, workstation,CLW queries or when using webview

     

    2)I have found the same stack trace in Defect DE142508, but without a heapdump it not possible to confirm that the root cause is the same:

     

    +++++++++++

    “We took a heap dump and examined the object graphs for the database connections on the MoM, which pointed to a Commons DBCP pool used by AppMap.  The pool instantiated internally by the EM, and is configured with an initial size of 10 connections, and a “maxIdle” size of 40 connections.  What this means is that even though the connections are not being used, the pool will hold them open.  What it looks like is a series of events, starting with the db aggregation job being configured on the wrong collector combined with the extra connections being held open in an IDE, which led to a burst of activity on Feb 18.  This caused the connection pool in question to grow, and at that point the extra connections are never released, causing the problems to continue.

     

    So there are two issues – 1) reduce the <property name="maxIdle” setting value so that the pool can return to normal after a burst, 2) add the pool to the sustainability metrics so that we can accurately track what is happening.

     

    Specifically, in em\com.wily.introscope.appmap.em\src\main\resources\spring\appmap-server-context.xml:

    <bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">

      <property name="driverClassName" value="${hibernate.connection.driver_class}" />

      <property name="url" value="${hibernate.connection.url}" />

      <property name="username" value="${hibernate.connection.username}" />

      <property name="password" value="${hibernate.connection.password}" />

      <property name="initialSize" value="10" />

      <property name="maxActive" value="250" />

      <property name="maxIdle" value="40" />

      <property name="poolPreparedStatements" value="true" />

      <property name="maxOpenPreparedStatements" value="250" />

    </bean>                            

     

    The quick workaround would be to reconfigure the setting in the xml file by unpacking the jar, editing the file, and repackaging the jar in the MoM installation, and then restart the MoM.]                                 

    +++++++++++

    The above file is located inside EM_HOME\product\enterprisemanager\plugins\ com.wily.introscope.appmap.em_10.5.1.jar\spring\appmap-server-context.xml

    You can use 7zip or similar utility to update the file, try to reduce maxIdle to 20

     

    If the problem persists, open a support case and attach EM logs + a series of threadumps from MOM + apm db logs.

     

    I hope this helps,

    Regards,

    Sergio



  • 4.  Re: What does this error mean?

    Posted 09-19-2017 10:40 AM

    Hi Sergio,

     

    Thanks for your insight! Based off the logs it seems to be occurring after an user is using CLW. We currently have version 10.3, does the jar file have the same naming convention however with the different version? Additionally when you state unpacking and re-packing you mean unzipping and rezipping correct?



  • 5.  Re: What does this error mean?

    Posted 09-19-2017 11:26 AM

    Hi Samantha,

    Yes, the filename is EM_HOME\product\enterprisemanager\plugins\com.wily.introscope.appmap.em_10.3.0.jar you can update it directly using 7zip, stop the EM first.

    Also, try to run a small CLW query, if the problem doesn't persists, enable DEBUG logging on the EM for a few minutes, ask the user to run the problematic CLW query again, you can switch back to INFO after the test is completed, send us all the above mentioned diagnostic files.

    Thank you,

    Sergio



  • 6.  Re: What does this error mean?

    Posted 09-22-2017 03:39 AM

    Hi Janelle,

     

    "Internal Server Error happend during controller invocation! org.mortbay.jetty.EofException" is a generic error I am afraid, we would need DEBUG log to identify the root cause.

     

    If you are using 10.5.1, I strongly suggest you to apply the latest 10.5.2HF#39 as it contains major performance improvements for ATC/EV (it is the latest HF as of now).  Open a support case to obtain a copy and details of the hotfix. 

     

    Please note that the new APM hotfix mechanism in 10.5.x allows you to upgrade your Introscope EMs in a matter of seconds, there is no longer need to reinstall your EMs or database!

     

    I hope this helps,

    Sergio