DX Application Performance Management

Expand all | Collapse all

Getting Errors in IntroscopeAgent.log on new install of 10.7 tomcat agent

Jump to Best Answer
  • 1.  Getting Errors in IntroscopeAgent.log on new install of 10.7 tomcat agent

    Posted 10-16-2018 10:58 AM

    Hi

    I have installed a freshly downloaded version of the tomcat agent , and only changed three settings from default

    1) set the name to find my MOM

    2) set

    introscope.agent.jmx.enable=true

    and added :
    com.wily.use.platform.mbeanserver=true

     

    at the request of the Application team.  When they start their tomcat app, I get the correct log files but get this reported every 15 seconds in the IntroscopeAgent.log:

     

    10/16/18 10:40:16 AM EDT [ERROR] [IntroscopeAgent.HttpStatusMsgFetcherTracer] getHttpStatusMessage() Unable to get status message: java.lang.NullPointerException
    10/16/18 10:40:31 AM EDT [ERROR] [IntroscopeAgent.HttpStatusMsgFetcherTracer] getHttpStatusMessage() Unable to get status message: java.lang.NullPointerException
    10/16/18 10:40:46 AM EDT [ERROR] [IntroscopeAgent.HttpStatusMsgFetcherTracer] getHttpStatusMessage() Unable to get status message: java.lang.NullPointerException
    10/16/18 10:41:01 AM EDT [ERROR] [IntroscopeAgent.HttpStatusMsgFetcherTracer] getHttpStatusMessage() Unable to get status message: java.lang.NullPointerException
    10/16/18 10:41:16 AM EDT [ERROR] [IntroscopeAgent.HttpStatusMsgFetcherTracer] getHttpStatusMessage() Unable to get status message: java.lang.NullPointerException

     

     

    Has anyone else seen this as a google search, and search here did not yield anything with that HttpStatusMsgFetcherTracer method?

     

    Thanks

    Lee



  • 2.  Re: Getting Errors in IntroscopeAgent.log on new install of 10.7 tomcat agent

    Posted 10-16-2018 11:40 AM

    Dear Lee:

    Looking at another case having the same error, placing logs in DEBUG should give you more details

     

    Dear Community:

      Can you help Lee out with next steps?



  • 3.  Re: Getting Errors in IntroscopeAgent.log on new install of 10.7 tomcat agent

    Posted 10-16-2018 11:53 AM

    The error comes from httpbackendmetrics.pbd, this was introduced in 10.x to monitor standard HTTP communications out of applications.

    The HTTPStatusMsgFetcherTracer maps to URLConnectionStatusMsgTracer

     

    SetTracerClassMapping: URLConnectionStatusMsgTracer com.wily.introscope.agent.trace.backend.http.tracer.HttpStatusMsgFetcherTracer com.wily.introscope.probebuilder.validate.ResourceNameValidator

     

    When we search on that, we get these two calls.

     


    TraceOneMethodWithParametersIfFlagged: HttpURLConnectionBackend getHeaderField(I)Ljava/lang/String; URLConnectionStatusMsgTracer "THIS STRING IS NOT USED"

    TraceOneMethodWithParametersIfFlagged: OC4JHTTPClientBackend getHeaderField(I)Ljava/lang/String; URLConnectionStatusMsgTracer "THIS STRING IS NOT USED"

     

    So you can see it is trying to make a getHeaderField call to get the status value

    I would assume the value is used to decorate transaction traces with the exact HTTP status code.

    Perhaps the method signature is different in the version you are using, or that getHeaderField method is accessible at all.

    If you want to stop the error you can comment either of the lines above that start with "Trace", most probably the first one will be enough, unless you know for sure the application uses OC4J/Weblogic HTTPClient library.

     

    It will fairly inevitably lead to a support case to investigate further, as Hal says, DEBUG logs would be useful.

     

    We would need to know what sort of HTTP library is being used in the application to see if there could be a compatibility issue. For example, when this was first released, we supported Apache HTTP client v4, but not v3 and we saw similar issues in the beginning where v3 was being used.

    I also can't comment on the metric specifier saying "THIS STRING IS NOT USED" but I assume it doesn't create a metric and only decorates transaction traces.



  • 4.  Re: Getting Errors in IntroscopeAgent.log on new install of 10.7 tomcat agent

    Posted 10-16-2018 12:43 PM

    I tried commenting out one, and then both of these and still continued to get the message every 15 seconds with an additional stack trace dropped in a single time :

     

    10/16/18 12:41:58 PM EDT [INFO] [IntroscopeAgent.Agent] re-loading configuration resource /opt/caapm/wily/core/config/IntroscopeAgent.profile
    10/16/18 12:41:58 PM EDT [INFO] [IntroscopeAgent.Agent] re-loading configuration resource /opt/caapm/wily/extensions/Extensions.profile
    10/16/18 12:41:58 PM EDT [ERROR] [IntroscopeAgent.Agent] IntervalHeartbeat.execute threw executing: Configuration Watcher
    java.lang.NoClassDefFoundError: Could not initialize class com.wily.introscope.agent.intelligent.detection.StackTraceRecorder$SingletonHolder
            at com.wily.introscope.agent.intelligent.detection.StackTraceRecorder.getInstance(StackTraceRecorder.java:66)
            at com.wily.introscope.agent.intelligent.detection.StackTraceRecorder.getExitPointRecorder(StackTraceRecorder.java:124)
            at com.wily.introscope.agent.intelligent.exitpoint.ExitPointDetectionConfiguration.setExitPointDetectionEnabledStatus(ExitPointDetectionConfiguration.java:364)
            at com.wily.introscope.agent.intelligent.exitpoint.ExitPointDetectionConfiguration$1.set(ExitPointDetectionConfiguration.java:100)
            at com.wily.util.properties.hot.ConfigurationProperty.setValue(ConfigurationProperty.java:635)
            at com.wily.util.properties.hot.ConfigurationProperty.setValueAndLog(ConfigurationProperty.java:593)
            at com.wily.util.properties.hot.ConfigurationManager.setProperties(ConfigurationManager.java:627)
            at com.wily.util.properties.hot.ConfigurationManager.processConfiguration(ConfigurationManager.java:241)
            at com.wily.util.properties.hot.ConfigurationManager.onChange(ConfigurationManager.java:224)
            at com.wily.util.ConfigurationWatcher.processContentChange(ConfigurationWatcher.java:97)
            at com.wily.util.ConfigurationWatcher.ITimestampedRunnable_execute(ConfigurationWatcher.java:129)
            at com.wily.util.heartbeat.IntervalHeartbeat$BehaviorNode.execute(IntervalHeartbeat.java:972)
            at com.wily.util.heartbeat.IntervalHeartbeat.executeNextBehaviorAndCalculateSleepTime(IntervalHeartbeat.java:491)
            at com.wily.util.heartbeat.IntervalHeartbeat.access$2(IntervalHeartbeat.java:445)
            at com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:668)
            at java.lang.Thread.run(Thread.java:748)

     

    I will turn on debug and look at the logs also.



  • 5.  Re: Getting Errors in IntroscopeAgent.log on new install of 10.7 tomcat agent
    Best Answer

    Posted 10-17-2018 06:42 AM

    The error here is not necessarily related to the first one

    Looking at this error:

     

    java.lang.NoClassDefFoundError: Could not initialize class com.wily.introscope.agent.intelligent.detection.StackTraceRecorder$SingletonHolder

     

    It seems we are missing IntelligentInstrumentation.jar, which should be in the core/ext folder of the agent.

    This is part of the smart instrumentation feature that adds interesting uninstrumented classes/methods to transaction traces.

     

    It is unlikely this file is missing but possible, otherwise maybe the user that runs Tomcat does not have sufficient permission over the agent folder - read/write for all subfolders to be sure

     

    This error appears after the Extensions.profile has been (re)loaded but configuration for this feature should not be in an extension, it should be part of the basic Java agent itself.

     

    I would suggest to raise a case on this, and, if possible, to share the complete agent folder you are using, so we could run it against a Tomcat app here and see if we get the same errors.

    Also, depending on exactly how you downloaded and got the file onto the Tomcat machine, you may want to try downloading again, making sure you transfer in binary mode if using FTP.