DX Application Performance Management

 View Only

New APM KBs for August (Many are for 9.6)

  • 1.  New APM KBs for August (Many are for 9.6)

    Broadcom Employee
    Posted Aug 29, 2014 09:49 AM

    Here are the KBs for this month. There are eight more
    awaiting to be published. I thank my colleagues for taking the time to write
    and review these.



    APM New KB Articles for August 2014



    1. IBM Java 7 agent exception JVMVRFY012 reported






    Description: When instrumenting IBM Java 7 based application
    with 9.5.5 or 9.6 agent, an exception is reported similar to:


    Caused by: java.lang.VerifyError: JVMVRFY012 stack shape





       at java.lang.J9VMInternals.verifyImpl(Native Method)

       at java.lang.J9VMInternals.verify(J9VMInternals.java:93)

       at java.lang.J9VMInternals.initialize(J9VMInternals.java:170)

       at java.lang.Class.forNameImpl(Native Method)

       at java.lang.Class.forName(Class.java:181)


    2. How does APM CE identify components? Does order of the components matter?



    Description: This explains the relationship between the order of components in the APM CE Business Transaction signature and which data
    is seen by the TIM. It answers these questions:


    Does APM CE need to see the JSPs invoked in the same order as the APM CE Business Transaction signature or does the order not matter ?


    When a JSP is called more than once in a single transaction, how does CEM treat this?


    Does the order have to be the same always or does it look or the number of times that JSP is called within the transaction?


    3. Error message seen during the Postgres database restore step while upgrading to APM 9.6.




    Description: NA



    4. Installation steps for a APM 9.6 TIM running on the  MTP Appliance (CA ADA Multi-Port Monitor)




    Description: Although the new 9.6 TIM installer ships now as an RPM package, the 9.6 TIM running on the MTP Appliance still needs to
    installed using the MTP Web interface.


    5. After upgrading Websphere, four of our twelve agent instances are no longer reporting Frontend or Servlet metrics




    Description:  After upgrading Websphere from release 7 to 8.5.5, some of the instances on one Server stopped reporting Frontend or Servlet metrics.



    The following is the same on both systems:


    Agent configuration on each system was not changed during the upgrade.


    The Websphere configuration on each system


    Each WAS instance serves the same application via a load balancer. It has also been verified that the application is working correctly on all WAS instances


    6. A Guide to Solving Common Stats and Defects Aggregation Problems  (Update)




    Description: Applies to APM 9.1 - 9.6



    7. Overriding the default login host for Workstation WebStart




    Description: When downloading the Workstation WebStart from the Enterprise Manager, it is auto-populated with a login hostname.


    However due to varying network configurations, that hostname may not be valid. is there a way to override the default value to provide a correct hostname?



    8. Are synchronizing TIMs versus Introscope Agents
    treated differently?




    Description: During synchronizing, two different operations are taking place. This document provides more details on each operation.




    9. Seeing duplicate Dot Net Agent names in investigator.





    Description: Problem: Duplicate Agent names in the investigator appended with %1,%2,%3......





    When the Agent restarts it has a new process id.


    In Windows, if the Agent about to restart still has a running transaction, then the process will continue analyzing the transaction
    and afterwards, the process terminates. However, this does not stop the Windows from creating a new process because of the restart. The result is there are two
    processes associated with the same application. In the Investigator, this would be represented as Agent and Agent%1.


    On the EM side, Once the original Agent has been terminated completely, then Agent%1 will take the Agent node and Agent%1 node will become



    On the Agent side, both the agents with the old process Id and the one with the new process Id will have the same name. However, because
    the EM already has this name established, the new process Id will have its name appended with "%1". The Agent, recognizes its name without the
    "%1" and sends this to the EM to place in the Investigator tree. Thus, the Agent naming feature is active and the Agent name has been properly
    retrieved. But users want to stop the EM from creating the Agent with the "%1" appended.



    The problem with setting the property "introscope.agent.agentAutoNamingMaximumConnectionDelayInSeconds ",
    is that it delays the connection until is receives it name. Using the information provided above, we see that the Agent gets its name properly and
    relatively quickly. Once the Agent gets its name, it connects to the EM and the EM applies the "%1".



    Note: If you try setting this property "introscope.agent.agentAutoRenamingIntervalInMinutes=10 ", this will
    not work because the Agent name will not change.


    10. How does Deep Inheritance work and what is its Impact?





    For normal instrumentation, the agent only sees one inheritance level on the bytecode level. Let us use the following inheritance
    structure as an example:


    Class A 

    |- Class B

    |-Class C

    |- Class E


    Let us assume, that you have a PBD directive which instructs the ProbeBuilder to monitor everything inheriting from "Class A".



    On startup, the application sends "Class E" through the Classloader and the ProbeBuilder. At this stage, since the
    ProbeBuilder has no idea that "Class E" inherits from "Class A", it would not instrument it.


    However, with deep inheritance enabled, when all the other classes are going through the Classloader some time after the startup, it
    always checks the complete structure and tries to instrument it using Dynamic Instrumentation. Without Deep Inheritance enabled, this would not happen. This
    is because it would not check the full inheritance once a new class comes up. So, in the above example, "Class E" would not be instrumented.


    Deep Inheritance Impact


    Although the Deep Inheritance feature has played a crucial part in providing a complete monitoring solution, the mechanism of checking the
    full inheritance tree and performing Dynamic Instrumentation often comes with a cost.



    Due to its described nature, Deep Inheritance causes some overhead, in which on most engines, usually happens at startup. It is
    especially significant in frameworks using dynamic proxy classes. For example,  frameworks like Spring which are created on the fly, are generally dangerous
    for Deep Inheritance. In scenarios like this, the AutoProbe.log can often be used to identify these classes, and a skip of these classes in the PBD could
    help addressed the overhead issue.



    On the other hand, removing/disabling the Deep Inheritance feature will mean, on most frameworks, having less metrics and more gaps in
    transaction traces.



    11. CEM - Internal Application Error: JDBC exception on
    Hibernate data access



    Description: Using the "Group Identification" feature under Business Applications to associate a Group with specific customers, you receive a JDBC error while trying to identify the associated Group using XML.



    When creating "User identification" under Business Applications and using XML, it works as expected. When using the same XML for both User ID and Group ID, and trying to "save" the Group ID (XML location), you receive this JDBC error:



    Internal Application Error: JDBC exception on Hibernate data access; nested exception is org.hibernate.exception.GenericJDBCException: Could not execute JDBC batch update.


    12. Which APM releases are supported for MTP 10?





    Description: Lists TIM/MTP release compatibility for MTP 10.


    13. APM 9.6 Transaction Impact Monitor (TIM) ports and
    how to change them post-install.


    Description: If needed, APM 9.6 TIM ports for a private  httpd web server and the internal communication with that web server can be  changed post-install.

    14. Frontends missing when monitoring WebSphere 8.5.5
    with Introscope 9.1 agent.





    Description: What to do if not seeing Frontends reported when monitoring WebSphere 8.5.5 (using IBM JDK 1.6) with an APM 9.1 agent.


    You are able to see Servlets and JSPs reported but not Frontends


    15. An overview of installation failures on Linux after selecting a new schema for Postgres.





    Description: This document describes a potentially troublesome area in APM installations -- choosing to create a new schema for an
    existing Postgres installation. The solution describes the various errors that can be displayed and how to avoid them.



    16. EPAgent support for AIX 7.1.




    Description: This document clarifies if there is any support for EPagent on AIX 7.1. This is not listed in the Compatibility Guide.



    17. A configuration example of the Shell Command Action.






    Description: An example of Shell Command Action configuration and use is provided.



    18. EM running on a 64-bit Windows 2003 server suddenly becomes






    Description: PM EM has just been installed as a service on a 64-bit Windows platform. The server is running and does not appear to be heavily utilized but the EM is unusable.



    19. Solaris 10-based APM Java agent reports fatal UTF error for "iconv_open()" at start-up






    Description:On start up, seeing this this fatal error :



    ["../../../src/solaris/instrument/EncodingSupport_md.c":66]: Failed to complete iconv_open() setup



    The environment is a Java Application Server running on Solaris 10 with the APM agent installed


    20. Custom tracer only shows Backends on the Application Triage Map when viewed by Frontend but not by Business Service.





    Description: The following custom tracer is defined:

    SetTracerClassMapping: AppMapCustomBackendTracer com.wily.introscope.appmap.agent.trace.hc2.AppMapMarkerTracer com.wily.introscope.probebuilder.validate.ResourceNameValidator


    SetTracerParameter: AppMapCustomBackendTracer nodeType  GenericBackend



    SetTracerParameter: AppMapCustomBackendTracer nodeLevel MethodClass



    SetTracerParameter: AppMapCustomBackendTracer ownerType Application



    SetTracerParameter: AppMapCustomBackendTracer boundaryType Backend



    SetTracerParameter: AppMapCustomBackendTracer useblame false



    SetTracerParameter: AppMapCustomBackendTracer newmode true



    SetTracerOrdering: AppMapCustomBackendTracer -1000001



    This works correctly when viewing the App Map by Frontend -- the called backends instrumented by this tracer are displayed as expected.



    However, when viewing by Business Service those backends are not shown.




    21. Running an APM CE (CEM) scheduled report generates database error messages in the Introscope Enterprise Manager log.






    Description: After running APM CE daily scheduled reports each morning, there are database errors in the Introscope logs (including key
    constraint errors.) The messages in the log look like the following:



    2/10/14 06:00:03.238 AM EST [ERROR]
    [QuartzScheduler_Worker-8] [org.quartz.core.JobRunShell] Job DEFAULT.jobDetailBean threw an unhandled Exception:
    Invocation of method 'pruneData' on target class [class
    com.wily.apm.model.pruning.DataPruner] failed; nested exception is org.springframework.dao.DataIntegrityViolationException:
    CallableStatementCallback; SQL [{? = call prune_apm_data(?, ?)}]; ERROR: update or delete on table "apm_vertex" violates foreign key constraint "apm_parent_id_fkey" on table "apm_vertex" Detail: Key
    (id)=(801) is still referenced from table "apm_vertex"



    22. After upgrading to 9.5, we are unable to launch CEM   console and NonUniqueResultException errors in MOM log file





    Description:  After upgrading APM to 9.5.x from 9 with an Oracle APM database, on rare occasions, the following errors appear in the MOM Enterprise Manager log file:



    8/12/14 04:47:43.398 PM EEST [ERROR]
    [InitializationPool.Thread1] [Manager.com.timestock.tess.util.DbUtils] Select failed for com.timestock.tess.data.objects.

    Setting  org.hibernate.NonUniqueResultException: query did not return  a unique result:2


    at  org.hibernate.impl.AbstractQueryImpl.uniqueElement(AbstractQueryImpl.java:813)


    at  org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:804)


    at com.timestock.tess.util.DbUtils.getObjectWithRetries(DbUtils.java:4533)


    at com.timestock.tess.util.DbUtils.getObjectWithRetries(DbUtils.java:4486)


    at com.timestock.tess.util.DbUtils.getSettingValue(DbUtils.java:8969)



    at com.timestock.tess.services.processors.BizEventProcessor.initialize(BizEventProcessor.java:118)



    at com.timestock.tess.services.service.ServiceBase.initialize(ServiceBase.java:62)


    at com.timestock.tess.services.service.ServicesManager.initialize(ServicesManager.java:123)


    at com.timestock.tess.services.initialization.RunInitializers$IntializeWrapper.call(RunInitializers.java:87)


    at com.timestock.tess.services.initialization.RunInitializers$IntializeWrapper.call(RunInitializers.java:1)


    at  com.timestock.tess.util.ThreadPool$ConcurrentCallableTask.call(ThreadPool.java:218)


    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)


    at java.util.concurrent.FutureTask.run(Unknown Source)


    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)


    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)


    at java.lang.Thread.run(Unknown Source)





    If the environment uses APM CE (CEM) then it may be also not  possible to launch the CEM console.


    This problem has been reported when upgrading from 9.1.2 to 9.5.2 and from 9.1.2 to 9.5.3. It has not been possible to recreate the problem in CA testing environments




    23. How to configure the MOM and Collector when processing VuGen trace files by the APM CE (CEM) Script Recorder.






    Description: The APM Transaction Definition Guide provides details on using VuGen Trace files as input to the Script Recorder for generating transaction definitions. However, some undocumented additional steps are needed for the transaction to display in the APM CE UI under the Recording Session tab.



    The Guide does not mention using different webserver ports for the MOM and Collector. Failure to do this results in no recording in the
    APM CE UI.



    24. APM CE(CEM)Analyst Responsibilities





    Description: One of the security roles for APM CE reports is that of a CEM Analyst. The security only says the following about this role: has
    access to CA CEM reports and views only



    The following are suggested APM CE (CEM) Analyst (non-operational, non-administrative) responsibilities. The range and frequency
    of these tasks may vary at each site depending on job responsibilities, complexity of environment, type of integration employed, number of users and
    support resources, and growth rate of the APM CE-monitored application.



    25. What are some of the best practices for APM CE (CEM) Web Server Filters?






    Description: This provides a rough framework to follow when implementing web server filters.



    26. Wily Technical Advisory: Native Memory Leak on IBM Java 1.6 when using -javaagent (Update)






    Description: CA Wily Technical Advisory: Native Memory Leak on IBM Java 1.6 when using -javaagent



    27. Information about accessing CEM data (not aggregated) and on how long it is retained after capture.






    Description: Three types of raw APM CE (CE) data (Defects, Stats and BtStats) may be accessed via the APM CE (CEM) UI and extracted for reports based on specific data retention rules for each data type.



    28. Which version is the WTG Introscope Agent?






    Description: Clarifying WTG/Introscope version compatibility.



    29. Does SiteMinder Manager and APM TG (WTG) share the same module for Native Data Reporter?






    Description:  Clarifies Native Data Reporter and the two components that use it -- SiteMinder Memory Manager and WTG.



    30. [Japanese] Incorrect value in historical mode



    -- No Link






    31. Using unsupported characters in a Java class name produce "Invalid Class path" errors in the agent log .






    Description:  Agent logs are filled with errors as the following:



    DATE/TIME [ERROR] [IntroscopeAgent.Agent] A problem occurred while attempting to Introscope-enable



    org/jboss/weldx/transaction/UserTransaction$-11111111$Proxy$_$$_Weld$Proxy$:  Invalid class file



    DATE/TIME [ERROR] [IntroscopeAgent.Agent] A problem occurred while attempting to Introscope-enable



    org/jboss/weld/security/Principal$-11111111$Proxy$_$$_Weld$Proxy$  Invald class file



    32. TIM installation breaks the working TCP Wrapper access control for specific hosts when using SSH and  /etc/hosts.allow.






    Description: This article discusses a scenario where the TCP Wrapper access control no longer works as configured. This happens usually
    these after two installation steps:



    The Redhat Enterprise Linux Kickstart image is installed. The TCP Wrapper configuration in /etc/hosts.allow & /etc/hosts.deny works as expected for configured hostnames.



    The third-party and tim-complete images are installed. Afterwards, TCP Wrapper access is open to all hosts. This is because the
    openSSH version of sshd installed replaced the previous version.