Layer7 API Gateway v10.1 with AppDynamics Monitoring Agent

[Muzeer Vaseem Abdul]

Hello there,

The Customer has upgraded the API Gateway vApp instances from v9.3 to v10.1 and the upgrade was successful. However, the AppDynamics agent wasn't coming up on v10.1, but it was running fine on the v9.3

I have done the below configuration on the v10.1 API Gateway, similar to the v9.3 configuration:

# AppDynamics agent
NODE_OPTS="$NODE_OPTS -Dappdynamics.http.proxyHost=xyz-proxy.corp.dmz -Dappdynamics.agent.applicationName=Gateway_PROD_NEW -Dappdynamics.agent.nodeName=hostname_gateway -Dappdynamics.agent.tierName=Gateway -Dappdynamics.http.proxyPort=8080 -javaagent:/opt/appdynamics/app-agent/javaagent.jar"

I get the below log trace on the AppD logs when I restart the API Gateway with above configuration, and the API Gateway gets into WONT_START mode.

[AD Agent init] 04 Nov 2022 16:05:24,932 WARN XMLConfigManager - XML Controller Info Resolver found invalid controller host information [] in controller-info.xml; Please specify a valid value if it is not already set in system properties.
[AD Agent init] 04 Nov 2022 16:05:24,932 WARN XMLConfigManager - XML Controller Info Resolver found invalid controller port information [] in controller-info.xml; Please specify a valid value if it is not already set in system properties.
[AD Agent init] 04 Nov 2022 16:05:24,934 INFO XMLConfigManager - XML Agent Account Info Resolver using account name [customer-prod]
[AD Agent init] 04 Nov 2022 16:05:24,934 INFO XMLConfigManager - XML Agent Account Info Resolver using account access key [****]
[AD Agent init] 04 Nov 2022 16:05:24,935 INFO XMLConfigManager - Using proxy settings [xyz.corp.dmz:8080]
[AD Agent init] 04 Nov 2022 16:05:24,935 INFO XMLConfigManager - Configuration Channel is using ControllerInfo:: host:[customer-prod.saas.appdynamics.com] port:[443] sslEnabled:[true] keystoreFile:[DEFAULT:cacerts.jks] use-encrypted-credentials:[false] secureCredentialStoreFileName:[] secureCredentialStorePassword:[] use-ssl-client-auth:[false] asymmetricKeysStoreFilename:[] asymmetricKeysStorePassword:[] asymmetricKeyPassword:[] asymmetricKeyAlias:[] validation:[UNSPECIFIED]
[AD Agent init] 04 Nov 2022 16:05:24,936 INFO XMLConfigManager - Keystore file /opt/appdynamics/app-agent-21.11.4.33358/ver21.11.4.33358/conf/cacerts.jks was not found
[AD Agent init] 04 Nov 2022 16:05:24,946 INFO XMLConfigManager - Using proxy settings [xyz.corp.dmz:8080]
[AD Agent init] 04 Nov 2022 16:05:24,948 INFO XMLConfigManager - Proxy authentication is not configured.
[AD Agent init] 04 Nov 2022 16:05:25,175 ERROR JavaAgent - Could Not Start Java Agent, disabling the agent with exception
java.lang.ExceptionInInitializerError: null
at com.l7tech.server.ServerConfig.getInstance(Unknown Source) ~[?:?]
at com.l7tech.server.log.JdkLogConfig.a(Unknown Source) ~[?:?]
at com.l7tech.server.log.c.run(Unknown Source) ~[?:?]
at com.l7tech.util.JdkLoggerConfigurator.configure(Unknown Source) ~[?:?]
at com.l7tech.server.log.JdkLogConfig.a(Unknown Source) ~[?:?]
at com.l7tech.server.log.JdkLogConfig.<init>(Unknown Source) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
at java.lang.Class.newInstance(Class.java:584) ~[?:?]
at java.util.logging.LogManager.readConfiguration(LogManager.java:1360) ~[?:?]
at java.util.logging.LogManager.readPrimordialConfiguration(LogManager.java:445) ~[?:?]
at java.util.logging.LogManager$2.run(LogManager.java:394) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at java.util.logging.LogManager.ensureLogManagerInitialized(LogManager.java:382) ~[?:?]
at java.util.logging.LogManager.getLogManager(LogManager.java:430) ~[?:?]
at sun.util.logging.internal.LoggingProviderImpl.demandJULLoggerFor(LoggingProviderImpl.java:408) ~[?:?]
at sun.util.logging.internal.LoggingProviderImpl.demandLoggerFor(LoggingProviderImpl.java:436) ~[?:?]
at jdk.internal.logger.DefaultLoggerFinder.getLogger(DefaultLoggerFinder.java:157) ~[?:?]
at jdk.internal.logger.LazyLoggers.getLoggerFromFinder(LazyLoggers.java:389) ~[?:?]
at jdk.internal.logger.LazyLoggers.getLazyLogger(LazyLoggers.java:444) ~[?:?]
at jdk.internal.logger.LazyLoggers.getLogger(LazyLoggers.java:414) ~[?:?]
at java.lang.System.getLogger(System.java:1662) ~[?:?]
at jdk.internal.event.EventHelper.isLoggingSecurity(EventHelper.java:145) ~[?:?]
at sun.security.provider.X509Factory.commitEvent(X509Factory.java:772) ~[?:?]
at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:108) ~[?:?]
at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:355) ~[?:?]
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:764) ~[?:?]
at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:243) ~[?:?]
at java.security.KeyStore.load(KeyStore.java:1479) ~[?:?]
at sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:365) ~[?:?]
at sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:313) ~[?:?]
at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:55) ~[?:?]
at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]
at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:278) ~[?:?]
at com.singularity.ee.util.httpclient.EasyX509TrustManager.<init>(EasyX509TrustManager.java:194) ~[appagent.jar:?]
at com.singularity.ee.util.httpclient.SimpleHttpClientWrapper.createInternalHttpClient(SimpleHttpClientWrapper.java:629) ~[appagent.jar:?]
at com.singularity.ee.util.httpclient.SimpleHttpClientWrapper.<init>(SimpleHttpClientWrapper.java:196) ~[appagent.jar:?]
at com.singularity.ee.util.httpclient.SimpleHttpClientWrapper.<init>(SimpleHttpClientWrapper.java:176) ~[appagent.jar:?]
at com.singularity.ee.rest.controller.request.AControllerRequest.createHttpClientWrapper(AControllerRequest.java:242) ~[?:?]
at com.singularity.ee.rest.controller.request.AControllerRequest.createHttpClientWrapper(AControllerRequest.java:193) ~[?:?]
at com.singularity.ee.rest.controller.request.AControllerRequest.setupHttpClientWrapper(AControllerRequest.java:185) ~[?:?]
at com.singularity.ee.agent.appagent.kernel.config.xml.XMLConfigManager.executeAgentResolver(XMLConfigManager.java:219) ~[appagent.jar:?]
at com.singularity.ee.agent.appagent.kernel.config.xml.XMLConfigManager.<init>(XMLConfigManager.java:165) ~[appagent.jar:?]
at com.singularity.ee.agent.appagent.kernel.AgentKernel.start(AgentKernel.java:131) ~[appagent.jar:?]
at com.singularity.ee.agent.appagent.kernel.JavaAgent.initialize(JavaAgent.java:451) ~[?:?]
at com.singularity.ee.agent.appagent.kernel.JavaAgent.initialize(JavaAgent.java:346) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at com.singularity.ee.agent.appagent.AgentEntryPoint$1.run(AgentEntryPoint.java:655) ~[?:Server Agent #21.11.4.33358 v21.11.4 GA compatible with 4.4.1.0 r77cc5f5f533203ddde31423514c2270e84f3238f release/21.11.0]
Caused by: java.lang.ClassCastException: class com.l7tech.util.ConfigFactory$DefaultConfig cannot be cast to class com.l7tech.server.ServerConfig (com.l7tech.util.ConfigFactory$DefaultConfig and com.l7tech.server.ServerConfig are in unnamed module of loader 'app')
at com.l7tech.server.aw.<clinit>(Unknown Source) ~[?:?]
... 53 more
[AD Agent init] 04 Nov 2022 16:05:25,176 INFO JavaAgent - Restoring properties, exception causing the resets was class org.apache.xerces.jaxp.DocumentBuilderFactoryImpl cannot be cast to class javax.xml.parsers.DocumentBuilderFactory (org.apache.xerces.jaxp.DocumentBuilderFactoryImpl is in unnamed module of loader 'app'; javax.xml.parsers.DocumentBuilderFactory is in module com.appdynamics.appagent of loader com.singularity.ee.agent.appagent.kernel.classloader.Post19AgentClassLoader @4ae3c1cd)
[AD Agent init] 04 Nov 2022 16:05:25,176 INFO JavaAgent - Clearing property javax.xml.parsers.DocumentBuilderFactory because none was set before config
[AD Agent init] 04 Nov 2022 16:05:25,176 INFO JavaAgent - Clearing property org.apache.commons.logging.LogFactory because none was set before config


Did anyone else out there configured AppD agent with v10.1 API Gateways ? If so, am I missing anything else ?

Thanks
Vaseem

[Lalit Kumar Soni]

Did you get the answer for above?

[Lalit Kumar Soni]

Hi Vaseem, I was able to solve the problem. I have made two changes: 

1) Upgraded the App agent to 22.*. Previously we were using 21 version and which is incompatible with Java 11. 
2) Added two parameters in Node.properties under node.java.opts
      -Dappdynamics.agent.use.agent.classloader.context=false
      -Dappdynamics.agent.log4j2.disabled=true

[Muzeer Vaseem Abdul]

Thanks Lalit. I will try that.

Regards,
Vaseem

[Minal Mallawat]

Hi Lalit, I also recently migrated the gateway to v10.1 from v10.0 and I am facing the same issue. I tried adding the 2 parameters as you suggested but it still doesn't work for me. Any suggestions?

Java version - openjdk version "11.0.15" 2022-04-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.15+9-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+9-LTS, mixed mode, sharing)

AppD agent version - ver22.1.0.33445

[Muzeer Vaseem Abdul]

Classification: Confidential
Hello Minal,

I added the same two parameters in the NODE_OPTS section on the appliancedefs.sh file as well. So the NODE_OPTS looks like below:
# AppDynamics agent
NODE_OPTS="$NODE_OPTS -Dappdynamics.http.proxyHost=xyz-proxy.corp.dmz -Dappdynamics.agent.applicationName=Gateway_PROD_NEW -Dappdynamics.agent.nodeName=hostname_gateway -Dappdynamics.agent.tierName=Gateway -Dappdynamics.http.proxyPort=8080 -javaagent:/opt/appdynamics/app-agent/javaagent.jar -Dappdynamics.agent.use.agent.classloader.context=false -Dappdynamics.agent.log4j2.disabled=true"

Make sure the entire NODE_OPTS part is a single line without line breaks and restart the Gateway using "service ssg restart". You can check the Gateway logs (ssg_0_0.log) and AppD agent logs for more information on what is causing the failure.

Thanks and Regards,

Muzeer Vaseem Abdul
Senior Consultant | (Identity and Access Management & API Management)
HCL Enterprise Studio (CA Techonologies / Broadcom Center of Excellence)
M: [phone] | [e-mail]
www.hcltech.com/enterprise-studio<http: www.hcltech.com/enterprise-studio="">

[signature_1743155637]

::DISCLAIMER::
________________________________
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.

[Manju Vivek]

Hi Minal, Did you resolve the issue?