Registry and Portal Outage

[SoumyaA]

Is any one aware of error-

Error 500 - JMS Messaging problem with ServerRequestHandler for tcp://<host>:2011/Coordinator Wire format negotiation timeout: peer did not send his wire format.; nested exception is: com.itko.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format

This error is thrown and Portal and Registry services are getting stopped time and again.

Any input will be much appreciated

[Sankar Natarajan]

Hi Soumya

        This error typically indicates an overwhelmed Coordinator service. Please check coordinator.log and share the exception messages found there for a better assessment of the problem. 

--

regards

Sankar Natarajan

[SoumyaA]

Attached Coord VSE and Portal Log. Attached is Portal and Registry log as well.
Registry machine and portal machine are different.
Also while starting the portal service back we noticed error-

while starting portal, we notice the following error in portal.log

java.lang.RuntimeException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'indexConsumer': Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchService': Invocation of init method failed; nested exception is org.elasticsearch.indices.IndexAlreadyExistsException: [sv] already exists

And Registry is taking huge heap space. Error from registry log-

2020-07-28 01:11:50,815Z (03:11) [Event Sink Thread Pool Thread 3] INFO  com.itko.lisa.coordinator.TestRegistryImpl - Coordinator Servers: 5 Simulator Servers: 6 VSEs: 7 Running vusers: -1 Labs: 1 Memory used 968mb, allocated 1,417mb, max 1,476mb (65%) Our cpu usage 16%, system cpu used 17% GC time 0% db ping: 0 ms

2020-07-28 01:11:52,703Z (03:11) [Event Sink Thread Pool Thread 5] ERROR com.itko.lisa.coordinator.TestRegistryImpl -

java.lang.OutOfMemoryError: unable to create new native thread

        at java.lang.Thread.start0(Native Method)

        at java.lang.Thread.start(Thread.java:714)

        at com.itko.activemq.transport.TransportThreadSupport.doStart(TransportThreadSupport.java:50)

        at com.itko.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:462)

        at com.itko.activemq.util.ServiceSupport.start(ServiceSupport.java:53)

        at com.itko.activemq.transport.InactivityMonitor.start(InactivityMonitor.java:127)

        at com.itko.activemq.transport.TransportFilter.start(TransportFilter.java:58)

        at com.itko.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)

        at com.itko.activemq.transport.TransportFilter.start(TransportFilter.java:58)

        at com.itko.activemq.transport.TransportFilter.start(TransportFilter.java:58)

        at com.itko.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:289)

        at com.itko.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:228)

        at com.itko.activemq.ActiveMQConnectionFactory.createQueueConnection(ActiveMQConnectionFactory.java:191)

        at com.itko.lisa.net.ActiveMQFactory.getRemoteQueueConnection(ActiveMQFactory.java:323)

        at com.itko.lisa.net.Transporter.getOrMakeConnection(Transporter.java:427)

        at com.itko.lisa.net.Transporter.init(Transporter.java:398)

        at com.itko.lisa.net.Transporter.<init>(Transporter.java:326)

        at com.itko.lisa.net.Transporter.createTransporter(Transporter.java:285)

        at com.itko.lisa.net.Transporter.getTransporter(Transporter.java:215)

        at com.itko.lisa.net.Transporter.transport(Transporter.java:161)

        at com.itko.lisa.net.Transactor.transact(Transactor.java:68)

        at com.itko.lisa.net.ClientBase.transact(ClientBase.java:361)

        at com.itko.lisa.net.ClientBase.invokeMethod(ClientBase.java:271)

        at com.itko.lisa.net.ClientBase.invokeMethod(ClientBase.java:244)

        at com.itko.lisa.net.CoordinatorServerClient.getCoordinators(CoordinatorServerClient.java:95)

        at com.itko.lisa.coordinator.TestRegistryImpl$1.pulse(TestRegistryImpl.java:1256)

        at com.itko.util.Pulse.execPulse(Pulse.java:521)

        at com.itko.util.Pulse.eventReceipt(Pulse.java:491)

        at com.itko.util.EventThread.execEvents(ThreadedEventSink.java:79)

        at com.itko.util.EventThread.run(ThreadedEventSink.java:59)

2020-07-28 01:12:01,354Z (03:12) [ActiveMQ Transport: tcp://eln004003.corp.telenet.be/10.4.41.243:2011] ERROR System.err                     - Exception in thread "ActiveMQ Transport: tcp://eln004003.corp.telenet.be/10.4.41.243:2011" java.lang.OutOfMemoryError: unable to create new native thread

portal service looks to be running but while launching portal uri, we receive 500 error

Attachment(s)

portal.log   8.87 MB 1 version

coordinator-Portal.log   1.08 MB 1 version

registry.log   2.01 MB 1 version

coordinator-VSE.log   636 KB 1 version

[Sankar Natarajan]

Hi Soumya

          I see entries like this in registry.log file. It looks like you haven't replaced a place holder with actual host name or IP address. PLease check vmoptions files for your Registry server and local/site/lisa properties files. Search for this placeholder  VSE_Hostname and analyze.

getSimulator - simulator not found for tcp://<VSE_Hostname>.corp.telenet.be:2014/Simulator@Default

java.rmi.RemoteException: JMS Messaging problem with ServerRequestHandler for tcp://<VSE_Hostname>:2011/Coordinator Wire format negotiation timeout: peer did not send his wire format.; nested exception 

--

regards

Sankar Natarajan

[SoumyaA]

Hi Shankar,

All the actual machine names have been replaced. To share a file in this platform I have must removed the actual names and replaced with VSE_Hostname and Portal_Hostname

[Sankar Natarajan]

ok got it. Then, please follow the steps below:

1. Stop running dev test services

2. Go to the folder lisatmp folder and locate the amq folder inside it

3. Delete all the contents of amq folder

4. Start DevTest services

Repeat these steps on all machines where you have Devtest components distributed. These steps ensure that there are no residual messages in the underlying AMQ broker used by each of the Devtest components. Let me know how it goes.

--

regards

Sankar Natarajan

[SoumyaA]

Hi Sankar, we followed these steps but got below error now-

2020-07-28 14:28:52,973Z (16:28) [main] ERROR com.itko.lisa.coordinator.VirtualServiceEnvironmentImpl - Cannot register with the DevTest Registry [Timeout (60,000 ms, actual wait 60,000 ms) on server object tcp://<Registry_Host>:2010/Registry on method putVirtualServiceEnvironment]
java.rmi.RemoteException: Timeout (60,000 ms, actual wait 60,000 ms) on server object tcp://<Registry_Host>:2010/Registry on method putVirtualServiceEnvironment
        at com.itko.lisa.net.Transactor.transact(Transactor.java:109)
        at com.itko.lisa.net.ClientBase.transact(ClientBase.java:361)
        at com.itko.lisa.net.ClientBase.invokeMethod(ClientBase.java:271)
        at com.itko.lisa.net.RegistryClient.putVirtualServiceEnvironment(RegistryClient.java:452)
        at com.itko.lisa.coordinator.VirtualServiceEnvironmentImpl.setTestRegistry(VirtualServiceEnvironmentImpl.java:827)
        at com.itko.lisa.coordinator.VirtualServiceEnvironmentImpl.main(VirtualServiceEnvironmentImpl.java:231)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)
        at com.install4j.runtime.launcher.UnixLauncher.main(Unknown Source)
2020-07-28 14:28:52,976Z (16:28) [main] ERROR System.err - Cannot register with the DevTest Registry [Timeout (60,000 ms, actual wait 60,000 ms) on server object tcp://<Registry_Host>:2010/Registry on method putVirtualServiceEnvironment]
2020-07-28 14:28:52,976Z (16:28) [main] ERROR System.err - java.rmi.RemoteException: Timeout (60,000 ms, actual wait 60,000 ms) on server object tcp://eln002741.corp.telenet.be:2010/Registry on method putVirtualServiceEnvironment
2020-07-28 14:28:52,976Z (16:28) [main] ERROR System.err - at com.itko.lisa.net.Transactor.transact(Transactor.java:109)
2020-07-28 14:28:52,976Z (16:28) [main] ERROR System.err - at com.itko.lisa.net.ClientBase.transact(ClientBase.java:361)
2020-07-28 14:28:52,976Z (16:28) [main] ERROR System.err - at com.itko.lisa.net.ClientBase.invokeMethod(ClientBase.java:271)
2020-07-28 14:28:52,976Z (16:28) [main] ERROR System.err - at com.itko.lisa.net.RegistryClient.putVirtualServiceEnvironment(RegistryClient.java:452)

[Vaibhav Jain]

Hi Soumya,

Mentioned error is for timeout between connection between registry and VSE. Make sure all the components are up and running.

Thanks

------------------------------
Regards,
Vaibhav Jain
Capgemini
------------------------------

[SoumyaA]

Hi Shankar, Vaibhav,

Thanks for your reply. Everything seems to be running now, but we can constantly getting this error logged in registry logs and registry is stopping after few hours with error "TCP connections to registry is reaching maximum". Because of this we had to restart the registry daily. Do you know if it is related to something. Attaching complete portal and registry logs.  

Caused by: java.rmi.RemoteException: JMS Messaging problem with ServerRequestHandler for tcp://<VSE Machine>:2014/Simulator Wire format negotiation timeout: peer did not send his wire format.

Caused by: java.rmi.RemoteException: JMS Messaging problem with ServerRequestHandler for tcp://<VSE Machine>:2014/Coordinator Wire format negotiation timeout: peer did not send his wire format.


Appreciate your response

Regards,
Soumya

Attachment(s)

portal.log   5.51 MB 1 version

registry.log   5.15 MB 1 version

[Vaibhav Jain]

Hi Soumya,

The number of active connections to a registry are limited, please check .

Thanks

------------------------------
Regards,
Vaibhav Jain
Capgemini
------------------------------

[Vaibhav Jain]

Hi Soumya,

Are you running multiple monitors in CVS ?

Try stopping the components and clear the temp logs and restart. For memory error you can try using Xmx & Xms as params as well.

Thanks

------------------------------
Regards,
Vaibhav Jain
Capgemini
------------------------------

[SoumyaA]

Hi Vaibhav,

No we are not running any monitors as of now

Regards,
Soumya