VSE crashing because of AMQ

[Akshit koul]

Hi All

Our vse is crashing becuase of jvm heapsize issue.Recentlywe did OSupgrade from 2016 to 2019 windows.And after a day we started observing this issue.We then reverted to 2016 but still didnt help.On observing the vse.pf file we figured out 2 amq services which used to work fine before,but now suddenly causing the heap memory utilisation go above 90 5 and this happens as soon as the service is started within 1min.All the other amq services seem to be working fine .Is there any setting or any jar at server or service level, which might have got corrupted?
Also we tested the test connection in workstation it seems to be connecting fine, only when it is deployed the services is causing this issue.Could you please suggest

Thanks
Akshit

[Marcy Nunns]

Akshit,

What version of DevTest are you running?

Can you upload your vse.log file and the log file of your virtual service?

Also what heap size are you specifying for your VSE? (normally would be set in the VirtualServiceEnvironmentService.vmoptions file)

Regards,
Marcy Nunns
Broadcom Technical Support

[Akshit koul]

Hi Marcy

We are using 10.6 version of devtest
And the heap size we have 16 gb max.Unfortunately ,I will not able to attach the log files here , however the case 33125059 i have raised. We could only be able to identify the 2 amq services which has caused this issue but not able to find why it has happened.
Also for the short time i could keep the services online ,i have observed this property in inspection which is very high as compared to other amq.not sureif it is something which is causing the issue.

Regards
Akshit

[Premalatha Gadde]

Hi Akshit,

I would like you to verify the VS's mar files size in DevTest_Home/vseDeploy/VSE_2013 folder on your VSE server. If they are zero bytes, they might have corrupted and caused the issue.

Did you restart VSE after facing the memory issue? If not, I would like you to do the below:
. Stop the VSE.
. Clear all the logs in lisatmp_10.x.x folder. If you have other components running on the same server then delete the logs related VSE, lads and amq folders.
. Restart VSE.
. Redeploy your MQ services which were causing the issue.

Retest the issue and update us. If you still face the issue, please open a support ticket and support will schedule a WebEx and look at the issue.

Meanwhile, share the available vse log files for reference.

Regards,
Prema

[Akshit koul]

Hi Prema @Premalatha Gadde

I checked the mar files and they are of 113 kb each.I tried redeploying with the mar files which were taken before the os upgrade but the issue remained.Also i did a complete restart of server, but as soon as it was up the, vse  went down again.We have portal and registry also on the same server
 Also I tried deleted only the application specific logs, but not sure about the lads folder .
I have raised the support case 33125059 where i have attached the log files , but still not sure why the service is causing this issue.
I checked in the portal for the short time services was up and i could see this property in inspection view which i observed is high as compared to other services.

Regards
Akshit​

[Vaibhav Jain]

Hi Akshit,

Is this observed for the previously created services or with the new ones deployed post upgrade as well?

Thanks

------------------------------
Regards,
Vaibhav Jain
Capgemini
------------------------------

[Akshit koul]

Hi Vaibhav

Actually the issue is seen with only 2 amq services which were working fine before.All the other services seem to be working fine

[Karl Miller]

Akshit, you said these are amq services, what type of AMQ specifically? Are they Redhat AMQ or is this ActiveMQ? What jars do you have in Hotdeploy? Also what are the settings in both the Performance VSE vmoptions file and the local.properties file related to performance. For the settings please refer to https://techdocs.broadcom.com/us/en/ca-enterprise-software/devops/devtest-solutions/10-6/installing/installing-and-configuring-devtest-server/post-installation/sample-performance-benchmarks-for-ibm-mq-service-virtualization.html 

------------------------------
[JobTitle]
[CompanyName]
------------------------------

[Akshit koul]

Hi @Karl Miller
There are the 2 active amq services​ which seem to be causing the issue.The jars which gets redeployed everytime after our server refresh are the below ones however they are placed in lib folder.Also we have around max 16gb memory  reserved for vse in service file.

and also for the short time i could keep the 2 amq running before vse went down i observed very high latency time in the portal which is low for other amqs.Is this an issue with the service config or something wrong with the backend/queues.

[Karl Miller]

Ashkit, probably something wrong with the ActiveMQ service config in the VSE. Check that you have only the activemq-client-5.15.13.ja and do not also have the activemq-all-5.15.13.jar that can cause issues. Make sure also you do not have old jars in the CA\DevTest\hotDeploy\.cache folder.

------------------------------
Karl M.
DevOps Capability Specialist
Broadcom
------------------------------