Route Through JMS Performance

[Ozan Okan]

Hello All,

in one of our clients, we are migrating from Gateway 8.4 to 9.4.  Requests are routed to TIBCO JMS. Our problem is that, performance decreased around 25% compared to version 8.4. 

Connection Pooling is not an option, customer has more than 150 Queues and connection pooling  creates too many connections and causing TIBCO to hang. 

Using "same" configuration (# of session pools) ,  we are getting around 45 TPS in new environment while old environment can reach 60 TPS easily. 

We tried to replace TIBCO jars with the latest drivers but this did not help as well. 

I would like to know whether someone has faced similar issue.

Thanks in advance.

[Maurizio Garzelli]

Hi Ozan, 
Quick question, is this JSM using SSL/TLS for the transport? The new 9.4 gateway is using a new SSL/TLS library that is FIPS104.2 compliant and is slower than the 'old' BouncyCastle one, so that would explain the slowness in this case.

I hope this helps

Maurizio


Maurizio Garzelli
APIIDA Principal Consultant
https:/apiida.com

[Ozan Okan]

Dear Maurizio,

thanks for the information but we are not using SSL, it is plain tcp connection ( tcp:// ). 

Regards
Ozan Okan AVCI

[Maurizio Garzelli]

thank you for the reply, 
so SSL/TLS s not the issue here, About same issue, I have not personally seen it, but also, I know a customer that went from 8.3 to 9.4 directly and they have JMS towards weblogic, but they did not detect issues at all with that performance part (at least, not yet).

So, configuration-wise, there are NO changes? did you upgrade from 8.4 to 9.4 or did you 'replace' the gateway and simply reimport the database and upgraded?

Thanks

------------------------------
APIIDA Principal Consultant
https://apiida.com
------------------------------

[Ozan Okan]

Dear Maurizio,

it is fresh installation, we migrated services/policies from old environment. As far as I read, JMS send function is synchronized for TIBCO driver, so basically using single connection per destination and synchronized calls we have a bottleneck , which might not be the case for weblogic. I hope you don't face performance problems but in our case, I made several load tests by changing possible parameters ( session count, idle count, jms cluster properties etc. ) in all test cases 8.4 performs better. 

Regards
Ozan Okan AVCI

[Maurizio Garzelli]

It is quite strange indeed, 
but if it is a fresh installation, then I would suggest looking at the old one to see if there are any settings that were set up to help with JMS performance, any clusterwide properties, or in the system.properties, or even in the connection itself, the channel used, you could try and test at an OS level with a ping, curl or trace to see if the times are the same.

I hope this helps

------------------------------
Maurizio Garzelli
APIIDA
APIIDA Principal Consultant
https://apiida.com
------------------------------

[Zhijun He]

Dear Ozan,
Many factors can impact the performance, in your test case,
1. are the 8.4 and 9.4 gateways located at the same LAN and connecting to the same JMS server?
2. what's the specs of the gateway server (gw9.4 requires more cpu/mem than gw8.4, to compare, you can put at least 4 cpu for each gateway, and ensure SWAP is 0 usage on all servers)
3. other factors -- assume that you're using same policy, same configuration of jms conneciton, you have no issue on database, and so on.

Regards,
Mark

[Ozan Okan]

Dear Zhijun,

new server is way stronger than the old one, we are testing new production machine performance against old test machine. ( 24 GB Ram 8 CPU vs 8 GB Ram 2 CPU ) 

However network latency is bigger than the old one. This might be the issue.

Thanks for insight. 
Regards
Ozan Okan AVCI

[Maurizio Garzelli]

Hi Ozan,glad to have been of help, 
That is why I asked you to try pingin, curling or tracert'ing, sometimes the causes are the wildest ones from where one does not expect
Regards, Maurizio

------------------------------
Maurizio Garzelli
APIIDA
APIIDA Principal Consultant
https://apiida.com
------------------------------

[Andreas Krueger]

Hi Ozan, 
we saw that the average CPU usage after migrating to the 9.4 Version were higher as in the 9.1. This may one point, why the JMS throughput decreases. 
One possibility to increase the JMS speed may be to switch from persistent messages to non-persistent messages. In our case we increased the throughput singnificant. But this change may have other implications for your Client in case of an error. 
Another possible idea is to increase the amount of CPU for your Gateway and test again. 
regards
Andreas