ErrorMsg: AwE-5001 Database Query Error

[VineetSharma610459]

​Hi Everyone
While upgrading AM V8.x to AM V9.3 upgrade got stuck on waiting for java awapi to start. Is anyone has Idea how to resolve this error:

ErrorMsg: AwE-5001 Database Query Error (4/6/20 11:02 AM)
Details: getConnection
      null
java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
      at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:743)
      at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666)
      at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
      at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566)
      at java.sql.DriverManager.getConnection(DriverManager.java:664)
      at java.sql.DriverManager.getConnection(DriverManager.java:208)
      at com.appworx.server.sql.AxConnectionPool.getConnection(AxConnectionPool.java:378)
      at com.appworx.server.sql.DBAccess.getConnection(DBAccess.java:1541)
      at com.appworx.server.sql.CommonDB.A(CommonDB.java:565)
      at com.appworx.server.sql.CommonDB.A(CommonDB.java:541)
      at com.appworx.server.sql.CommonDB.B(CommonDB.java:264)
      at com.appworx.server.sql.CommonDB.getTableData(CommonDB.java:413)
      at com.appworx.server.sql.CommonDB.getTableData(CommonDB.java:384)
      at com.appworx.agent.AgentService.getAgents(AgentService.java:1714)
      at com.appworx.agent.AgentStack.fileExists(AgentStack.java:986)
      at com.appworx.agent.AgentService.masterFileExists(AgentService.java:889)
      at com.appworx.agent.AWServer.<init>(AWServer.java:179)
      at com.appworx.agent.ServerStack$3.run(ServerStack.java:358)
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
      at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:470)
      at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:510)
      at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:595)
      at oracle.net.ns.NSProtocol.connect(NSProtocol.java:230)
      at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1439)
      at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:496)
      ... 17 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
      at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
      at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
      at java.net.Socket.connect(Socket.java:589)
      at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:161)
      at oracle.net.nt.ConnOption.connect(ConnOption.java:159)
      at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:428)
      ... 22 more

Thanks
Vineet

[Heidi Schuerger]

Prior to the upgrade, ensure both TNS ping and SQL*Plus connection to the DB from the AppMgr host.
We have seen these errors if there is any disruption of connectivity between the two.

------------------------------
DBA / Sr Enterprise Applications Administrator
Franklin & Marshall College
------------------------------

[Richard_Blumlein_8490]

Hi everyone, 

We had the same issue with the Java api wait, and timeout. The problem is related to the Java you are using..
You have to change the dev statement for the random to urandom. 

We had to change the statement from randome to urandom.

I believe there is another way to do this globally on the system for some service. 

[root@isifp601 security]# pwd
/usr/java/latest/jre/lib/security

[root@isifp601 security]# diff java.security.ORIG java.security
117c117
< securerandom.source=file:/dev/random
---
> securerandom.source=file:/dev/urandom


Look up the rngd services.  I pasted some  information I had on it.  Remember, when you change your Java, you have to 
update the java.security file. I had to do that when I went to Java 8 231. 

Good luck, 

Rich 

man rngd

 

-i     Ignore repeated fips failures.

 

-o    file, Kernel device used to random number output (Default: /dev/random)

 

-r     file, Kernel device used for random number input (Default: /dev/hwrandom)

 

-t    Interval written to random-device when the entropy pool is full.  Note that this option

       exists for backward compatibility in RHEL6, but is ignored. Rngd now correctly waits until

       writing to a random-device will not block.

 

-W  Once  we  start  doing it, feed entropy to random-device until at least fill-watermark bits

       of entropy are available in its entropy pool (default: 2048).  Setting this too high will

       cause rngd to dominate the contents of the entropy  pool. Low values  will  hurt system

       performance during entropy starves. Do not set fill-watermark above the size of the entropy

       pool (usually 4096 bits).

 

 

I found numerous web site which document this same exact setting we used.  

 

EXTRAOPTIONS="-i -o /dev/random -r /dev/urandom -t 10 -W 2048"

 

 

http://blog.hussaindba.com/java-programs-are-running-slower-in-newer-servers-due-to-lack-of-entropy-in-the-system/

 

yum install rng-tools

echo 'EXTRAOPTIONS="-i -o /dev/random -r /dev/urandom -t 10 -W 2048"' > /etc/sysconfig/rngd

chkconfig rngd on

service rngd restart

 

 

 

https://felixcentmerino.wordpress.com/oracle-fusion-middleware/webcenter-portal-content-cluster-infraestructure/installation-java-process/

We can improve your entropy installing the rng-tools

 

Install rng-tools (it should be installed by default)

yum install rng-tools

Modify /etc/sysconfig/rngd file adding

EXTRAOPTIONS="–i –o /dev/random –r /dev/urandom –t 10 –W 2048"

Make rngd start when booting the O.S

sudo chkconfig rngd on

Restart the service

sudo service rngd restart

 

 

 

https://developers.redhat.com/blog/2017/10/05/entropy-rhel-based-cloud-instances/

rngd -r /dev/urandom -o /dev/random

 

 

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/

sect-security_guide-encryption-using_the_random_number_generator

 

Similarly, you can use the -o (or --random-device) option to choose the kernel device for

random-number output (other than the default /dev/random). See the rngd(8) manual page for

a list of all available options.

 

 

 

https://quotidian-ennui.github.io/blog/2012/06/15/slow-java-crypto-performance-on-linux/

 

It's always SecureRandom; if you have a problem with speed during a cryptographic operation you can

point the finger at SecureRandom. It all stems there being not enough randomness in /dev/random; the

virtual machine isn't doing enough network reads or disk accesses or something to generate any more

entropy and it's blocking.

 

Make java actually use /dev/urandom

 

We checked ${java.home}/jre/lib/security/java.security, and as always the securerandom.source property

points to file:/dev/urandom. The notes in the file itself says that the NativePRNG will use /dev/urandom;

however, it lies, lying like a Gregor MacGregor as Bug 6202721 will testify; file:/dev/urandom is treated

as magic, and it ends up using /dev/random regardless. That's pretty frustrating; it's not the fact that

it's magic and it points to /dev/random, it's the fact that the lazy fools haven't even updated the

documentation; it's been like that, for what, 8 years, and they haven't modified the stupid java.security

documentation. Still, all you really have to do is to make the URI not magic anymore.

 

[root@linux ~]# yum install rng-tools

[root@linux ~]# echo 'EXTRAOPTIONS="-i -o /dev/random -r /dev/urandom -t 10 -W 2048"' > /etc/sysconfig/rngd

[root@linux ~]# chkconfig rngd on

[root@linux ~]# service rngd restart

 

The man page for rngd is pretty comprehensive so you can check that for the exact meanings for each parameter.

The only thing that is a must is the -i flag as your source device (/dev/urandom) isn't going to FIPS compliant

and you don't want rngd to terminate unexpectedly.

 

[VineetSharma610459]

Hi Everyone & specially Carmen
This issue has been resolved. I have gone through everyone's response and also Carmen's detail response email and did double check again
1. If database Host name is correct.
2.If I can telnet to database port [on server where we installing Master Agent] [tnsnames.ora entry is best place to look into]
3. Run sqlpus command on server where we installing Master Agent.

Issue I think was database Host name. In our environment we have vtier concept [virtual server]. Which has it's own IP address and on one physical server we can have multiple vtiers. So in DB host name after using vtier server information [after doing telnet to it first] installation went fine. Earlier I think I used main server name itself and that was not right and was only good Master agent Host name.

Thanks again everyone.
Vineet

[Richard_Blumlein_8490]

Hello, 

we ran into the same issue, and the solution is to change the Java you are using to /dev/urandom.
This should fix it. It sure did for us.  When I upgrade the Java I have notes to do this to the new version. 

There is also a service that can make the randomness system wide.  I don't want to complicate this so 
I will just mention it is the "rngd service to increase system entropy"..    I had to do this on one system. 
But for my Applications Manager systems I still use the change to java.security. 

Rich 

[root@isifp601 jdk1.8.0_231-amd64]# pwd
/usr/java/jdk1.8.0_231-amd64

[root@isifp601 jdk1.8.0_231-amd64]# find . -type f | grep java.security
./jre/lib/security/java.security.ORIG
./jre/lib/security/java.security

[root@isifp601 jdk1.8.0_231-amd64]# diff jre/lib/security/java.security.ORIG jre/lib/security/java.security
117c117
< securerandom.source=file:/dev/random
---
> securerandom.source=file:/dev/urandom