AppWorx, Dollar Universe and Sysload Community

 View Only
Expand all | Collapse all

Applications Manager 9.1 Installation - awapi timed out

  • 1.  Applications Manager 9.1 Installation - awapi timed out

    Posted Jun 25, 2020 12:34 PM
    Hi,

    I'm installing Applications Manager 9.1 on Linux connecting to Oracle 12c. I eventually want to upgrade it to AM 9.3 and upgrade Oracle 19c.

    The install seems to go fine but after doing the database stuff, after it starts the AgentService, awapi times out:

    agentservice.lock file is /apps/appworx/status/agentservice.lock
    AgentService pid is 112123
    waiting for java awapi to start

    Error: java awapi timed out.

    netstat shows 1099, 2136 and 5050 aren't being used and the firewall's disabled.

    What could be causing awapi to time out?

    Thanks.


  • 2.  RE: Applications Manager 9.1 Installation - awapi timed out

    Posted Jun 25, 2020 03:27 PM
    John,

    We have a similar issue. After data dump from previous database and re-run install script, the java awapi timed out. We put the keystore files in data directory, We increased java memory. However, the java timed out issue still cannot be solved.


    Regards,

    Peter Wen


  • 3.  RE: Applications Manager 9.1 Installation - awapi timed out

    Posted Jun 25, 2020 06:41 PM
    Hi Peter,

    Thank you for your reply but I'm confused about the keystore files. This AM install is OOTB Applications Manager so the database is completely empty and we've never used any Java keystore files. Or are you saying that because AM 9.1.3 uses JRE 1.6 by default that the JRE 16 certificates have expired?

    Thanks.


  • 4.  RE: Applications Manager 9.1 Installation - awapi timed out

    Posted Jun 25, 2020 06:51 PM
    Hi John,

    In your case, do not worry the keystone files. The keystone files are needed when you upgrade to V9.3. However, please double check with your DBA. I remember the oracle 12c need at least java 7. Maybe I am wrong.

    Regards,

    Peter Wen


  • 5.  RE: Applications Manager 9.1 Installation - awapi timed out

    Posted Jun 26, 2020 09:15 AM
    There are a lot of scenarios that generate the "java awapi timed out" error message.  Look at RmiServer and AgentServer log for the error message that prevents the awapi from starting.

    ------------------------------
    Sr. Delivery Director - Services
    HCL Americas
    ------------------------------



  • 6.  RE: Applications Manager 9.1 Installation - awapi timed out

    Posted Jun 26, 2020 06:47 PM
    Hi Ray,

    I've checked the AgentService and RmiServer logs and the former shows handshake failures errors while the latter shows SSL cipher errors. That leads me to think it's an SSL cipher issue which seems odd because our existing production AM 9.1.3 installation doesn't have the issue. At this point I've tried three different JREs (1.6 from the installation, 1.7 from the existing environment and 1.8 freshly downloaded) and they all fail to start. (I've also adjusted the random/urandom value that Richard suggested below, but also to no avail.)

    I've also turned on maximum debugging in the awenv.ini file and see if that can help shed some light on this.

    Thanks for your help.

    John


  • 7.  RE: Applications Manager 9.1 Installation - awapi timed out

    Posted Jun 27, 2020 06:51 PM
    John,

    For the SSL handshake with cipher suite issue, it could be needed to adjust jdk.tls.server.protocol (jdk.tls.server.protocols=TLSv1). Maybe you can ask your network administrator about how to test the ssl protocol for particular port. (We got help from somebody else. therefore I do not know the detail).

    BTW if you are using jre 1.8, do not using anything above 1.8_201. For our previous AM V9.2. servers, we run successfully at jre 1.8_191  

    Regards,

    Peter Wen



  • 8.  RE: Applications Manager 9.1 Installation - awapi timed out
    Best Answer

    Posted Jun 26, 2020 09:16 AM
    Hi everyone, 
    This is a interesting conversation. When I saw the error about " Error: java awapi timed out." it reminded me of when we got this error. 
    The solution was to change /dev/random to /dev/urandom in the java.security file.  Find this file for the Java you are using for the master/agent systems 
    and change it. It fixed it nicely. 

    I know the keys have to be in place, but I don't remember getting the Java awapi, but this is worth a shot. The keys had me going, but I finally got it working. 

    Rich 

    securerandom.source=file:/dev/random

     To

     securerandom.source=file:/dev/urandom

    I think there is also a way to change the system entropy for all Java on the system. It's been a while but I helped the DBA's use the rngd service. 

     This is from some of the documentation I had for the rngd service. 


    man rngd

     

    -i     Ignore repeated fips failures.

     

    -o    file, Kernel device used to random number output (Default: /dev/random)

     

    -r     file, Kernel device used for random number input (Default: /dev/hwrandom)

     

    -t    Interval written to random-device when the entropy pool is full.  Note that this option

           exists for backward compatibility in RHEL6, but is ignored. Rngd now correctly waits until

           writing to a random-device will not block.

     

    -W  Once  we  start  doing it, feed entropy to random-device until at least fill-watermark bits

           of entropy are available in its entropy pool (default: 2048).  Setting this too high will

           cause rngd to dominate the contents of the entropy  pool. Low values  will  hurt system

           performance during entropy starves. Do not set fill-watermark above the size of the entropy

           pool (usually 4096 bits).

     

     

    I found numerous web site which document this same exact setting we used.  

     

    EXTRAOPTIONS="-i -o /dev/random -r /dev/urandom -t 10 -W 2048"

     

     

    http://blog.hussaindba.com/java-programs-are-running-slower-in-newer-servers-due-to-lack-of-entropy-in-the-system/

     

    yum install rng-tools

    echo 'EXTRAOPTIONS="-i -o /dev/random -r /dev/urandom -t 10 -W 2048"' > /etc/sysconfig/rngd

    chkconfig rngd on

    service rngd restart

     

     

     

    https://felixcentmerino.wordpress.com/oracle-fusion-middleware/webcenter-portal-content-cluster-infraestructure/installation-java-process/

    We can improve your entropy installing the rng-tools

     

    Install rng-tools (it should be installed by default)

    yum install rng-tools

    Modify /etc/sysconfig/rngd file adding

    EXTRAOPTIONS="–i –o /dev/random –r /dev/urandom –t 10 –W 2048"

    Make rngd start when booting the O.S

    sudo chkconfig rngd on

    Restart the service

    sudo service rngd restart

     

     

     

    https://developers.redhat.com/blog/2017/10/05/entropy-rhel-based-cloud-instances/

    rngd -r /dev/urandom -o /dev/random

     

     

    https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/

    sect-security_guide-encryption-using_the_random_number_generator

     

    Similarly, you can use the -o (or --random-device) option to choose the kernel device for

    random-number output (other than the default /dev/random). See the rngd(8) manual page for

    a list of all available options.

     

     

     

    https://quotidian-ennui.github.io/blog/2012/06/15/slow-java-crypto-performance-on-linux/

     

    It's always SecureRandom; if you have a problem with speed during a cryptographic operation you can

    point the finger at SecureRandom. It all stems there being not enough randomness in /dev/random; the

    virtual machine isn't doing enough network reads or disk accesses or something to generate any more

    entropy and it's blocking.

     

    Make java actually use /dev/urandom

     

    We checked ${java.home}/jre/lib/security/java.security, and as always the securerandom.source property

    points to file:/dev/urandom. The notes in the file itself says that the NativePRNG will use /dev/urandom;

    however, it lies, lying like a Gregor MacGregor as Bug 6202721 will testify; file:/dev/urandom is treated

    as magic, and it ends up using /dev/random regardless. That's pretty frustrating; it's not the fact that

    it's magic and it points to /dev/random, it's the fact that the lazy fools haven't even updated the

    documentation; it's been like that, for what, 8 years, and they haven't modified the stupid java.security

    documentation. Still, all you really have to do is to make the URI not magic anymore.

     

    [root@linux ~]# yum install rng-tools

    [root@linux ~]# echo 'EXTRAOPTIONS="-i -o /dev/random -r /dev/urandom -t 10 -W 2048"' > /etc/sysconfig/rngd

    [root@linux ~]# chkconfig rngd on

    [root@linux ~]# service rngd restart

     

    The man page for rngd is pretty comprehensive so you can check that for the exact meanings for each parameter.

    The only thing that is a must is the -i flag as your source device (/dev/urandom) isn't going to FIPS compliant

    and you don't want rngd to terminate unexpectedly.







  • 9.  RE: Applications Manager 9.1 Installation - awapi timed out

    Posted Jun 27, 2020 06:57 PM
    Rich,

    Thank you for your post. I will try this method on Monday. I will update after that. Thanks again.

    Regards,

    Peter Wen


  • 10.  RE: Applications Manager 9.1 Installation - awapi timed out

    Posted Jun 30, 2020 03:29 PM
    Hi Rich,

    We implemented the change java security method. After DB bounce, the awapi timed out issue go away. Thanks for help.


    Regards,

    Peter Wen


  • 11.  RE: Applications Manager 9.1 Installation - awapi timed out

    Posted Jun 30, 2020 03:47 PM
    Hi John,   That is really nice to hear.  We hit that issue after one of the kernel patches that went on. 
    And changing /dev/random to /dev/urandom in the java.security file fixed it. 

    Great, 

    Rich