AutoSys Workload Automation

 View Only
Expand all | Collapse all

Communication attempt with the CA WAAE Application Server has failed

  • 1.  Communication attempt with the CA WAAE Application Server has failed

    Posted Feb 27, 2020 09:18 AM

    Hi all

    We are running an AE server with WCC connected as well as EEM. AE is version 11.3.6 SP8. The landscape was commissioned by CA implementation consultants who have now rolled off the project. 

    Currently the AE server is still up and running and thankfully still running SAP workloads but WCC cannot connect and for example running autoping or autorep on the AE server itself results in the error message below:

    CAUAJM_E_10029 Communication attempt with the CA WAAE Application Server has failed! [stwprdvsaae01.internal.isuzu.co.za:9000]
    CAUAJM_E_10221 Exhausted list of available Application Server(s). Failing request.
    CAUAJM_E_50033 Error initializing tx subsystem: CAUAJM_E_10062 Failed to get initial configuration from CA WAAE Application Server(s).

    I can see the below messages for example in csam.log on the AE server:

    Thu Feb 27 16:14:17.109 WARN CSAM.BROK [4146657600] [4131] [2] processAcceptRequest: - caller [::ffff:10.91.0.18]:50440 wants us to connect to port 9000. No client available
    Thu Feb 27 16:14:22.548 WARN CSAM.BROK [4146657600] [4131] [2] processAcceptRequest: - caller [::ffff:10.91.0.18]:50448 wants us to connect to port 9000. No client available
    Thu Feb 27 16:14:27.986 WARN CSAM.BROK [4146657600] [4131] [2] processAcceptRequest: - caller [::ffff:10.91.0.18]:50456 wants us to connect to port 9000. No client available
    Thu Feb 27 16:14:33.438 WARN CSAM.BROK [4146657600] [4131] [2] processAcceptRequest: - caller [::ffff:10.91.0.18]:50466 wants us to connect to port 9000. No client available
    Thu Feb 27 16:14:34.480 WARN CSAM.BROK [4146657600] [4131] [2] processAcceptRequest: - caller [::ffff:10.91.0.18]:50470 wants us to connect to port 9000. No client available

    In the above example the .18 IP is the WCC server. 

    Any suggestions or ideas? I have opened a support case but to date no working solution was offered. I was directed at article 145355 but I do not think that is relevant. I may be wrong...






    ------------------------------
    Thanks
    Ruan
    ------------------------------


  • 2.  RE: Communication attempt with the CA WAAE Application Server has failed

    Broadcom Employee
    Posted Feb 28, 2020 08:43 AM
    Ruan,

    KB article 145355 is very relevant to the problem. It identifies the common problems related to the connection error you're receiving.

    I assume this was working at one point and something has changed. Given that, let's look at the most common configuration problems. 

    1. CSAM configuration on both client (WCC) and application server machines must be configured the same. This can be reviewed using 'csamconfigedit port=9000 display' command. The EnableSSL and EnablePmux settings are config settings customers most frequently change. Port 9000 assumes that this is port specified in the AutoSys $AUTOUSER/config.$AUTOSERV file for the AutoServerPort setting.
    2. Assuming CSAM EnablePmux=True is set, TCP port 7163 is used for communication, by default. The port used can be confirmed by looking at the global configuration, 'csamconfigedit display'. This port must be open to through the firewall to both machines.
    3. Network routing. Both WCC and App server machines need to be able to connect. This typically can be verified using OS ping command.

    Assuming this was working at one point and the AutoSys/WCC configuration wasn't changed my guess would be this is a firewall issue.

    Please review the KB article and the items listed above to see if this helps.

    Dan Shannon

    ------------------------------
    Broadcom - CA Technologies
    ------------------------------



  • 3.  RE: Communication attempt with the CA WAAE Application Server has failed

    Posted Feb 28, 2020 08:48 AM
    Hi Dan

    Thanks for the response - what I find confusing is that e.g. autorep fails with the same error message locally on the AE server so no firewall etc. 

    Any ideas?


  • 4.  RE: Communication attempt with the CA WAAE Application Server has failed

    Broadcom Employee
    Posted Feb 28, 2020 09:06 AM
    Ruan,

    So running autorep on the machine where the app server is running also fails? Are there any errors in the $AUTOUSER/out/as_server.$AUTOSERV log file? Sounds like the app server may be having problems. The app server is not involved in scheduling.

    ------------------------------
    Broadcom - CA Technologies
    ------------------------------



  • 5.  RE: Communication attempt with the CA WAAE Application Server has failed

    Posted Feb 28, 2020 09:19 AM
    Hi Dan 

     What I see there is

    [02/08/2020 17:38:46] CAUAJM_I_30032 Client [CA WAAE API JNI:32253][13][stwprdvsawe01:49746:10.91.0.18] [0xa8ffde58][02/08/2020 17:38:46.9443][1:S_AUTOSYS_EEM<@stwprdvsaae01.internal.isuzu.co.za> 1] API ID [300] execution completed. Total time: 0.002 seconds.
    ORA-24550: signal received: [si_signo=11] [si_errno=0] [si_code=128] [si_int=0] [si_ptr=(nil)] [si_addr=(nil)]
    kpedbg_dmp_stack()+397<-kpeDbgCrash()+104<-kpeDbgSignalHandler()+107<-skgesig_sigactionHandler()+291<-F66094F6<-_ZN10Safe_0_0_05Cache12unlockEventQEv()+35<-_ZN10Safe_0_0_05Cache13sendoutEventsEv()+603<-_ZN10Safe_0_0_05Cache7evtLoopEv()+172<-evtthreadfunc()+58<-_Z12ithread_funcPv()+59<-start_thread()+216<-clone()+94

    Does not look good...


  • 6.  RE: Communication attempt with the CA WAAE Application Server has failed

    Broadcom Employee
    Posted Feb 28, 2020 09:25 AM
    Ruan,

    I agree. The root cause of the problem appears to be related to accessing the Oracle database.

    ------------------------------
    Broadcom - CA Technologies
    ------------------------------



  • 7.  RE: Communication attempt with the CA WAAE Application Server has failed

    Posted Feb 28, 2020 09:33 AM
    Hi Dan 

    The log ends at that point. Is there a way to restart the app server? I checked and I can connect to the database listener on the database server

    Thanks in advance
    Ruan


  • 8.  RE: Communication attempt with the CA WAAE Application Server has failed
    Best Answer

    Broadcom Employee
    Posted Feb 28, 2020 09:47 AM
    Ruan,

    Since the scheduler is running fine, I would try to restart the application server. It's probably worth looking at Oracle logs to see if any problems were reported too.

    ------------------------------
    Broadcom - CA Technologies
    ------------------------------



  • 9.  RE: Communication attempt with the CA WAAE Application Server has failed

    Posted Feb 28, 2020 09:55 AM
    Thanks Dan

    Is the correct command "unisrvcntr start waae_server.$AUTOSERV"? After running it I see the output below, but as_server.PRD is not updated

    autosys@stwprdvsaae01:/opt/CA/AE/autouser.PRD/out> unisrvcntr start waae_server.$AUTOSERV
    redirecting to systemctl start waae_server.PRD.service
    Executed waae_server.PRD start...................................OK

    Best regards




  • 10.  RE: Communication attempt with the CA WAAE Application Server has failed

    Posted Feb 28, 2020 10:00 AM
    Dan you are the man! 

    I rather ran "unisrvcntr restart waae_server.$AUTOSERV" and that did the trick.

    Appreciate the kind and prompt assistance!



  • 11.  RE: Communication attempt with the CA WAAE Application Server has failed

    Posted Mar 02, 2020 07:17 AM

    Ruan,

     

    Please read what Dan said: look at your MUX settings.. if you turned on security on then its on for everything.

    Make sure your mux settings are correct and make sure that as_server are actually functioning correctly..

    autosyslog -s

     

    Good luck.

     

     

     

     

    Steve C.

     



    Nothing in this message is intended to constitute an electronic signature unless a specific statement to the contrary is included in this message.

    Confidentiality Note: This message is intended only for the person or entity to which it is addressed. It may contain confidential and/or privileged material. Any review, transmission, dissemination or other use, or taking of any action in reliance upon this message by persons or entities other than the intended recipient is prohibited and may be unlawful. If you received this message in error, please contact the sender and delete it from your computer.