Automic Workload Automation

 View Only
  • 1.  CALLAPI - Not Launching JOBS/JOBP

    Posted Oct 22, 2019 07:44 PM
    Hello,

    We are in a bit of a situation (though not permanent) where we have one of our Linux web servers that has an agent, and we also use CallAPI from this server as well.  There are ondemand JOBS/JOBP executed from various websites on this server which are now not working.  This past weekend, this server has moved to our new data centre, and unfortunately, we are not able to move our Automic server to this new data centre yet as we have to wait for our other major workloads to move and move at this time.

    The problem we are encountering is that when we run the ondemand jobs now, they are not executing and the following error appears (we have tried from command line and from the ondemand links on the web server):
    U00003308 Error while receiving (error code '11'). The connection '10.100.222.159:2217' was terminated.

    We did several tests with support (they have been very helpful) and several packet captures with our network team, however, this is still not working.  I am just wondering if anyone has had this type of issue before and if there is any advice on what we could do (even if it's on the network side, as our team are not seeing any issues coming up).

    Thanks!


  • 2.  RE: CALLAPI - Not Launching JOBS/JOBP

    Posted Oct 23, 2019 02:17 AM
    Hello Tina,

    you can activate traces on the agent and uc4-server to get much more detailled logs written on both sides. I think the call api component will write logs, too.

    We have got many problems with our network in the past and always our network team was saying "we see no problems"... But in all cases it turns out that it was network.

    You can try to telnet from the web-server running the call api to the server. If you get a connection that´s the first part. If not, then you know the team ;-)

    In our environment we had additional problems in our lan firewalls that are terminating our uc4-sessions sometimes as where was some kind of session timeout set that was killing our sessions after a few minutes. Maybe this is a problem on one of the internal firewalls?!

    Best regards,

    Toni

    ------------------------------
    Administrator and Developer Jobautomation
    BNP Paribas S.A. Niederlassung Deutschland
    ------------------------------



  • 3.  RE: CALLAPI - Not Launching JOBS/JOBP

    Posted Oct 23, 2019 02:53 AM
    Hey Tina,

    as an addition to what Toni said here is the concret telnet command

    telnet 10.100.222.159 2217

    You may also check if there is a communication process running on server 10.100.222.159 and port 2217 in Automic (just to be shure).

    Regards, Matthias


  • 4.  RE: CALLAPI - Not Launching JOBS/JOBP
    Best Answer

    Posted Oct 23, 2019 05:47 AM
    Edited by Diane Craddock Oct 23, 2019 09:00 AM

    Hi Tina.

    Traces should work​, you should likely enable a TCP trace at level 9 in the agent properties, then look at the trace files. I do, however, find that network people are sometimes like: "Your trace information means nothing to me". If push comes to shove, this should work:

    These error codes (i.e. error code 11) in Automic messages are always passed straight through from some kernel function. You didn't say what platform your server is, but if it's a Linux, you can attach strace to the call API process. As root:

    strace -f -p <pid_of_call_api>

    Then capture that output until the error happens. Look for the errror message , it will be in the output because that's also printed via kernel function. Then, look up from that in the output and find the kernel function ahead that has returned code 11 (it will have " = 11" at the end). It will probably be some basic TCP function. You can then go to your network people and tell them what operation fails.

    I find "outsiders" can usually parse "the TCP syscall that Linus made ..." more favourably than "that proprietary line in the Automic trace says ...".

    (edit: oh, and also possibly tcpdump/wireshark).

    Hth,