Automic Workload Automation

 View Only
Expand all | Collapse all

10053 - An established connection was aborted...

  • 1.  10053 - An established connection was aborted...

    Posted Oct 02, 2019 10:13 AM
    We have some agents in Singapore that were working fine when the servers were on a different domain, but they decided to move these servers onto the same domain that our AE exists and since they migrated them we have been getting the below error every five minutes on the agents when they reconnect.  We have thousands of agents that have no issue, except for these 10 or so servers that were changed.

    I have researched online, I have searched the community, but nothing that seems to point to a cause.  I saw mention about Windows firewall and virus protection, but the agent directory is excluded.

    Does anyone have additional ideas?  



    20191002/215946.614 - U02000042 Connection aborted. Error code '10053', error description: 'An established connection was aborted by the software in your host machine.'.


  • 2.  RE: 10053 - An established connection was aborted...

    Posted Oct 02, 2019 11:12 AM
    ​This message is a true classic.

    It is usually caused by a client firewall or something else sending RST TCP packages and thus, terminating your connections. "Excluding a directory" probably means all kinds of things for virus scanning and anti malware mechanisms, but I doubt this on it's own lets the process communicate through the firewall. Try the connection with "telnet.exe <hostname> <port>", this is the true test. The port is usually 2300 for agent data connections, 8871 for service manager control connections or whatever port your server-side CP uses.

    Beyond that, '10053' has also been seen when you run certain other client software. For instance, Automic has always maintained that they refuse any support if you're running any McAfee Anti Virus component. They insist you uninstall and replace it with something else (though I've shown that disabling McAfee WebIntelligence sufficiently solves the actual issues).

    But in most cases, you'll find that the telnet.exe test shows it's usually a clear-cut firewall or network issue.

    Hth,


  • 3.  RE: 10053 - An established connection was aborted...

    Posted Oct 02, 2019 12:30 PM
    I am able to telnet from the agent server to the AE server using our defined CP ports.  I am able to telnet from our AE server to the agent server using the Service Manager port.

    As for the agent data connections, 2305 in our case, we don't have that open except between servers for file transfers.

    These agents are up and those that have jobs do run, but they just reconnect every five minutes with the error.



  • 4.  RE: 10053 - An established connection was aborted...
    Best Answer

    Posted Oct 02, 2019 12:44 PM
    Are both the server and the agents Windows?

    The only thing I can really think of, since the telnet test works, is looking at all and any software that might interfere. Look for patterns with anything that runs with admin permissions or installs kernel modules, such as anti virus, security suites, lower level network drivers such as vmware drivers etc. - anything that those machines have but not the others?

    If all those agents are now in the same e.g. network segment and that's a pattern in itself, maybe talk to the network people as well, see if they possibly have some security appliance or anything in there that could break up established connections.

    Failing that, you'd either need to try to debug this by enabling TCP=9 traces in the agents (Automic management tab, agent properties) or possibly with tcpdump/wireshark. Or eventually go the ticket route at Automic support.

    Best regards,
    Carsten

    p.s. having a bank holiday here, therefore won't be responding further here now until at least Monday.


  • 5.  RE: 10053 - An established connection was aborted...

    Posted Oct 02, 2019 02:55 PM
    Hi

    just my 2 cents on that.
    It could be a network timeout - of any networking component - you can run a ping to the target server in parallel - do the reconnects persist?
    in theory it could be an entry in INI file (CP list) are they up to date or are orphaned entries in it?

    cheers, Wolfgang

    ------------------------------
    I know I do really know it!
    ------------------------------



  • 6.  RE: 10053 - An established connection was aborted...

    Posted Oct 09, 2019 09:52 AM
    I did run a ping and had no drops.  This is frustrating as in our non-prod system it is causing system trace dumps.


  • 7.  RE: 10053 - An established connection was aborted...

    Posted Oct 06, 2019 05:42 PM

    Hello,

    We had similar issue and ive spent hours and hours trying to figure out what exactly is happening.

    There was no antivirus, not FW, nothing...
    We was running 11.2.9 and those agents was 11.2.3 (we upgraded from 3 to 6 to 9 the AE but not all of the agents).

    The solution for me was to upgrade the agents to the current AE version. 

    What version of AE/Agents are you using?




  • 8.  RE: 10053 - An established connection was aborted...

    Posted Oct 09, 2019 09:52 AM
    These are v12.0.8.


  • 9.  RE: 10053 - An established connection was aborted...

    Posted Nov 25, 2019 09:21 AM
    I just wanted to post that I have this resolved.  Not sure if this was resolved using the best method but I ended up reducing the KEEP_ALIVE setting in the UC_HOSTCHAR_* setting for these Singapore servers.


  • 10.  RE: 10053 - An established connection was aborted...

    Posted May 13, 2020 05:03 AM
    Hi @Jared Kessans,

    I was going through your post, thanks for the resolution on this issue.
    Could you please provide a screenshot how you have set up for specific servers. (providing the server names in Value1, Value2 .... ) 
    How does it affect the other agents.


    Thanks
    Shani


  • 11.  RE: 10053 - An established connection was aborted...

    Posted May 13, 2020 05:09 AM
    > How does it affect the other agents.​

    Why would it? You can create a specific UC_HOSTCHAR class for just the select agents (in OP's case those in Singapur). You don't need to apply this setting to all agents. No need to alter UC_HOSTCHAR_DEFAULT.


  • 12.  RE: 10053 - An established connection was aborted...

    Posted May 13, 2020 06:32 AM
    Carsten is correct.  I just created a separate UC_HOSTCHAR_SING object with the updated settings and then updated UC_EX_HOSTCHAR for each specific agent.  We only have about 15 agents in Singapore so it's not too much of an issue to maintain.