Harvest

 View Only
Expand all | Collapse all

Deploying to a remote agent / PEC error

  • 1.  Deploying to a remote agent / PEC error

    Posted Aug 20, 2019 03:20 PM
    We've been using Harvest / ALC ( PureSCM ) and Openmake for a long time. We're in the process of upgraded to the last versions of each on new servers, but have an issue with the current , out of support version.  A server that we've deployed to for years with a remote agent has been moved behind a firewall. We've opened up a firewall rule to allow traffic on port 5400 ( the agent's port) between the target server and the Harvest application server. Other than the firewall, this is nothing new and we have hundreds of applications that deploy in this way. 

    Trying to test a deployment from a package in Harvest to the server, we get one of two results. The Harvest UDP finishes successfully and the log we get back from Openmake just kind of ends like this ----

    Fetching parameters from URL
    Use of uninitialized value $buildRefDirList in string ne at D:/Build Scripts/CASCM/Util.pm line 136.
    Use of uninitialized value $buildProductsList in string ne at D:/Build Scripts/CASCM/Util.pm line 146.
    Build mode not specified - skipping build step
    Check in mode not specified - skipping check in build products step
    Build package clean up mode not specified - skipping step

    When the expected outcome is similar to this -----

    Attempting to log into remote agent...
    Connected
    I00020110: File \pureSCM_Auto\lib\plugins\cascmIntegration.jar;0 checked out to serverxxx\\C:\inetpub\wwwroot\application1\test.jar .
    I00060080: Check out summary: Total: 1 ; Success: 1 ; Failed: 0 ; Not Processed: 0 .
    Checkout has been executed successfully.

    The other result is when the deployment doesn't end successfully and the log ends like this ----

    Executing hco to stage files to serverxxx:C:\inetpub\wwwroot\application1
    connect: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
    I00060040: New connection with Broker harvestserver established.
    Attempting to log into remote agent...
    Unable to connect to the agent [serverxxx] (PEC ec=10060)

    Failed to stage build products

    Any ideas on what could be the issue? or where to look?





  • 2.  RE: Deploying to a remote agent / PEC error

    Broadcom Employee
    Posted Aug 21, 2019 04:28 AM
    Hello David,

    Could you please try the below steps

    A. The reason for PEC ec=10065, is not able to reach the machine or connect to the machine. To check the connectivity we can try below steps
        1. Execute ping to be able to reach out to the agent machine from the machine where you are executing the command.
        2. Execute ping to be able to reach out to the machine where you are executing the command from agent machine.
        [Connectivity test between both the machines]

    If you are able to ping in the above 2 steps and still facing the agent connectivity error, please try to connect to the agent using Fully Qualified Domain Name(FQDN)

    B. The reason for PEC ec=10060, is not able to connect to that particular port. This may be due to agent not running or port got blocked.In this case please verify whether the agent is running properly or the agent running port is not blocked by any other application.

    Please verify this and let us know the outcome.
    You may kindly respond to
    Swapna.akkaladevi@broadcom.com
    Balakrishna.shantamurthy@broadcom.com


    Thanks & Regards
    Swapna.


  • 3.  RE: Deploying to a remote agent / PEC error

    Posted Aug 21, 2019 09:13 AM
    On thing that I can do -- is I can connect to the agent via that port in the Remote Agent Neighborhood on the Harvest server. I always works, every time. So you would think connectivity isn't the issue, unless it needs the port opened from the various other servers in play ---
    Harvest Application Server ( has access to port through the firewall )
    Openmake KB Server ( no rule in place )
    Openmake Remote Build Server ( no rule in place )
    The Target Server

    When using an OM workflow to perform the actual deployment by executing an HCO command, 
    where is the connectivity actually needed?







  • 4.  RE: Deploying to a remote agent / PEC error

    Broadcom Employee
    Posted Aug 21, 2019 10:30 AM
    Edited by Melinda Skelton Aug 21, 2019 10:31 AM
    You might check to see if the port is open for traffic in both directions.

    You could also use one of the following powershell commands from the broker/openmake machine to see if there is any trouble with the port:

    For newer versions of Windows (works on my Windows 2012 R2 test box)

    Test-NetConnection -ComputerName agenthostname -port agentportnumber

    (replace agenthostname with the hostname of the agent machine and replace agentportnumber with the actual port number)

    If you are using an older version of Windows and Powershell where Test-NetConnection isn't available, here is a command that will accomplish the same thing.  It will return an error message or "OK" if connection test was successful.

    $t = New-Object System.Net.Sockets.TcpClient 'agenthostname', agentportnumber; if($t.Connected) {"OK"}

    Let me know if this helps. :)

    Melinda


  • 5.  RE: Deploying to a remote agent / PEC error

    Broadcom Employee
    Posted Aug 21, 2019 11:02 AM
    Actually, the command for the older versions of Windows and Powershell give us more information.

    I ran some tests from my SCM Client machine targeting my SCM broker machine (olnx75) over port 5101.

    1. The first test, broker is up and listening over port 5101 and firewall is down.  Expecting a successful connection:

    PS C:\Users\Administrator> $t = New-Object System.Net.Sockets.TcpClient 'olnx75', 5101; if ($t.Connected) {"OK"}
    OK

    2. The second test, I stopped the broker, so it's no longer listening on port 5101, but the firewall is still down.  This should simulate the 10065 error code:

    PS C:\Users\Administrator> $t = New-Object System.Net.Sockets.TcpClient 'olnx75', 5101; if ($t.Connected) {"OK"}
    New-Object : Exception calling ".ctor" with "2" argument(s): "No connection could be made because the target machine actively refused it 192.168.56.105:5101"
    At line:1 char:6
    + $t = New-Object System.Net.Sockets.TcpClient 'olnx75', 5101; if ($t.Connected) { ...
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : InvalidOperation: (:) [New-Object], MethodInvocationException
    + FullyQualifiedErrorId : ConstructorInvokedThrowException,Microsoft.PowerShell.Commands.NewObjectCommand

    3. The third test, I restarted the broker but put the firewall back up.  So this would simulate a port blockage :

    PS C:\Users\Administrator> $t = New-Object System.Net.Sockets.TcpClient 'olnx75', 5101; if ($t.Connected) {"OK"}
    New-Object : Exception calling ".ctor" with "2" argument(s): "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 192.168.56.105:5101"
    At line:1 char:6
    + $t = New-Object System.Net.Sockets.TcpClient 'olnx75', 5101; if ($t.Connected) { ...
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : InvalidOperation: (:) [New-Object], MethodInvocationException
    + FullyQualifiedErrorId : ConstructorInvokedThrowException,Microsoft.PowerShell.Commands.NewObjectCommand

    The same tests should work with the agent machine as well.  Just substitute the hostname of the agent machine instead of the broker hostname, and substitute the agent's port number instead of 5101.

    Let me know if this helps :)


  • 6.  RE: Deploying to a remote agent / PEC error

    Posted Aug 21, 2019 11:26 AM
    The Test-Connection that I have didn't recognize the -port or -TCPPort, so I ran the other script.
    It returned Ok






  • 7.  RE: Deploying to a remote agent / PEC error

    Broadcom Employee
    Posted Aug 21, 2019 11:55 AM
    Going back to your original message, am I reading correctly that sometimes the connection works and other times it does not?  Or is it more like sometimes you get the early-on "Fetching parameters from URL" error messages, and other times, you don't get those errors, and the build makes it as far as the agent connection and then it fails?  End result is the build fails every time?  Or sometimes the build completely succeeds, and other times it fails?

    If you haven't already, would it be possible to run a stand-alone hco command from a command prompt window on the broker/openmake machine targeting the agent machine?  Does this fail every time?  Or sometimes it succeeds and other times it fails?

    command would look like:

    hco -b brokername -usr harvestuserid -pw harvestpassword -br -en projectname -st statename -vp viewpath -cp clientpath -op pc -replace all -s "*" -rm agenthostname -rport agentportnumber -rusr agentuserid -rpw agentpassword & type hco.log

    Where:
    brokername = harvest broker name
    harvestuserid = harvest login userid (same as you would use for Workbench)
    harvestpassword = harvest password
    projectname = name of your project
    statename = name of your state
    viewpath = pathname of a folder in that project and state
    clientpath = name of a folder on the broker machine to which you can write the files
    agenthostname = hostname of the agent machine
    agentportnumber = port number the agent is listening on
    agentuserid = userid you would use to login to the agent machine
    agentpassword = password for the agent machine

    (Adding "& type hco.log" at the end lets it automatically display the contents of the hco log file.  Saves a step :) )

    Since the broker/openmake machine is the one having trouble connecting to the agent, we need to try this from that machine.

    Let me know if this makes sense.

    Thanks :)


  • 8.  RE: Deploying to a remote agent / PEC error

    Posted Aug 26, 2019 03:14 PM
    Sorry for the delay - the swamp of other projects got in the way 

    When I go to the remote build server and run the hco command as described below --
    I get the error ---

    connect: A connection attempt failed because the connected party did not properly respond after
    a period of time, or established connection failed because connected host has failed to respond.
    I00060040: New connection with broker (harvest broker name) established.
    Attempting to log into remote agent ....
    Unable to connect to the agent [target server name] (PEC ec=10060) 


    SO -- with the remote agent machine being behind a firewall, do I have to open port 5400 on the Harvest broker AND the remote build server? 







  • 9.  RE: Deploying to a remote agent / PEC error

    Broadcom Employee
    Posted Aug 27, 2019 01:45 AM
    Hi David,

    Is 5400 is your agent running port? If yes please open port 5400 on Harvest Agent running machine.

    Thanks & Regards
    Swapna.


  • 10.  RE: Deploying to a remote agent / PEC error

    Posted Aug 27, 2019 11:15 AM
    I can't telnet to that port from the OM RBS , so it is being blocked by something, I have the server admin looking at it for me.







  • 11.  RE: Deploying to a remote agent / PEC error

    Posted Sep 05, 2019 03:17 PM
    It was open between the target server and the Harvest application server .. once 
    they opened it between the target server and the OM remote build server that executes the deployment workflow ( which runs on the application server so I'm not certain of the reason) 
    it will connect..   now we're having the old "permission denied" error 

    I00060040: New connection with Broker HHHHHHH01 established.
    Attempting to log into remote agent ...
    Connected
    E03020028: The file agent error for item TTTTTTTTT\\c:\inetpub\wwwroot\test\scripts\testscript.pl : Permission denied. It might have no access, can't replace read_only file or the file is being used by another process. .
    E03060019: Process Execute failed. Process Name: Check Out for Browse .
    I00060080: Check out summary: Total: 1 ; Sucess: 0 ; Failed: 1 ; Not Processed: 0 .
    Checkout.Execute() failed






  • 12.  RE: Deploying to a remote agent / PEC error
    Best Answer

    Posted Sep 16, 2019 01:21 PM
    We finally narrowed it down. Turning UAC off allows the agent to write to the filesystem.
    Turn it on, and you get permissions errors. 

    Thanks for everyone's suggestions.