Time taken to copy a 48 GB file from one Linux Agent to Another.

[Mohit__9840]

Hello All,

We are transferring a log file of around 48 GB from one Linux agent to another using the FTP Job of AWA. The time taken for transfer is 26 min,which looks to me very high. When I transfer the same file through SCP it took around 10 Min. Is there any way to speedup the transfer done using FTP Job of automic ?? Some configuration changes or modifications which can help ??

-Many Thanks
     Mohit
 

[Matthias Schelp]

Hi Mohit,

You can try to deactivate the generation of checksums and activate compression in UC_HOSTCHAR_DEFAULT or  UC_HOSTCHAR_*

Just set :

FT_COMPRESS = "Normal" (or better: activate compression via the JOBFs "file transfer" tab)
FT_USE_MD5 = "N"

You need to restart the agent after setting theses options. Be aware that settings made in UC_HOSTCHAR_DEFAULT apply to all filetransfers. MD5 sums check if the file is still consistent after transfer. So deactivating md5 sums may leave you with a bit of a risk. I would acivate compression first and deactivate md5 sums only if performance is still bad.

Matthias

[Anon Anon]

Stop everyone to avoid misunderstanding:

Just clearify: Mohit is talking about FTP, Matthias about common FT (Filetransfer object)

:-)

cheers, Wolfgang

[Matthias Schelp]

Hm,

Quite legitimate objection. Still i'm not shure. He sais "FTP Job of AWA" but he also sais "from one agent to another". 

Mohit__9840 Are you talking about JOBF or real FTP (via RA FTP Agent I suppose)

Mohit said:

from one Linux agent to another using the FTP Job of AWA

[Anon Anon]

If there is no security limitation, firewalls etc. I would use a simple OS job, thats the fastest way (if possible).

[Matthias Schelp]

True

[Legacy User]

If it's the JOBF object you might want to tweak the following values in the agent's ini files:

sendBufferSize =     

The size of the TCP/IP input buffer for the messages that should be sent (in bytes).

The default value that depends on the system environment is used when you do not define this setting or when you define the value 0.

Default: 1024k

recvBufferSize =     

The size of the TCP/IP input buffer for the message that should be received (in bytes).

The default value that depends on the system environment is used when you do not define this setting or when you define the value 0.

Default: 1024k

FileBufferSize =     

Size of the input buffer for files that are transferred with a FileTransfer object .

Bytes are used by default. Alternately, you can also append the letter "K" or kilobytes or "M" for megabytes to the number. You can use uppercase or lowercase letters for this purpose. The OS setting is used if 0 is specified.

Examples (the following definitions are identical):

Specification in bytes: 1048576

Specification in Kbytes: 1024K or 1024k

Specification in Mbytes : 1M or 1m

Default: 0

As well as this parameter in the UC_HOSTCHAR_DEFAULT

FT_ASYNC_QUIT_MAX, 

FT_ASYNC_QUIT_MIN, 

FT_ASYNC_QUIT_NUM     

The settings for synchronous / asynchronous transfers between a sending and a receiving agent of file transfers. Asynchronous transfers are the default setting.

Synchronous transfer: Data packages are only sent when the receipt of sent packages has been confirmed. Sender and receiver adjust their transmission speeds. A synchronous behavior is achieved when the setting FT_ASYNC_QUIT_NUM equals FT_ASYNC_QUIT_MAX.

Asynchronous transfer: Data packages are sent although confirmations are still missing for sent packages. In doing so, data can be transferred faster because a faster sender or a slower connection are used more efficiently. The agents' TCP/IP buffer (INI-file parameter: SendBufferSize= and RecvBufferSize=) should be increased accordingly. 

An asynchronous behavior is achieved when FT_ASYNC_QUIT_MAX is a multiple of FT_ASYNC_QUIT_NUM.

The higher the value of FT_ASYNC_QUIT_MIN, the more the receiving agent will be unburdened when FT_ASYNC_QUIT_MAX has been reached.

These options are only effective in OS/400, NSK, UNIX, Windows and z/OS agents.

FT_ASYNC_QUIT_MAX 

Maximum number of non-confirmed data packages that the sender sends to the receiving agent. 

Minimum value: 1, default value: 60, maximum value: 100

FT_ASYNC_QUIT_MIN 

Number of non-confirmed data packages as of which the sender sends them to the receiver. 

Minimum value: 0, default value: 30, maximum value: 50

FT_ASYNC_QUIT_NUM 

Number of data packages that are confirmed per receipt. 

Minimum value: 1, default value: 5, maximum value: 50 

Note that the value that is specified as the MIN setting must not exceed the MAX setting.

For example:

FT_ASYNC_QUIT_MIN / MAX / NUM: 

0 / 1 / 1: Synchronous file transfer 

0 / 10 / 5: Weak asynchronous transfer. The sending agent may send a maximum of 10 non-confirmed data packages. If the maximum has been reached, it waits until all receipts have been confirmed before it continues the sending process. 

30 / 60 / 5: Strong asynchronous transfer. The sender may send a maximum of 60 non-confirmed data packages to the receiver. Receipts are confirmed per 5 packages. The sender always waits until 30 non-confirmed data packages are open before it starts the data transfer.

[Mohit__9840]

Matthias Schelp said:

Hm,

Quite legitimate objection. Still i'm not shure. He sais "FTP Job of AWA" but he also sais "from one agent to another". 

Mohit__9840 Are you talking about JOBF or real FTP (via RA FTP Agent I suppose)

Mohit said:

from one Linux agent to another using the FTP Job of AWA

Hello Matthias,

I am talking about the JOBF in AWA. We have installed Linux OS agents in 2 different machines and transferring the file from one agent to other.

-Thanks
  Mohit

[Srujan_Pathuri_9871]

Hello Alexander_Trenker_120

We have tweaked the values of both sendBufferSize and recvBufferSize in the agent's INI file. They are set to 60M now and restarted the agent. But still we don't see any difference.

As well as this parameter in the UC_HOSTCHAR_DEFAULT is set as follows by default,which according to you is the strong asynchronous transfer. 

FT_ASYNC_QUIT_MAX : 60

FT_ASYNC_QUIT_MIN : 30

FT_ASYNC_QUIT_NUM : 5

So we still have no clue how could we make this file transfer faster. Please help us in doing any changes in order to achieve our goal.

Many Regards,
Srujan.

[Matthias Schelp]

Mohit__9840

so did you try the file compression I suggested? If not activated allready this should significantly speed up the transfer.

[Legacy User]

The Secure Copy Protocol is designed for transferring data.

The “Automic Automation Engine” Protocol is used for all AE purpose. That means it’s used for communication between all AE Components (Engine, Agent, CallAPI, etc.) and for all different actions (executing tasks like Jobs, File transfers, Events, all User actions done vie the UI and much more).

So it’s not specialized for data transfer only, it’s best optimized for all these different actions. So it’s somehow understandable, that it doesn’t reach the same transfer speed as other pure transfer protocols.

 

Another difference, which is also relevant for the performance, might the data encryption and the data integrity checks. Data code page conversion and compression are another differences, which might influence the transfer speed.