DX NetOps

 View Only
Expand all | Collapse all

CA Spectrum FT-pair database sync performance on Linux install

  • 1.  CA Spectrum FT-pair database sync performance on Linux install

    Broadcom Employee
    Posted Jun 30, 2017 06:47 AM

    Dear all, 

     

    introduced with CA Spectrum R10 and higher the 64bit memory addressing and further CA SpectroSERVER improvements will allow to monitor more devices. Due to this the SpectroSERVER database will increase in size and this bigger SpectroSERVER savefile needs to be copied over to the Secondary_SpectroSERVER.

     

    For Linux hosted Spectrum install you may find, that the "data-transport" performance is bad even the network layer may allow much higher transfer speed (i.e. when using "scp" tool for testing). Find the current Spectrum RCPD performance by output written to Secondary_SpectroSERVERs ./SS/RCPD.OUT file. 

     

    Background for this could be the default TCP socket buffer sizes - which are per default set to smal values. Increasing these to i.e. 512kB will significantly increase the "RCPD"-performance. You may find many advised when searching for this - as a sample we did modify the Linux/RedHat configuration file "/etc/sysctl.conf" to cover/updated to:

     

    net.core.wmem_max=524288
    net.core.rmem_max=524288
    net.ipv4.tcp_rmem= 10240 87380 524288
    net.ipv4.tcp_wmem= 10240 87380 524288

     

    So - just modifying the 524288 value per all 4 entries above to 1048576 (1MB per socket buffer) for a second test 
    - so config then looks like:
    net.core.wmem_max=1048576
    net.core.rmem_max=1048576
    net.ipv4.tcp_rmem= 10240 87380 1048576
    net.ipv4.tcp_wmem= 10240 87380 1048576

     

     

    Procedure for testing is then to:

    • reconfigure the /etc/sysctl.conf and doinig by "root" a "sysctl -p".
    • Then edit on the Secondary_SpectroSERVER edit the ./SS/.rcpdrc file - now increasing the buffer here to
       match per single/multiple to the socket-buffer size - i.e. use multiples of 1048576.

     

    Sample ./SS/.rcpdrc content it then:

    tcp_buffer_size=4194304
    send_buffer_size=4194304

     

    The you can re-trigger an OnLinebackup for testing then. The testing does not require to stop/restart CA Spectrum service or SpectroSERVER - as the configuration at kernel/TCP-level and also the "./SS/.rcpdrc" file are honored dynamically at next time.

     

    Greetings, Joerg 

     

    PS: 

    Do not increase the default socket buffer size too much as this will force the IP-stack to allocate much more memory "for any default TCP-session". You may find advises to increase buffer to 10MByte or higher which we do not recommend. 



  • 2.  Re: CA Spectrum FT-pair database sync performance on Linux install

    Posted Sep 20, 2017 07:33 AM

    for info: default configuration values can be found under  /proc/sys/net/core/*