Asynchronous replication in ca directory

[Diligent]

HI All,

 

I am trying to configure dsa's in asynchronous replication mode. Set has been done via command line. In documentaion I found that, if multi-write-group  is different than , dsa replication is not synchronous. Please let me know if any settings are needed for the same.

 

Thanks in Advance,

Ankush

[mcdju01]

Hi Ankush,

 

You are correct, when using text-based configuration files, the multi-write-group item in the "set dsa" command (knowledge) indicates which DSAs will replicate to each other synchronously or asynchronously.

 

In the example below when dsa1 receives an update:

* dsa1 will replicate to dsa2 synchronously as both DSAs are in the multi-write-group "UK"

* dsa1 will replicate to dsa3 asynchronously as DSAs are in different groups, "UK", versus "US"

* dsa1 will only send to one other DSA in the other group (US), therefore, dsa3 will propagate the update to dsa4 as they are in the same group

 

When using DXmanager to configure DSAs, asynchronous replication occurs automatically between regions.

 

dsa1 (UK)	dsa2 (UK)	dsa3 (US)	dsa4 (US)
set dsa "dsa1" = {     prefix       = <o ca>     dsa-name     = <o ca><cn dsa1>     dsa-password = "password"     address = tcp host1 port 10000     disp-psap = DISP     snmp-port = 10000     console-port = 10001     auth-levels  = clear-password     multi-write-group = "UK"     dsa-flags    = multi-write     trust-flags  = allow-check-password };	set dsa "dsa2" = {     prefix       = <o ca>     dsa-name     = <o ca><cn dsa2>     dsa-password = "password"     address = tcp host2 port 10000     disp-psap = DISP     snmp-port = 10000     console-port = 10001     auth-levels  = clear-password     multi-write-group = "UK"     dsa-flags    = multi-write     trust-flags  = allow-check-password };	set dsa "dsa3" = {     prefix       = <o ca>     dsa-name     = <o ca><cn dsa3>     dsa-password = "password"     address = tcp host3 port 10000     disp-psap = DISP     snmp-port = 10000     console-port = 10001     auth-levels  = clear-password     multi-write-group = "US"     dsa-flags    = multi-write     trust-flags  = allow-check-password };	set dsa "dsa4" = {     prefix       = <o ca>     dsa-name     = <o ca><cn dsa4>     dsa-password = "password"     address = tcp host4 port 10000     disp-psap = DISP     snmp-port = 10000     console-port = 10001     auth-levels  = clear-password     multi-write-group = "US"     dsa-flags    = multi-write     trust-flags  = allow-check-password };

[Diligent]

Hi Justin,

 

There is another option that i came across:--

 

multi-write-async : Makes the DSA update asynchronously, even though it is in a multiwrite group. Not sure in which file this needs to be defined , whether in knowledge file ( *.dxc ) or main dsa file in DSAHOME/config/servers (*.dxi ). Also what value should be given to it ( true / false ) or rather how to set it ?

 

Is there any way to print in logs or check what kind of replication is place for two dsa ?

 

Thank & Regards,

Ankush

[mcdju01]

Hi Ankush,

 

Q1: Yes you can set 'multi-write-async' in the DSAs knowledge under the dsa-flags list (this can't be set in a .dxi file). This flag pre-dates multiwrite groups. The main issue with this flag is that if you add it to a DSA, no DSAs will replicate to it synchronously. Not even co-located DSAs. The issue with asynchronous replication between co-located DSAs is that if an application hits DSA1 with an add request, it is possible that the application can receive a response, perform a modify which is directed to DSA2 before the add has been replicated. This causes the subsequent modify to fail.

 

In short, I wouldn't recommend using both asynchronous configuration methods. If using MW groups, stick with that. If you have a single DSA that you don't want to replicate to synchronously then just put it in a group on its own.

 

Q2: Unfortunately, there is no logging to indicate whether 2 DSAs are replicating synchronously or asynchronously. The only way is by inspecting the configuration files.

 

Note: There are other times when synchronous replication can switch to asynchronous mode, for example, multiwrite peer cannot be contacted.

[Diligent]

Hi Justin,

 

What prompted me to look into asynchronous mode is, while going through bookshelf, I found below description :--

 

Administration Guide › Set Up Replication › Multiwrite Replication › How Multiwrite Replication Works › Multiwrite Groups

 

Multiwrite Groups

Multiwrite replication works well if all DSAs are connected with high-speed high-bandwidth links.

If some of your DSAs are connected by latent links, updates to the entire directory will be slower. This is because multiwrite replication is synchronous by default. In synchronous replication, an update operation is not confirmed until all the multiwrite peers have applied it, which provides for loadsharing and failover.

If your backbone includes any slow network connections, you should set up multiwrite groups (or regions if you use DXmanager). DSAs connected by slow links should be in different groups.

Within a multiwrite group, replication is synchronous. Between groups, replication is asynchronous.

 

What method should I use for dsa replication for dsa replication which is best in terms of performance, high availability and scalability ?

 

Thanks in Advance,

Ankush

[mcdju01]

Hi Ankush,

 

There are many parameters involved in answering this question . Typically, CA Services are engaged to evaluate a customer's environment, use cases and make recommendations on configuration, topology, size and data layout.

 

High Availability:

• We would recommend 3 multi-write DSAs (dsa1, dsa2, dsa3) per namespace (prefix) per site (synchronous replication). If a DSAs needs to be taken down for maintenance, having another 2 running provides redundancy.
• For these 3 DSAs run MW-DISP recovery (set multi-write-disp-recovery = true; in *.dxi). This allows DSAs to recover missed updates when taking offline or handle network outages without requiring DR.
• For these 3 DSAs run one or more router DSAs. I would also "set write-precedence = dsa1, dsa2, dsa3;" on the routers to ensure updates are sent to a preferred master. Having a preferred master has many benefits. Routers take care of failover/failback should a DSA/machine need to be stopped.
• For these 3 DSAs set dsp-idle-time to 30 seconds. When machines are shutdown, occasionally the DSA is not notified by the host operating system that the other end of the network connection it is sending to has gone away in a timely fashion (tcp-keepalive). This setting causes the DSA to cut the link if it doesn't receive responses for 30 seconds.

 

Performance: For these 3 DSAs enable load-sharing (dsa-flags) to share the load of search requests across all 3.

 

Scalability/Performance: There are 2 questions with regard to scalability, how many entries do we have now? how many entries are we likely to have in the future? There are 2 ways to scale data. 1) Distributing data across multiple DSAs, 2) Using horizontal partitioning to automatically distribute data across multiple DSAs. Note: Distributing also improves performance as it allows for better write throughput. If you are likely to have more the 10 million entries in the future, then you should consider distribution.

 

Replication Performance: Synchronous/Asynchronous? If your have a number of data centers spread across the state/country/global with applications sharing data from your DSA backbone, then add an additional 3 multi-write DSAs per namespace (prefix) per data center. If the network  between data centres is unreliable or latent then I would put each set of 3 DSAs in there own multi-write-group. That way a client application is stalled waiting for an update to be replicated globally. For each DR you should set up local routers with write-precedence set to these local DSAs (dsa4, dsa5, dsa6). Load sharing can be kept local to a data center by using load-share groups. This prevents the router from sending requests across data centres. For this topology, our recommendation is to use mult-write group hubs. This is a case of marking each preferred master (write-precedence) with "dsa-flags = multi-write-group-hub", there is more information here:  Multiwrite Groups Hubs - CA Directory - 12.0.16 - CA Technologies Documentation

 

In summary

• HA use at least 3 data DSAs service by a routing layer
• Scale using distribution
• Synchronous replication when DSAs are collocated
• Asynchronous replication between Data Centers

 

Thanks,

 

Justin

[Stephan vanBeerschoten]

I am trying to implement async replication to a remote (WAN) site, but as soon as I add multi-write-group to my dxc file, the DSA will die during an init or a start with a syntax error pointing to the line after the multi-write-group line. DXC:

   auth-levels   = clear-password    multi-write-group = "overthere"    dsp-idle-time = 3600    dsa-flags     = multi-write, no-service-while-recovering    trust-flags   = allow-check-password, trust-conveyed-originator    link-flags    = ssl-encryption-remote };

Logs:

[108] 20160929.181400.637 WARN : 'clear schema;' has been disabled as it is not required [108] 20160929.181401.713 ERROR : Syntax Error: Line 1 in  E:\Programs\CA\Directory\dxserver\config\knowledge\<removed>-impd-notify.dxc near 'dsp-idle-time'  Expected a '}'

Any ideas?

[mcdju01]

The knowledge file configuration items are order dependent. Please try moving the multi-write-group definition after dsp-idle.time.

 auth-levels   = clear-password   
 dsp-idle-time = 3600     
 multi-write-group = "overthere"   
 dsa-flags     = multi-write, no-service-while-recovering   
 trust-flags   = allow-check-password, trust-conveyed-originator   
 link-flags    = ssl-encryption-remote };

[Stephan vanBeerschoten]

Awesome. That did it. Thank you.