Verification and Help needed: ISL Setup, 2 Fabrics, 6 Switches, 2 datacenters

[DukeR]

Hi all

Tomorrow I am going to be on site, and I would like to verify once again before ;) I am mainly referring to this article, where I already got some really goog help (thanks again for this).

https://community.brocade.com/t5/Fibre-Channel-SAN-Forums/Connect-Brocade-6505-with-a-Brocade-300-Licensing-and-Design/m-p/96422#M28082

Basically we do have the following.

• Two datacenter
• Distance between datacenters approx.  2 KM
• Connection between datacenters single mode
• SFP type for connection between datacenters (Long Wave 8G)
• SFP for internal ISL in datacenter 1 between fabric members (Short Wave 16G)
• Datacenter 1 Hosts and Storage (failover via datacore)
• Datacenter 2 Hosts and Storage (failover via datacore)

***Datacenter 1***

Existing datacenter with existing hardware and equipment

Fabric A

• Existing 12 Ports 6505 Switch with Short Wave SFP, standard licensing
• A new 12 Port 6505 Switch with Short wave SFP and ONE Long Wave SFP, standard licensing

The reason for the new one is, that we do not have enough ports on the first one so we would like to extend the ports via a new Switch here.

Fabric B

• Existing 12 Ports 6505 Switch with Short Wave SFP , standard licensing
• A new 12 Port 6505 Switch with Short wave SFP and ONE Long Wave SFP, standard licensing

The reason for the new one is, that we do not have enough ports on the first one so we would like to extend the ports via a new Switch here.

***Datacenter 2***

 New datacenter with new hardware and equipment

Fabric A

A new 24 Port 6505 Switch with Short wave SFP and ONE Long Wave SFP,extenden Fabric license

Fabric B

A new 24 Port 6505 Switch with Short wave SFP and ONE Long Wave SFP,extenden Fabric license

******The idea*******

Fabric A

Connecting the existing 6505 in datacenter 1 with the new 6505 in datacenter 1 via Short Wave ISL. Connect the new 6505 in datacenter 1 via Long Wave SFP with the new 6505 in datacenter 2.

Fabric B

Connecting the existing 6505 in datacenter 1 with the new 6505 in datacenter 1 via Short Wave ISL. Connect the new 6505 in datacenter 1 via Long Wave SFP with the new 6505 in datacenter 2.

All new switches will have NO zoning configuration and a different ID at the time i connect them.

I will plug in and extend the fabrics in datacenter 1 first and will check the situation. After that i will connect Fabric A  in datacenter 1 with Fabric A in  datacenter 2 by using the long wave SFP / ISL connection. The same after that for Fabric B.

******Questions******

1) Does this make good sende for you or is there a major issue in this setup or action plan?

2) Is it true that I have to set the speed to 8 Gig on the "local" ISL because the SFPs allow 16G but the ISL between the datacenter uses 8 Gig SFP? If so, how do I do that?

3) Is there any other setting I neeed to configure to make it work?

4) Any other remark or recommendation about layout, design, logic, setup, configuration, licenses?

I would be more than happy if you could quickly give me some feedback on this.

Thank you very much and all the best

PS: Do you have a good article or thread about how to configure long distance setups above 10K? My next setup is going to be such a situation and I have never done before.

[Marian Bezeg · Accepted solution]

******Answers******

1) Does this make good sende for you or is there a major issue in this setup or action plan?

Looks fine for me

2) Is it true that I have to set the speed to 8 Gig on the "local" ISL because the SFPs allow 16G but the ISL between the datacenter uses 8 Gig SFP? If so, how do I do that?

Yes you have to setup it to 8Gig.

portcfgspeed [slot/]port 8

3) Is there any other setting I neeed to configure to make it work?

You have to setup longdistance port on your ports between switches and DWDM.

LD mode is dynamic settings of buffers based on distacne but if you know distance you may use LS mode.

portcfglongdistance [slot/]port LS 1 2

4) Any other remark or recommendation about layout, design, logic, setup, configuration, licenses?

No.

[DukeR]

Hi Marian

Thanks for this great feedback!

I do have some questions with regards to long wave (it is my first long wave configuration).

1. Is there a good guide or thread about theory, configuration and setup
2. Can you explain a little bit about the long wave sitation?
   1. Which distance means LONG distance based on Brocade?
   2. For wich distance do I need to use Long Distance SFP?
   3. There are multiple modes for long distance WITHOUT Extenden Fabric license?
   4. There is an exntenden Fabric License which gives me configuration settings for Buffer and others for which distance do I need to setup this and with which mode?

Last question here: What happens to a switch and its zoninig configuration when I physically unplug it from the Fabric. From my past knowledge the swtich is automatically removed from the Fabric.

When I later on plug it in for example I changed the SFP from Short Wave to Long Wave what happens to the Aliases and configuration? Will it remember its configuration and role?

Or is there a certain way to avoid the loss like exporting config first, setup 2nd ISL first oder anything like this?

Cheers and thanks

Michael

[Antonio Bongiorno T…]

@DukeR

The portCfgLongDistance command supports the following long-distance link modes:
• Normal Mode (L0) — L0 is the normal (default) mode for an E_Port. It configures the E_Port as a standard (not long-distance)
ISL. A total of 20 full-size frame buffers are reserved for data traffic, regardless of the E_Port’s operating speed. The maximum
supported link distance is up to 5 km at 2 Gbps, up to 2 km at 4 Gbps, and up to 1 km at 8, 10, and 16 Gbps.

• Extended Mode (LE) — LE configures the distance for an E_Port when that distance is greater than 5 km and up to 10 km. LE does not require an Extended Fabrics license.

The baseline for the buffer credit calculation is one buffer credit per km at 2 Gbps.
This allocation yields the following values for 10 km:
– 10 buffer credits per port at 2 Gbps
– 20 buffer credits per port at 4 Gbps
– 40 buffer credits per port at 8 Gbps
– 50 buffer credits per port at 10 Gbps
– 80 buffer credits per port at 16 Gbps
– 160 buffer credits per port at 32 Gbps

with MM = SWL SFP+ the Maximum Distance with OM4 cabling is 125 Meter.

If both Datacenter distance exceed 125 Meter you have to use LWL SFP+

[DukeR]

Hi Antonio

Thank you so far. In our case we do have 2x 6505 with 8GB LWL SFP+ modules (1 per switch). The distance between both datacenter is 2 km. Based on your feedback I could use L0 but only with a speed of 4 Gbit, correct?

We would like to have 8 Gbit speed for this 2 km distance do we need to configure LE Mode now and set 40 credits even though you mentioned LE is for ditances greater 5km and up to 10? Or is there another way to get 8Gbit?

You also mentioned no license is required for this, when do I need extenden Farbic license?

What I realized yesterway was strange though. Fabric A Switch 0 iin datacenter 0 s a 12 Port 6505 without any licensens and Fabric A Switch 1 in datacenter is a 24 Port 6505 with the following 

Extenden Fabric license
Fabric Wathc license
Performance Monitor license
Trunking license
Fabric Vision license

When I edited the speed LWL SFP+ on Switch 0 without licenses I could not change any setting except of speed and some basic thing BUT on Switch 1 with all the licenses I was able to change the modes you descriped? Do I get something wrong?

Cheers and thank you very much Antonio

[Antonio Bongiorno T…]

@DukeR

here a simple design, if I've understand you request.....

From DC_A Switch_1 <--SWL_ISL> SWITCH_2 <-----LWL-----> SWITCH_2 <--SWL_ISL--> Switch_1 to DC_B

Both, SWITCHES 2 at both DC port' is Set as LE

at this point it is not required - as per you request Port expand - to ISL inside the Datacenter also in the same Fabric / DC Side with LWL SFP, not matter if both DC are connect trough LWL.

see Table below about allowed distance and License

Buffer Credit Allocation_Distance

Transceiver_Speeds_Cable_Distance_2_4_8_10_16_32_Gbps

[Marian Bezeg]

Hello DukeR,

I've forgotten yesterday that you don't have extended license, sorry for that .

Becuase of that you are allowed to use only LE mode only.

So it's normal behaviour. But its not an show stopper as there is no greather distance you will have enough BB credist between your links. As Antonio mentioned LE with 40BB should cover 8Gb.

About some open questions which i catch:

1.) You need extended fabric license if you want to go over 10KM.

2.) If you will replace SFP there wont be any miss config. Once you will enable ISL again switch will compare both cfg if someone will change zooning during split, fabric will not merge and you will got segmentation. Then you may check via command bellow whats happened and why segmentation occured.

fabstatsshow

To go deeper try to check at link bellow:

https://community.brocade.com/t5/Design-Build/Data-Center-Infrastructure-Storage-Design-Guide-SAN-Distance/ta-p/36627

[DukeR]

Hi both

Great responses and very helpful! Thank you so much, I think I am quite sure now that our design will work as expected. I think the additional settings with regards to buffer credits was to the fact that the new switches in datacnter 2 do have extenden fabric licenses and the ones in datacenter 1 dont have this license so less settings.

I will keep you posted!

Cheers and many thanks