For more details, please see ourCookie Policy.


Fibre Channel (SAN)

Reply
Highlighted
Occasional Contributor
Posts: 5
Registered: ‎02-20-2019
Accepted Solution

SAN design considerations

[ Edited ]

Hi,

I'm a beginner SAN administrator and I have a couple questions about design.

Please check our diagram: https://go.gliffy.com/go/share/su5rqq5vuy8lyfyw2k2f

 

  1. At the moment we have only blue links between devices, so like I understand we have 2 independent fabric zones (I added exemplary zones in blue fields  (FABRIC1 and FABRIC2 - correct?)
  2. I want to add red links between SAN switches, so after that, all switches will be in one domain, so our zones should look like in a red filed (FABRIC 3)? If so, whether such architecture it's recommended at all? I understand that the error in the configuration is propagated throughout the whole domain (all switches)?
  3. With only blue links I should see 4 paths on each server. But after adding red links I should see 8 pats on each server. Am I right?

thanks for your answers

Super Contributor
Posts: 286
Registered: ‎04-04-2018

Re: SAN design considerations

[ Edited ]

Hello 

 

 

At the moment we have only blue links between devices, so like I understand we have 2 independent fabric zones (I added exemplary zones in blue fields  (FABRIC1 and FABRIC2 - correct?

With add RED links in tology you will disable redundancy in fabric so in case that one switch will start flooding fabric ALL connection to the storage will be disrupted. Main reason to have two sepparate fabric is that when failure will ocure in first fabric you have twin fabric which will hold production. So this is not good approach at all and this will remove main design of SAN redundancy

 

I want to add red links between SAN switches, so after that, all switches will be in one domain, so our zones should look like in a red filed (FABRIC 3)? If so, whether such architecture it's recommended at all? I understand that the error in the configuration is propagated throughout the whole domain (all switches)?

 

Read Question 1. Yes its true that switches will have same config.

This architecture is not recommended.

 

With only blue links I should see 4 paths on each server. But after adding red links I should see 8 pats on each server. Am I right?

 

Path between server and storage is defined by LUN masing. So in case that you've add another cross ISL links as you've described, the number of path will doesnt change.

 

Could you please explain what you want to achive ? Do you have performance problems in current topology ?

Please mark this response as "Accept as Solution" if it answers your question.

Marian Bezeg
Super Contributor
Posts: 286
Registered: ‎04-04-2018

Re: SAN design considerations

[ Edited ]

Hello,

 

1. With this config you will lost redundancy which is main desing for SAN infrastructure. This is not recommended as with the panic at one switches it may affect all switches in fabric. Based on that you will lose all connections to your storage.

 

2. Config is still same on all switches in fabric, so yes its true that it will be disctributed across all switches. This tolopogy is not recommended check Q1.

 

3. Path to luns are defined by lun mapping. If you will add another ISL between switches as you've described this will be not propagated to the host. You will still see 4 paths.

 

Could you please explain what you would like to achive ? Do you have any performance problems?

 

Please mark this response as "Accept as Solution" if it answers your question.

Marian Bezeg
Occasional Contributor
Posts: 5
Registered: ‎02-20-2019

Re: SAN design considerations

I still do not understand something. So adding links (red ones) will increase redundancy or not? (consequently, all switches will start to see each other and the configuration change will be propagated throughout the network as you wrote).

So it is recommended or not topology? if not, what topology I should create having 4 san switches (all servers are connected to 2 switches and to the rest 2 all storage arrays)*.

 

*each of the connection lines represents one physical cable connections

 

 

Super Contributor
Posts: 286
Registered: ‎04-04-2018

Re: SAN design considerations

[ Edited ]

Hello,

 

so from begining. The main idea of SAN network is to have at least two INDEPENDET networks. So in case that one of the fabric will get panic the second one will hold produciton.

 

In your case you will merge BOTH fabrics together so there is single point of failure. 

If the one switch will got in panic whole fabric will stuck and your storage will be not reachable till recovery.

 

So and now your questions:

If you will add new RED links (connections between switches are caled ISL links) you will merge fabric together. So you will have single point of failure if one switch will got panic. Second thing ist that currently you have two diferent ZONE configs as you have two separate fabrics so if you will cable it like that FABRIC will segment untill both ZONE config will be the same and to be honest this is not always easy job to merge it together.

 

Your described topology ist worst approach which you may choose.

 

In your scenario you may keep its as it is. Currently you have "EDGE-CORE" topology which is mostly recommended. 

So i will it keep as its is, only thing which you may check is if the current ISL connections between switches have enough bandwith to serve storage from array to servers. But if you don't see any performance problems you may keep it as it is.

 

I've attached guide which best praticies for SAN design.

 

 

Please mark this response as "Accept as Solution" if it answers your question.

Marian Bezeg
Super Contributor
Posts: 286
Registered: ‎04-04-2018

Re: SAN design considerations

Hello,

 

so from begining. The main idea of SAN network is to have at least two INDEPENDET networks. So in case that one of the fabric will get panic the second one will hold produciton.

 

In your case you will merge BOTH fabrics together so there is single point of failure. 

If the one switch will got in panic whole fabric will stuck and your storage will be not reachable till recovery.

 

So and now your questions:

If you will add new RED links (connections between switches are caled ISL links) you will merge fabric together. So you will have single point of failure if one switch will got panic. Second thing ist that currently you have two diferent ZONE configs as you have two separate fabrics so if you will cable it like that FABRIC will segment untill both ZONE config will be the same and to be honest this is not always easy job to merge it together.

 

Your described topology ist worst approach which you may choose.

 

In your scenario you may keep its as it is. Currently you have "EDGE-CORE" topology which is mostly recommended. 

So i will it keep as its is, only thing which you may check is if the current ISL connections between switches have enough bandwith to serve storage from array to servers. But if you don't see any performance problems you may keep it as it is.

 

 

 

Please mark this response as "Accept as Solution" if it answers your question.

Marian Bezeg
Occasional Contributor
Posts: 5
Registered: ‎02-20-2019

Re: SAN design considerations

Thanks for the explanation. I still do not know why one switch failure affects the stability of the entire network, but I need to read more about the protocol.

 

My last question: because I have only one physical cable between switches (ISL links), can I easily add another link without any consequences (green one in the diagram:  https://go.gliffy.com/go/share/su5rqq5vuy8lyfyw2k2f) and I understand that it will be okay. Doing it I go back to the idea of having two independent networks as you wrote and therefore I need to continue zoning configuration like an example in blue boxes?

Super Contributor
Posts: 286
Registered: ‎04-04-2018

Re: SAN design considerations

Hello,

 

its not failure of whole switch but situation when the affected switch is flooding all switches in fabric for example when the port is flapping. In case that the port do logoff and login in to the fabric in short period switch floods these "information frames" (CLASS F) to all switches in fabric. These frames have hihgest priority as per FC standard. Based on that you "data frames" will be delivered with latency or discarded as per timeout tresholds. There is an various situation when the one switch may bring whole fabric down.

 

Yes, you can add another link as the ISL. But now depends if you have trunking license installed in to the switch (check ouput of licenseshow command).

If yes, you can add another cable in to the same port group, 0-7, 8-15 -- etc. to form trunk.

The main reason to use trunking is, in case that one of ISLs in trunk group will fail FSPF will not recalculate routes so traffic will be not disrupted.

 

If you don't have trunking licenset, you can have another "basic" ISL. If you have exchange based routing policy enalbed (check outpuy of aptpolicy command) the trafic between ISL will be distribuded by FSPF and DLS. Be sure that DLS will be enabled with losless feature which will prevent frame loss when the E-port go offline or online.

 

For zooning configuration yes you have to define all members of zone per fabric. 

 

 

 

Please mark this response as "Accept as Solution" if it answers your question.

Marian Bezeg
Occasional Contributor
Posts: 5
Registered: ‎02-20-2019

Re: SAN design considerations

Fantastic help. Thanks!

Occasional Contributor
Posts: 5
Registered: ‎02-20-2019

Re: SAN design considerations

Unfortunately, we do not have trunk license. But like I understood adding another ISL I should run DLS?

aptpolicy
Exchange Based Routing Policy

dlsshow
DLS is set with Lossless disabled
E_Port Balance Priority is not set

Marian, You wrote that after adding even one red link, I should not see more paths. But in the test environment after adding one red link (see diagram: https://go.gliffy.com/go/share/su5rqq5vuy8lyfyw2k2f) I got 8 paths (from 4) on ESXi (I did not change anything in the mapping). Of course, as you wrote, the configuration merged.

Join the Broadcom Support Community

Get quick and easy access to valuable resources across the Broadcom Community Network.