Brocade Fibre Channel Networking Community

Expand all | Collapse all

FCIP Tunnel up no connection: NPVID -> E -> EX ->VEX ---VEX <-EX <-E <- NPVID

  • 1.  FCIP Tunnel up no connection: NPVID -> E -> EX ->VEX ---VEX <-EX <-E <- NPVID

    Posted 07-14-2013 06:05 PM

    I have a POC I am trying to setup and am not seeing the issue in what I have.  The connection is a bit "odd" I have a pair of Brocade 8Gb 24 port switches at each "site" (physically linked with 1Gb CAT 5 cables between two racks) with a blade chassis of servers which the customer purchased QLogic FC switches so I put them into NPV mode.  Zones within site work fine.  I have implemented best practices of seperate "A" and "B" fabrics (and for sake of example I call them by colors "red", "blue", "green", "yellow") 

    Now I have a storage controller attached directly to the 24port Brocade SAN switches (two controllers with two connections to each switch). I want to link primary site "PSCFC01"  storage ports to remote site "RSCFC01" storage so I can do SAN replication. (and then repeat to join the "PSCFC02" storage to "RSCFC02").

    I have the tunnels up and linked and can ping from both directions..

    I created LSAN zones which contain the cross connected site WWPNs from each storage controller.  I don't think I have ever done single FCR at each site with expectation to set tunnel to join "GE0" -> Red / blue SANs    and  "GE1" -> yellow and green SANs.  I think their is some step I am missing to set link between the EX to VEX association?

    I am likely missing something simple.  It has been a while sense I did FCIP...

    FCIP_Diagram.png

    Any help and comments would be appreciated.    

    ***************************

    Draft Lab Guide

    Fibre Channel Router

    Port layout

    Port

    Switch

    Domain ID

    Mode

    0

    lcpscfc01-00

    101

    EX

    1

    lcpscfc02-00

    102

    EX

    2

    3

    16

    lcrscfc03-ge0*

    121

    VEX

    17

    lcrscfc03-ge1*

    122

    VEX

    18

    19

    20

    21

    * Make sure the ports are in the copper (default) ports, if optical is desired mode must be changed.

    Step 1: Activate License Keys for FCIP and Routing

    Step 2: Login to fibre router and set up port

    Select fc port and rename port description

    Set mode on port to EX (SAN Routed Port) Set port to "EX-Port" with matching fabric ID to Native mode.

    Now enable the "Gi" ports for VEX mode for ports 16, 17 which will bridge FCIP to remote switch.

    Choose switch and "FCR"

    Set the FCR ID by choosing "Set Fabric ID" (switch will have to be set to "disabled" to change this ID) then select "Enable FCR"  and set Backbone ID to 20

    (For lcrscfc03 site, ID will be 20,21,22,23,24  while lcpscfc03 site IDs will be 10,11,12,13,14)

    Choose "VE/Vex Ports" and choose the first Ethernet port and set to VEX and set Domain ID

    Port Summary of descriptions (upstream connections) as well as Fabric Domain IDs

    Step 2:   Setup Zones Aliase and routed Zone

    Under Zone Admin

    Create aliases for each WWPN. Example is for "lcrscv70001" system Canister 1, port 1

    Now create zone for storage

    ** Repeat above step exactly but this time name zone "LSAN_******".  This creates a Zone that can be routed.

    Now add both zones created to the zone database, save to switches

    Then activate by clicking "Enable Config" and then select desired version and click ok

    Step 3:   Setup FCR IP and routes on Gigibit interfaces

    Example: lcrscfc01 with below WAN IP mapping

    PSC Ports

    RSC Ports

    lcpscfc03w1

    1. 10.101.42.1

    0/16 (ge0)

    lcpscfc03w2

    1. 10.101.42.2

    0/17 (ge1)

    lcrscfc03w1

    1. 10.101.42.3

    0/16 (ge0)

    lcrscfc03w1

    1. 10.101.42.4

    0/17 (ge1)

    Display mapping of ports: portcfgpersistentenable

    portcfgpersistentenable SlotO 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+--- Enabled YES YES YES YES YES YES YES YES YES YES YES YES YES YES YES YES Slot 0 16 17 18 19 20 21 22 23 +---+---+---+---+---+---+---+--- Enabled YES YES YES YES YES YES YES YES Slot O geO gel ge2 ge3 ge4 ges Enabled YES YES YES YES YES YES" height="192" width="594">

    This sets IP to port "16" which is the first SFP based Ethernet Port.

    lcrscfc03:admin> portshow fciptunnel all

    No FCIP Tunnels found

    lcrscfc03:admin> portshow 0/16

    portIndex:  16

    portName: lcpscfc03-16

    portHealth: No Fabric Watch License

    Authentication: None

    EX_Port Mode:   Enabled

    Fabric ID:      23

    Front Phantom:  state = Not OK  Pref Dom ID: 160

    Fabric params:  R_A_TOV: 0      E_D_TOV: 0      PID fmt: auto

    Authentication Type: None

    Hash Algorithm: N/A

    DH Group: N/A

    Edge fabric's primary wwn: N/A

    Edge fabric's version stamp: N/A

    portDisableReason: None

    portCFlags: 0x0

    portFlags: 0x4021        PRESENT VIRTUAL U_PORT EX_PORT DISABLED LED

    LocalSwcFlags: 0x0

    portType:  12.0

    portState: 2    Offline

    Protocol: FC

    portPhys:  255 N/A     portScn:   2 Offline

    port generation number:    0

    state transition count:    2

    portId:    cb1000

    portIfId:    4302002a

    portWwn:   20:10:00:27:f8:4a:7e:e4

    portWwn of device(s) connected:

    Distance:  normal

    Port part of other ADs: No

    lcpscfc03

    lcrscfc03

    # Disable port then bind IP and set Gateway interface

    lcpscfc03:admin> portdisable ge0

    lcpscfc03:admin> portcfg ipif ge0 create 10.101.42.1 255.255.255.0 1500

    Operation Succeeded

    lcpscfc03:admin> portcfg iproute ge0 create 10.101.42.0 255.255.255.0 10.101.42.230

    Route is an interface subnet

    # Create FCIP tunnel from psc to rsc @ 512MB and 1Gb max rate

    lcpscfc03:admin> portcfg fciptunnel 16 create 10.101.42.3 10.101.42.1 --min-comm-rate 512000   --max-comm-rate 1000000 -c 3 -d  lcpscfc03-ge0_lcrscfc03-ge0

    # Re-enable interfaces

    lcpscfc03:admin> portenable ge0

    lcpscfc03:admin> portcfgpersistentenable ge0

    lcpscfc03:admin> portenable 16

    lcpscfc03:admin> portcfgpersistentenable 16

    # Test via ping to remote site

    lcrscfc03:admin> portcmd --ping ge0 -s 10.101.42.1 -d 10.101.42.3

    # Display FCIP tunnel

    lcpscfc03:admin> portshow fciptunnel all

    -------------------------------------------------------------------------------

    Tunnel Circuit  OpStatus Flags    Uptime TxMBps  RxMBps ConnCnt CommRt  Met

    -------------------------------------------------------------------------------

    16 -         Up      a------     1m5s    0.00 0.00    3      - -

    -------------------------------------------------------------------------------

    Flags: tunnel: c=compression m=moderate compression a=aggressive compression

                     A=Auto compression f=fastwrite t=Tapepipelining F=FICON

                     T=TPerf i=IPSec l=IPSec Legacy

    # Disable port then bind IP and set Gateway interface

    lcrscfc03:admin> portdisable ge0

    lcrscfc03:admin> portcfg ipif ge0 create 10.101.42.3 255.255.255.0 1500

    Operation Succeeded

    lcrscfc03:admin> portcfg iproute ge0 create 10.101.42.0 255.255.255.0 10.101.42.230

    Route is an interface subnet

    # Create FCIP tunnel from psc to rsc @ 512MB and 1Gb max rate

    lcrscfc03:admin> portcfg fciptunnel 16 create 10.101.42.1 10.101.42.3 --min-comm-rate 512000   --max-comm-rate 1000000 -c 3 -d  lcpscfc03-ge0_lcrscfc03-ge0

    # Re-enable interfaces

    lcrscfc03:admin> portenable ge0

    lcrscfc03:admin> portcfgpersistentenable ge0

    # Test via ping to remote site

    lcrscfc03:admin> portcmd --ping ge0 -s 10.101.42.3 -d 10.101.42.1

    # Display FCIP tunnel

    lcrscfc03:admin> portenable 16

    lcrscfc03:admin> portshow fciptunnel all

    -------------------------------------------------------------------------------

    Tunnel Circuit  OpStatus Flags    Uptime TxMBps  RxMBps ConnCnt CommRt  Met

    -------------------------------------------------------------------------------

    16 -         InProg a------    1m59s 0.00    0.00    2 -      -

    -------------------------------------------------------------------------------

    Flags: tunnel: c=compression m=moderate compression a=aggressive compression

                     A=Auto compression f=fastwrite t=Tapepipelining F=FICON

                     T=TPerf i=IPSec l=IPSec Legacy

    Research Frame support for site to support jumbo frames across WAN: http://wikibon.org/wiki/v/How_to_configure_FCIP_on_Brocade_7500%27s

    LC-Router >>portcfg ipif ge0 create 10.255.2.3 255.255.255.0 2250

    2250 is the MTU size (2148, is FC Frame size)

    Repeat of above but this time to bind ge1 and ge1

    lcpscfc03

    lcrscfc03

    # Disable port then bind IP and set Gateway interface

    lcpscfc03:admin> portdisable ge1

    lcpscfc03:admin> portcfg ipif ge1 create 10.101.42.2 255.255.255.0 1500

    Operation Succeeded

    lcpscfc03:admin> portcfg iproute ge1 create 10.101.42.0 255.255.255.0 10.101.42.230

    Route is an interface subnet

    # Create FCIP tunnel from psc to rsc @ 512MB and 1Gb max rate

    lcpscfc03:admin> portcfg fciptunnel 17 create 10.101.42.4 10.101.42.2 --min-comm-rate 512000   --max-comm-rate 1000000 -c 3 -d  lcpscfc03-ge1_lcrscfc03-ge1

    # Re-enable interfaces

    lcpscfc03:admin> portenable ge0

    lcpscfc03:admin> portcfgpersistentenable ge0

    lcpscfc03:admin> portenable 16

    lcpscfc03:admin> portcfgpersistentenable 16

    # Test via ping to remote site

    lcrscfc03:admin> portcmd --ping ge1 -s 10.101.42.2 -d 10.101.42.4

    # Display FCIP tunnel

    lcpscfc03:admin> portshow fciptunnel all

    -------------------------------------------------------------------------------

    Tunnel Circuit  OpStatus Flags    Uptime TxMBps  RxMBps ConnCnt CommRt  Met

    -------------------------------------------------------------------------------

    16 -         Up      a------       3s 0.00    0.00    5 -      -

    17 -         Up      a------    1m12s    0.00 0.00    1      - -

    -------------------------------------------------------------------------------

    Flags: tunnel: c=compression m=moderate compression a=aggressive compression

                     A=Auto compression f=fastwrite t=Tapepipelining F=FICON

                     T=TPerf i=IPSec l=IPSec Legacy

    # Disable port then bind IP and set Gateway interface

    lcrscfc03:admin> portdisable ge1

    lcrscfc03:admin> portcfg ipif ge1 create 10.101.42.4 255.255.255.0 1500

    Operation Succeeded

    lcrscfc03:admin> portcfg iproute ge1 create 10.101.42.0 255.255.255.0 10.101.42.230

    Route is an interface subnet

    # Create FCIP tunnel from psc to rsc @ 512MB and 1Gb max rate

    lcrscfc03:admin> portcfg fciptunnel 17 create 10.101.42.2 10.101.42.4 --min-comm-rate 512000   --max-comm-rate 1000000 -c 3 -d  lcpscfc03-ge1_lcrscfc03-ge1

    # Re-enable interfaces

    lcrscfc03:admin> portenable ge1

    lcrscfc03:admin> portcfgpersistentenable ge1

    # Test via ping to remote site

    lcrscfc03:admin> portcmd --ping ge1 -s 10.101.42.4 -d 10.101.42.2

    # Display FCIP tunnel

    lcrscfc03:admin> portenable 16

    lcrscfc03:admin> portshow fciptunnel all

    lcrscfc03:admin> portshow fciptunnel all

    -------------------------------------------------------------------------------

    Tunnel Circuit  OpStatus Flags    Uptime TxMBps  RxMBps ConnCnt CommRt  Met

    -------------------------------------------------------------------------------

    16 -         Up      a------       9s 0.00    0.00    5 -      -

    17 -         Up      a------    1m18s    0.00 0.00    1      - -

    -------------------------------------------------------------------------------

    Flags: tunnel: c=compression m=moderate compression a=aggressive compression

                     A=Auto compression f=fastwrite t=Tapepipelining F=FICON

                     T=TPerf i=IPSec l=IPSec Legacy

    Step 4: Now we create a zone for v7000s across sites

    The router setup is to route between the WWP connections of the connected sites per table below.

    Zones Routed

    lcpscfc01_lcrscfc01

    lcpscfc02_lcrscfc02

      (50:05:07:68:02:35:b7:83) IBM Corp

      (50:05:07:68:02:45:b7:83) IBM Corp

      (50:05:07:68:02:15:b7:84) IBM Corp

      (50:05:07:68:02:25:b7:84) IBM Corp

      (50:05:07:68:02:35:b7:ab) IBM Corp

      (50:05:07:68:02:45:b7:ab) IBM Corp

      (50:05:07:68:02:15:b7:ac) IBM Corp

      (50:05:07:68:02:25:b7:ac) IBM Corp

      (50:05:07:68:02:15:b7:83) IBM Corp

      (50:05:07:68:02:25:b7:83) IBM Corp

      (50:05:07:68:02:35:b7:84) IBM Corp

      (50:05:07:68:02:45:b7:84) IBM Corp

      (50:05:07:68:02:15:b7:ab) IBM Corp

      (50:05:07:68:02:25:b7:ab) IBM Corp

      (50:05:07:68:02:35:b7:ac) IBM Corp

      (50:05:07:68:02:45:b7:ac) IBM Corp

    The above table will be created on each switch as a LSAN (routed) zone.  Below are the commands for each of the four switch sides.

    lcpscfc01

    lcpscfc02

    lcrscfc01

    lcrscfc02

    Step 5: Now with the use of the Brocade Network Assistant GUI, we can quickly add switches and setup routed zones between the fabrics

    Install 70day evaluation version of Brocade Network Advisor

    na1202_brocade_windows

    Username

    Password

    Administrator

    password

    Launch GUI login

    Click to add new device

    Input each of the six FC switches

    Note that the three switches of each site are diagramed as being in a single connection

    Now we can create a routed zone between the two sites for the v7000 controllers

    Create LSAN zone

    2)Typeinzone 4: 141 Ibi ÇIIP_New Zone New Member... OK Cancel Help Potential Members All Levels El lcrsctcO3 S 4 lcrsctcOl El lcrsctcO2 4 Type VMsN Name Product Type Class lcrsctcO3 lcrsctcOl lcrsctco2 site vl000s ¿3 Activate... :h la" height="374" border="0" width="576">

    CLI LSAN zoning

    lcpscfc03_lcpscfc01

    lcpscfc03_lcpscfc2

    lcrscfc03_lcrscfc01

    lcrscfc03_lcrscfc2


    #BrocadeFibreChannelNetworkingCommunity


  • 2.  Re: FCIP Tunnel up no connection: NPVID -> E -> EX ->VEX ---VEX <-EX <-E <- NPVID

    Posted 07-15-2013 12:01 AM

    have you created ONE or TWO LSAN config ?

    I'm not sure, but i think such config you want/have created can not work with more than one LSAN


    #BrocadeFibreChannelNetworkingCommunity


  • 3.  Re: FCIP Tunnel up no connection: NPVID -> E -> EX ->VEX ---VEX <-EX <-E <- NPVID

    Posted 07-15-2013 04:59 AM

    Yes there is / will be two VSAN segments.

    The LSAN zone for the storage will be the interfaces in an LSAN that have "pscfc01" and "rscfc01" controller ports.  And a seperate LSAN containing controller ports of interfaces in "pscfc02" and "rscfc02".

    Example attached of Aliase / WWPN from switch where I manually added the cross WWPN IDs to the LSAN_ zone

    pscfc01rscfc01

    zone:  lcpscv70001

                    lcpscv70001_1_3; lcpscv70001_1_4; lcpscv70001_2_1;

                    lcpscv70001_2_2

    zone:  LSAN_lcpscv70001_lcrscv70001

                    50:05:07:68:02:35:b7:83

                    50:05:07:68:02:45:b7:83

                    50:05:07:68:02:15:b7:84

                    50:05:07:68:02:25:b7:84

                    50:05:07:68:02:35:b7:ab

                    50:05:07:68:02:45:b7:ab

                    50:05:07:68:02:15:b7:ac

                    50:05:07:68:02:25:b7:ac

    zone:  lcrscv70001

                    lcrscv70001_1_3; lcrscv70001_1_4; lcrscv70001_2_1;

                    lcrscv70001_2_2

    zone:  LSAN_lcrscv70001

                    50:05:07:68:02:35:b7:83

                    50:05:07:68:02:45:b7:83

                    50:05:07:68:02:15:b7:84

                    50:05:07:68:02:25:b7:84

                    50:05:07:68:02:35:b7:ab

                    50:05:07:68:02:45:b7:ab

                    50:05:07:68:02:15:b7:ac

                    50:05:07:68:02:25:b7:ac

    alias: lcpscv70001_1_3

                    50:05:07:68:02:35:b7:83

    alias: lcpscv70001_1_4

                    50:05:07:68:02:45:b7:83

    alias: lcpscv70001_2_1

                    50:05:07:68:02:15:b7:84

    alias: lcpscv70001_2_2

                    50:05:07:68:02:25:b7:84

    alias: lcrscv70001_1_3

                    50:05:07:68:02:35:b7:ab

    alias: lcrscv70001_1_4

                    50:05:07:68:02:45:b7:ab

    alias: lcrscv70001_2_1

                    50:05:07:68:02:15:b7:ac

    alias: lcrscv70001_2_2

                    50:05:07:68:02:25:b7:ac

    alias: lcpscv70001_1_3

                    50:05:07:68:02:35:b7:83

    alias: lcpscv70001_1_4

                    50:05:07:68:02:45:b7:83

    alias: lcpscv70001_2_1

                    50:05:07:68:02:15:b7:84

    alias: lcpscv70001_2_2

                    50:05:07:68:02:25:b7:84

    alias: lcrscv70001_1_3

                    50:05:07:68:02:35:b7:ab

    alias: lcrscv70001_1_4

                    50:05:07:68:02:45:b7:ab

    alias: lcrscv70001_2_1

                    50:05:07:68:02:15:b7:ac

    alias: lcrscv70001_2_2

                    50:05:07:68:02:25:b7:ac

    What I don't follow is if their is any other step, besides creating the LSAN,  to join the two zones over FCR.


    #BrocadeFibreChannelNetworkingCommunity


  • 4.  Re: FCIP Tunnel up no connection: NPVID -> E -> EX ->VEX ---VEX <-EX <-E <- NPVID

    Posted 07-15-2013 01:31 PM

    I don't think you can create a Routing with/between two LSAN, or better to say any such config is unknown to me.


    #BrocadeFibreChannelNetworkingCommunity


  • 5.  Re: FCIP Tunnel up no connection: NPVID -> E -> EX ->VEX ---VEX <-EX <-E <- NPVID

    Posted 07-17-2013 12:29 AM

    Hi there,

    I have taken a look at the diagram and it is conceptually wrong:

    You cannot connect two VEX port together, a VEX port only can be connected to a VE port (as well as a EX port can only be connected to a E port). The only way to perform the config you want to implement is to configure a BB fabric made up of the two fcip tunnels.

    Rgds


    #BrocadeFibreChannelNetworkingCommunity


  • 6.  Re: FCIP Tunnel up no connection: NPVID -> E -> EX ->VEX ---VEX <-EX <-E <- NPVID

    Posted 07-17-2013 05:32 AM

    Thanks,

    I figured that out via some RTFM and poking around and got the environment working.

    What I miss configured is that the brocade R06 is "routing"  what it is actually doing is "Bridging"

    NPVID -> E -> EX ->VE ---VE <-EX <-E <- NPVID

    That works.  Because what I needed to do is merge the FC03 switches (R06 router switch fabrics) together such that they were "one switch"  from which I could then "Bridge" the "red to blue" and "green to yellow".

    Here is updated diagram

    FCIP_Diagram.png

    That works as expected.


    #BrocadeFibreChannelNetworkingCommunity