VSAN: Design

[arnab125]

Hello All,

I am in process of designing a production infrastructure based on VSAN. Please review my hardware and suggest me about the networking devices that would be required:

a. 8 Servers in a cluster

b. Server model : SYS-2027R-N3RFT+ (Supermicro)

c. Drive Bays : Hot-swap 16x Hot-swap 2.5" SAS / SATA HDD trays

Two Disk groups; each group having (1 SSD and 7 SATA)

d. Network Controller (Four)

Intel® X540 Dual Port 10GBase-T

Intel® i350 Dual Port Gigabit Ethernet

Please suggest me about the networking devices that would support the above the servers. I am aware that each 8 servers would require a dedicated 10 gigs of network card. I am looking for cisco or juniper devices.

Thanks in advance.

[Vikas Shitole]

depping  and CHogan would be best to answer your queries.

[Cormac Hogan]

I don't think we make any statements regarding the network switch. That is typically left to the customers discretion. They key point is to make sure it can handle true multicast traffic. Some lower end switches convert multicast traffic to broadcasts, and we'd recommend avoiding those. Otherwise you can use whatever switches you are comfortable with.

[jgover]

Hi,

Waiting for your book to arrive, I hope you cover switching/networking decisions in more depth and with suggested models. It's a huge price decesion for a small vSAN 4-Node 10TB.

What do you mean by True Multicast in reference to this post? I am looking at the switches below for a small 4-Node vSAN. Switch decision is slowing me down and either 10GBase-T or 10GbE SPF+.

NETGEAR ProSAFE 16-Port 10G Web Mng Switch 1 Copper/SFP+ Combo (XS716E)

Compliant Standards: IEEE 802.1p , IEEE 802.1Q , IEEE 802.3 , IEEE 802.3ab , IEEE 802.3ae , IEEE 802.3an , IEEE 802.3aq , IEEE 802.3az , IEEE 802.3u , IEEE 802.3x , IEEE 802.3z

   

Features:2 fans , 2MB packet buffer , Auto-negotiation , Broadcast Storm Control , Cable test , Class of Service (CoS) , DHCP client , DiffServ Code Point (DSCP) support , DoS attack prevention , Energy Efficient Ethernet , Firmware upgradable , Flow control , Full duplex capability , IGMP snooping , IPv4 support , Jumbo Frames support , Layer 2 switching , Link Aggregation , Loop prevention , Multicast Storm Control , Port mirroring , Quality of Service (QoS) , Security lock slot (cable lock sold separately) , Store and forward , Trunking , Type of Service (ToS) , VLAN support , Weighted Round Robin (WRR) queuing

Form Factor: Desktop , Rack-mountable

Jumbo Frame Support:9KB

MAC Address Table Size:16K entries

Manageable:Yes

Ports Qty: 16

Thanks

Jeff

[jgover]

Hi,

How can I tell from these features if this switch supports "True Multicast"

https://www.netgear.com/business/products/switches/smart/XS728T.aspx?cid=wmt_netgear_organic#tab-fea...

Specs' but I am not certain from these features.

• Multicast groups: 512
• Broadcast, multicast, unknown unicast storm control
• Block unknown multicast
• Multicast VLAN Registration (MVR): XS708T/XS716T: Yes ----- XS728T/XS748T: No
• Broadcast, multicast, unknown unicast storm control

P.S. I see on page 254-255 of your book I can perform a vSAN Test and Multicast traffic test but I do not have the switches in-hand.

Thanks

Jeff

[jgover]

Hi,

I received this from the vendor.

<

Yes, our 10G Smart Managed Switch XS728T is perfectly capable of line-rate, true multicast traffic - at Layer 2, so using IGMP Snooping for forwarding multicast packets to interested receivers. Smart Managed switches come with intuitive Web GUI only for configuration, so under Switching \ Multicast at the top and IGMP Snooping on the left, you just need to:

• Enable Admin Mode for IGMP Snooping Configuration
• Enable all other IGMP functions there, including Proxy Querier mode so that you DON'T need external Multicast router on your network
• In Interface Configuration on the left, you finally need to enable IGMP Snooping Admin Mode and Fast Leave and Proxy Querier on all physical interfaces you need.

>

Jeff

[Duncan Epping]

arnab125 wrote:

Hello All,

I am in process of designing a production infrastructure based on VSAN. Please review my hardware and suggest me about the networking devices that would be required:

a. 8 Servers in a cluster

b. Server model : SYS-2027R-N3RFT+ (Supermicro)

c. Drive Bays : Hot-swap 16x Hot-swap 2.5" SAS / SATA HDD trays

Two Disk groups; each group having (1 SSD and 7 SATA)

d. Network Controller (Four)

Intel® X540 Dual Port 10GBase-T

Intel® i350 Dual Port Gigabit Ethernet

Please suggest me about the networking devices that would support the above the servers. I am aware that each 8 servers would require a dedicated 10 gigs of network card. I am looking for cisco or juniper devices.

Thanks in advance.

Which disk controller are you planning on using? And you sure you want to use SATA and not NL-SAS?

[arnab125]

Hello All,

Thanks a lot for all your replies.

@Depping and CHogan: We are using “LSI SAS HBA 9201-16i” as Disk controller and Seagate 1TB ST91000640NS Constellation SATA 6GB/s 7200RPM 2.5" HDD and Intel 800GB S3700 Series SATA 6GB/s MLC 2.5" SSD Drive per disk-group.

I have couple questions and would be grateful if you could help me out in them as well. 

• What would be the recommended number of I/O controllers per Host? I/o controller could be one the critical component as its failure would bring down the complete storage on the host.
• I have also read in your pre-release book “Essential Virtual SAN”, that if a recovery occurs within 60 minutes has elapsed or before the creation of the replica has completed, the VSAN will start writing all the data in the rest of the available disk groups and VSAN will decide if it makes sense to complete the creation of the replica or if synchronizing the “old” components makes more sense.

What would happen if the host comes online after 60 minutes and suppose the data rebuilding process had begun, vSAN intelligently decides that it’s better to re-sync the data on the lost host ? Will it discard the rebuilt data on the cluster?

• Also initially if we go ahead with three host cluster(All storage contributing and No other host) and one host fails, what will happen to my VSAN Cluster and the data on the failed host ?
• Suppose, if on one of the host in the cluster, one of the component besides the Storage part fails and the host is Dead. VSAN will take 60 mins to detect this failure and start the rebuild. Is it possible that before the 60 mins period, I add a new ESXi host in the cluster with same physical Disks I take out from the failed(Dead) host ? Will this work, detect all the storage policy settings and sync the data  on this newly added host?

I appreciate your time and help.

[Duncan Epping]

arnab125 wrote:

Hello All,

Thanks a lot for all your replies.

@Depping and CHogan: We are using “LSI SAS HBA 9201-16i” as Disk controller and Seagate 1TB ST91000640NS Constellation SATA 6GB/s 7200RPM 2.5" HDD and Intel 800GB S3700 Series SATA 6GB/s MLC 2.5" SSD Drive per disk-group.

I have couple questions and would be grateful if you could help me out in them as well.

• What would be the recommended number of I/O controllers per Host? I/o controller could be one the critical component as its failure would bring down the complete storage on the host.
• I have also read in your pre-release book “Essential Virtual SAN”, that if a recovery occurs within 60 minutes has elapsed or before the creation of the replica has completed, the VSAN will start writing all the data in the rest of the available disk groups and VSAN will decide if it makes sense to complete the creation of the replica or if synchronizing the “old” components makes more sense.

What would happen if the host comes online after 60 minutes and suppose the data rebuilding process had begun, vSAN intelligently decides that it’s better to re-sync the data on the lost host ? Will it discard the rebuilt data on the cluster?

• Also initially if we go ahead with three host cluster(All storage contributing and No other host) and one host fails, what will happen to my VSAN Cluster and the data on the failed host ?
• Suppose, if on one of the host in the cluster, one of the component besides the Storage part fails and the host is Dead. VSAN will take 60 mins to detect this failure and start the rebuild. Is it possible that before the 60 mins period, I add a new ESXi host in the cluster with same physical Disks I take out from the failed(Dead) host ? Will this work, detect all the storage policy settings and sync the data  on this newly added host?

I appreciate your time and help.

1) Most solutions I see have a single disk controller so far. I can see your point, but you need to ask yourself how often a disk controller fails. It is not really common.

2) Correct, VMware VSAN will make the decision whether it is smart to continue resyncing or stop and bring the old components back up to speed. Or discard the data if needed.

3) The data on the failed VSAN host will be inaccessible right. When the host comes back the data will be resynced

4) Depends on the type of failure. There is a "degraded" and "absent" state. So depending on how it failed you will get a 60 minute timer or an instant rebuild. If you add your old disks to a new host then no this will not resync those old components as far as I know!

[arnab125]

Thanks a lot for clarifying my doubts @Depping and CHogan:.

One more query I have:

As per your blogs, http://www.yellow-bricks.com/?s=Queue%20depth the Queue depth should be equal to 256 or more. The disk controller LSI SAS HBA 9201-16i has queue length of 32 (As informed by their technical support. Also they mentioned queue depth information are private and they were reluctant in sharing the same). Could you suggest us some disk controllers that I could use in our design having recommended queue length and that can handle 16 internal disks in the HBA/ Pass through mode as we are planning to have two disk group per host (1 SSD + 7 HDD).

Thanks in advance.

[Duncan Epping]

Supermicro typically offers their own LSI based controllers like the SMC2208 and the SMC2308 for instance. In my lab I use the SMC2308 which works fine.

[JohnNicholson]

Using a 2208 in production and liking it a lot.  Just have to go into the MegaRAID CLI to enable JBOD is the only minor annoyance. 

[Duncan Epping]

JohnNicholson wrote:

Using a 2208 in production and liking it a lot.  Just have to go into the MegaRAID CLI to enable JBOD is the only minor annoyance.

I tested the procedure in our own lab, it works great though and just a couple of steps you have to take just once!

[admin]

amab125 - please make sure you use components from the HCL. The Seagate SATA drive is not on the HCL.

Thanks.

Kiran

[arnab125]

Thanks a lot for your reply.

Depping: Thanks for guiding and sharing your knowledge on the hardware designing part.

@kmadnani: Sorry i mentioned SATA controller its SAS controller (ST91000640SS) and its in the HCL of vmware.

[Duncan Epping]

arnab125 wrote:

Thanks a lot for your reply.

Depping: Thanks for guiding and sharing your knowledge on the hardware designing part.

@kmadnani: Sorry i mentioned SATA controller its SAS controller (ST91000640SS) and its in the HCL of vmware.

Kiran mentioned the seagate magnetic disk not the disk controller!

[arnab125]

My apology. We are using seagate SAS drives (ST91000640SS).

[Duncan Epping]

arnab125 wrote:

My apology. We are using seagate SAS drives (ST91000640SS).

Excellent...

I hope you are avoiding that disk controller... go for a proper one!