Brocade Switch Over-Subscription
Frequent poster and commenter to the Ruptured Monkey sites is Stephen2615. He posted over in the forums this morning that he was little upset by a statement that a Brocade SAN Engineer (Chip Cooper) made during an interview with SearchStorage back when the McData and Brocade merger was taking place.
What's your strategy for oversubscription? Cisco has made so many inroads here.
[It's an old debate. Cisco says oversubscription of ports saves users money by letting them use less bandwidth for applications that don't need full bandwidth. Brocade does not oversubscribe ports saying it causes contention.]
Cooper: Our 32-port card is running at 4 Gbit completely nonoversubscribed … we can help you with the correct fan-in density … Let's face it, we've had switches that have been running longer than Cisco's been in the Fibre Channel business.
It turns out that in the Brocade 48000 technical guide it describes the over subscription that actually does take place on the 48000 32 port and 48 port blades.
32 Port Blade Over Subscription Description:
The 32-port blade is designed with a 16:8 subscription ratio at 4 Gbit/sec for non-local traffic, and a 1:1 ratio at 2 Gbit/sec for any traffic pattern. If some or all of the attached servers and storage devices run at 2 Gbit/sec, or if I/O profiles are “bursty,” the 32-port blade typically provides the same performance as the 16-port blade.
48 Port Blade Over Subscription Description:
At 24:8, the 48-port blade has a higher backplane over-subscription ratio but also has larger port groups to take advantage of locality. The backplane connectivity of this blade is identical to the 32-port blade. The only difference is that, rather than just 16 ports per ASIC, the 48-port blade exposes 24 outward-facing ports (96 Gbit/sec or 192 Gbit/sec full duplex of localswitching per ASIC).
This blade is especially useful for high-density SAN deployments where:
- Large numbers of servers need to be connected to the director
- Some or all hosts are running below line rate much of the time
- Potential localization of most traffic flows is achievable
So the question is, does all this really matter? I guess in a high performance environment where there is a potential for all the servers to run at a full 4 Gb on each port, this would matter. But in the real world, where users can barely push 2 Gb speeds I don't think that end users are going to really care whether or not Cisco or Brocade port cards are over-subscribed. I have seen only a few high end servers push to a full 2 Gb of bandwidth, but most often the amount of I/O in the SAN is actually more the restricting factor than the actual bandwidth on a switch. Disk subsystems can only do so much I/O before their processors are maxed out, and that usually comes way before the bandwidth in a switch is the limiting factor. Rarely have I seen some Tru64 clusters or Oracle databases using the full bandwidth in a switch, but I have seen it happen (It all depends on the size of your I/O right? So size really does matter. 
). For those high bandwidth needs I would recommend the following configuration scenario if you want to use a Brocade 48000 in your high performance environments.
- Use a 16 port card for your high throughput ISL connections between your core and edges.
- Use a 32 port card for your disk and tape connections and keep them as localized within the same ASIC as possible.
- Use a 48 port card for your server connections.
Following that you should be able to get the most out of your switch as possible with the most overall connection:performance ratio. If you don't really care about maximizing the number of ports in a chassis, then just buy 16 port cards and forget about it. 
Snig
May 2nd, 2007 at 8:36 pm
What a can of worms! I’ve had this on/off discussion for about the last 18 months. I don’t think it matters that either Cisco or Brocade oversubscribe if the price is right; if it isn’t then there’s a problem. I bet 100% that the per port cost of a 24 port Cisco blade is not 1/2 of the cost of a 48 port blade. Then vendors will bleat on about how expensive GBICs are blah blah and if you take into account you need to load the chassis cost in there too, it doesn’t work. I have a lot of other views on oversubscriptions – again, I haven’t space to publish here – I’ll blog again separately, sorry, don’t want to be seen to be stealing traffic!!
May 3rd, 2007 at 3:58 am
Or you could get a QLogic Sanbox 9xxx with 256 (I believe) non-oversubscribed 4Gb ports
May 3rd, 2007 at 12:24 pm
Oh Tim. I think I just threw up in my mouth a little.
May 4th, 2007 at 4:37 am
It’s funny – I cut my teeth on the QLogic SANBox switches back when they were doing the 8 and 16 port 1gbit switches.
When I worked for the hellhole (hereafter referred to as MTI) they dumped the QLogic switches based on the sound engineering reasoning that "Brocade has a better name in the industry"
So what? Brocade has a better marketing department. So does Microsoft, it doesn’t mean their products are better built, just better sold.
I would *LOVE* to see QLogic switches make a major comeback, but they are going to have to break some barriers to overcome the Cisco effect.
May 4th, 2007 at 4:54 am
My first SAN switches were Qlogic and I had no problems with them at all other than the occasional power supply failure which was a bummer as they only had one power supply. We purchased the 64 port blade switches when they first appeared and only within the last two years went the way of Cisco Directors but that was only for business decisions. When I asked Sun if the ports were oversubscribed, the "answer" was no. They probably went to the same school of misinformation as Chip Cooper but so far, I have not evidence that the ports are not.
On a side note, I personally believe that Cisco and Qlogic are in bed somehow. There is no interop issues between the two of them and you never hear either vendor bad mouth each other. Brocade does enough of that to make up for all the vendors. Cisco producing the 91xx series MDS must have some impact on Qlogic but so far I have not seen any real complaints.
For the average small to medium business, Qlogic have the right gear and price as far as I am concerned. Cisco has to make IVR free….
May 4th, 2007 at 11:14 am
Interesting post. Its been awhile since I looked at this topic and it immediately brought back to focus the challenges in stucturing large SANs. I always thought smaller SANs were better than large ones because of these sorts of challenges as well as the risks and difficulties with change management and the time required to troubleshoot/fix them.
I have several questions/comments. 1) Don’t you want to put most disk subsystem links on 16 port cards due to the concentration of traffic and oversubscription these links tend to see? This provides performance overhead and lets you increase the ratio of server ports to disk ports for low I/O servers. 2) I think Q Logic’s ASICS support the largest numbers of ports on a single piece of silicon – do they not? I agree that trying to localize traffic within individual ASICS will provide the best performance overhead – even if it does make for more difficult design and change management work. So, the highest performing systems and storage ports should be connected through the same ASIC, and if that isn’t possible, within the same switch and as infrequently as possible over an ISL. 3) If you have a core-edge topology, don’t you want to put disk subsystem ports on the core switch (director)? 4) I don’t think Cisco switches do any routing in port ASICS instead, I thnk it is all done in their supervisor modules. 5) Cisco’s switching architecture doesn’t hurt them all that much because so few servers actually need anything close to wire speed – even at 1Gb/sec. If you do have systems pumping out wickedly high I/O rates – don’t you know which ones they are and can’t you limit their exposure to traffic congestion by isolating them on their own SANs? 5) Can’t a single management tool provide visibility into multiple, discrete SANs? If so, why build large, interconnected SANs – the money saved on port costs are overwhelmed by the administration effort required.
This is why I think iSCSI makes so much sense. Keep ultra-high performance servers and storage on Fibre Channel while lowering the port and overall ownership costs of all other servers by putting them on iSCSI.
May 4th, 2007 at 2:14 pm
Great responses. Just to clarify, i was just giving TimC a hard time and hopefully made him laugh.
Personally the only QLogic experience I have is with their HBAs, which I’m really not a big fan of. I prefer Emulex.
As far as their switches go, from what I’ve heard and from you guys have said they seem like a decent fit. I looked at their director switch the other day when Tim posted his comment and they seem like they on their way to a decent product. Once they get hot code loads and proper failover between controllers, I think they’ll be able to make a good play in some Data Centers.
To answer Marc’s questions:
1) You could. From my experience you are going to over run the disk subsystem port with IOPs before it hits it’s bandwidth constraint, so I wouldn’t really worry about the switch in that case. Only in the highest performance environments would I worry about that.
2) I’m not sure. I’m not real familiar with QLogic’s switches or ASIC architecture. Maybe someone from QLogic could comment.
3) That is one approach, and that is the way I used to do it at my old company. The only reason was to save money on switches though, not for performance.
4) I’m not sure about the routing in Cisco switches, Stephen?
5) I’m sure Cisco switches are fine with their over subscription as long as you pay attention. I don’t really know though. Any customers out there that care to comment?
6) Personally I have always found it simpler to manage fabrics with a lot of switches rather than a bunch of fabrics with less switches. Less zoning databases to worry about, easier to troubleshoot, and reduces the amount of spreadsheets I have to maintain.
I definitely think IP storage has it’s place in the Data Center. For the right applications it totally makes sense.
May 4th, 2007 at 11:00 pm
Cisco does all its routing inside the supervisor modules. I quite liked the idea of Brocade’s inter port group routing until I figured that most hosts don’t talk to other hosts directly. I don’t share storage and heavy usage hosts in the same port group so its probably just another marketing ploy.When I first got the MDS, they were gen 1 which caused a couple of issues with ISL’s and port groups in higher density cards but now they have fixed it with the gen 2 modules. Basically I still figure out what goes where in the switches.In a recent exercise in my previous job, we had a number of HP Blade enclosures and there were 74 blades in each one running RHEL. The project designer demanded SAN for each one which surprised me a bit. I think it was a tick in the project box. I suggested iSCSI but he would not wear that idea. The cost of putting in SAN for literally hundreds of hosts was expensive and time consuming. I think that 80% of the hosts could have used iSCSI and the rest normal SAN.But recently I have heard some horror stories with iSCSI and multlipathing in Windoze environments that makes me nervous. If our "architect" wants it, I will give it to him but he has to be aware of the risks. So, while this is still an issue, I still count the ports on the switch. HTnM is my friend.I still like the rusty disks that Windozs use..
May 4th, 2007 at 11:36 pm
Hi Stephen2615,
I’m glad that you posted this entry. I’ve received a lot of comments pertaining to the SearchStorage article, and your post gives me a great opportunity to respond to what may be a misunderstanding of what was said.
First and foremost, your excerpt of the 48000 Technical Guide does indeed correctly describe the over subscription behavior of the 32 and 48 port blades for the 48000. I appreciate your taking the time to research this and for including the complete description of the behavior of these blades in your post.
And so what about the quote? "Cooper: Our 32-port card is running at 4 Gbit completely non-oversubscribed…we can help you with the correct fan-in density…." It would help for you to have a little more context in which this was said. I was speaking at an event that was being held in a Manhattan hotel. Audience members included Brocade customers, partners, Systems Engineers, Sales Representatives, and the attendees from SearchStorage. The question of oversubscription on the 32 and 48 port card comes up often. When it came up this time in this large forum, I gave an answer which was consistent with the technical description you included in your post.
The difficulty is in the ellipses (…).
The complete quote was to the effect: "Our 32-port card is running at 4 Gbit completely non-oversubscribed if all the external ports are attached to 2 Gbit devices or if local switching is used to insure that no more than 64 Gbit per second of data moves through the core of the 48000." The statement which followed ("we can help you with the correct fan-in density") was said to emphasize the fact that we are willing to consult with customers to help them design their SANs in such a way as to avoid any oversubscription in the 32 and 48 port cards.
If you have attended Brocade events, you know that our partners, customers, Systems Engineers, and Sales Representatives are quick to correct any inaccuracies that are stated about our products. Had I represented the 32 port card as being non-oversubscribed under any circumstances, I would have been quickly corrected by any one of a number of people in the audience. They are not a shy group!
And oh, one more thing. My last name is Copper, not Cooper. I’m glad I had a chance to address that as well!
Thanks again for your post, Stephen2615, and if you happen to see me at a Brocade event, I hope you’ll say "Hi."
Take care,
Chip Copper
Brocade
May 6th, 2007 at 12:15 am
Chip,I am very sorry to have mistakenly referred to you as Cooper. Ever since I was about 8, people have mucked up my name which is irritating.I had to point out the same over subscribed feature to a Brocade senior SE here when he took the time to visit. I am in Australia so we are a small piece of the cake in the global business but we take pride in being technically very savvy and dislike misleading information.I don’t go to one day events but perhaps I might get to something interesting outside of Australia.Thanks for taking the time to expand on the context of the article.Stephen (almost a CCIE in Storage Networking)