blogs.rupturedmonkey.com

A while ago I posted an article demystifying the relationship between HP and Hitachi – in particular the Engineering Agreement that HP have in relation to the StorageWorks XP line of arrays.  So…. as a follow-on I thought I’d do something similar, this time on the relationship between HDS and Hitachi.

To help with this I spoke with Roberto Basilio at HDS.  Roberto heads up Product Management for Platforms and knows a thing or two about how HDS fits into the larger Hitachi organisation.  Oh and he knows plenty about the HDS product line-up too!

(more…)

The following is reposted from my new blog site http://blog.nigelpoulton.com  As a result, comments are disabled on this site, but feel free to visit my new site and leave a comment. Thanks to Snig for allowing me to post here for a short transition period…

One of the key technologies or principles in Rack Area Networking (RAN) is I/O Virtualisation (IOV).  In fact, IOV is about to rock the world of physical server and Hypervisor design.

If you work deploying VMware, Hyper-V, XenServer etc or if you have anything to do with the so called Virtual Data Centre, then you need to be all over IOV.

This is the second post in my mini-series on RAN and IOV.  In this particular post Im going to talk about the concept virtual adapters – Virtual NICs and Virtual HBAs.

(more…)

The following is reposted from my new blog site http://blog.nigelpoulton.com  As a result, comments are disabled on this site, but feel free to visit my new site and leave a comment. Thanks to Snig for allowing me to post here for a short transition period…

Ever heard of a Rack Area Network?

The term, as well as the concept, of Rack Area Networking is one I’m hearing more and more often.  As a result of this, as well as the fact that I’m convinced this is going to be one of the most interesting and important areas of Data Center computing over the next few years, I’ve decided to write a mini-series on the topic. 

This post is instalment number 1 and is intended to introduce the concept and get the ball rolling.  The whole thing is a bit of me thinking out-loud and attempting to generate some awareness and conversation around the topic, as .  So please pitch in!

(more…)

If you ever want to get up the nose of an HP bod, a sure-fire way is to tell them that the XP is just an OEM’d HDS array.

(more…)

UPDATED 19/11/09 minor corrections/clarifications in purple:

OK, if you don’t already know Xsigo Systems, and what they do, then you are seriously missing out!

(more…)

Im very fortunate to be one of the people invited to the first ever GestaltIT Tech Field Day and its going to be great.  This is a short post but I think worthwhile so please read on

What the H*!! is this GestaltIT Tech Field Day thing?

The crux of the event is this – taking a handful of bloggers and a handful of smaller vendors, locking them in the same room for two days with only blogging and tweeting as their connection to the outside world………. And seeing who comes out alive!

Presentations will be given, demos will be done and most importantly, no holds barred hard questions will be asked, and asked, and asked….. 

In fact, Stephen Foskett the event organiser, and industry legend, told me the other week that he had warned the vendors that the only way he could guarantee that we won’t say they suck, is if they don’t suck!

Why should you care?

Answer: Same reason I care…… It will be one hell of a learning opportunity.

Granted its an invite only event, so we cant all go, but I will be there and will be blogging and tweeting about everything I learn.  So if you want to know more about the technologies and strategies of the companies who will be presenting, then stay tuned this blog and tune in to my Twitter account (@nigelpoulton).

I promise to blog and tweet about everything I learn.  The good, the bad and the ugly.

Proxy

As everybody cant be there in person, feel free to drop me a line via any of the many means by which Im contactable, with any questions you want me to ask the vendors.  If they are good questions then I promise to ask them.

Who will be there – Vendors

Who will be there – Bloggers

Finally

I am genuinely excited about this event. Same kind of excitement I would have if my football team ever made the FA Cup final or England made the World Cup final (well….. may be not quite that excited).  But seriously, this is a landmark event and a great opportunity to learn about some of the less well known vendors and technologies out there.

Feel free to contact me with any questions you would like me to ask…….

Nigel

PS.  Im really looking forward to meeting all in attendance but in particular Xsigo, MDS and 3PAR.

While at SNW Europe I had the opportunity to visit the Arista Networks booth with my video camera in hand, and the guys on booth duty were good enough to allow me to shoot some footage while we talked about their products.

Enjoy the video and then feel free to comment on my thoughts below –

First up, Arista is start-up networking company, and like all networking companies…….. they have Cisco in their crosshairs.

Although low cost seems to be their trump card, to be fair to Arista they are also touting performance as another key advantage.  Better performance, literally at a fraction of the Cisco price – if the guys at the booth are to be believed.  However, as I mention in a previous post, there is more to value than price.  It may turn out to be very short sighted to jump all over short term cap-ex savings at the true expense of the long term good.

However, the current down economy is probably creating ‘too good to pass-up’ opportunities for companies like Arista, who come in at a value price-point- most companies being forced to think more and more about cost, thus creating windows of opportunity for those coming in at significantly lower price-points.

However, when at the booth, the Arista guys did a great job ensuring that their products didn’t have that ‘value’ feel.  Despite the fact that they are apparently way cheaper than Cisco, they weren’t over-stressing that point and wanted to talk a lot about performance.

Technology and performance overview

So a quick overview of what Arista have on the table from a product and performance point of view….

Basically they are offering high-speed low latency 10Gbps Ethernet products.  Oh and apparently they are cheap…cheap enough for my garage??.

However, if you listened closely to the video, they only have DCB/CEE “on the roadmap”.  This, in my opinion is a huge shortfall in the current shipping products.  As each day goes by I am becoming more and more of the opinion that to be a true data center switch, you need CEE and FCoE.  Yes, 10Gbps without the goodness of CEE and FCoE might suffice for now, however, it will be found wanting in the very near future.  More on this later though….

I am impressed at the published latencies and port count as well as the dual hot-swap power supplies and reversible airflow (to change the airflow from front-to-rear, to rear-to-front requires swapping out the fans).  However, my gut reaction is that this is almost a gimmick in comparison to features like CEE and FCoE. Sorry that I keep coming back to this but I can’t stress it enough.

NOTE:  In the interest of fairness, should point out that when at HP Colorado Springs recently I was knocked back when I was told that one of the hardest things to engineer for new servers and blade chassis is the power supply units (PSU).  So I am careful not to totally dismiss the reversible airflow and redundant hot plug PSUs as totally insignificant.

7100 Series Data Center Switches

The 10Gbps 7100 series SFP+ based switches come in either 24 or 48 port-count models and you can choose between a non-oversubscribed and oversubscribed models.  You can also buy similar 10GBASE-T models which take standard RJ45 connectors and work with existing Cat 5/6/… cables.

When first at the booth (without my camera) I raised my eyebrows when introduced to the 10GBASE-T model.  As I raised my eyebrows the guy giving me the tour (different guy to the one on the video) pre-empted me and said “your are going to say ‘Wow’ aren’t you. But I had to admit that I was actually going to say “Why”…..

My initial reaction was why would anybody want to do that from a power cooling and latency standpoint.  Essentially SFP+ up to 10 metres has people talking about ~0.1W nominal power draw whereas 10GBASE-T RJ45 cables are closer to 4W at approx 10metres, but obviously offers greater distances at the expense of more power.  And that’s not to mention the inferior transceiver based latency that 10GBASE-T has compared to SFP+. 

Obviously a 10GBASE-T model may make the jump to 10Gbps Ethernet simpler for some people, I personally believe this approach is a false economy of sorts and hedges up the route to the full benefits of 10Gbps, and greater, based on CEE. 

Pass-Through module

They also had a 10Gbps Pass-through module on display.  This pass-through module has 14 internal and 14 external linespeed connection, with no oversubscription and is designed for the IBM BladeCenter H.  The module only has 4 physical ports on the external faceplate and achieves the 14 external port count via 10GBASE-CR QuadSFP to SFP+ splitter cables.  This is the copper twinax 4-way splitter cable seen in the video.  The module and is apparently very competitively priced compared to competitive offerings from the likes of BLADE Network Technologies.  Although features and functionality might be less than that of the BLADE Networks offering, apparently performance is not.

Nice, but no game changers

So, a lot of talk about feds and speeds.  Interesting for some, but talking feeds and speeds, so Im told, is sooooo last year.  EMC and HDS used to do it all the time until analysts and customers told them they didn’t care.  Feeds and speeds are fine, but not everything (at least not to the vast majority).

So for feeds and speeds the Arista kit seems to be very good and very competitively priced. If latency and performance are critical for you then Arista are definitely worth a look.

However, away from feeds and speeds was where there were signs of weakness. 

In my opinion, for a networking product to be a true Data Center product it has to play in the converged networking space (IP, FC, RDMA..).  At the moment, Arista products do not.

First, there is no Converged Enhanced Ethernet (CEE/DCB).  For me this is vital to the future of Data Center networking.  I know they guy on the video said that Arista doesn’t do anything that is not standards based, alluding to DCB/CEE standards not yet being fully ratified.  However, Im willing to bet they do jumbo frames, and I’d like to see them produce the IEEE 802 standards documents for jumbo frames

Sorry, that was a cheap shot, but I think comments like these are poor and will not help you out in the future when you are playing catch up.

Second, these are pure Ethernet products.  No Fibre Channel ports.  This is understandable as they have no FC legacy or expertise. 

In the very short term, while people use the edge in approach for implementing a unified fabric, this means that Arista products will not be an option.  But more importantly, beyond this, once end-to-end FCoE becomes mainstream, where will this leave Arista?  Having no FC expertise that I’m aware of will make implementing FCoE (FCF and all the other FCoE related functions) in their products very time consuming and difficult.  They will likely have to acquire a company that has FCoE or hire some engineers who know it, and hire them fast!

Although Data Center networking is converging on Ethernet, it is not converging on the Ethernet that is widely deployed today.  In fact, the Ethernet that it is converging on (CEE) bares less resemblance to todays 1Gbps lossy Ethernet than most realise.  So just because you are an Ethernet company today, and Data Center networking is converging on Ethernet, does NOT automatically mean you are well placed for the future.

Looking to the future

So…….. normally, it is the large companies, those with the large market share and legacy architectures that struggle to change with the times (think Cisco).  The newer and more agile companies (think Arista) often beat them out of the blocks and are tearing away around the first bend before the likes of Cisco are even out of the blocks.  Not so this time! 

Cisco are actually heading around the first corner with early deployments of CEE and FCoE on the new and expanding Nexus platform and NX-OS.

Far from being caught flat-footed or even found trying to protect their existing markets, Cisco have made bold moves into the Blade Server market with UCS as well coming to market with the new and ever expanding Nexus platform and NX-OS.  If anything Cisco are already heading around the first bend with early deployments of CEE and FCoE.

Its one thing to make a better or faster “switch” than Cisco.  Buts altogether another thing to build better solutions.  Right now Hypervisors are driving change and shaping the future.  Addressing the challenges that Hyperviors bring are vital.  From my perspective Im not seeing a lot from Arista to help here

Management Big Hitters

Before finishing, it is worth quickly mentioning the very impressive list of networking legends that is heading up the Arista management team.  A quick scan of the management team listed on the Arista website reads like a who’s who of Data Center networking and is nicely capped off with Andy Bechtolsheim who has pedigree when it comes to picking winners.  These guys know their networking, oh ……. and they know the competition   So don’t write them off just because of what I say :-S

Finally

With the lack of a networking group at IBM, it would be remiss of me not to suggest that Arista may be a future acquisition target for Big Blue   I had to say it.

I’d love to have been more impressed with Arista.  Who knows, may be they have a game changer up their sleeve.

Nigel

You can follow me on Twitter where I talk about storage technologies (@nigelpoulton) and I am also available for hire as a consultant.

Here are some tech related photos I took at the recent SNW Europe in Frankfurt Germany.

In my opinion the show was a real success with over 1,500 attendees, of which over 1,100 were end users and reseller delegates and the remainder made up of general riff-raff such as vendors, press and the likes….

One of the things I like to see at shows like these is hardware.  What can I say, Im just the kind of guy that gets a kick out of seeing hardware.  So for the rest of you out there like me – sit back and enjoy……..

First up, I was impressed to see a Symmetrix V-Max, blue strip light ablaze.  The only disappointment was that whenever I popped by to have a chat, somebody else was always being given an overview

Next up the IBM HS22 BladeCenter that was used to demo the new IBM Virtual Fabric, which is based on technology from IBM, BLADE Networks and Emulex.  A much needed addition to the IBM portfolio in my opinion.

Oh and while on the theme of IBM, here is another rack of IBM kit

As the IBM Virtual Fabric solution has an Emulex CNA in it, next up is Shawn Walsh from Emulex showing us that the Emulex UCNA is real and not a myth.

And a close up on a desk with a pen for lined up to give an idea of size

Then if you followed the girls with “Thin” written on their T-shirts you couldn’t miss the 3PAR InServe kit on show.  I plan on writing about 3PAR RAID MP and Persistent Cache, both of which are potentially very interesting technologies.  But seeing as 3PAR are attending the upcoming GestaltIT Field Day I might wait and see if I can glean some deep tech info from them.

Brocade also turned up with a rack load of kit, although hugely disappointing for me was the lack of an FCoE 10-24 blade in the DCX director.  Not to worry though, there was a B8000 top of the rack CEE/FCoE switch to keep me happy.

And a Brocade dual port CNA

I have some video footage from the Brocade booth that I will post some time next week.  So stay tuned.

Even the internet connected laptops were also of decent spec.  Below is a smart little HP laptop alongside my personal 11.1” Sony job – check out the grease marks on my trackpad and spacebar

And last but not least, the quality of freebies was good.  Aside from the standard pens and stress balls, I was particularly impressed with –

iPod Shuffle
Solio solar powered USB charger
3-in-1 pen/laser pen/1GB memory stick (James Bond style)
2GB micro SD with standard SD adapter 

The economy must be recovering!

As well as the above mentioned video footage from the Brocade booth, I also got some video footage from the Arista networks booth.  Keep an eye out for that as I plan to post them in the following week…

Nigel

You can follow me on Twitter where I talk about storage technologies (@nigelpoulton) and I am also available for hire as a consultant.

The below video is something I shot yesterday while at Storage Networking Europe in Frankfurt Germany. 

In the video, William Lloyd Scull Senior Network Architect at BLADE Network Technologies, demos a feature of the IBM BladeCenter Virtual Fabric (comprised of Emulex OneConnect UCNA, BLADE Network Technologies Blade Switches and IBM BladeCenter hardware).

In the video William dynamically reduces network bandwidth allocated to a vSwitch from 2500Mbps down to 1900Mbps, as well as talks us through some of the other features and some of the hardware involved in the Virtual Fabric solution.

Its an interesting video if you want quick and dirty overview of Virtual Fabric for IBM BladeCenter and a glimpse at what it looks like with the lid off.  The video was not rehearsed, and although Id gone through the same thing with William the day before he had no idea I would turn up again the next day with my video camera and ask him to perform to the world

So thanks to William for being a good sport.  Enjoy……..

First impressions for me are similar to my thoughts on HP Virtual Connect Flex-10…….  In fact this is very similar to VC Flex-10 but with a more clear roadmap for FCoE and is built on a solid Converged Network Adapter (OneConnect UCNA) from Emulex.  However, as good as it is, its very generation 1.  But that’s not a huge issue because these are early days in this area.  Everybody else is in a similar position.  To my knowledge there is nobody doing hairpin turns in silicon yet and we are all a long way off MR-IOV and the new rack area landscape which that makes possible……..

So…. an important technology for IBM, may be slightly better than Virtual Connect Flex-10 but is probably not quite as good as some of the stuff Cisco is doing with Palo… But a solid foundation to build on (see my previous post on the Emulex OneConnect Universal Converged Network Adapter for more info).

Thoughts and comments welcome as usual…..

Nigel

Oh and you can follow me on Twitter where I talk about storage technologies (@nigelpoulton) and I am also available for hire as a consultant.

Today I attended the Emulex launch of their OnceConnect Universal Converged Network Adapter (UCNA) at SNW Europe, and I liked what I saw.

Most will know by now that Im all for converged Ethernet unified fabrics.  Still, Im well aware that some of the vendor offerings are very “Generation 1” and have that version 1.0 feel to them.  Todays launch from Emulex honestly didn’t have that feel to it.  In fact it seems that the products shipping around CEE are getting more and more mature by the day.  So let me take a few minutes to discuss some of the product highlights…..

Architectural Overview

At a high level, the OnceConnect UCNA is a high performance single chip CEE adapter.

Lets qualify that statement…..

High Performance:  Sure, anybody can call their products, especially 10Gbps Ethernet adapters, “high performance”.  However, Emulex can back this up with the fact that this product provides hardware offloads for TCP/IP, iSCSI and FCoE.  This is marketed under the name vEngine (add a “v” to any technology and it automatically sounds cooler). 

Basically this adapter is a workhorse.  It will do the protocol related work for you, freeing up your CPU to do other tasks.  This enables it to be truly high performance, not just for FCoE but also for iSCSI and TCP/IP.  This is great – especially in Hypervisor estates.

Single Chip:  There is a single ASIC on the card that does the above.  No requirement for dedicated ASICs for each protocol.  ASICs also add to the High Performance claim, as ASICs are still very much the way to go with high performance I/O adapters – so called merchant silicon doesn’t quite cut the mustard. 

At the time of writing this article this is the only adapter on the market with all of the above hardware offloads on a single chip.  Although it is only fair to mention that FCoE functionality will be released later this year, but as its already late October, this will be soon.

Pay-As-You-Go and future proofing

The above is all good, but a bit overkill if you don’t need it all just yet.  Well……. You can buy the adapter as a 10Gbps Ethernet adapter, without any of the additional hardware offloads etc, at the pricepoint of a 10Gbps Ethernet adapter.  But in the future, if/when you require iSCSI offload or FCoE then you can easily unlock these features with a license.

Sounds almost perfect for the kind of staged deployments of CEE and FCoE that I think a lot of companies will adopt – Buy future proofed hardware now that allows you to keep your options open.  Deploy the UCNA now as a 10Gbps Ethernet adapter and be in a position to press ahead with FCoE etc in the future without the need to rip and replace your I/O adapters!  What is there not to like about it? 

Remember that we already do this with our switches and storage arrays, and it works well.  Why not apply the same model to I/O adapters.

This is all of course assuming that the progressive licensing doesn’t make the overall cost more expensive!!  From tweeting about this at SNW Europe today this is the biggest concern from fellow tweeters.

Oh and you can run software initiated iSCSI etc over the adapter without licensing the iSCSI offload….. 

All-in-all it seems very flexible to me.

NOTE: Of particular interest to me was the fact that the core features, as well as the base cost, of this adapter are 10Gbps Ethernet.  This is very interesting when you consider Emulex are traditionally a Fibre Channel company.  Clearly Emulex are moving with the market here and recognising Ethernet as the dominant technology and building on that.  Emulex also have people on IEEE 802.1 committees such as DCB.  Now that’s what I call not betting against Ethernet.

A must for the Virtual Data Center

In my last post regarding CEE I talked about the importance of CEE and its associated speeds and intelligence in the light of advances being made in Hypervisor based environments. 

I would add to that – features and functions such as those seen on the Emulex OnceConect UCNA are equally as vital….

As the number of VMs per CPU core increases to greater than 10, the I/O and networking components need to keep step.  What is the point of CPU technologies enabling huge numbers of VMs per CPU core if your I/O subsystems can’t keep up!

I think most people can accept the need for FCoE offloads as this is the norm in FC environments – and FCoE is Fibre Channel, just over a different Layer 2.  But….. offloads for iSCSI also become more and more important for iSCSI shops as they also deploy more and more VMs per physical server.

RuptureMonkey opinion:  Protocol offloads are absolutely vital to Virtual Data Centers.

Don’t Blink

There is no doubt that the I/O world is changing in front of our very eyes.  Be careful not to blink as you might miss something important.  Technologies such as CEE and protocol offloads (as well as many others) are key. 

Other related stuff

Also mentioned in the launch were technology agreements with IBM regarding 10Gbps Ethernet NICs and 16Gbps native FC HBAs, both for IBM Power Systems.  The 16Gbps FC design win being an industry first!  Technology is marching on and Emulex are certainly up at the front.

This is on top of the recent announcement around the IBM Virtual Fabric for BladeCenter – where Emulex, BLADE Network Technologies and IBM have collaborated to bring to market a very good blade based I/O solution, comparable, and possibly superior to, HP Virtual Connect Flex-10.  I saw IBM Virtual Fabric demonstrated in a VMware environment today and it looks a great technology.  The guys were also kind enough to rip a blade out and let me see inside one of the blades.

Finally, Emulex also announced OneCommand Manager which replaces HBAnywhere, but I havent seen this yet to be able to comment.

Random info:  Apparently World of Warcraft has ~15,000 blade servers. Cool!

Nigel

Wherever you look these days there’s no shortage of talk about FCoE.  However, sometimes I think a little too much attention is given to FCoE and people sometimes overlook the underlying DCB/CEE Enhanced Ethernet – In my opinion, this where much of the real work and enabling is happening.  So lets spend a couple of minutes talking about CEE……

NOTE: Im using the term CEE. I should probably use DCB, but Im not.  For disambiguation of the interchangeable terms CEE, DCB and DCE see here.
 

 
 
Half baked

First up, and in the spirit of honesty, I have to mention that the DCB/CEE related standards are not yet fully baked.  Formal ratification is expected early to mid 2010.  However, enough is defined for major vendors to be shipping the technology.  And more than merely shipping it, some are betting large chunks of their business on it. 
 
 
CEE 10 second overview

Really quickly, CEE is currently a 10Gbps full duplex lossless Ethernet with dynamic prioritisation/bandwidth allocation and some other goodies.

Specifically -

• Priority based Flow Control (802.1Qbb)
• Enhanced Transmission Selection (ETS 802.1Qaz)
• Congestion Notification (802.1Qau)

If you know the above technologies, then you will know that they are compelling and pervasive.  If you don’t know them then trust me, they are good.
 
 
The Grand Key

CEE is a key technology of the future.  For starters it is absolutely key to FCoE.  In fact without CEE there would be no FCoE, at least no compelling FCoE.  But CEE is not just for FCoE, its features and benefits can and will increasingly be leveraged by many areas of IT infrastructure…  iSCSI as well as NFS over UDP are just a couple that are commonly talked about.

Point being, FCoE is only one of the many new options made possible by CEE.
 
 
Smarter AND faster

Yes there is non-CEE 10Gbps Ethernet.  For the purposes of this post Im going to refer to it as “Dumb 10Gbps Ethernet” (and Im writing it in pink because its for girls).  This Dumb 10Gbps Ethernet is 10Gbps Ethernet without the goodness of CEE that we just mentioned above.  Sure its cheaper than the CEE version.  But, we all know that cheaper usually isn’t better.  Pay peanuts, get monkeys.   Trust me, in the long run you will want CEE….

Not convinced?  Let me draw a parallel that I hope will help create an “Aha” moment –

Just like processor technology, Ethernet can and will get faster and faster and faster.  But, like processor technology, there comes an inflection point – where getting faster and faster, but not smarter and smarter, becomes almost pointless.  The change in focus needs to be towards smarter and not just faster.

Ask yourself the following question – what are the real game changers and compelling aspects of Intel’s current raft of “Nehalem” processors?  Is it the GHz?  Or is it the built-in virtualisation intelligence and all the associated benefits (Im thinking Intel VT….)?

Same goes for Ethernet, speed is not everything, intelligence matters too.  CEE brings both speed and intelligence to the network.

Ask Hypervisor architects and admins if they would like to go back to the old days of non-hypervisor aware CPUs.  Im pretty sure they will tell you where to go.  Same goes, and will go, for people deploying and using CEE.  Once bitten forever smitten.

RupturedMonkey statement: “CEE brings intelligence to the network.”

 
 
Hubs versus Switches

In my opinion, the whole Dumb 10Gbps Ethernet versus CEE smacks of the old Hubs .vs. Switches debate of days gone bye. 

Is there anybody wishing they still deployed hubs at the centre of their networks?  No.

Point being…. CEE is here and its changing the game. 
 
 
Cable once and most of most

All of a sudden the possibility of cable once is a reality.  CEE has the capability to run most, if not all, of most companies portfolio of network traffic over a single cable.  But more than that – over a single PCIe card and to single edge switch port.  Who wouldn’t want that? 

But there’s even more – it will simplify changes and network management, as well as bring down the cost of power, cooling and all of that jazz too!
 
 
Unabridged opinions and conclusions

So if you didn’t know previously, you know now – Im liking the look of this whole converged network unified fabric thing.  I see CEE as an essential building block of every modern Data Centre!

So with that, let me finish with a word or two to the naysayers and FUD-slingers –

Do yourselves a favour and don’t waste your time and energy trying to slow the unstoppable forward march of technology.  Believe me, it will roll right over you like a steamroller over a cockroach rat something tiny and insignificant   Where would Intel, AMD or any of the major server vendors be if they had resisted the march of VMware?  After all, they would all ship more units if every server image required dedicated hardware…..  Short answer is, they would be in a world of hurt.

RupturedMonkey advice: Don’t get left behind… or rolled over by a steamroller.

Its a brave new world out there, and we as IT pro’s as well, as the enabling technologies, need to move with the times.  CEE is doing just that – moving things forward. 

By all means, choose to limp forward with bog standard 10Gbps Ethernet “sans” the goodies of DCB/CEE.  If you do, then good luck to you, but dont look on enviously as the rest of us race forward into the light.

Nigel

Please feel free to share your thoughts below. You can also follow me on Twitter @nigelpoulton.  I only talk about storage and technology and the conversation is often very interesting.

My current series of posts on FCoE and CEE have generated a lot of questions and feedback via the comments section and on Twitter.  In this post I will attempt to address some of the questions and misunderstandings – especially around encapsulation and framing efficiency……

I received a comment on the site today from a confused reader.  The point of confusion is a common one and not something to be embarrassed about.  Part of the comment read as follows

“…..but it’s [FCoE] problem is that it consists of 3 different protocols (IP, FC, SCSI) just enlarging it’s frame size with unneccessery information. This is a step backwards."

The above is clearly a misunderstanding.  It would be accurate to say that DCB/CEE is designed to transport all three of the above protocols, but FCoE does not consist of these.  So if it doesn’t consist of these then what is it…….

FCoE in a single sentence

Put in to a single sentence “FCoE is the encapsulation of Fibre Channel frames within Ethernet frames”.

If you have a good quality dive suit then put it on…… we are about to perform a deep dive…………..

Squeezing it in

Some will know and some will not, but a standards based Ethernet frame with VLAN Tagging, including Tag Control Information can be 1522 bytes in length.  On the other hand a Fibre Channel frame can be up to 2148 bytes in length.  Clearly, this poses a problem – How can an Ethernet frame be wrapped around a fibre channel frame, if the fibre channel frame is bigger?

Two of the most viable and obvious solutions to this challenge include –

1. Fragment the FC frames so that they fit within standard Ethernet frames.
2. Make bigger Ethernet frames

From a performance and simplicity point of view, option 1 should be avoided at all costs due to the fact that fragmentation adds complexity to the encapsulation and extraction process.  Complexity brings overheads, which in turn would slow the protocol down – which if we remember back to my initial FCoE post, is highly undesirable in networks transporting SCSI as a payload. 

Option 2, on the other hand, keeps the encapsulation/extraction process very simple – 1 FCoE frame for every FC frame. 

No fragmentation and reassembly overhead, no tinkering with the headers or payload, and vitally, no unnecessary protocol induced latency.  These are big wins for networks transporting SCSI.  This seamless handling of frames offered by option 2 keeps the encapsulation/extraction process very simple, very lightweight and ultimately, very fast!

To put this in perspective, here is as good a place as any to point out that the encapsulation and extraction process occurs at every hop on the FCoE network (we won’t talk about multi-hop FCoE here).  In other words, every time an FCoE frame traverses and FCoE switch, it is re-encapsulated ready to be moved on to its next hop. 

NOTE: Due the broadcast nature of the Ethernet networks, frames need a new source and destination MAC address for each hop.  Point being, if the encapsulation/extraction process was arduous, it would significantly impact the performance of FCoE.

Option 2 also keeps hardware designs simpler, allowing for cheaper hardware than would be required if complicated encapsulation and extraction techniques had to be implemented.

Fortunately the folks on the standards body (INCITS T11 FC-BB-5) decided on option 2.

Jumbo to the Rescue

But in order to go with option 2, we need to invent larger Ethernet frames.  Well actually we don’t need to invent them because they already exist and are called Jumbo Frames. Baby Jumbo frames (~2.5KB) provide enough space, plus some headroom for FCoE frames.
 
The diagram below shows an FC frame encapsulated within an FCoE frame.

Please feel free to correct the above diagram if its incorrect or unclear….

Although Jumbo Frames are not an officially recognised IEEE 802 standard, in fact due to technical reason they likely never will be, they are extremely widely implemented, well understood and about as standard as anything non-standard can be.  In fact, if jumbo frames did not exist, they would have to be invented to enable FCoE networks.

The same comment that came in from a reader talked about poor framing efficiency in FCoE.  Lets take a quick look at this while we’re here.

Data is protocol overhead

By “framing efficiency” we mean the number of control bytes sent versus total number of data byes sent.  A protocol with a high percentage of control bytes would be seen as inefficient.  Because of going with option 2 from above, FCoE actually has quite good framing efficiency, as it doesn’t carry baggage required to reassemble an FC frame from multiple FCoE frames.

I’ve heard it said that to techie people data is protocol overhead

Keep your hands off my bandwidth

Another reader has recently asked the following question

“….how is the IP traffic and the FCoE traffic kept from consuming one another’s bandwidth…”.

The answer is to do with Ethernet Priorities and I touched on them in a previous post and in the comments to that post, but while we’re here with the diagram above lets add a bit more meat to the answer.

I’ve written VLAN Tag in the above diagram, lets break it open a little more –

The above is a representation of an 802.1Q VLAN field.  This field also lists the Ethernet Priority (technically referred to as the Priority Code Point or PCP).  This is leveraged by Priority based Flow Control as discussed in earlier posts and is absolutely critical in the implementation of a lossless fabric, without which FCoE would be dead in the water.  PCP can be between 0-7 and each type of traffic will be encoded with its own PCP allowing features such as losslessness and bandwidth allocation etc to be implemented.  FCoE would have one value, iSCSI another etc…..

Hope that helps clear some things up.  Let me know if you have any more questions.  I’m enjoying the discussion that this is generating, and don’t be embarrassed about asking questions(obviously I don’t know all the answers but Im sure somebody out there will).

Nigel

I am currently on the market for FCoE related work and can be contacted at nigel AT rupturedmonkey DOT com

Last week at HP Tech Day in Colorado Springs, invitees were given a guided tour of the impressive HP StoprageWorks EVA tast lab.  Initially we were told "no cameras", but after being so impressed with the tour and facility, I badgered HP and they reversed their original decision and decided to allow the cameras in.  This was a really good move from HP, in my opinion, and kudos to HPStorageGuy Calvin Zito for not just fobbing me off on my request).  Not only do HP have nothing to hide, they actually have an impressive facility that they worth bragging about!
 
The actual video below is of a the second guided tour that we had from Tony Gregory the lab manager.  In the first video section Tony explains that the lab has -

• Over 1,200 servers
• 500 EVA arrays (although there didn’t seem quite that many from where I was looking*)
• Over 20PB storage

Also, based on those numbers, way over 1,000 fabric ports.  As Calvin puts it just before we enter the room "its your momas data centre".

* I should point out that the far end of the lab was off limits to us, and no I didn’t try and get a sneak peak.  From memory the off limits area was about 4 or 5 rows deep.  More than that I cannot say.

So the video below is an exclusive peak into the bowels of a major storage platforms support lab environment.  May I suggest that if you dont like this video then you dont like storage……………………… or you are a comptetitor

Enjoy the footage.

 
Although video footage is not quite the same as being there in person, I hope you get a glimpse of what an impressive facility this is and appreciate exactly what goes into delivering one of the best storage platforms in the world (and for the record Ive always been a fan of the EVA). 

For further written analysis and some great stills, I recommend you pop over to Ray Lucchesi’s writeup.

Two final points –

1. This is not the only EVA testing facility HP have.  On our first, non-videod tour, Tony mentioned at least one other facility, where they do "level 1 testing".
2. Sadly there was no SSD available for us to shoot.  But remember that we werent allowed to view the entire facility, so may be they had some in the "off limits" area at the back of the lab??

I was impressed!  Thanks again to Calvin and the folks at HP for allowing us in with the cameras so we can share with everyone else – hope you enjoyed it.

Finally, for the record, the EVA currently supports up to 8 SSDs.

Nigel
 

One of the rumours doing the rounds today is that Brocade are apparently up for sale or at least considering the possibility – I stress at the moment of writing this that it is pure speculation.  Clearly rumours like this must be taken with a bucket of salt and are often either blatantly untrue and sometimes calculated in their nature.

Anyway, the interesting thing from my point of view is that it was just last week that I cheekily said to somebody at HP, via email, that I thought HP should buy Brocade.  So with the current rumour in mind I might as well take a minute to explain why I think this would be a great move, not only for HP and Brocade, but also the wider industry……

First up, there is no doubt that there is a huge push towards convergence in the industry.  One topic Im thinking a lot about at the moment is the converged network (aka unified fabric) bringing together the likes of IP, iSCSI, FC and more, all on to the same cable, PCI adapter, switch ports…..  In this area Cisco are flexing their muscle and driving much of both the changes as well as leading the standards bodies.  Brocade and others are at the table, especially in the designing of the standards etc, but struggle to compete with the might of Cisco when it comes to getting products to market and gaining market share.

Being Brutally Honest.

Question 1: Is it realistic to expect Brocade to compete with Cisco in the unified fabric arena?

My thoughts are heavily influenced by the fact that I have never seen a Brocade Foundry switch (IP) out in the wild.  Now if Ive never seen one in my professional life, and Ive seen hundreds and hundreds of Cisco switches, I have to wonder whether they can ever compete.

Sure Brocade have a great FC capabilities and are very strong in that area of the market.  But they are relatively poor in the IP arena. 

Honest answer – Probably not good enough to compete with Cisco.

Question 2: Is it realistic to expect HP to compete with Cisco, based solely on their existing portfolio in ProCurve?

While I HAVE seen HP ProCurve switches in the wild, I have not seen many. 

Based on the way things have gone and are going in IP networking, and the fact that HP ProCurve does not offer FC, FCoE or DCB then they will probably struggle to compete with Cisco.

Interestingly though, on the unified fabric strengths and weaknesses comparison chart, HP  ProCurve is almost the exact opposite of Brocade – it has relatively good Ethernet/IP capabilities but not FC or FCoE.  When you compare them, its almost as if they are screaming out to be married up.

Introducing the HP ProCade platform

Bring the two together though – the unquestioned market leading capabilities of Brocade in FC and FCoE, and the number 2 player in the IP networking space, merge the technologies together and suddenly a potential challenger to Cisco materialises. 

Im calling it HP ProCade!

May be the HP ProCade Unified Network System (HP PUNS)!  OK may be not, but ProCade definitely has potential.

NOTE: Based on existing market share, and the fact that it would be HP doing the buying (I must re-iterate that this is 110% pure speculation) and Brocade doing the dancing, I would expect ProCurve to win over Foundry.

Granted takeovers and technology mergers take time (dont even ask about the Oracle SUN thing which is seeing the SUN side of the bargain destroyed while the European competition people drag their heels) and of course it would not be easy – but then again not many things in life really worth while are easy.  But I think for the industry this would be extremely worthwhile.  Competition is is what drives the industry forward.

Sadly, aside from this potential, I dont see any other serious challenge to Cisco.

Interestingly though, Im willing to bet that Cisco would be worried if HP were to buy Brocade.  Some of the guys on Twitter seem to be coming across as if they feel HP purchasing Brocade would pose a threat to Cisco.  At least far more likely to threaten than anything else on the table at the moment.

In summary, I would LOVE to see HP and Brocade come together and I think it would really give Cisco something to think about.  And for any Brocaders out there reading this, I think HP StorageWorks is in good hands, and I dont think being taken over by HP would be the worst thing in the world.  Im sure we’d all love Brocade to stay independant, but in order to survive and live on and challenge, may be a change is needed!

Thoughts and comments encouraged.

Nigel

Continuing with my current theme of Fibre Channel over Ethernet, and at the request of several people, this post will take a close look at Converged Network Adapters (CNA).

Now then, there is no easy way to put this, but to do justice to a topic like this will require a lot of words.  I’ll do my best to keep it succinct, but if you are looking for a high level overview in under 1,000 words then this is probably for you, but thanks for stopping by………  However, if you want a deep dive and opportunity for technical discussion, then this might be what you’re looking for.

Still here? Magic, let’s go…..

First things first, its best to have a quick look at what things look like without CNAs, in order to more fully appreciate some of the problems CNAs are resolving.

 
Before CNAs

In a traditional server with no CNA cards and not connected to a unified fabric it is not uncommon to see the following “network” related sprawl (sorry about the terrible diagram but I wrote most of this post while on an aeroplane) –

Diagram 1

 
The FC HBAs tend to run at either 2, 4 or 8Gbps with the NICs usually running at either 100Mbps or 1Gbps.

The multiplicity of NICs is common to meet the demands of multiple traffic types such as; production, backup, management etc.

It is also worth noting that most servers are built with two physical HBA cards installed to provide both redundancy and increased aggregate bandwidth/performance – commonly referred to as I/O multi-pathing and is a must in 99.99% of FC SANs.

But that was then and this is now….. or nearly now

Say hello to the Converged Network Adapter, or CNA for short.

The Converged Network Adapter is exactly what it says it is – an HBA and a NIC converged on to a single PCIe adapter –

Diagram 2 – Picture courtesy of Emulex

Digging into the technical detail, some CNAs have a single ASIC that performs both the HBA and the NIC functionality, whereas other have a separate ASIC for each of the distinct functions.  This really depends on the vendor and model.  The important point being – CNAs provide HBA and NIC functionality in hardware, making them fast and reducing CPU overhead.  Implementing functionality without thieving CPU cycles is a big plus-point in server virtualisation environments.

Offload EVERYTHING!!

Considering the fact that stealing CPU cycles is undesirable at the best of times and a federal offence punishable by death in virtual server environments, providing hardware offloads is vital.  To help out in this area, the latest raft of Generation 2 Emulex OneConnect Universal Converged Network Adapters, such as the LP2100x, provide full CPU offload for all protocols on a single chip design and are FIP compliant!  That’s not just FCoE offload, we’re talking about TOE and iSCSI as well – the iSCSI one is interesting and may be a chat for another day!

NOTE: I will point out that I had a chat recently with Emulex and was very impressed.  They are laser focussed (pun intended) on FCoE and really understand the market and value position of FCoE.  A real pleasure to chat with passionate like minded people, oh and they didn’t hang up on me when I mentioned "Broadcom"   VERY unprofessional of me but I was unbelievably jetlagged when we spoke, so I thank them for overlooking my lack of tact and poor humour.

So with the fact that CNAs provide both HBA and NIC functionality, as well as connecting to 10Gbps Enhanced Ethernet fabrics, it shouldn’t take a rocket scientist to figure out that we could swap out our 6 PCI adapters from Diagram 1 and replace them with just 2 CNAs (2 for redundancy).  If you think about it, this has the potential to reduce PCI adapter count and cable sprawl by crazy%

Saving Space

Early generation CNAs are installed as PCIe adapters in servers and as PCIe mezzanine cards in blade servers.  However, the way forward is to eventually have them embedded on the motherboard – a la’ LOM – vendors currently have this on their roadmap.  But this is no biggy right?  Well actually this has the potential to hugely reduce the amount of space consumed by PCI adapters inside of servers paving the way for higher density in blade servers where real-estate is at a premium!

Now this brings up another interesting benefit.  Not only is FCoE and its associated technologies resolving todays problems, it is actually enabling a better future.  How good would it be to have 6 or more CNAs per blade server!  That’s 60Gbps+ of flexible bandwidth based on today’s current 10Gbps Enhanced Ethernet!

Now this becomes a real winner in light of the current raft virtualisation optimised CPUs on the market – think Intel Nehalem and associated  virtualisation technologies like Intel VT-x VT-c, VT-d VMDq – xpect the ratio of VMs per core to rocket skyward!  In order to keep pace with the advancements in chip technology the I/O subsystem needs to move in step.  CNAs and Lossless Enhanced Ethernet are vital to this.

This point aside, I recently had a hand in a design involving HP BladeSystem c7000 technology.  I remember during the design wishing that the system had CNAs instead of the Flex10 and Virtual Connect technologies.  Could have saved space and cabling as well as potentially given more flexibility.  However, this was prior to even FCoE being ratified by INCITS.

Obvious benefits summarised

So some obvious benefits that CNAs bring include – reduced number of PCI adapters, cables, space, power and cooling.  Nice!

Secondly, by supporting 10Gbps Enhanced Ethernet they provide greater bandwidth than existing adapters allowing more traffic types and volume to travel down the same stretch of cable.  Suddenly the wire once nirvana is now a reality (actually wire twice if you want true I/O multi-pathing).

NOTE: A quick heads up on throughput.  While CNAs provide access to 10Gbps CEE networks, not all services, including FCoE, are supported at full bandwidth.  For example, CNAs normally support Ethernet at 10Gbps but FCoE only runs at a max of either 4Gbps or 8Gbps.  This is in line with 4Gbps and 8Gbps FC – there is currently no standard for 10Gbps FC N-Port to F_Port connections.

<OK so this is about the half way point, so you might want to come up for some air here before discussing the interesting stuff>

Clever things with CNAs

Now to the deeper technical stuff………..

With 10Gbps Enhanced Ethernet, FCoE, CNAs and other associated technologies being relatively new, the standards bodies (such as INCITS T11 FC-BB-5 for FCoE) as well as the vendors are able to design with virtualisation in mind.  This is great!

Let’s mention a couple of technologies that bring a lot to the table in Hypervisor environments.

NPIV

The first technology worth mentioning is N_Port ID Virtualisation, otherwise known as NPIV.  NPIV is a T11 INCITS and ANSI standard initially developed by IBM and Emulex, and yes I know it’s not exactly new.

Because NPIV is not a new technology I wont spend much time on it here other than to say NPIV makes it possible for a single HBA to have several N_Port IDs and WWPNs, allowing virtual machines to be uniquely addressable on the SAN and therefore able to be managed on the SAN just like normal physical servers.

If it would be useful I can put something up on NPIV.  Just leave a comment at the bottom asking and if enough people ask I’ll post on it.

I/O Virtualisation (SRIOV)

So while NPIV allows multiple VMs to be uniquely addressable on the SAN side of an HBA, on the other side, the side of the servers PCI tree, there is still a 1:1 mapping between physical ports and addressable PCI devices.

The Middle-man

Early virtual server implementations see the hypervisor act as middleman, between the I/O adapter and the Virtual Machine (VM).  The VM never actually talks directly with the I/O adapter, always through a middle-man – the hypervisor.  But like any middleman scenario, it has its advantages and disadvantages.  The middleman would argue that he adds “value”, but is rarely so keen to point out that he also always takes a generous percentage of the spoils (in our case, I/O performance and CPU cycles).

Some of the advantages and disadvantages of having the hypervisor as the middleman might include –

Middle-man Advantages

• Hypervisors often provide snapshot capabilities
• Hypervisors often provide thin provisioning capabilities
• Many existing Hypervisor technologies are currently designed around this model

Middle-man Disadvantages

• Lower I/O performance.  Because the hypervisor handles all I/O to and from a VM, as well as the associated interrupt handling etc, this injects server-side latency in to the I/O path as well as stealing CPU cycles form the main role of the Hypervisor.
• Security concerns.  The fact that all I/O, to and from VMs, is seen and touched by the hypervisor may be a concern to some people :-S
• Limited feature set.  You are restricted to the features and functions provided by the Hypervisor and not exposed to the full feature set available from the manufacturers native driver….

A common example where having the Hypervisor act as the middle man is not seen as desirable is a high throughput OLTP type system.  These are rarely virtualised due to, among other things, the I/O performance impact associated with having the Hypervisor as the middle-man.

So what does SRIOV do to help?

NOTE: Single Root I/O Virtualisation (SRIOV) and Multi Root I/O Virtualisation (MRIOV) are extensions to PCIe brought to us courtesy of the electronics industry consortium known as the PCI Special Interests Group (PCI-SIG).

SRIOV implementations allow an I/O adapter, in co-operation with the Hypervisor, to be sliced up in to multiple virtual adapters.  Each of these uniquely addressable on the servers’ PCI tree.  Each virtual adapter can then be mapped directly to a virtual machine and in turn is directly addressable by that virtual machine – no more middle-man.  SRIOV based adapters may have dedicated I/O paths in silicon and work alongside other related technologies such as Intel VT-c, VT-d, VMDq…. to provide a huge variety of offloads and assists aimed at reducing the load on main CPU and memory systems and thus increasing system performance.  This mode of operation is sometimes referred to as “hypervisor bypass” mode (we should be thinking VMDirectPath by now) and offers close to full line rate, making virtualisation a more realistic option for transaction based and other I/O intensive systems like our example OLTP system.

This also opens the door to things like VMs running drivers provided by the manufacturers making new functionality immediately available without waiting for the Hypervisor middle-man to support it.  Obviously it all needs buy in from your Hypervisor …..

Of course SRIOV is a semi-open standard (I say semi-open because several months ago when I researched it you had to pay to get access to the standards docs) and like most standards each vendor is free to implement the specifics in their own unique and value-add way as well as to add more features and requirements around it.

So in a nutshell, NPIV enables a single port to have multiple discrete addresses on the SAN side, whereas SRIOV allows a single I/O adapter to have multiple discrete addresses internally on the servers PCI bus/tree.  The diagram below from a long time ago on the Cisco website shows an I/O card that can be partitioned in to 128 (0-127) virtual interfaces.

Diagram from the Cisco website

Say goodbye to hardware rip and replace

By implementing the above mentioned technologies it becomes possible to run a single cable to a server and then use management software to configure the virtual interfaces according to current requirements.  For example, removing a NIC function and adding an HBA function becomes just a software change – no requirement to physically swap out a PCI card or lift the floor and run new cables.  Simply make the change in software and the new device will be presented to the servers PCI tree, job done!  Sound good to anyone else?

There is more, but I doubt anyone would read more than this in one post.  Other topics can always be discussed in the comments section below….

So if you made it all the way to hear, thanks and I hope it was useful.  Please feel free to join the discussion either here on the blog site via the comments section below, or by following me on Twitter @nigelpoulton.  I only talk about storage and related technologies.

Nigel

PS.  I am an independent consultant and can be contacted at nigelATrupturedmonkeyDOTcom

Amid all the discussion of FCoE and the many requests to cover more aspects and underlying technologies, I thought it probably a good idea to take a step back for a moment and recap flow control in native FC networks.  After all, FCoE is almost a transplant of FCP onto Ethernet.

Granted, to some people this will be teling you how to suck eggs, but for others clearly not!  So………. native FC Class 3 networks achieve reliable frame delivery/lossless behaviour through the use of a link layer flow control mechanism called buffer-to-buffer credits (aka BB_Credits) 

NOTE:  Class 3, in FC parlance, basically refers to a connectionless network without acknowledgements.

Class 3 is the most common class of service in todays FC SANs.  It is connectionless without notification – somewhat similar to UDP in LAN environments.  However, and very importantly, if a SAN is well designed and quality components and installation practices used, the lack of acknowledgements is not an issue – in fact it’s actually a bonus.  Why have an ACK related overhead when the network does not lose frames.

At a high level, the FC BB_Credit scheme works on the principle that a sender cannot send frames unless it explicitly knows that the receiver is able to receive and process it.  This behaviour ensures that congestion does not arise and therefore frames will not be discarded.  See the diagram below –

NOTE:  I should point out at this stage that link layer flow control in Enhanced Ethernet networks is based on PFC (802.3Qbb) and not BB_Credits – see my previous post here.  This is one of a small number of differences between FCoE and FC, and as a result BB_Credits are not part of the FLOGI process in FCoE networks.

The FC BB_Credit system is a hop-by-hop system, it has no notion of an end-to-end connection and only works between N_Ports and F_Ports and not between two N_Ports.

Each device on a native FC network establishes its initial BB_Credit pool with the fabric switch that it logs in to (FLOGI).  E.g. An HBA that is plugged in to the fabric and establishes an initial BB_Credit pool with the switch port it logs in to.  Each BB_Credit then represents a single frame that the HBA is able to send.  Correspondingly, each time a frame is sent, the sender must decrease the number of BB_Credits in its pool by 1. 

The number of BB_Credits a device can issue to connected devices is based on how many buffers they have (HBAs and switch ports have physical buffers, send buffers and receive buffers).  More buffers = more credits.  However, more buffers also = more ££$$.

At the other end, receivers transmit R_RDY primitive signals to notify senders of the availability of receive buffers.  This is the means by which senders replenish their BB_Credits.

If senders only had 1 buffer credit and had to wait for that to be replenished after every transmit, this would lead to horrendously low link utilisation.  To more efficiently move data on the FC SAN, multiple frames are sent consecutively so that senders do not have to wait for their pool to be replenished after every frame sent. 

Data can be sent on the transmit line and credits replenished, via R_RDY primitives, on the receive line in full duplex fashion.

And FCoE?

Although the FC Class 3 buffer credit system is different to Priority based Flow Control (PFC) in Enhanced Ethernet and used by FCoE, the end result is essentially the same – a lossless network!  In fact, if the truth be known, the FCoE system of using PFC is actually superior

It appears that people have a natural tendency, especially FC folks, to associate Ethernet with congestions and routinely discarded packets.  But such comments and feelings are referring to your grandmothers Ethernet, and we need to come to understand that FCoE networks are not built on your grandmothers Ethernet, they are built on Data Centre Ethernet/Converged Enhanced Ethernet – which is another beast altogether.  In fact I may go so far as to suggest that if they have Ethernet in heaven it will be DCE/CEE!

Nigel

You can follow me on Twitter (I only talk about storage and virtualisation) –
http://www.twitter.com/nigelpoulton – or @nigelpoulton 

You can also reach me at nigel at rupturedmonkey dot com – Im available to hire as a freelance consultant

In keeping with my current focus of attempting to shed some light on FCoE and help people learn and realise some of the beneits, the link below is to a video that Cisco will be putting up tomorrow (10am PDT).

Its an interview with Sr Director of IT at the University of Arizona, where they have had Cisco Nexus 5000 kit installed for nearly a year and a half and running FCoE in production.  Towards the end of the video they promise to cover off some FCoE myths too – and there are plenty of them around!!

Promises to be interesting.  If you cant make the live broadcast then it should be available as playback after the event.  If its of interest I’ll keep the link up, if not then in a day or two this post will have mysteriously disappeared in the ether, never to be seen again

The following link has been updated.  Cisco had some technical difficulties and the old link did not work.  This one does!  This post refers to the video featuring Derek Masseth.  Apologies for the old broken link.

http://tools.cisco.com/cmn/jsp/index.jsp?id=92757 

Nigel

You can follow me on Twitter (I only talk about storage and virtualisation) –
http://www.twitter.com/nigelpoulton – or @nigelpoulton 

You can also reach me at nigel at rupturedmonkey dot com – Im available to hire as a consultant for speaking, writing and consulting

Ewan Leith (@ewantoo) asked if I could whip something up re cables and infrastructure required for FCoE (DCE/CEE).  So here goes. 

I think there has been a lively discussion on FCoE over the last 24-48 hours and it would be good to keep it going…… Before I start I’d be interested in any feedback, updates comments and the likes – I havent had a chance to properly research this and my schedule for the next few days makes it difficult.

Anyway……………… in order to accommodate the enhancements and improvements that come as part of what I have been calling “Enhanced Ethernet” (DCE/CEE), as well as facilitating the goal of consolidation to a single unified fabric, there are obviously some physical infrastructure changes required.  In the next few paragraphs we will discuss some of them.

NOTE:  When I say Enhanced Ethernet I’m referring to 10GigE, lossless, low latency, ETS, congestion notification as seen in DCE and CEE…

First up, cables!

Unfortunately our existing install base of Cat5 and Cat6 unshielded twisted pair and the likes, for the most part do not meet the demands of the unified fabric.  They don’t kcik it when it comes to latencies and Bit Error Rate etc.  In order to deploy and use Enhanced Ethernet, and therefore FCoE, we need to lay shiny new cables.  No problems though, that’s cheap and easy, right? 

<cough cough>

In order to keep with some of the major goals of convergence (drive down costs and power consumption), a cable and transceiver combination with low power demands and that has a good cost value is required.  This predominant combination is a passive or active twinaxial copper cable with SFP+ transceivers – sometimes referred to as “SFP+ Copper”. 

Twinaxial cables, usually referred to as Twinax or twin-ax, and named as such because they have a dual core.  The specification generally allows for cable runs of up to ~10 metres, although some kit combinations may support longer or shorter runs.  Twinax copper cables are typically ~6mm in diameter.

Twinax can transmit at 10Gbps full duplex (half duplex is not specified or supported in the IEEE DCB standards for 10GigE) over distances of up to 10 metres.  Although at first glance this is a relatively short distance, it is actually fairly well suited to Data Centre environments, which are typically short range high speed networks (remember we are not running these cables to workstations).  Such run lengths are especially suited for routing within a single cabinet or adjacent cabinets, such as from a server to an access layer switch mounted in the top of the rack. 

If longer runs are required then fibre can be used but at a higher cost. 

Twinax copper has been rated with a bit error rate in the region of 10−18 making it ideal for FCoE. 

Bit error rate:  or BER for short, is the ratio of erroneous bits received divided by the total number of bits transmitted.

SFP+, or to give its full name Small Form-factor Pluggable Plus, is an extension of the popular SFP standard seen in Fibre Channel SANs and legacy Gigabit Ethernet.  The design of SFP+ is relatively simple for a transceiver. 

In saying the design is simple, I refer to the fact that much of the signal processing circuitry and logic, often found in the transceiver module (such as with XFP), is removed from the transceiver and relocated to the switch and CNA (Converged Network Adapter).  This allows SFP+ to be smaller and cheaper in comparison to the more complex XFP, allowing for higher density switches.

As can be seen by the two diagrams below, SFP+ modules exist for both copper and glass and can transmit at 10Gbps.

SFP+ Optical transceiver

SFP+ copper transceiver with casing removed

Obviously this transferring of logic (silicon) from the transceiver to the switch and CNA may merely offset the cost from the transceiver to the switch.  While this may be the case it is generally agreed that such logic is better placed in the switch and CNA and usually allows for overall cheaper manufacturing costs. 

Backplanes

While on the topic of cables, copper and glass…..  As well as standards for carrying 10GigE over copper and fibre cables, the IEEE has also defined standards for backplane implementations.  One such commonly implemented standard is 10GBASE-KR.  This is commonly seen in blade servers as well as routers and switches and utilises a single lane running at a baud rate of 10Gbps, sometimes referred to as 10Gbps Backplane Ethernet.  It supports distances of up to 1m on copper based circuit boards which is plenty of distance for intra-chassis communications.

Other Infrastructure Requirements

At this point I suppose I should mention CNAs.  However, I can’t realistically talk about CNAs without talking about things like NPIV, SRIOV which is a topic and a half in and of itself.  So, for now I’ll skip over CNAs and say a quick word or two about FCoE capable switches.

FCoE capable Switches

A ton of new Ethernet standards as well as a new ULP, new network adapters and new cables inevitably leads to one thing…….. new switches.

Fortunately FCoE and the aforementioned enhancements are evolutionary.  By saying that, I am implying that implementing them in your Data Centre does not have to cause huge upheaval.  Some upheaval yes, but there is no requirement for large scale rip-and-replace.  In fact FCoE and her attendant technologies will happily site side-by-side with the likes of native Fibre Channel and 1Gbps Ethernet and at many levels the adjacent technologies will not even bat an eyelid.

Starting at the Edge

In order to simplify and expedite the adoption of Enhanced Ethernet and especially FCoE, some switch vendors provide edge switches at the access layer that allow companies to start deploying CNAs, at the edge of the network, and have them feed in to existing FC and Ethernet backbones via edge switches. 

For example, a company may deploy a new blade farm fully equipped with FCoE capable 10GigE Converged Network Adapters.  These CNA ports can be connected to the network via edge switches that support GigE, 10GigE, FCoE and FC.  This allows the blades to connect via 10GigE Enhanced Ethernet and then branch out to the existing network core via the 1Gbps Ethernet ports and to the FC SAN via the native FC ports.

These switches tend to be 2u pizza box style switches that are deployed within the server rack.  For some, they are a good place to start, especially in situations where ripping out your existing core and replacing it with shiny new 10GigE Enhanced Ethernet kit is not an option (i.e. just about everywhere).

Also at the Core

Some switch vendors also offer ultra-high performance ultra-scalable 10GigE Enhanced Ethernet FCoE aware switches for the network core.  These are typically modular blade based switches supporting multiple interface types and scaling to hundreds of ports.  Interface types include;10GigE Enhanced Ethernet, 1Gbps Ethernet, 4Gbps FC and 8Gbps FC.  These switches represent the next generation of data centre switches and represent a huge move toward the virtual data centre and I/O consolidation.

As always, comments and thoughts welcome.

Nigel

You can follow me on Twitter @nigelpoulton – I only talk about storage and virtualisation.

I’m a freelance consultant and can be contacted at nigel at rupturedmonkey dot com

Following on from Hu Yoshidas apparent blunder over FCoE and lossy networks, I thought I’d do my bit to clear things up and shed some light.  Knowing a thing or two about FCoE I’m regularly amazed at how little some people know…….

So the following is a mini tutorial on the flow control and lossless Ethernet in FCoE networks.  If you don’t care about FCoE, then point your browser somewhere else and don’t come back   On the other hand, if you are interested and want to know more, then put your feet up for 10 minutes and read on……..

Not your grandmothers Ethernet

FCoE requires an Enhanced 10GigE Ethernet network – it does not run over standard gigabit Ethernet.  Depending on who you speak to, this Enhanced Ethernet usually goes by one of two names –

1.    Converged Enhanced Ethernet (CEE).  Commonly used by IBMers and Brocadites.

2.    Data Centre Ethernet (DCE).  This is trademarked by Cisco and according to the Cisco website refers to the company’s architecture for next generation Ethernet in the Data Centre and is a superset of the DCB standards with some additions including L2MP.

As my wife often tell me that I live in my own little world, and to remain neutral, I’ll refer to it as simply Enhanced Ethernet.

Whatever you decide to call it, it encompasses a collection of new technologies required in order for it to be able to transport FCoE traffic (frames).  Although FCoE is not the only driving force behind Enhanced Ethernet, it is certainly a major force.  Some of the technology changes encompassed in Enhanced Ethernet include the following –
 
•    Priority Based Flow Control (PFC) / Lossless behaviour
•    Low latency
•    Improved aggregate bandwidth
•    New link level negotiation protocols
•    Congestion Notification
•    Enhanced Transmission Selection

In this post I’ll concentrate on how lossless behaviour is implemented and achieved.  To do this its best to start somewhere near the beginning –

Its all goes back to SCSI

As the following very simple diagram shows, in FCoE networks, SCSI is encapsulated in FC frames, and FC frames are encapsulated in FCoE frames. 

So indirectly, SCSI is encapsulated in FCoE frames, meaning that FCoE and the underlying Ethernet network must keep SCSI happy.

So how do we keep SCSI happy?  If we cast our minds back, we should hopefully remember that SCSI was originally designed to be used within the confines of a physical server chassis, running uncontested over short parallel cables.  Uncontested = zero contention.  As a result SCSI was not designed to deal well with delays or transmission errors.  In fact, when either occurs, SCSI deals with them poorly.

In a nutshell – drop frames carrying SCSI payloads without efficient recovery capabilities (further up the stack) and you will be in a world of hurt! 

To keep SCSI happy you really need low latency and lossless behaviour.

What is a Lossless network?

Put very simply, a lossless network is a network that does not drop frames. 

The corollary being a “lossy” network – a network that drops frames when congestion occurs.  Your grandmothers GigE (1Gbps full duplex) network is lossy, it drops frames under congestion.

Making Enhanced Ethernet Lossless

The Data Centre Bridging Task Group has decided to implement link layer flow control in Enhanced Ethernet via a mechanism called Priority based Flow Control (802.3Qbb), or PFC for short.  PFC is the how losslessness is achieved on Enhanced Ethernet networks.  I might have just invented the word “losslessness” :-S

However, before we dig into PFC it is worth taking a side-step to briefly talk a little about Ethernet priorities.

Ethernet Priorities:  IEEE 802.1p defines 8 priorities for Ethernet.  These priorities allow for the implementation of Classes of Service at the link layer by tagging certain traffic types with an encoded priority.  Implementing Classes of Service allow available bandwidth to be divided in to logical lanes, or virtual links.  These virtual links can then be leveraged by other protocols and services, such as Priority based Flow Control which we will talk about.

The diagram below shows how a physical link between two switches can be divided in to 8 logical lanes/virtual links labelled CoS 0 through CoS 7 –

PFC is IEEE 802.1p Ethernet priorities “aware” and applies intelligence in the form of selective enforcement of the Ethernet PAUSE condition.  This is where the pause condition is selectively applied to particular Classes of Service.  This makes PFC perfect for converged unified fabrics where multiple traffic types and classes of service share a common network.  It also makes PFC superior to native FC BB_Credits which can only apply arbitrary conditions that affect all traffic on the link.

Interestingly, while PFC achieves the same, albeit superior, results as FC BB_Credits – that of creating a lossless network – the technical specifics of the two implementations are vastly different.  On the one hand FC BB_Credits require the sender to assume it cannot send frames until it explicitly knows that the receiver is ready.  On the other hand PFC allows senders to assume that they can always send unless explicitly told not to.

Hitting the Pause Button

PFC enforces the Pause condition per Class of Service by issuing a PFC frame with 8 time fields, one for each previously mentioned CoS/Ethernet priority.  When a switch issues PFC frames it is instructing the connected node to apply the PAUSE condition (i.e. stop sending) to frames with particular CoS values.  The diagram below shows the PAUSE condition applied to all Classes of Service except CoS 3 –

As well as the classes to which it is applied, the duration of the pause is also specified within the PFC frame.  This allows PFC to be selective in which class of traffic it will apply the Pause condition to, making it possible to enforce the Pause condition on a single, or a subset of all 8 classes.  It is also possible to selectively lift the pause condition, such as when congestion has dissipated and there is no need to wait until the Pause timeout expires.  This adds to the functionality of PFC.

PFC frames are handled at the MAC layer similar to how R_RDY’s are handled by FC-1 in native FC networks.  The PFC frame is a standard non-tagged MAC Control frame identified by Ethertype 0×8808 with Op-code 0×0101.  For best performance and efficiency all good FCoE switches handle flow control in hardware.

In Enhanced Ethernet networks FCoE will be assigned to a Class of Service which in turn will be treated a high priority so that when congestion starts to occur in the network (per switch) it will be allowed to continue to operate while other Classes of Service (protocols) might be paused.

Net result – lossless behaviour for FCoE frames on shared Enhanced Ethernet networks (unified fabrics).  All implemented at the link layer by the FCoE capable switches.  Voila!

Comments and thoughts welcome.

Nigel Poulton

I’ve recently moved to a new website http://www.nigelpoulton.com where future articles and discussion will be posted.

I’m a freelance consultant and can be contacted at nigel at rupturedmonkey dot com

UPDATE:  I have removed this post due to glaring innacuracies – I trusted my engineer (never again!!) and didnt do my background checking.  Apologies and no harm intended.