Linkage

Background: This customer has a very simple SMB setup as far as VMware goes. 3 servers in a single cluster, only a few vSwitches per host, and a HDS AMS disk subsystem. They wanted to upgrade to vSphere and are capable of taking some downtime to do it. (Not that downtime is required.) They bought 3 new Dell 710 servers to run vSphere on.

There are at least 20 different ways to go about this and I wanted to keep this upgrade as simple as possible Since the customer can take some downtime for their VMs I decided to do a Cold Migration of the VMs. This is by far the simplest and runs the least amount of risk for having problems with the VMs. The customer, while understanding VMware and the administration of it, could not troubleshoot issues once I was no longer onsite.

Here is the process that we walked through for a successful upgrade and migration:

1. Upgrade vCenter to version 4 – You need to read through the upgrade guide before you attempt this upgrade. There are some specific permissions changes to networks and datastores that could bite you after an upgrade if you don’t understand the way that vCenter changes read-only attributes. We’ll leave it at that. RTFM!

2. Install new ESXi servers on new hardware and add them to the existing cluster. Ensure all updates and patches have been applied. – Let’s discuss a few key reasons that I pushed this user to go with ESXi. All new implementations I have done over the past year have all been done with ESXi.

A) They were not running any fancy monitoring tools in the Service Console.
B) ESXi is a much smaller attack surface for hackers since RedHat is no longer running underneath.
C) Because there is no RedHat, VMware has full control over all patches which will allow for much quicker turn around on bug fixes, etc.
D) The user didn’t have any scripts or jumpstart servers that required the Service Console to run. Even if they did, we could have re- written them to run against the vMA (Virtual Management Appliance).
E) No Service Console! Personally I see more people get on the Service Console and make a mistake that causes a lot of problems. “Let’s see what this command does.”
F) THERE IS NO TECHNICAL REASON NOT TO GO TO ESXi!!!!!!

3. Map out all networks and datastores attached to 3.5 cluster – Self explanatory. You need to know what you need to know.

4. Create identical networks on new vSphere cluster – Again SE. The VMs will have to connect to the network after the migration. If the networks aren’t there, migration validation will fail.

5. Test cold migration to local storage and test networks with a test VM – Just test things out prior to the big outage.

6. Remediate any changes that need to be made. – SE. Fix any problems that pop up.

7. Present all datastores that reside on the old ESX servers to new ESXi servers. Verify connectivity, specifically LUN IDs. – Since the new ESXi boxes are part of the same cluster we can present the same LUNs/Datastores to them without any potential problems. Scan them and bring in the VMFS Datastores and verify everything looks OK.

8. Shut down all VMs – SE.

9. Cold migrate all VMs to new ESXi servers. – SE.

10. Remove all ESX servers from cluster. – This will ensure that during the next step DRS won’t try and spread the workload to the old servers. Just make things cleaner and less checking on hosts for DRS as a whole.

11. Upgrade the VMware Tools and Virtual Machine Hardware on all VMs. – Using Update Manager go ahead and upgrade things prior to putting them back online for your users. A reboot is required so the users would see interruption anyway. I’m not going to walk you through this. RTFM!

12. Install the vMA for command line management of the ESXi hosts. – SE.

On the ESX vs. ESXi front, I truly believe that once you try it you won’t even notice a difference and you’ll probably like ESXi better. If you’re an old ESX shop that loves the Service Console, then I would challenge you to use the vMA for a month and I would bet that you won’t go back.

Well now that I’m a Mac guy I can’t simply run the batch script anymore. So here is what I did. I’m running VMware Fusion on my Mac and I have a management Windows XP VM which is used for various things. I shared my Mac HD with that VM and was able to browse to the location where I store the script and all of the VMware docs. Next I had to add a path to the Environment variables in Windows so that it could find wget.exe. I simply added “; \\vmware-host\Shared Folders\Macintosh HD\Users\username\Stuff\vmware stuff\Documentation;” to the path statement. After that you’re good to go. Double click the batch file and it will download the files to the c:\windows directory. When complete, copy the files to your Mac HD.

Hope this helps some of you.

To help with this I spoke with Roberto Basilio at HDS.  Roberto heads up Product Management for Platforms and knows a thing or two about how HDS fits into the larger Hitachi organisation.  Oh and he knows plenty about the HDS product line-up too!

Urgh? HDS and Hitachi are the same company right?

Actually, they’re not!

Starting from the beginning…… Any press release or official company announcement from HDS is tagged with the following “Hitachi Data Systems Corporation, a wholly owned subsidiary of Hitachi, Ltd (NYSE: HIT) and the only provider of…”.  So, that’s that, clear as….. mud!  Well, that’s probably unfair, but with all of the FUD and underhand comments that get spouted from the competition, one could be forgiven for thinking that HDS is some b@$tard child that Hitachi, Ltd is ashamed to own.  Others would have you believe that HDS as a mere sales division with absolutely zero influence on product development and futures.  While none of which are true, they do a pretty good job of confusing some people.

Subsidiary Family

One of the first impressions I got from speaking with Roberto was that he is not a fan of the term subsidiary.  He feels that HDS are an integral part of the larger worldwide Hitachi IT products group – a valued member of the larger tightly-knit Hitachi family.

A couple of examples of the closeness and intimacy of the relationship include the following –

1.  Minoru Kosuge, current Chairman of the Board of HDS, has previously served as division president of RSD (RAID Storage Division at Hitachi, Ltd) where he headed the development of the Universal Storage Platform (USP).

2.  Yoshinori Okami, current Executive Vice President and Chief Strategy Officer at HDS, has previously served as President of the Disk Array Systems division of Hitachi, Ltd.

According to Roberto, senior management folks moving around within the greater Hitachi IT products divisions is commonplace.

Personally I’d be willing to bet that HP, as tight as HP’s relationship is with Hitachi, Ltd is, are not as tight as with Hitachi Ltd as that.  HDS are family!

I should point out that Roberto was quick to acknowledge the value of the Hitachi and HP relationship and was in no way derogatory toward HP.  He even pointed out that Hitachi, Ltd also resell some HP products….

We’ve got more than you!

In the HP and Hitachi article, I quoted James R. Wilson, HP StorageWorks XP Disk Array PRoduct Manager “Hitachi has 6-8 engineers on site in California to work with HP’s engineering teams on a continual basis…”.  On that topic, Roberto was able to confirm what I already knew – the number of Hitachi Ltd engineering type staff on site at HDS corporate offices far exceeds the 6-8 at HP! 

Roberto also confirmed that HDS staff are regularly over in Japan and that he himself had recently been over in Japan.

We designed that

Something that some of the competition would like to believe is that HDS have no involvement in product development and futures.  They would have you believe that the reason HDS don’t talk futures is because they don’t know futures.  They would have you believe that the first HDS knows of a new product is when it arrives in a large box from Japan with a note on the side saying “Unpack carefully, take photos, upload to web, start selling tomorrow”.  From what Roberto tells me, this is a long loooong way from the truth.

Taking the USP family as an example…. While it is true that the hardware development is done in Japan, HDS have been directly responsible for the design of some of the most important and well known USP features including the following –

• Hitachi Universal Volume Manager – Controller-based external array virtualization
• Hitachi Dynamic Provisioning – Thin Provisioning, wide striping…..
• Hitachi High Availability Manager – USP V Clustering

To be clear, while the above were developed by RSD in Japan, the design and requirements were lead by HDS in the US.  According to Roberto, USP V clustering was 100% driven by HDS.

Other examples of products under the remit of HDS in the US are –

• Hitachi Storage Command Suite (oh how I wish they would stop renaming things every other week)
• HCP (formerly HCAP). 

HCP is 100% owned and developed by HDS.

Speaking with Roberto I quickly got the picture that there is a lot of collaboration, knowledge and staff sharing between the two.  It certainly doesn’t seem like HDS are a mere sales division or disowned child!

SANRISE Universal Storage Platform

One final point.  While it is true that HDS do not sell in Japan, HDS are now responsible for worldwide marketing of the USP and AMS family of products.  What this means is that while Hitachi Ltd sell in Japan, due to cultural reasons (fair enough), they now sell disk arrays under the same marketing names as HDS.  So whether you buy your DKC610I from Hitachi in Japan or HDS anywhere else in the world, it will be called the Hitachi Universal Storage Platform V.

Previously Hitachi Ltd had marketed the RAID400, RAID500… family of arrays SANRISE 9900 etc while HDS had marketed them under the names Lightning 9900, Universal Storage Platform etc.  While there is very little in a name, the competition occasionally touted this as an example of the two companies apparently not speaking with each other   I guess we can put that one to bed now!

As well as worldwide product marketing, HDS are also responsible for worldwide product management and product requirements.  That’s a fair amount of trust and responsibility in my book.

When I asked Roberto if there were ever collaboration meetings between Hitachi Ltd, HDS AND HP…. he hesitated, laughed and left me feeling that indirectly there might be

Comments are disabled for this article as it is a re-post from my new site.  Feel free to leave a comment on my new site over at Technical Deep Dive.

Nigel

One of the key technologies or principles in Rack Area Networking (RAN) is I/O Virtualisation (IOV).  In fact, IOV is about to rock the world of physical server and Hypervisor design.

If you work deploying VMware, Hyper-V, XenServer etc or if you have anything to do with the so called Virtual Data Centre, then you need to be all over IOV.

This is the second post in my mini-series on RAN and IOV.  In this particular post Im going to talk about the concept virtual adapters – Virtual NICs and Virtual HBAs.

The vNIC and the vHBA

The concept is simple: Take a physical NIC, perform some magic on it, and make it appear to the OS as multiple NICs.  Same goes for HBAs.

The diagram below shows a single physical NIC carved into 4 virtual NICs (vNIC) and a single HBA carved in to 4 virtual HBAs (vHBA).

The benefits of such technologies should be obvious – higher utilisation, requirement for fewer physical NICs, fewer cables, and fewer edge switch ports – just to name a few.

Another added benefit is flexibility.  Assume you have a 10Gbps NIC in a server which you have carved in to 2 vNICs.  That server now has a requirement for an additional NIC.  You no longer have to power down the server, open it up, install a new physical card and then wait for new cables to be laid.  Instead, you can simply create a new vNIC, from the already installed physical NIC, and have it dynamically discovered and initialised by the OS.  All done in software – no cracking the server open and no waiting for cables!  Talk about reducing the time taken to implement a change, not to mention reducing the risk (there is always added risk when opening up servers and messing around in the floor void….).

The CNA

In the above diagram we labelled the vNIC solution as Good.  If we swap out that IOV capable NIC and replace it with a CNA (Converged Network Adapter) that can act as both a NIC and a CNA, then we suddenly have the ability to carve vNICs and vHBAs from a single physical Adapter.  The diagram below has been expanded to now include a CNA based solution.  The CNA solution is labelled as “Better”.

NOTE: I should point out that in most IOV solutions most of the  legwork is done in hardware.  The vNIC and vHBA devices are created in hardware, as well as most modern CNAs providing protocol offloads…..

Single Root

The above approach – of creating multiple virtual adapters from a single physical adapters located within a single server – falls under the category of Single Root (SR).  Single Root being another way of saying single server (PCIe root complex).  Single Root approaches are limited to presenting their virtual adapters to a single PCIe root complex – operating systems executing within a single physical server.

While talking about Single Root technologies I need to mention SR-IOV.  SR-IOV is a semi-open PCI-SIG standard for SR style I/O Virtualisation.  As with all standards, it will take time to take-off and become widely deployed, and is open to implementation interpretation (some vendors may implement SR-IOV slightly differently to others).

True PCI-SIG SR-IOV requires the following components to be SR-IOV aware in order to support it -

• BIOS
• OS/Hypervisor
• Physical I/O Adapter
• Driver

Changes to the above components are required due to the fact that SR-IOV changes the architecture and model for PCIe adapters.  It introduces the concept of Virtual Functions (VF) which look and feel like a normal physical I/O adapter.  However, VFs are a lightweight version of a physical I/O adapter and inherit some configuration options from their parent physical I/O adapter.  As a result, vNICs and vHBAs are enumerated on a servers PCIe device tree as VFs, and the BIOS, OS and driver must understand this.

Citrix recently demoed XenServer working with SR-IOV NICs and Intel VT-d technology.

While SR-IOV is a great technology and is destined to play a role in driving IOV forward, it is very early days for the technology and the many of the currently shipping IOV technologies are using proprietary techniques and not PCI-SIG SR-IOV.  Some of these technologies include -

• HP Virtual Connect Flex-10
• IBM Virtual Fabric

NOTE: While the Virtual Fabric for IBM BladeCenter solution is not currently SR-IOV, the chip that powers the Emulex CNA that sits at the heart of the solution, is SR-IOV capable….. just waiting for the other components (BIOS, OS, Drivers…) to catch up.

Good, better, BEST!

So far we have talked about SR style solutions where the vNIC and vHBA devices are only available to Operating Systems executing on the same physical server that the adapter is installed in.  While these technologies are all good and a step in the right direction, there exists a superior solution – Multi Root (MR).

Taking IOV to the next step involves removing the physical I/O adapters from the physical server chassis and re-house them in an external device that I am generically referring to as the I/O Aggregator.

The diagram below has been expanded to include an example I/O Aggregator approach.

Such technologies can be referred to as Multi Root (MR).

There are already Multi Root I/O Aggregator style solutions shipping from the likes of NextIO, VirtenSys and Xsigo – all are delivering next generation IOV benefits today!

Of the currently available solutions, these MR technologies offer the greatest levels of virtualisation and flexibility and for me represent the future.  By removing the I/O adapter from within the physical confines of the server chassis, you enable any vNIC or vHBA to be assigned to any server.  Your physical server becomes entirely stateless from an I/O perspective!

I used to be excited about LOM style CNA implementations…. until I discovered I/O Aggregators.

NOTE: PCI-SIG also have a specification for MR-IOV.  However, I do not know of anybody deploying it at the moment

Moving Home

Opinion time here, but they way I see it, the I/O adapter is folding its underwear and packing its bags ready to ship out of the server chassis into a bigger, better and more comfortable new home – the I/O Aggregator.

PCIe adapters in servers…… don’t be so yesterday

The following is reposted from my new blog site http://blog.nigelpoulton.com  As a result, comments are disabled on this site, but feel free to visit my new site and leave a comment. Thanks to Snig for allowing me to post here for a short transition period…

Nigel

You can follow me on Twitter. I’m @nigelpoulton and I only talk about technology.

I am also available as a consultant on any of the topics I write on.

Ever heard of a Rack Area Network?

The term, as well as the concept, of Rack Area Networking is one I’m hearing more and more often.  As a result of this, as well as the fact that I’m convinced this is going to be one of the most interesting and important areas of Data Center computing over the next few years, I’ve decided to write a mini-series on the topic. 

This post is instalment number 1 and is intended to introduce the concept and get the ball rolling.  The whole thing is a bit of me thinking out-loud and attempting to generate some awareness and conversation around the topic, as .  So please pitch in!

Rack Area Network – the concept

For me, Rack Area Networking, or RAN for short, is an umbrella term for most of the clever networking and I/O virtualization stuff that goes on within a rack – a 42u rack. 

With it being “Rack Area”, it is a close proximity network and as a result operates over very high-speed low-latency interconnects.  

Physically, RAN technologies include a new generation of at least the following: I/O adapters, cabling, Top of Rack (ToR), and may be even End of Row (EoR), switches.  However, for reasons which will become clear, the emphasis is heavily on the clever – technologies that enable the flexible, the dynamic and the virtual aspects.

For example, the I/O Adapters driving the RAN evolution are not just faster than the legacy adapters they are replacing, they have built-in cleverness – hardware virtualization and huge flexibility! 

Some of the other technologies that define and operate within the RAN include – SR-IOV, MR-IOV, vNIC, vHBA, CNA, FCoE, Hairpin-turns, switching in the adapter, VNTag, VN-Link…. just to name a few.  In future posts we will discuss most of them.

As well as the above new hardware and technologies, the RAN also requires and includes a new generation of management software and functionality.  True value is often in the software – the glue that holds it all together and makes it all happen.

The best part being, there are early RAN technologies already out there in the wild.  And they are already delivering real-world tangible benefits.

Some technologies driving the evolution…

It’s really important to note that while technologies in the RAN are experiencing a period of accelerated evolution, it is most definitely an evolution.  The changes are happening fast, but they are not huge disruptive changes.  For the most part, they are improvements and enhancements, albeit major, on what we already know and are comfortable with.  E.g. take PCIe adapters and create multiple virtual adapters (vNIC and vHBA) in hardware….

Just a few of the currently shipping RAN technologies include -

• HP Virtual Connect Flex-10
• IBM Virtual Fabric Solution w/ Emulex UCNA
• Cisco UCS w/ Palo adapter
• Xsigo I/O Director
• Virtensys VIO switches

NOTE: Let me know if I’ve missed any major RAN technologies off the list

Some of the above technologies are very much generation 1 and only a small step towards the RAN, bringing only small benefits.  Whereas others are a major step with huge benefits.  All vendors are scrambling to take the lead in this evolving area.  In later posts we’ll dig deep into most of them.

Blurring the Lines and Causing Havoc

Naturally, many of these technologies are challenging and threatening the traditional server/network edge configurations we are used to. 

Hairpin turns, switching in the adapter and avoiding edge switches are just some of the paradigm shifts that RAN technologies might force us to consider.  Such topics are the subject of intense and engaging debate.  All very interesting and some of the concepts are very cool!

In upcoming posts we’ll talk about the likes of – SR-IOV MR-IOV Hairpin-turns VirtenSys Flex-10 VEB Xsigo VNTag NextIO VN-Link InfiniBand PCIe

Thanks for dropping by and feel free to throw in your penny’s worth.

Nigel

You can follow me on Twitter. I’m @nigelpoulton and I only talk about technology.

I am also available as a consultant on any of the topics I write on.

So what is going to happen to this site? Nothing. We will keep going with our initial mission, to let anyone that wants to blog about storage have an avenue to do that without restriction. Being one of the first five people (I’d say second to only Drunkendata.com) to begin blogging about storage back in May 2005 I started this site with the goal to let other people have an avenue to give their thoughts and feelings about the storage vendors in their shops and what they were doing right or wrong. The secondary goal was to get others talking about storage period! Too many people were hoarding information and unwilling to share it because they thought they would be giving away valuable IP and end users suffered greatly in those days. I’d say we were successful in those goals. You now have at least one hundred blogs dedicated to talking about storage. Most primarily marketing focused, some not.

As for myself, I plan on posting a few articles here and there. I’ve begun focusing more on the entire data center infrastructure over the past few years, but especially in the last 18 months. I don’t deal with just storage anymore and I really like it that way. Storage is a commodity now (to me) and it was time to move on to the next big thing. Having a ten and nine year old doesn’t give me much time outside of work to write many articles either.

So, here is the casting call. If you have a passion for storage and would like a place to write about it that already has a lot of dedicated readership then send me an email and we’ll get an author account setup for you. snig [at] rupturedmonkey.com.

Such a slur insult comment is almost guaranteed to make the blood of any true-blue HP employee boil.  Once their pulse has slowed back down and the veins in their necks have returned to normal, you will likely get the a response that includes the following three remarks –

1. HP has nothing whatsoever to do with HDS
2. Very specific, over-emphasised and repeated mention of Hitachi Ltd, Japan
3. Reference to the much vaunted and almost mythical “Engineering Agreement”

Points 1 and 2 are fairly basic. It is true that HP does not resell or OEM from HDS (a wholly owned subsidiary of Hitachi Ltd).  At the time of writing, Sun has an agreement with HDS.  HP, on the other hand, have an agreement directly with the source, the mother company – Hitachi Ltd.  In fact it is not uncommon of HP folks to speak very slowly, clearly and occasionally loudly when they stress the JAPAN part

However, it has always been with point 3, the almost myth like “Engineering Agreement” that I have had issue with. 

In my experience, the HP people spouting this line have never been able to provide any substance to the statement.  “So what does this engineering agreement entail” has always been met with either blank looks, or the  occasional stupid comment suggesting they cant tell me as is involves industry secrets.  Kind of like, I could tell you, but then I’d have to kill you.

Demystifying the Engineering Agreement

Well finally some light has been shed on the topic.  James R. Wilson, XP Disk Array Product Manager has replied to a couple of my Questions over on the HP Storage Blog.

In essence, James explains that while HP obviously doesn’t have full control over product development, they do have regular and active participation that enables them to be influential.  Specifically, James reveals the following –

“Our HP XP storage architects and lead engineers have regular detailed design planning, analysis and reviews with Hitachi engineers onsite in Japan. These meetings take approximately 3-5 days per meeting and are held on a regularly scheduled basis. We also have regularly scheduled management, engineering, and marketing planning and review meetings with respect to future technology with Hitachi. I just returned from one of these held in Japan just last week. These meetings alternate on a regular basis between Japan and our HP XP headquarters in Roseville, CA. These meeting typically take 3 days per event. 

Hitachi has 6-8 engineers on site in Calilfornia to work with HP’s engineering teams on a continual basis. HP engineers travel to Japan on a regular basis to work with counterparts there. HP and Hitachi also have weekly and daily conference calls as needed to manage the engineering work as required. HP and Hitachi also cooperate on operating system connectivity testing, sharing the work load between teams. Collectively, the engineering partnership between Hitachi Japan and HP serves very well to provide HP sufficient opportunities to impact the development of the XP family to meet the needs of our customers in a most satisfactory way”

While still not a great insight, interesting nonetheless, and thanks to James and Calvin Zito for answering my questions. 

In fact, ianhf has posted additional questions that might get answered now that the Thanksgiving break is over.  I also recommend Calvins blog in general, as it is slowly becoming a half-decent place of HP storage info and insights – although it is quite generic covering the many aspects of the HP StorageWorks portfolio.

Nigel

You can follow me on Twitter where I talk about storage and Data Center technologies (@nigelpoulton)

I am also available for hire as a free-lance consultant.

OK, if you don’t already know Xsigo Systems, and what they do, then you are seriously missing out!

The way that I see it, Xsigo (pronounced “see-go”) are of particular interest for two reasons –

Firstly, they are playing in the steaming hotbed that is the I/O consolidation and virtualisation space.  This area of Data Center computing is probably experiencing its biggest period of change and upheaval since the birth of Local Area Networking.  Also, the I/O subsystems of modern servers and blades are becoming increasingly important in modern data centers.

In a nutshell, just about everything is changing in the Data Center I/O space!

RupturedMonkey advice to vendors: Now is not a time to stand still or try and defend your traditional core competencies. Move with the times or risk falling behind!

RupturedMonkey advice to consultants and architects: Now is not a good time to take a professional snooze. If you do, you might find that you dont recognise the world you wake up to. Stay awake!

The second reason Xsigo Systems are of interest is because they have an absolutely kick-ass product – the Xsigo VP780 I/O Director.  So lets talk about it……

The Xsigo VP780 I/O Director

Before digging in to the specs and architecture, I should point out that the VP780 I/O Director is the only offering from Xsigo!  Also, as well as being the only product they currently offer, it is also pitched squarely at enterprise customers.  I suppose one of their responses to that would be that it allows them to be laser focussed, but for me, my initial reaction was that this makes them a bit of a one trick pony……  Compare their I/O consolidation portfolio to the likes of Brocade, and especially Cisco, and you will see what I mean.

During their presentation at the recent GestaltIT Tech Field Day they did say that they are working on similar but scaled down offerings for the SMB, but nothing to announce at the moment.

However, despite being the only noteworthy member of the Xsigo family, the VP780 is no wimp!  On the contrary, in a one-on-one it would probably fancy its chances against any of its competitors.  I certainly wouldn’t bet against it from a technology point of view! 

Specs and Techs

The VP780 is a 4RU high-speed low-latency 780Gbps I/O consolidation platform that was over two and a half years in the making.  It provides PXE boot and boot from SAN across 20Gbps connections to your servers and makes cable once and do the rest in software a reality!

Below is a picture of the front panel of the one on display at GestaltIT Tech Field Day –

From a high level architecture point of view the VP780 has –

• Server-side connectivity via 20Gbps Infiniband XFP ports
• Network-side connectivity via 15 hot-plug slots that can be loaded with 1Gbps Ethernet, 10Gbps Ethernet and 4Gbps FC.
• Passive midplane
• High-speed low-latency internal switching fabric

Hopefully the scribble below will be helpful as I attempt to dig deeper and explain some of the main components. Later in the week I will record a whiteboard session and upload as a complimentary post.

What does it do – in a nutshell

In a nutshell, the Xsigo I/O Director does for I/O what VMware does for CPUs.  Only it has an added benefit that it removes the physical limits of the server chassis –> instead of installing your NICs and HBAs in your servers and then carving them into virtual adapters that can only be used by that server, you install your NICs and HBAs in the Xsigo VP780 chassis so that they can be carved up and dynamically allocated to any connected server.  In doing this, you are effectively moving the edge of the network out of the server, enabling servers to be entirely stateless from an I/O perspective. Cool.

Connections to your servers… the hardware stuff

The VP780 has 24 x 20Gbps Infiniband ports for connections to your servers (server-side in the above diagram). They utilise copper or optical CX4 cables and XFP interfaces and are terminated at the server side on Host Channel Adapters (HCA in Infiniband parlance and yes each connected server needs an HCA). These HCAs are not used directly by the OS, instead, host-side drivers work together with the Xsigo I/O Director to ensure that the appropriate vNIC and vHBA devices are available to the OS.  Of critical importance is that thes vNIC and vHBA devices work exactly as physical NICs and HBAs and the OS is none the wiser.

Some quick comments on these physical aspects –

1. The 20Gbps XFP interfaces are not hot swappable and not upgradeable to 40Gbps QDR Infiniband.  These Infiniband HCA and switch ports are more energy efficient, lower-latency and higher-throughput than their 10GigE counterpoarts.  They also support longer distances over copper meaning that copper is an option more than it is for 10GigE which currently has practical limits between 5-10 meters.

2. As these are CX4 Infiniband connections, you will need Infiniband Host Channel Adapters (HCA) in your servers.

3. XFP and copper CX4 is more power hungry than SFP+ and copper commonly used with 10Gbps CEE. However, XFP optical is not more power hungry than SFP+ optical.

On point 1 from above – this should not be seen as a major issue. 20Gbps to your servers is lightning fast by todays server standards, and with the current wave of PCIe 2.0, you’re unlikely to be pushing beyond 20bps anyway. 40Gbps models are planned, as well as faster HCA cards, although this will be down to Mellanox as the silicon is 3^rd partied form Mellanox.

Also, in reality,  how many people are racing to crack open servers and blades to upgrade I/O cards? Most people seem to be opting to buy newer servers and blades when higher bandwidth I/O is required – may be Nehalem-EX…  20Gbps is more than fast enough for the vast majority of todays applications and servers.

On point 2. Don’t be put off by the word Infiniband! 

FUD watch: Infiniband is not a disease, nor is it dead!  It is actually a rock solid ultra high performance low-latency channel interconnect designed for data center use and high performance computing. In fact many of the worlds fastest supercomputers use and are built around Infiniband.  So its definitely alive and well.

You don’t have to learn a boat-load of new Infiniband skills. You will run a copper CX4 cable from the HCAs in your servers to the Xsigo director and that is about as much Infiniband as you will ever see or need to configure in the solution. The rest is normal Ethernet and FC.

Connections to your servers… the clever stuff

So, if all of this talk about Infiniband hasn’t scared you off, well done.

The Xsigo VP780 I/O Director allows you to carve its NIC and HBA resources in to virtual NICs (vNIC) and virtual HBAs (vHBA). Each of these virtual vNICs and vHBAs acts exactly like a normal physical NIC or HBA and thanks to some clever work in the server side drivers, Operating Systems (ESX, Windows, Linux etc) see just like they would physical NICs and HBAs.

Another thing not to be underestimated is that fact that the physical NIC and HBA hardware sits outside of the physical server or blade chassis.  This enables the physical servers to be stateless from and I/O point of view, and for virtual resources to be moved around from physical server to physical server with great ease.  The VP780 owns the server profiles which include MAC addresses and WWPNs etc and allows up to 16 vHBAs and 32 vNICs to be assigned to a single physical server, all of which can be created and deployed in literally seconds with no reboots.   Ideal for VMware and the c c cl cll cllllll clllllll cloud!?  Think thats the first time Ive said the “c” word in a blog.

In my opinion, the previous paragraph is of huge importance.  If you don’t think this is huge, then I suggest that you re-read it and take a minute or two to think about it.  This is flexibility like no other solution I know of.

Is it just me, or does this look and feel very much like MR-IOV (PCI-SIG Multi-Root I/O Virtualisation)? 

Does anybody else do anything like this?

NOTE: I’ll post on this in the near future, but I personally think that MR-IOV has huge potential to rock the I/O consolidation world, and I’m not alone in thinking that! However, there is a case for Infiniband being a better Rack Area Networking (RAN?) interconnect than PCIe. One for a future post if people are interested.

Connecting to existing backbones

On the network side of the VP780, there are 15 slots that can be populated with various modules.  Currently 3 module types are available –

1. 1 x 10Gbps Ethernet module
2. 10 x 1Gbps Ethernet module
3. 2 x 4Gbps HBA module

This gives you traditional LAN And SAN Connectivity, with FCoE and iSCSI offload being on the map.  So ,connecting to your existing LAN and SAN is “as easy as organising a tweet-up at TechFieldDay”   Your up-stream LAN and SAN is oblivious to the fact that the I/O is not initiated at the server chassis and just hums away as normal (NPIV is implemented on the SAN side of the HBAs).  The diagram below shows native FC connections coming out of the network side of the VP780 at the demo lab at VMware. 

At the moment, the VP780 has no support for FCoE.  Not a huge drawback as the standard and shipping products are still young, however, if they drag their heels over this they will fall behind in an important new and emerging market.  Something like the Emulex UCNA with its 10Gbps Ethernet, FCoE and iSCSI offload all on a single module would be like the cherry on the cake for this.

FUD Watch:  Be careful to note that the VP780 is not a switch.  True, it can switch frames between servers without passing traffic to the upstream network switch, but it does not HAVE to. It can forward the frames to the upstream switch if the switch supports hairpin switching.  So it does not have to alter existing network management models. However, there are several scenarios such as HPC or RDMA where performing hairpin switching over the internal IB fabric is beneficial for performance reasons. Choice is a good thing!

Nice Management GUI

While visiting with Xsigo at GestaltIT Tech Field Day I got my hands on some Xsigo kit including the management interface.  I was able to present vNIC’s and vHBA’s to ESX servers and have them picked and recognised on the fly by virtual machines. I was also able to play a little with some simple QOS features – increasing and decreasing bandwidth is very simple and also dynamic.  All simple stuff and worked a treat.  Oh and it has a CLI.

There is a ton more I could say, but this is already pretty long so I’ll wrap up with some final thoughts…..

Conclusion

The VP780 does some interesting stuff and in some respects is ahead of the curve. For instance –

• Performance to the server is 20Gbps over Infiniband
• Flexibility. Removing the physical NICs and HBAs from the server or blade chassis makes this a hugely flexible solution. 
• Number of vNIC and vHBA devices that can be carved per physical card and presented to each host is more than most of the competition. E.g. HP VC Flex-10 and IBM Virtual Fabric can only create 4 virtual functions per port. This offer superior utilisation as a result.

As always though there is no perfect solution.  There is currently no FCoE, and both the Infiniband and the 10Gbps Ethernet options use XFP transceivers which are not as good as SFP+ when it comes to the likes of size, cost and power consumption.

There is also the fact that Xsigo are a relatively small and new company pitching to the enterprise.  From a technology point of view they are brilliant, but one has to wonder if they will still be around in 10 years time supporting and developing their products?

However, when all is said and done, I really like what they are offering. My final question for Camden Ford after his presentation was “can I have one for my garage”.  Says it all really.

If you are looking in to I/O consolidation and virtualization then you should definitely at least check out Xsigo… unless you’re too chicken! 

Thoughts and comments welcome.

Nigel

You can follow me on Twitter where I talk about storage technologies (@nigelpoulton)

I am also available for hire as a free-lance consultant.

What the H*!! is this GestaltIT Tech Field Day thing?

The crux of the event is this – taking a handful of bloggers and a handful of smaller vendors, locking them in the same room for two days with only blogging and tweeting as their connection to the outside world………. And seeing who comes out alive!

Presentations will be given, demos will be done and most importantly, no holds barred hard questions will be asked, and asked, and asked….. 

In fact, Stephen Foskett the event organiser, and industry legend, told me the other week that he had warned the vendors that the only way he could guarantee that we won’t say they suck, is if they don’t suck!

Why should you care?

Answer: Same reason I care…… It will be one hell of a learning opportunity.

Granted its an invite only event, so we cant all go, but I will be there and will be blogging and tweeting about everything I learn.  So if you want to know more about the technologies and strategies of the companies who will be presenting, then stay tuned this blog and tune in to my Twitter account (@nigelpoulton).

I promise to blog and tweet about everything I learn.  The good, the bad and the ugly.

Proxy

As everybody cant be there in person, feel free to drop me a line via any of the many means by which Im contactable, with any questions you want me to ask the vendors.  If they are good questions then I promise to ask them.

Who will be there – Vendors

Who will be there – Bloggers

Finally

I am genuinely excited about this event. Same kind of excitement I would have if my football team ever made the FA Cup final or England made the World Cup final (well….. may be not quite that excited).  But seriously, this is a landmark event and a great opportunity to learn about some of the less well known vendors and technologies out there.

Feel free to contact me with any questions you would like me to ask…….

Nigel

PS.  Im really looking forward to meeting all in attendance but in particular Xsigo, MDS and 3PAR.