I see HP are at it again, this time blowing up a data centre <link here>.
Next time Im haggling a price with them I will be pointing out that they obviously have enough spare cash floating around for some of their guys to play around seeing how much fun they can have trashing the type of kit Im paying to buy!
Looked like fun though.
Don't miss the, not so subtle, point of the Windows server failing over quicker than the Linux. Hmmmmmmm
I run Hitachi Tuning Manager (HTnM) and have been battling with what to do with it for some time. The information it provides is good (if not a little too good) but I would like to know what to do with the performance data I see.
Today, I was doing real time monitoring of the top 10 array groups (AG) in one of our USP's. One AG sat on about 45 % busy for the hour or so I watched it. It has a couple of Exchange server's LUN's and they have been working hard for the last day or so. Yesterday, one 25 GB LUN was chewing up 600 IOPS of one AG. I figure that a 7D+1P AG can do about 1000 IOPS at best so what about the other LUNs sitting on that AG. Was I just lucky that no one was watching as I bet some other servers would have been paying the price for that Exchange LUN because I have over 900 GB of disk that has to be used on the AG.
Time and time again, I have asked HDS for some guidance on what to do with certain stats. I basically get a run around as they are not willing to give guidance on their hardware. HTnM has the capacity to create alerts/events. I tried it but got flooded with alerts. The thresholds were high but it seems that the USP can run hot especially during backup windows.
I think that the best bet is to wait until the customer/user complains and then look at the problem. I do try to be proactive in making sure the storage is behaving but I would dearly love to figure out a regime of when to do something.
Ignorance is bliss as before I got HTnM, I never had enough information to concern me. Now I want to do something but when it the right time to do it?
The most surprising thing about HTnM is that is uses Command Devices to query the USP. They can really skew metrics especially for logical devices. When I run a couple of real time reports, the Command Device often has 500 to 800 millisecond response time. That means the graph is useless as I am looking for 15 or above especially in our HSC clusters which use Sync True Copy.
Perhaps EMC are better at offering advice on what to do with their storage?
Comments on this subject would be more than welcome.
Stephen
Chris Evans (http://storagearchitect.blogspot.com) has raised the VTL question and Copan's were mentioned a couple of times. I find his blog to be very informative and it looks like he has the same issues as I do. I don't want to steal his thunder on this but I would like to open it up a bit regarding what the future of VTL is.
My management seems to hate Tapes. I recently moved all our backup hardware into our Director based SAN from standalone switches. I now have about 24 tape drives and a couple of small VTL's spread across a couple of data centres. The Brocade switches and HTnM now complain bitterly about a buffer credit zero metric. I used to have a couple of them across our SAN extension but now I get them though the fabrics including the USP ports. The tape drives are doing something to cause this issue but I can't figure it out.
Each tape drive can deliver about 30 MB's max throughput but you rarely see them all working. Our DR solution regarding tape backups is complex and needs some tweaking as I imagine all companies do.
So, now we are implementing Copan VTL's and archiving solutions (and perhaps tiered storage in some form or another) in a smallish stand alone environment just to see how well it goes.
The Copan VTL's will automatically replicate backups to our DR site in native fibre. So there is a solution to off site storage of data that is done in almost real time. The disks are cheapish and the VTL back's up at normal fibre speed of up to 200 MBs (umm.. lets say up to 150 but one VTL equals 5 tape drives in performance). I also don't have issues with tapes being tapes and going off line.
Tapes cost a fair amount of money and we use lots and lots of them. We also have people whose job is to look after them and that must be a mind numbing experience.
I can fit 448 TB in a single cabinet but as it is so heavy, it is going to fall through the floor one day. Its load rating is considerable even compared to a fully maxed out USP. I doubt most data centres can actually cope with it. But, 448 TB in one cabinet is impressive. My USP's have only 146 TB in five cabinets.
Along with archiving, we might be able to remove our tape libraries which are about the size of a small truck and replace them with a couple of Copans in each data centre. Not to mention, the Copan's are green.
I would like to see even more discussion on this especially from anyone doing what I am doing and even more. Will the humble Tape drive still be in use by 2010 other than to read old tapes?
Chris Evans Blog has a poll that says a good percentage use VTL's now. I would like to see how many aim to get rid of tape drives as the main backup strategy. In other words, no more disk to disk to tape except for special regulatory cases if that still occurs now.
Stephen
I looked at a link on the RM site about United Air Lines and it got me wondering about how much effort it takes to manage Storage.
<quote>
June 22, 2007 (Network World) — When you tell him that on average, storage admins manage between 30TB and 60TB each, Samuel Turner smiles. As United Air Lines Inc.'s manager of storage utility services, he has good reason: His staffers each manage triple that.
"They manage nearly 200TB per person," Turner says, noting that although his staff is pushed, such amounts are doable, at least for the time being.
</quote>
Over the last few years, I have looked after well over the 200 TB level and I can say I am not over worked in the least. Occasionally I can spend a few months planning and implementing something large but it was normally not just the storage. It is normally a mixture of designing new systems that just happen to use switches and storage.
So what do I do? Be mean. I am probably the meanest person where I work. Someone wants another TB for this. Show me the money. But.. no buts, just show me the money. Can I borrow a TB and give it back later. No. Give me the money upfront and you have a deal.
Other then being mean, I watch what the systems and the storage do. I probably spend about half an hour looking at Tuning Manager while having a coffee. If I see something bad, I think about fixing it. I like to see an even system so I may move data around depending on what happens.
I have some work coming up this year when I get rid of our 9980V's and 9960's as they are old. I expect to get a couple of USPV's to move the data onto. Moving mainframe storage is easy. Moving Solaris storage is easy. Moving Windows… not so easy especially as we don't use Veritas yet.
So, how many storage admins do it full time if they only manage 30 or 60 TB. I wonder if they are using lots of smaller arrays and perhaps SAN islands, you know that whole project thing where I pay for it so no one else can use it.
Prior to SAN switches, double disk failures in smaller storage arrays (such as Sun T3's) were the causes of nightmares for me. I could spend days mucking about trying to get fibre connections to hosts and move the data before we could change the failed disks. Since I got large systems, I don't have to worry any more. If my USP breaks, there is nothing I can do about it.
If I have say approximately 400 TB and I look after it myself with plenty of time to think of other things (don't tell my boss), am I doing a proper job of SAN management?
I should say I don't touch backup at all. That's a mugs game and someone else can do it. I provide features like True Copy, Shadow Image and Copy on Write. That's good enough for me.
Also, change management is some sort of help to me. Nothing is ever any emergency any more and I wont do anything without a weeks notice. It takes sooooooooooo darn long to create storage on the USP it is not funny. 10 x 100 GB LDEV's can easily take two days to create. If I don't have any free LDEV's, I can't give them away. If I have reclaimed LDEV's sitting around and they are willing to use LUSE LUN's, then they can have it immediately.
I don't work hard, I just like to plan….. oh and be mean.
Stephen
Very, very much off topic……
Just from speaking with various storage bloggers and reading around other storage related blogs its starting to amaze me how many people in this industry seem watch Heroes.
So Im wondering who else out there watches it, and if its some sort of unwritten pre-req for working in storage, that none of us are aware of (until may be now) ???
Im even starting to wonder if both HDS and EMC people watch it and agree that its a great TV programme – surely not 
I watch it, although I'm a few episodes behind (#18) so no spoilers please!
Nigel
He’s said it again! A while ago Mark over at Storagezilla said that he thought it might be a great idea if the storage industry rearranged its calendar making EMC World the industry event of the year . Well he’s said it again, this time in response to Snigs post on Vendor Marketing .
I’m sure he says it somewhat tongue in cheek, hoping people will bite and keep talking about EMC, a bit like what I’m doing now However, I couldn’t let it go unchecked again. Hey, may be he’s 100% sincere, if somewhat naïve? Who knows?
Basically he is suggesting the following –
Although I can see where HAHAHAHAHAHA! Sorry I couldn’t help laughing again. Anyway, where was I? Oh yes……. Although I can see where he might be coming from, and I do think there is some potential in it, I think there is also the distinct possibility that the event could become a farce. Running the risk of turning into something resembling the House of Commons during Prime Ministers Question time (for those of you who have not had the pleasure of witnessing this – basically you have at least two sets of people who disagree on absolutely everything, even the things that they really agree on. They shout, heckle, fall asleep and occasionally throw paper across the room at each other while people are speaking).
But the presumption that such an event could be called EMC World leaves me speechless! Why would everybody unite under such a banner? He of all people should know something about the difficulties of trying to unite different people under one country, or company, flag.
I see the idea as being similar to expecting every religion in the world to alter its calendar to celebrate their major festival at the same time the Christian world celebrates Christmas, oh and by the way, they also have to rename their festivals to “Christmas” as well. Duh!
Christmas is great and EMC World is great, but I just don’t see everybody else joining in the fun.
Nigel
Anyway Ive been out of action recently with a nice kidney infection so have missed out on some of the news and stuff going on in the storage world and have been playing catch-up the last few days.
One interesting thing that I noticed was that Dave Roberson has swapped HDS for HP. Obviously the he felt the grass was greener on the other side………. actually, is HP even classed as “the other side”? Anyway one obvious plus for Dave is the immediate increase in productivity he will realise by not wasting as much time spelling his email address to colleagues – @hp.com is full letter shorter than @hds.com!
The interesting thing about this move though, and Ive not seen anything written about it, is what impact this might have on the EVA. We already know that HP resells Hitachi’s high end storage as the XP. But as of this moment in time HP have their own midrange offering in the EVA.
There are obviously several directions this could go in, including, but not limited to, those listed below –
Ive tried to list them above in order of likelihood (the top one being the most likely) – in my humble opinion of course.
I personally would be very disappointed if the EVA was end-of-lifed. Ive said it before and I’ll say it again, I think when it was first released it was ahead of its time and still has some really cool features that we don’t see enough of elsewhere. One of my favourites is still the wide back end striping and the way it will re-distribute the load (re-level) if you add more disks on the back end. Of course I know its had its problems in the past and in many ways I think they were a reflection of the problems that the HP StorageWorks division was having in general. Problems at the top and in the background inevitably manifest themselves on the playing field.
Anyway, HP StorageWorks seems to be turning a corner these days and this could be a good thing for the EVA. However, sadly we live in a ruthless world where decisions are ultimately made by people who don’t understand cool stuff, like wide back end striping and dynamic re-levelling.
Number 3 in the list above is the most exciting, albeit unlikely, prospect in my opinion. A hybrid of the EVA and the AMS, call it the AVA (Adaptable Virtual Array). Stranger things have happened….. at least I think they have
Anyway, long live the EVA. If it does die, I might have to pick me one up from E-bay for posterities sake.
Thoughts welcome as always.
Nigel
Something that akro, an RM reader, said in response to one of my recent posts got me thinking. We were talking about wide striping and some other cool stuff that the EVA has been doing for a while now. Specifically akro mentioned the idea of sticking some EVA controllers in front of a high end array and seeing what magic could be done. Made me think a little.
First of all it made me think of 3PAR and their so called enterprise offering, the Inserv Storage Server – can you stick a bunch of modular arrays together with a fancy interconnect and call it enterprise? Anyway, a little more on that later. But eventually it got me thinking what actually qualifies an array as “enterprise”?
I thought of a few things that I had normally associated with enterprise storage arrays -
First up, reliability. While I was growing up in IT, “five nines” was always synonymous with expensive enterprise gear. Basically that sort of kit was built not to go down. If you were to ask the guy in charge of it how long it took to boot, you might well be met with a response along the lines of “why would I know that, we haven’t booted this thing in 10 years”. Whereas your typical modular kit would be up and down like a yo-yo in comparison. So, in my neck of the woods “five nines” was always synonymous with enterprise kit. Then along comes Chuck and his ramshackle mob, recently announcing “five nines” availability for the CLARiiON CX3. Of course Im also aware that Hitachi guarantee 100% data availability on their current crop of high end arrays. So it seems five nines no longer gets you automatic access to the “enterprise club”.
Next up – performance. At least three years ago now I knew of an EVA 5000 that cleaned up in a foot race against a vastly more expensive “enterprise array”. So even 3 years ago it blatantly wasn’t sheer speeds alone that qualified you. If only the enterprise kit on sale that day was as fast as its salesmen, those guys were like greased lightning to point out that their box scaled better than the EVA We’ll get to scalability in a minute……..
What about size then? Im going to resist all of the easy puns, but suffice it to say that the only people concerned with size are well……. Anyway, these days size (Im talking about storage capacity really) is in the eye of the beholder. For example, on initial inspection the DMX-3 appears to blow the competition away in this category. If there was one storage array out there that you wouldn’t want to bump into in a dark allay, it would be the DMX-3. But then all the folks out there externalising stuff off the back of a Tagmastore might offer a different view of errr …. size. Whereas the DMX-3 might have a set of shoulders like Arny, the Tag on the other hand may seem a little puny in comparison, only its not as simple as that. The Tag has “connections”, if we keep with the analogy of not wanting to cross one in a dark alley, we might say that the Tag “knows people” and put them together and you wouldn’t want to mess with it either.
And to be honest, Im seeing a lot of small USP’s going into companies these days, one or two frames to start with. Not very big but still classed as enterprise storage. In fact, I wonder how many fully loaded DMX-3’s have been shipped and how many USP’s have those infamous gazillions of Petabytes hung off the back?
Cost? Sure the official price tag will still make most companies step back in horror. But who pays list price these days? I hear a lot about deals being done extremely cheaply these days. It’s the in thing to play your current vendor off against the rest and you might be surprised at how cheap you can get your storage. Of course that’s the initial purchase, and as sure as eggs are eggs it will get a lot more expensive once the kit is on the floor In fact Ive heard “rumours” of one vendor giving some of their enterprise kit away just to get or keep their hairy sweaty foot in the door.
Lack of cool features. I used to find that the enterprise boxes were a big, cumbersome and a bit boring. Too big and bulky and based on ancient code, written and understood only by retired or dead people, that nobody dared to tinker with and add functionality to. Well that’s changing these days with Hitachi redefining their enterprise offering by putting loads of new functionality into the controller. Love or loath the Tag, it does quite a bit of cool stuff these days.
Now to scalability – Akro’s comment made me think a lot about 3PAR and what they are doing with their Inserv Storage Server, allowing you connect up to 8 modular arrays over a proprietary switching interconnect and letting you manage them as a single entity. Sounds good. Managing 8 modular arrays as one allows it to scale similar to an enterprise box both in capacity as well as processing power. May be this a modular offering with enterprise scalability?
But does it bring the flexibility and ease of use and maintenance that you (I) expect from modular? Apparently not. Im reliably informed that configuration is mainly manual, quite labour intensive, requires sound understanding of the underlying architecture and is prone to complex configurations and errors. A lot different from the bulk of my modular experience, the EVA, which was so simple I had to actively stop the Exchange admins from getting their hands on it in case they realised just how easy my job was.
Anyway, the single biggest thing that I can’t get passed when looking at the 3PAR kit is the use of commodity parts. And when I say commodity parts Im talking about bog standard Intel chips on a PCI-X architecture. To me it just looks too much like a PC to be enterprise. I wonder if I can partition one of the internal drives and boot Windows on it during free processor cycles (I hear it actually runs a version of BSD under the hood)? And with that commodity approach comes a lack of manly grizzly enterprise
stuff like cross-bar switches, dedicated parity generating ASICs and the likes.
But the 3PAR does sport one truly enterprise feature – you cant just send an operator into the machine room to replace a failed disk. Like all good enterprise storage arrays you’re supposed to be a trained engineer to be able to do that.
So for me a couple of things remain as hallmarks of enterprise storage –
Nigel
I was having a discussion with a customer over lunch yesterday about the nuances of the storage industry and we got on the subject of marketing. We began discussing some of the differences between EMC and HDS specifically and the perceived lack of marketing on HDS's part when compared to EMC.
We all know that EMC is an American company and that HDS has it's roots in Japan even though they are headquartered here in the U.S. It seems to me that what we see out of both of these companies goes to the root of each companies cultural diversity. EMC being the brash and bold Americans and HDS taking their marketing cues from the more humble Japanese culture.
I think we've all concluded that when you compare subsystem vs. subsystem there are few differences when it comes to raw workload in that they both can get the job done. If you want to go super huge in a single frame, go EMC. If you want to virtualize other storage behind your array and run leaner on capacity and meaner on performance, go HDS.
So why doesn't HDS break out of their mold a little and go head to head with EMC's marketing department? HDS has picked it up a little with a lot of the new releases they have coming out this year, but they are still no EMC when it comes to marketing. Why isn't there an HDS World? EMC has had that going for quite a few years and even before they went on their buying binge of the last few years. What better place to do a ton of marketing than at one event that talks about all your products with techies and the people with the purse strings? There are a ton of things that HDS could do, so why aren't they doing it?
I just wanted to throw these thoughts out there and would love to hear some comments from the rest of you. Maybe I have it wrong, but from what I've seen I think I'm pretty on target.
Here's a thought, maybe HDS should sponsor a car in Nascar. That would get a lot of heads turning wouldn't it? 
Its 10 years to the month since I put in my first Fibre Channel storage solution. I was working for the biggest stock broking firm (by sales) in Australia and we needed some new and large systems for a global trading initiative. I was running Sun E4000 servers and Sun RSM 2000 storage arrays (using 9 GB disks) with a Sun rebranded Veritas 2.3 managing the storage. I was impressed with the performance and so were the end users. I was in seventh heaven. All the money I wanted both in hardware and salary. The bonuses were big and so was the cost of hardware. As I was also a die hard DBA (Sybase), I was actually horrified at the 9 GB drives. They were too big and I actively only used the outside area of the disks so I could get my required speed and spindles.
Here I am 10 years older and still doing much the same thing with a few years break in the middle doing Enterprise Systems Management. Only real difference to 10 years ago is that I have lots more to look after. I am definitely not a specialist when it comes to forecasting the future of what? Some people out there want SAN to mean other things instead of purely fibre channel. When I have a free moment I look at all those online storage sites raving on about the future of Storage. They all have experts who say that fibre channel is dying quickly and we will all be using iSCSI or FCoE tomorrow if not next week.
So the next time I read the experts view on something, perhaps they can tell me why they think it is the next big thing. I think it has to be driven by costs. My SAN costs lots and probably equals the GDP of some African nations. My data is extremely important and that's why we have the current infrastructure. All the discussions about the future of IP or Ethernet based SAN's never really talks about the security of the data.
I love reading headline stories about John Doe who is the CIO of some company where he uses iSCSI for a hundred users of Exchange. We class Exchange as a Tier one solution and have tens of thousands of people hanging off it. We have invested lots of money in doing stretched clustering of Exchange and all the servers are part of a five node cluster using True Copy. The same goes for our database servers.
I am waiting for iSCSI or FCoE to be introduced to our Mainframe environment.
The problem with FCoE is that it is going to take a long time before it may get introduced into large business for important work. I am looking at replacing my Brocade Directors in two years as they will have reached the end of their costed life in our organisation which is 4 years. So, if FCoE is not up and running by then, it is likely that we wont look at it for 6 or 7 years into the future. That's not overnight or next week.
The most interesting thing about iSCSI or FCoE is that you still need hefty SAN switches to present it. I don't see any $5k switches from Brocade or Cisco that do it. The cheapest Cisco switch is the 9216i and that is not cheap if you are looking at saving money by using non FC devices. I have not done a lot of research into that so there could be much cheaper brands that can do iSCSI or FCoE.
There is that old saying that ex is a has been and a spurt is a drip under pressure. So is that the definition of an expert writing for some wanna be tabloid style web sites.
If the experts are going to make predictions, then at least quantify them with decent reasons and expectations.
Can I become an expert and say that we will be using 100 Gbps wireless networking for storage area connections by 2010?
Stephen
Well today is June 1st and there are clouds on the horizon here in FL. 80% chance for rain tonight (thank God) and I think we'll have at least two hurricanes hit us this year. The numbers of named storms are pretty big this year.
I wonder how many companies here in FL actually did something after the 2004 hurricane season? We had four hurricanes hit Polk county that year and I was without power for a total of 21 days. Did that spark companies to at least do some sort of business impact analysis? Do most companies know what their data and information is worth? I would venture to say no. With tight budgets and even tighter work pools, they just didn't have time or see the value in doing the exercise.
Well I hope everyone fairs okay this season and no storms hit the mainland U.S. Take care and as the annoying weather people say here in FL, "hunker down."
get your job site
at simplyhired.com