Jesse was mildly annoyed about being stuck on “low-end” CLARiiON projects recently. Well, I’m lucky enough to have been scraping a little lower down the barrel and have now deployed my first AX4-5. It’s branded as a CLARiiON, but just like its predecessors (the AX100 and AX150), it’s a little, er, different to your average CX or CX3. For a good overview of the tech specs and howtos, etc, have a look at the support site and click on “Learn”. I have only had the opportunity to initialise the array and haven’t loaded it up yet. The Navisphere Express interface is different to full Navisphere, but you can upgrade that if need be.

There are a few things to look out for when you configure one of these arrays. Firstly, it runs a variation of FLARE (Release 23 for those keeping score at home), and the first 4 disks (as opposed to the first 5 in the CX and CX3) are used for this code and other array features. This means that, when you configure a RAID Group (or Disk Pool), using these disks, you’ll lose approximately 17GB per spindle. It’s not as bad as the 33GB on the CX3, but it can still pooch your storage calculations. Especially when the type of customer deploying this array has little exposure to storage anyway, and might think that 1 TB equals 1024GB. Secondly, the hot spare needs to be on a disk other than a FLARE disk. This is nothing new, but again, these kind of deployments aren’t always done with these kind of constraints in mind. I have a bad feeling I’ll be doing a deployment shortly on a 4-disk SATA shelf that was sold without any hot spare at all. Thirdly, the only way you can assign LUNs (Virtual Disks) to different SPs is via Disk Pool membership. The manual suggests that you make at least 2 disk pools to balance the load across the SPs. Sounds good, except when you’re working on a 6-disk (total) deployment, one of which is already a hot-spare. Unless I did something nasty with 2 RAID 1/0 pools (I can’t afford the capacity penalty), I couldn’t do this. Watch out for storage designs that only look at capacity. Finally, if you’re getting it with SAS disks, make sure the “AX4-5 Expansion Pack” is ordered, as this allows you to actually use the SAS disks, rather than just look at them. A colleague of mine had to wait while this was shipped before he could finish a deployment. The job I did recently had the enabler pre-installed (I guess they rectified that issue). By the way, it looks like it’s manufactured by Foxconn (judging by the big foxconn label on the box and this news item). If nothing else, Foxconn’s press releases are hilarious.

Oh yeah, and we’re throwing a 4-node vi3 cluster and 18 guests at this little puppy. I can’t wait to watch it melt.

Has anyone out there used the LaCie Ethernet Disk? I am looking for some bulk storage for media and wanted something I can rackmount. If anyone has any real experience I’d like to hear about it. The local pricing seems reasonable given what I’d spend on a Maxtor / Seagate without the RM option. RAID be damned.

I really enjoy hearing about people using technology for the betterment of humankind. That’s why I’m so happy to share this link to an article on retrothing. Be sure to check the youtube videos as well. Enjoy!

I just watched Game 6 of the 1980 NBA Finals. It was a great game. Long socks, short shorts, afros and gold chains galore. Julius Erving and Mo Cheeks playing for the Sixers. Michael Cooper and Jamaal Wilkes in full flight for the Lakers. And Magic Johnson’s 42 points, 15 boards and 7 assists. A player in the Sixers team whom I’d forgotten about since I was a kid was Darryl Dawkins. Let’s not forget him again. Let’s never forget Darryl :)

This is very old news, but one of the neat things about ESX is that you can build clusters for testing and don’t have to shell out for a split-bus DAS or SAN space. My colleague, who’s been using ESX for good and evil (testing Veritas Cluster Server) needed to build one in our dev environment the other day. Last time I did this was to cluster VirtualCenter 2.0.2 and I was a bit rusty on the process.

So, for my reference, do this:

Create a shared vmdk to act as the quorum disk using vmkfstools (it needs to be in thick format)

vmkfstools -c 512m -a lsilogic -d thick /vmfs/volumes/datastore/quorum.vmdk

-c to create a vmdk and what size it should be
-a to specify the adapter type (lsilogic or buslogic)
-d can specify some neat things, like RDM settings, etc

Make another vmdk if you’d like to, well, share some data between the nodes.

Add the disk to the guest as an existing device and change the SCSI ID of the card to something like SCSI (1:0). This adds another SCSI adapter to the guest. Set the sharing mode on the adapter virtual or physical, depending on whether you want the cluster in the box or sharing with another physical / virtual host outside of the ESX host. If you don’t specify the -a option when you create the vmdk, it defaults to buslogic. Obviously if you want to share the disk with another physical host you can’t use a vmdk.

Do the same on the other guest / physical host / etc. Install MSCS or VCS or whatever passes as a clustering solution in your life. Enjoy. I also recommend the man page for vmkfstools - it’s an invaluable reference.

Short takes seem popular right now, so I’m thinking of wading in with a few notes that I’ve been meaning to blog about but haven’t had the time.

• I had to rebuild my laptop last week, and re-installed Sun Common Array Manager (CAM). I’ve bitched and moaned about this software in the past, but the current version – 6.0.1.11 – at least on my laptop and our management server, installed / upgraded without the usual slew of dramas.
• I’ve been doing a lot of Backup Exec lately. Which is funny if you know that I haven’t used it since version 8 was GA and I still haven’t worked out all of my NetWorker biases. But, if you keep it simple, it just works. Although of the two implementations I did recently, the first used NDMP to send files to a tape library backing on to a Celerra, and the other used a combination of SAS, iSCSI and VMware Consolidated Backup (VCB). FFS (!), is anyone just buying FC libraries and SAN-attaching them? Incidentally, the Backup Exec Integration Module for VCB and I didn’t get on very well. So I wrote some scripts of my own which do the same job, but without the flexibility and useability that some may crave.
• A customer’s P2V went wrong. Twice. And he couldn’t delete the associated files. esxtop showed that the VM was still running. There are, apparently, two ways to deal with this. Vincent at virtrix likes to use the old-school ps method. And Daniel made the suggestion to use the vm-support command. I favour the ps and kill -9 combo, but that’s because I’m a bit rough and ready.
• The exchange guy has learnt how to spell IOPS, which reminded me to point people with access to EMC Powerlink to a useful BPG from EMC regarding Exchange 2007 implementations. Log in to Powerlink and look for “EMC CLARiiON Storage Solutions: Microsoft Exchange 2007”. It’s a worthwhile read, particularly if you’re sitting in front of a customer who’s a little antsy about the whole Exchange 2007 thing.
• I’ve been playing around with VMware Update Manager, but haven’t gotten as deep as I’d like yet.
• I have Playstation 3 now.

I’m sure there’re a few other things I’ve been meaning to jot down but that’s all I have for now. I hope to elaborate on some of these topics in the next few weeks.

I’ve been doing some internal upgrades recently and noticed a few things that could cause some grief down the track. But firstly, this comic is really neat.  The release notes for ESX 3.5 and 3i have been out a little while and are being updated frequently with information regarding some slightly hairy situations that have arisen in the wild. Read them here. One of the issues I encountered (after I’d read about it) was on our Development environment - “ESX Server 3.5.0 Might Require Manual RPM Installation After Upgrading from ESX Server 3.0.1 or 3.0.2”. Read about it here. I’d seen this behaviour very early on in my home lab but put it down to my unsupported hardware or something that I did. Because software installations are normally faultless, right? Anyway, follow the instructions and everything’s fine, but I feel sorry for the poor people who launched right in late last year before this issue was identified. 

Another issue to look out for, which you’ll encounter first because you’re doing your VirtualCenter before building your ESX 3.5 environment (normally) is a minor issue relating to the use of the SQL Native Client ODBC Driver. Apparently the SQL Server driver is now longer good enough for VC and SQL 2005. More instructions can be found here - “Upgrades and new installations of VirtualCenter Server With Microsoft SQL Server 2005 Database Require SQL Native Client ODBC Driver”. If you’re scratching around for the files, go to the Microsoft site and get them from here.

 They’re minor issues, but no fun when you haven’t seen them before or are trying to look like a pro in front of the customer …

In a much earlier post, I went through a slew of esxcfg-* commands to attempt to provide an insight into how to use them in deployments or troubleshooting scenarios. What I didn’t provide, at least with my brief coverage of esxcfg-mpath, is a useful real-world example.

So if you’ve ever found yourself with a few CLARiiONs and 15 or so ESX hosts that are thrashing LUNs like nobody’s business, it might be time get dirty with esxcfg-mpath. Sure, you could try and do this from the GUI, but it will take, like, forever. And using the CLI impressess women. No, really.

So esxcfg-mpath -l will give you a list of paths to LUNs and what’s preferred and what’s active

[root@dev01 root]# esxcfg-mpath -l
Disk vmhba0:0:0 /dev/sda (139899MB) has 1 paths and policy of Fixed
 Local 4:0.0 vmhba0:0:0 On active preferred

Disk vmhba1:0:0 /dev/sdb (230400MB) has 2 paths and policy of Most Recently Used
 FC 15:4.0 210000e08b9c1e11<->200400a0b829871d vmhba1:0:0 Standby active preferred
 FC 15:4.0 210000e08b9c1e11<->200500a0b829871e vmhba1:1:0 On

Disk vmhba1:0:1 /dev/sdc (230400MB) has 2 paths and policy of Most Recently Used
 FC 15:4.0 210000e08b9c1e11<->200400a0b829871d vmhba1:0:1 On active preferred
 FC 15:4.0 210000e08b9c1e11<->200500a0b829871e vmhba1:1:1 Standby

Disk vmhba1:0:2 /dev/sdd (230400MB) has 2 paths and policy of Most Recently Used
 FC 15:4.0 210000e08b9c1e11<->200400a0b829871d vmhba1:0:2 Standby active preferred
 FC 15:4.0 210000e08b9c1e11<->200500a0b829871e vmhba1:1:2 On

But what if you want to set a different preferred path? 

[root@esxdev01 root]# esxcfg-mpath –preferred –path=vmhba1:1:0 –lun=vmhba1:0:0
Setting vmhba1:0:0 — vmhba1:1:0 as preferred path

[root@esxdev01 root]# esxcfg-mpath –path=vmhba1:1:0 –lun=vmhba1:0:0 –state=on
Setting vmhba1:0:0 — vmhba1:1:0 state to on

Now you can see the results:

[root@esxdev01 root]# esxcfg-mpath -l
Disk vmhba0:0:0 /dev/sda (139899MB) has 1 paths and policy of Fixed
 Local 4:0.0 vmhba0:0:0 On active preferred

Disk vmhba1:0:0 /dev/sdb (230400MB) has 2 paths and policy of Most Recently Used
 FC 15:4.0 210000e08b9c1e11<->200400a0b829871d vmhba1:0:0 Standby active
 FC 15:4.0 210000e08b9c1e11<->200500a0b829871e vmhba1:1:0 On  preferred

Disk vmhba1:0:1 /dev/sdc (230400MB) has 2 paths and policy of Most Recently Used
 FC 15:4.0 210000e08b9c1e11<->200400a0b829871d vmhba1:0:1 On active preferred
 FC 15:4.0 210000e08b9c1e11<->200500a0b829871e vmhba1:1:1 Standby

Disk vmhba1:0:2 /dev/sdd (230400MB) has 2 paths and policy of Most Recently Used
 FC 15:4.0 210000e08b9c1e11<->200400a0b829871d vmhba1:0:2 Standby active preferred
 FC 15:4.0 210000e08b9c1e11<->200500a0b829871e vmhba1:1:2 On

And you want to make the preferred path active now? Sure. Run esxcfg-rescan followed by the vmhba you want to scan.

Have fun.

If you’ve had the good (!) fortune of working with me for more than 5 minutes you’ll know I’m not the biggest fan of IT and the way it’s implemented at home and in the office to “improve” our lifestyles. Too often I’ve been on-site to implement or, perish the thought, have spent some of my own cash on software or hardware that promised so much and delivered so little. So I find it hard that I, like a number of my colleagues, friends and family invest so much in data availability. We put all of our music and movies and photos on computers. We don’t print out photos any more, we put them on digital displays. We don’t buy CDs, we buy the songs online. And happily ignore the prompt to back it up thinking that we’ll do that “some other time”. But then we have too much data to back up to DVD+-R, so we rely on external hard drives or just think it will never happen to us.

So when my father-in-law rang me a few weeks ago to say that he’d thought he’d lost “everything” I was thinking to myself that this was another typical end-user IT experience. He is a keen photographer and has been collecting photos, documents and other assorted data since Windows 95 was a twinkle in Bill Gates’ eye. He was an early adopter of Recordable CD so that he could back up his data, but was bitten a few times down the track by reliability issues. In the last few years he has been travelling quite a bit, and used an external drive as the primary repository for his data. He got nervous about having everything in one place and bought a second drive to act as backup for the first. Which is a good idea as far as it goes. Unfortunately, for reasons I’m still not entirely clear on, this second drive had become “full” and so he formatted it to make room for his next backup. Which generally just involves a Windows Explorer Drag and Drop activity. He had eschewed the bundled Retrospect HD Express in favour of a more granular approach which I kind of understand, having been the victim of semi-proprietary backup formats in the past.

The problems started when, after quick-formatting his backup drive, he went to copy his data to the backup drive and the primary drive failed. I asked him if he had copies on DVD, as we’d discussed this previously. He didn’t because the Nero software he had obtained had stopped working and he didn’t have time to look into it. I cussed under my breath (I think) and asked if he had anything he could recover from. No, he didn’t, and what can we do to get it back? So I asked Google what other poor schmucks had used and forum post results were variable to say the least.

I eventually settled on “GetDataBack for NTFS” from Runtime Software. It looked like it could do what we needed to and the cool thing was that it went through and did the analysis before you had to pony up for the software license. So we ran it over the primary drive for a few days and found nothing. Which wasn’t cool. And then the primary drive gave up on us. Which was less cool still. In hindsight, I should have taken an image of the drive before performing the analysis, but, well, I didn’t. So we ran the software over the backup drive and, long story short, retrieved most of the data up until about August last year. Funnily enough if I lost that much data at a client site I’d be in all kinds of pain, but when you put it in perspective, a few months out of 10 years is not too bad. I can’t recommended the Runtime Software enough. The price was great, and the have pre and post-sales support that actually picks up the phone and answers your questions. It may not be appropriate for every situation but for what we needed it did the job.

So, my father-in-law has since promised to copy his stuff to DVD-R regularly, bought himself a few new external drives and an iMac for good measure (I made him watch the podcast on Time Machine while we were waiting for his data to magically re-appear). Hopefully I won’t have to go through that experience any time soon. The good news is that we got a lot of the data back and I get to speak well of a software product, rather than deliver another shrill criticism of something that could have been so neat.