What have I been doing? – Part 3

CPU ID Masking

Sometimes vendors (in this case Dell) sell servers that have the same model name (in this case Dell PowerEdge 2950), but have CPUs in them from Intel that are incompatible as far as VMotion is concerned. In this case we had two existing nodes, purchased about 12 months ago, running Intel Xeon E5310 CPUs, and two new nodes running Intel Xeon E5410 CPUs. Even though they’re both dual-socket, quad-core hosts, and even though they are ostensibly the same machines, running the same BIOS revisions, VMotion doesn’t like it. This is Dell’s fault necessarily, they like to sell whatever CPU is cheap and performance focussed. It just so happens that, in the last 12 months, Intel have made a number of developments, and have started selling CPUs that do more for the same price as the older one. I have a friend who used to work at Intel and knew all of the model names and codes and compatibilities. I don’t though, so have a look here for the rundown on what Xeon chips are what. Basically, moving from Clovertown to Harpertown has caused the problem. Awesome.

When we tried to VMotion a Virtual Machine from an existing host to a new host, we got this error:

cpuid

VMware has a spiel on the issue here, and some possible solutions, depending on the version of ESX you’re running. A few other people have had the issue, and have discussed it here and here.

What’s frustating is that we were able to VMotion from the Acer hosts running dual-core CPUs to the existing quad-core Dell hosts with no problem. To think that we couldn’t then go from the Dell hosts to the new Dell hosts seems just, well, silly.

I didn’t want to setup the CPU ID masking on each Virtual Machine, so I elected to make the change on the VirtualCenter host. I edited the vpxd.cfg file, which is located by default in “C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter”. Here’s what I setup on the VirtualCenter host:

<guestOSDescriptor>
<esx-3-x-x>
<all-versions>
<all-guests>
<cpuFeatureMask>
<default-vendor>
<level-1>
<ecx>0000:0000:0000:xx0x:xxx0:0xxx:xxxx:xx01</ecx>
</level-1>
</default-vendor>
</cpuFeatureMask>
</all-guests>
</all-versions>
</esx-3-x-x>
</guestOSDescriptor>

I may have put too many settings in. But it worked fine. But I’d be feeling like if the consultant has to do that, I’d be chasing my sales guy for an upgrade to some compatible CPUs.

Upgrading VMware Tools on Netware 6.5

If you find yourself working on what some uncharitably call the dodo of network operating systems – NetWare 6.5 – and need to upgrade the VMware Tools – these instructions will help get you on your way.

To start, select “Install VMware Tools” on the VM.
The volume automounts on the NetWare guest.
Run vmwtools:\setup.ncf.
Some stuff happens (sorry I forgot to take some screenshots).
You’ll see “VMware Tools for NetWare are now running” once the upgrade is complete. You should then reboot the guests.

esXpress upgrade notes

esXpress from PHD Technologies is a neat backup and recovery tool. According to the website, it is “The ONLY scalable and completely fault tolerant backup, restoration and disaster recovery solution for Virtual Infrastructure 3.x. Whether you have 1TB of data or 50TB, esXpress makes a 100% complete backup of your entire virtual farm every night”. So maybe they’re blowing their own trumpet a little, but whenever I’ve had to use esXpress it’s been a pleasant experience. Moreso than some enterprise backup tools that I won’t name at this time. Well, okay, it’s nicer than reconfiguring devices in EMC NetWorker, or getting any kind of verbose logging out of Symantec Backup Exec.

There are some straightforward documents on the website that will get you started. The first thing you should look at is the Installation Guide. The next thing you should be looking at is the User’s Manual, and, well, you should probably consider reading up on the whole recovery process too.

The existing nodes already had an older version of esXpress installed, so a lot of the initial setup (ie the hard bit) had been done for me already. Hooray for other people doing work and saving me time!

To upgrade your version of esXpress, you need to uninstall (the old version) and re-install (the new version) rpm. This can be achieved by running the following commands:

rpm –e esxpress
rpm –e esxpressVBA

This will remove the application but not delete your backups.

Then install the new version of esXpress and the VBA and import the previous configuration. Running phd-import will import the previous esXpress settings, including licenses and so on. It can also import settings from another host, which will save some time.

It’s good stuff, and it works, so check it out.

What have I been doing? – Part 1

I recently had the “pleasure” of working on a project before Christmas that had a number of, er, interesting elements involved.  During the initial scoping, the only thing mentioned was two new arrays (with MirrorView/Asynchronous), a VMware ESX upgrade, and a few new ESX hosts.  But here’s what there really was:

– 4 NetWare 6.5 hosts in 2 NCS clusters;
– An EMC CLARiiON CX200 (remember them?) hosting a large amount (around 5TB) of NetWare and VMware data;
– A single McData switch running version 7 firmware;
– 2 new Dell hosts with incompatible CPUs with the existing 2950 hosts;
– A memory upgrade to the two existing nodes that meant one host had 20GB and the other had 28GB;
– A MirrorView target full of 1TB SATA-II spindles;
– A DR target with only one switch;
– Singley-attached (ie one HBA) hosts everywhere;
– An esXpress installation that needed to be upgraded / re-installed;
– A broken VUM implementation.

Hmmm, sound like fun? It kind of was, just because some of the things I had to do to get it to work were things I wouldn’t normally expect to do.  I don’t know whether this is such a good thing.  There’re a number of things that popped up during the project, each of which would benefit from dedicated blog posts.  But given that I’m fairly lazy, I think I’ll try and cram it all into one post.

Single switches and single HBAs are generally a bad idea

<rant> When I first started working on SANs about 10 minutes ago, I was taught that redundancy in a mid-range system is a good thing. The components that go into your average mid-range systems, while being a bit more reliable than your average gamedude’s gear, are still prone to failure. So you build a level of redundancy into the system such that when, for whatever reason, a component fails (such as a disk, fibre cable, switch or HBA), the system stays up and running. On good systems, the only people who know there’s a failure are the service personnel called out to replace the broken component in question. On a cheapy system, like the one you keep the Marketing Department’s critical morning tea photos on, a few more people might know about it. Mid-range disk arrays can run into the tens and hundreds of thousands of dollars, so sometimes people think that they can save a bit of cash but cutting a few corners by, for example, leaving the nodes with single HBAs, or having only one switch at the DR site, or using SATA as a replication target. But I would argue that, given your spending all of this cash on a decent mid-range array, why wouldn’t you do all you can to ensure it’s available all the time? Saying “My cluster provides the resiliency / We’re not that mission critical / I needed to shave $10K off the price” strikes me as counter-intuitive to the goal of providing reliable, available and sustainable infrastructure solutions. </rant>

All that said, I do understand that sometimes the people making the purchasing decisions aren’t necessarily the best-equipped people to understand the distinction between single- and dual-attached hosts, and what good performance is all about. All I can suggest is that you start with a solid design, and do the best you can to keep that design through to deployment. So what should you be doing? For a simple FC deployment (let’s assume two switches, one array, two HBAs per host), how about something like this?

Zoning

Notice that there’s no connection between the two FC switches here. That’s right kids, you don’t want to merge these fabrics. The idea is that if you munt the config on one switch, it won’t automatically pass that muntedness on to the peer switch. This is a good thing if you, like me, like to do zoning from the CLI but occassionally forget to check the syntax and spelling before you make changes. And for the IBM guys playing at home, the “double redundant loops” excuse doesn’t apply to the CLARiiON. So do yourself a favour, and give yourself 4 paths to the box, dammit!

And don’t listen to Apple and get all excited about just using one switch either – not that they’re necessarily saying that, of course … Or that they’re necessarily saying anything much at all about storage any more, unless Time Capsules count as storage. But I digress …