EMC – CLARiiON / VNX Disk Addresses

Mat came across this about 9 months ago, and I’d forgotten to post it here. I don’t know when it happened, or whether it’s always been like this, but EMC have apparently used letters to reference disks in VNX and CLARiiON systems for a while now. It looks something like this:

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

0

1

2

3

4

5

6

7

8

9

A B C D E

What we can’t work out is what it looks like when you’ve got a 60-disk DAE in play. In any case, I verified that it works with naviseccli on our CX4 arrays.

C:\Users\penguinpunk>naviseccli -h arraysp1 getdisk 1_3_14
Bus 1 Enclosure 3 Disk 14
Vendor Id: HITACHI
Product Id: HUS15606 CLAR600
Product Revision: C3A8
Lun: Unbound
Type: N/A
State: Enabled
Hot Spare: NO
Prct Rebuilt: Unbound
Prct Bound: Unbound

[snip]

Current Speed: 4Gbps
Maximum Speed: 4Gbps

And here it is but using E instead of 14.


C:\Users\penguinpunk>naviseccli -h arraysp1 getdisk 1_3_E
Bus 1 Enclosure 3 Disk 14
Vendor Id: HITACHI
Product Id: HUS15606 CLAR600
Product Revision: C3A8
Lun: Unbound
Type: N/A
State: Enabled
Hot Spare: NO
Prct Rebuilt: Unbound
Prct Bound: Unbound

[snip]

Current Speed: 4Gbps
Maximum Speed: 4Gbps

EMC – Unisphere / Analyzer weirdness

Mat came across a weird problem with Unisphere the other day while he was trying to retrieve some nar files for EMC to look at. Normally I like to post solutions up here but in this case I don’t know what the solution is. Previously, it was my understanding that we could retrieve access multiple nar files from multiple arrays in the same domain via the one Unisphere session. This doesn’t seem to be the case anymore. As background, we run 4 CX4-960s and a CX4-240 in a single Unisphere domain. These arrays were all upgraded to 30.525 recently.

Normally, I’d login to one of the arrays and go to Monitoring -> Analyzer and then Retrieve Archive. So far, so good.

analyzer1

But when I change the “Retrieve Archive From:” selection, I get the following.

analyzer2

Notice that I’m trying to retrieve files from the array serial number 0260, but I’m still seeing 0679, even after hitting Refresh. Maybe it’s Unisphere on that array, let’s try another.

analyzer3

Ok, I’ve logged on to array 0260 and can now retrieve the files I want. But do I still get weird behaviour? Yes, yes I do.

analyzer4

If anyone has any ideas, I’m all ears. I think Mat is going to log a call at some stage.

 

EMC – DIY Heatmaps – Updated Version

Mat has updated the DIY Heatmaps for EMC CLARiiON and VNX arrays to version 4.01. You can get it from the Utilities page. Any and all feedback welcome.

Updates and Changes to the script

  • Add database storage / retrieval for performance stats. The database size will be approximately 2.1 x the size of the NAR file based on the default interval of 30 minutes. On my PC it took a bit over 9 hours to process 64 NAR files into a database, the NAR files were 1.95GB and the resulting database was 4.18GB. However running the script over the database to produce a heatmap only takes seconds.
  • Changed to use temporary tables for transitional data.  This should slightly reduce the size of the database file, as the temporary data is not written to disk.
  • Changed the way the script processes multiple NAR files, the script previously bunched all NAR files into a single naviseccli process, this was problematic if you were processing multiple large NAR files, the script now processes them one at a time.
  • Add command line options:

–output_db                               Output the processed NAR file to the nominated database

–input_db                                  Use the nominated database as the source of data for the heatmap

–s_date                                       Specify a start date/time must be in the format (with quotes if specifying date and time “mm/dd/yyyy hh:mm:ss”

–e_date                                       Specify an end date/time

–retrieve_all_nar                     When retrieving NAR files from the array, you can now retireve all nar files (it wont overwrite files already downloaded)

–process_only_new                 If you are downloading NAR files, only process files that haven’t been downloaded previously

–max_nar_files                        Set the maximum number of files to download and process

 

Please let us know if you find any bugs or problems with the script, or if you have any further suggestions for changes and enhancements.

Thanks

Mat.

EMC – Sometimes it’s best not to pay too much attention to USM

Just a quick one to start the year off on the right note. I was installing updated Utility Partition software on our lab CX4s today and noticed that USM was a bit confused as to when it had started installing a bit of the code. Notice the Time started and Time elapsed section. Well, I thought it was amusing.

CX4_Utility_Partition

EMC – Maximum Pool LUN Size

Mat has been trying to create a 42TB LUN to use temporarily for Centera backups. I don’t want to go into why we’re doing Centera backups, but let’s just say we need the space. He created a Storage Pool on one of the CX4-960s, using 28 2TB spindles and 6+1 private RAID Groups. However, when he tried to bind the LUN, he got the following error.

err1

Weird. So what if we set the size to 44000GB?

err2

No, that doesn’t work either. Turns out, I should really read some of the stuff that I post here, like my article entitled “EMC CLARiiON VNX7500 Configuration guidelines – Part 1“, where I mention that the maximum size of a Pool LUN is 16TB. I was wrong in any case, as it looks more like it’s 14TB. Seems like we’ll be using RAID Groups and MetaLUNs to get over the line on this one.

Get cracking with Stormons 1.3.2 GA

I covered the release of Stormons 1.3.2 GA here and mentioned a few of the highlights of the new version. In between doing some vCD stuff and the general faffing about that seems to happen at this time of year, I’ve had a chance to install it on my lab PC and monitor our two CX4-120s. I’m not going to regurgitate the installation manual (which can be downloaded from here). I thought it might be more useful if I covered off the bits that weren’t obvious to me when I first installed the product. Note that if you’re used to deploying web apps and familiar with Apache configuration files, you won’t have as much of an issue with getting this working as I did. As it happens, I’m more idiot than savant when it comes to these things so it took me a little longer than it should have to get going.

Windows Server 2003 is listed as the supported OS. I used 32-bit Windows 7, as that was the OS on my laptop at the time. I used Apache 2.2 as the web server. There’s a bunch of supported models listed in the installation guide, but I’ve only used this product with EMC CLARiiON CX4 arrays. I can’t speak for its usefulness with NetApp FAS, EMC Symmetrix, EMC Celerra or Brocade FC devices.

I copied the extracted (and renamed) installation files to C:\tools\ and set the System Variable accordingly. Here’re the changes I made to my httpd.conf file to get things working. If I’ve done things really badly, feel free to chime in.

The first bit is to add a Directory for the location of the SM1.3 installation.

# STORMONS 
<Directory "C:/tools/SM1.3/http">
SetEnv STORMONS_HOME "C:/tools/SM1.3"
AllowOverride None
Options Indexes FollowSymLinks ExecCGI
Order allow,deny
Allow from all
</Directory>

I then added in Aliases for the /SM and other directories.

# STORMONS
Alias /SM "C:/tools/SM1.3/http"
Alias /capa_spk "C:/tools/SM1.3/repository/capacity/spk"
Alias /perf_spk "C:/tools/SM1.3/repository/performance/spk"

I also uncommented the .cgi handler under

AddHandler cgi-script .cgi

And that was it from the Apache side of things. After that I ran through the Stormons configuration as per the installation guide, setting paths to various binaries, etc. Note that if you’ve just copied RRDtool from the internet it’s worth running it first. I was missing the msvcr100.dll and it was crapping out without my knowledge (re-installing the Visual Studio 2008 redistributable fixed that). When you’re installing the Stormons polling service, make sure that you’re running a command prompt with sufficient privileges to install services, or you won’t get any useful results. I’m still having problems with RRDtool drawing pictures, but I’m working on that.

I’m hoping to do a more thorough run-through on some of the stuff you can do with Stormons in the next week or two.

Stormons 1.3.2 GA Released

Didier from Stormons recently got in contact (I knew a degree in French would eventually come in handy for something) to let me know that Stormons 1.3.2 GA has now been released. I’ve been meaning to give this tool a run in the lab for a while now, and I’m hopeful that I can do something in the next few weeks when work gets a bit quieter. You can download pre-compiled Windows (32 and 64-bit) binaries or Linux source code here. I won’t go into all of the features now, but here’re a few highlights:

In terms of the code, it is comprised of 43000 lines of Perl 5.14, 70 HTML reports, a multi-threaded engine and a scheduler.

Some reports offer the ability to compare configurations between two dates (helpful for trending and capacity planning).

Detailed alerts can be created.

“Storage Classes” can be created and used as a basis for Chargeback reports.

There’s a bunch of other features, including support for Brocade fabric traffic monitoring and some Nagios integration. And, as mentioned previously, there’s 70 HTML reports covering RAID Groups, CX Storage Pools, MirrorView, NetApp Volumes, Celerra LUNs, and so on. The program has been written for storage admins by a storage admin. I’m looking forward to testing it out on our CX4s, and Mat might even have a little less work to do as a result.

EMC – MirrorView Secondary Image States

I sometimes get asked what the definition of these states is, and I frequently have trouble defining it clearly. Fortunately, EMC’s MirrorView Knowledgebook has been updated to incorporate Release 32, and Appendix A has some succinct definitions. If you can’t be bothered looking them up for yourself, here they are.

  • Synchronized – The secondary image is identical to the primary. This state persists only until the next write to the primary image, at which time the image state becomes Consistent.
  • Consistent – The secondary image is identical to either the current primary image or to some previous instance of the primary image. This means that the secondary image is available for recovery when you promote it.
  • Synchronizing – The software is applying changes to the secondary image to mirror the primary image, but the current contents of the secondary are not known and are not usable for recovery.
  • Out-of-Sync – The secondary image requires synchronization with the primary image. The image is unusable for recovery.
  • Rolling Back (MV/A only) – A successful promotion occurred where there was an unfinished update to the secondary image. This state persists until the Rollback operation completes.

I think one of the key things here is to pay attention to the various image states, particularly if you’re seeing a lot of out-of sync states on your secondaries. You don’t want to have to explain to people why they can’t recover secondaries in the event of a serious failure. And, more importantly, get on Powerlink and check out the MirrorView Knowledgebook (H2417).

EMC – CX4 Configuration – a few things I’d forgotten

I’ve been commissioning some new CX4-960s recently (it’s a long story), and came across a few things that I’d forgotten about for some reason. If you’re running older disks, and they get replaced by EMC, there’s a good chance they’ll be a higher capacity. In our case I was creating a storage pool with 45 300GB FC disks and kept getting the following error.

This error was driving me nuts for a while, until I realised that one of the 300GB disks had, at some point, been replaced with a 450GB drive. Hence the error.

The other thing I came across was the restriction that Private LUNs (Write Intent Log, Reserved LUN Pool, MetaLUN Components) have to reside on traditional RAID Groups and can’t live in storage pools. Not a big issue, but I hadn’t really planned to use RAID Groups on these arrays. If you search for emc254739 you’ll find a handy KB article on WIL performance considerations, including this nugget “Virtual Provisioning LUNs are not supported for the WIL; RAID group-based LUNs or metaLUNs should be used”. Which clarifies why I was unable to allocate the 2 WIL LUNs I’d configured in the pool.

*Edit* I re-read the KB article and realised it doesn’t address the problem I saw. I had created thick LUNs on a storage pool, but these weren’t able to be allocated as WIL LUNs. Even though the article states “[The WIL LUNs] can either be RAID-group based LUNs, metaLUNs or Thick Pool LUNs”. So I don’t really know. Maybe it’s a VNX vs CX4 thing. Maybe not.

EMC – DIY Heatmaps – Updated Version

Mat has updated the DIY Heatmaps for EMC CLARiiON and VNX arrays to version 3.0211. You can get it from the Utilities page here. Any and all feedback welcome. Changes below:

Add –min_colour, –mid_colour, –max_colour options (just a change of spelling of colour)

Remove case sensitivity for colours

Added FC SSD drive type