EMC – Some new scripts

Mat has come up with a few new scripts – FlipFASTTiering and ReplicationCapacity. They’re PERL scripts that you can use to list / modify FAST Tiering scheduling and report on MirrorView replication data respectively. Hopefully you’ll find them of some use. Further information can be found on the Utilities page.

EMC – MirrorView Secondary Image States

I sometimes get asked what the definition of these states is, and I frequently have trouble defining it clearly. Fortunately, EMC’s MirrorView Knowledgebook has been updated to incorporate Release 32, and Appendix A has some succinct definitions. If you can’t be bothered looking them up for yourself, here they are.

  • Synchronized – The secondary image is identical to the primary. This state persists only until the next write to the primary image, at which time the image state becomes Consistent.
  • Consistent – The secondary image is identical to either the current primary image or to some previous instance of the primary image. This means that the secondary image is available for recovery when you promote it.
  • Synchronizing – The software is applying changes to the secondary image to mirror the primary image, but the current contents of the secondary are not known and are not usable for recovery.
  • Out-of-Sync – The secondary image requires synchronization with the primary image. The image is unusable for recovery.
  • Rolling Back (MV/A only) – A successful promotion occurred where there was an unfinished update to the secondary image. This state persists until the Rollback operation completes.

I think one of the key things here is to pay attention to the various image states, particularly if you’re seeing a lot of out-of sync states on your secondaries. You don’t want to have to explain to people why they can’t recover secondaries in the event of a serious failure. And, more importantly, get on Powerlink and check out the MirrorView Knowledgebook (H2417).

EMC – Manipulating the Write Intent Log LUNs with naviseccli

Last week Mat was wanting to migrate some WIL LUNs on one of our CX4s to a different RAID Group. Unfortunately, one does not simply migrate WIL LUNs. Firstly, once they’re allocated, they’re treated as Private LUNs by FLARE, so you can’t use Virtual LUN technology to migrate them. Secondly, if you have active MirrorView mirrors in place, you’ll get the following error when trying to de-allocate the WIL LUNs via Unisphere: “Unable to deallocate WIL since in use (0x71058199)[0x71058199]“. Note that fracturing the mirrors is not sufficient to avoid this error.

So there’s a few things you need to do to resolve this. Firstly, confirm the WIL LUNs in use on the array (there’s only 2).

naviseccli -user User_ID -scope 0 -password xxxxxxxx -h mirror -sync -listlog
Storage Processor: SP A
Lun Number: 10000
Storage Processor: SP B
Lun Number: 10001

Now list all the mirrors that are using the WIL. Note that it lists the primary and secondary mirrors, so you’ll need to extraxt the primary mirrors from the list.

naviseccli -user User_ID -scope 0 -password xxxxxxxx -h mirror -sync -list -usewriteintentlog
MirrorView Name: DC1_FC_R5_DATA_0002_SRM
Write Intent Log Used: YES

Now the simplest thing to do is create a script that runs the command below for each of the mirrors in your list of primaries. This command disable the use of the WIL, and -o won’t prompt for confirmation.

naviseccli -user User_ID -scope 0 -password xxxxxxxx -h mirror -sync -change -name DC1_FC_R5_DATA_0002_SRM -usewriteintentlog no -o

Now that you’ve done that, you can allocate the new WIL LUNs.

naviseccli -user User_ID -scope 0 -password xxxxxxxx -h mirror -sync -allocatelog -spA lun_ID1 -spB lun_ID2

Now run your script again, changing -usewriteintentlog from no to yes.

naviseccli -user User_ID -scope 0 -password xxxxxxxx -h mirror -sync -change -name DC1_FC_R5_DATA_0002_SRM -usewriteintentlog yes -o

And now you can get rid of those old WIL LUNs. Huzzah!

EMC – Configure the Reserved LUN Pool with naviseccli

I’ve been rebuilding our lab CLARiiONs recently, and wanted to configure the Reserved LUN Pool (RLP) for use with SnapView and MirrorView/Asynchronous. Since I spent approximately 8 days per week in Unisphere recently performing storage provisioning, I’ve since made it a goal of mine to never, ever have to log in to Unisphere to do anything again. While this may be unattainable, you can get an awful lot done with a combination of Microsoft Excel, Notepad and naviseccli.

So I needed to configure a Reserved LUN Pool for use with MV/A, SnapView Incremental SAN Copy, and so forth. I won’t go into the reasons for what I’ve created, but let’s just say I needed to create about 50 LUNs and give them each a label. Here’s what I did:

Firstly, I created a RAID Group with an ID of 1 using disks 5 – 9 in the first enclosure.

C:\>naviseccli -h createrg 1 0_0_5 0_0_6 0_0_7 0_0_8 0_0_9

It was then necessary to bind a series of 20GB LUNs to use, 25 for each SP. If you’re smart with Excel you can set the following command to do this for you with little fuss.

C:\>naviseccli -h  bind r5 50 -rg 1 -aa 0 -cap 20 -sp a -sq gb  

Here I’ve specified the raid-type (r5), the lun id (50), the RAID Group (1),  -aa 0 (disabling auto-assign), -cap (the capacity), -sp (a or b), and the -sq (size qualifier, which can be mb|gb|tb|sc|bc). Note that if you don’t specify the LUN ID, it will automatically use the next available ID.

So now I’ve bound the LUNs, I can use another command to give them a label that corresponds with our naming standard (using our old friend chglun):

C:\>naviseccli -h chglun -l 50 -name TESTLAB1_RLP01_0050

Once you’ve created the LUNs you require, you can then add them to the Reserved LUN Pool with the reserved command.

C:\>naviseccli -h reserved -lunpool -addlun 99

To check that everything’s in order, use the -list switch to get an output of the current RLP configuration.

C:\>naviseccli -h reserved -lunpool -list
Name of the SP:  GLOBAL
Total Number of LUNs in Pool:  50
Number of Unallocated LUNs in Pool:  50
Unallocated LUNs:  53, 63, 98, 78, 71, 56, 88, 69, 92, 54, 99, 79, 72, 58, 81, 5
7, 85, 93, 61, 96, 67, 76, 86, 64, 50, 66, 52, 62, 68, 77, 89, 70, 55, 65, 91, 8
0, 73, 59, 82, 90, 94, 84, 97, 74, 60, 83, 95, 75, 87, 51
Total size in GB:  999.975586
Unallocated size in GB:  999.975586
Used LUN Pool in GB:  0
% Used of LUN Pool:  0
Chunk size in disk blocks:  128
No LUN in LUN Pool associated with target LUN.

If, for some reason, you want to remove a LUN from the RLP, and it isn’t currently in use by one of the layered applications, you can use the -rmlun switch.

C:\>naviseccli -h reserved -lunpool -rmlun 99 -o

If you omit the override [-o] option, the CLI prompts for confirmation before removing the LUN from reserved LUN pool. It’s possible to argue that, with the ability to create multiple LUNs from Unisphere, it might be simpler to not worry about naviseccli, but I think that it’s a very efficient way to get things done quickly, particularly if you’re working in a Unisphere domain with a large number of CLARiiONs, or on a workstation that has some internet browser “issues”.

EMC – getting the status of MirrorView operations with naviseccli

 Tired of sifting through a Consistency Group to see the status of MirrorView synchronizations? Tire no more with the mirror -sync -listsyncprogress command!

c:\>naviseccli -address mirror -sync -listsyncprogress

MirrorView Name:  MIRROR_FC_R5_DAT02_0030_SRM
Has Secondary Images:  YES
Image UID:  50:06:01:60:BB:20:36:C1
Image State:  Synchronizing
Synchronizing Progress(%):  97

MirrorView Name:  MIRROR_FC_R5_MNT01_0042_SRM
Has Secondary Images:  YES
Image UID:  50:06:01:60:BB:20:36:C1
Image State:  Synchronized
Synchronizing Progress(%):  100

MirrorView Name:  MIRROR_FC_R5_DAT04_0046_SRM
Has Secondary Images:  YES
Image UID:  50:06:01:60:BB:20:36:C1
Image State:  Synchronizing
Synchronizing Progress(%):  72

MirrorView Name:  MIRROR_EFD_R5_MSMQ01_0055_SRM
Has Secondary Images:  YES
Image UID:  50:06:01:60:BB:20:36:C1
Image State:  Synchronized
Synchronizing Progress(%):  100

EMC – naviseccli – how to identify the MirrorView ports

With the introduction of Flexports the location of the MirrorView front-end port is not always the last port on the SP. Sometimes you’ll find it quicker to identify the MirrorView port through the use of naviseccli. Here’s how to do it.

If you don’t have the MirrorView enabler installed on the array, but want to keep the port aside for when you do, use the ioportconfig command to identify any ports marked as Special in Port Usage – these ones are MirrorView ports.

C:\>naviseccli -h ioportconfig -list > c:\tmp\array_ports.txt

SP ID:   A
I/O Module Slot:   1
I/O Module Type:   Fibre Channel
I/O Module State:   Present
I/O Module Substate:   Good
I/O Module Power state:   On
I/O Carrier:   No

Information about each port on this I/O module: 
Physical Port ID:   0
Port Role:   BE
Logical Port ID:   2
Port Usage:   Normal
Port Type:   Fibre Channel
Port State:   Enabled
Port Substate:   Good
Is Persisted:   Yes

Physical Port ID:   1
Port Role:   BE
Logical Port ID:   3
Port Usage:   Normal
Port Type:   Fibre Channel
Port State:   Enabled
Port Substate:   Good
Is Persisted:   Yes

Physical Port ID:   2
Port Role:   FE
Logical Port ID:   2
Port Usage:   Normal
Port Type:   Fibre Channel
Port State:   Enabled
Port Substate:   Good
Is Persisted:   Yes

Physical Port ID:   3
Port Role:   FE
Logical Port ID:   3
Port Usage:   Special
Port Type:   Fibre Channel
Port State:   Enabled
Port Substate:   Good
Is Persisted:   Yes

If you have the MirrorView enabler installed the port command is all you need. This is a handy command with some pretty verbose output, and provides information on the total number of attached initiators, amongst other things.

C:\>naviseccli -h port -list -sp -all > c:\tmp\array_port_list.txt


SP Name:             SP B
SP Port ID:          3
SP UID:              50:06:01:60:BB:20:36:C1:50:06:01:6B:3B:20:36:C1
Link Status:         Up
Port Status:         Online
Switch Present:      YES
Switch UID:          20:09:00:0D:EC:5A:FA:81:20:CB:00:0D:EC:5A:FA:80
SP Source ID:        852207
ALPA Value:         0
Speed Value :         4Gbps
Auto Negotiable :     NO
Available Speeds:    
Requested Value:      Auto
MAC Address:         Not Applicable
SFP State:           Online
Reads:               0
Writes:              641452077
Blocks Read:         0
Blocks Written:      3126242880
Queue Full/Busy:     0
I/O Module Slot:     1
Physical Port ID:    3
Usage:     Mirrorview

Of course you can always look at the ports in Unisphere – check out an old post I did a little while ago for information on how to do that.

EMC – naviseccli – Basics – Part 1

For those of you unfamiliar with the EMC tool naviseccli, I thought I’d do a few posts on some useful commands that can be run to save you some time if you’ve been doing a lot of repetitive tasks. The naviseccli tool is very powerful and can be unforgiving, so as always I recommend you read the manual and be sure that you know what you’re doing before you do it. The tool is ostensibly used to send status or configuration requests to a storage system (specifically the VNX or CX4) via the command line and can be installed on a Windows, Linux, Solaris or whatever platform.

Typing naviseccli at the prompt will provide the following:

[-address IPAddress|NetworkName|-h IPAddress|NetworkName]
[-f filename]
[-password password]
[-port port]
[-scope 0|1|2]
[-timeout |-t timeout]
[-user username]
CMD [optional_command_switches]

There’s a (metric) tonne of other stuff it can do via the CMD switch, but I thought for an introduction we’ll start with the basics. I’m going to try and avoid regurgitating the user manual, and instead focus on real world examples where it’s come in useful for me.  Okay, so maybe I’ll regurgitate a portion of the manual. So let’s get down to it.

-f filename. This switch specifies to store data in a file. If you’re working inside a script you may find this is a more useful option than other output options you have available.
-m. Suppresses output except for values. This option is most useful when used as part of a script. Note that this is only supported for commands that originated in Classic CLI.
-nopoll|-np. Directs the feature provider not to issue a poll request. This switch significantly increases performance when dealing with large or multiple storage systems. The feature provider automatically polls unless this switch is specified. Note that when the -nopoll switch is set, get commands may return stale data and set commands may erase previously changed settings. Use caution when the -nopoll switch is set.

-user. If you don’t want to create a security file on the machine you’re working on (I’ll cover this in a future post), you’ll need to specify a username that works on the system you’re addressing. At this point you should also provide the scope (0|1|2), with 0 being global, 1 being local to the SP you’re addressing, and 2 being LDAP credentials.

-port portnumber. Sets the port number (type) of the storage system. The default is 443. If you choose to change the default port number, management port 2163 will be supported; however, you will need to specify the -port switch and number 2163 in every subsequent command you issue. That doesn’t sound like fun does it?

EMC MirrorView – New Article

I wrote a brief article on configuring EMC MirrorView from scratch through to being ready for VMware SRM usage. It’s not really from scratch, because I don’t go through the steps required to load the MirrorView enabler on the frame. But I figured that’s more a CE-type activity in any case. I hope to follow it up with a brief article on the SRM side of things. You can find my other articles here. Enjoy!

CX4 I/O Modules and MirrorView

When the CX4 was released some time ago, I sat through on-line training and learnt about the differences between the CX3 and CX4. One of the big things for the CX4 was the introduction of I/O modules that enabled you to scale out front-end performance for the array. But, like any training I’ve undertaken, I filed away the information about how they work in the real world and forgot all about it.

So when we went to test MV/S connectivity between our two 960 arrays, we had some connectivity issues. When we went to establish a MirrorView connection, this is what we got on the first array.

On the second array, it looked like this.

Not so cool. But the boneheaded thing I’d forgotten from my CX4 What’s New training was that, when an array is first initialised, the MirrorView ports are assigned to the last port. If you add I/O modules after the fact, those MV ports don’t change.  So while one of the array’s was a new CX4-960, the other was an upgraded CX3-40f, which had different ports when it was upgraded. Here are some pictures that serve to illustrate my words.

You’ll notice that on the second array the MirrorView ports are A1 and B1, not A3 and B3.

And finally, it’s important to respect the zoning rules with MirrorView – something my colleague didn’t understand when he had B1 zoned to A3, and so forth. Good times.

MV/S consistency groups and multiple secondary images

I recently completed a migration for a client from a CX3-20 to a CX4-240. I’d done similar work for them in the past, moving their primary site from a CX500 to a CX4-240. This time things were simpler, as the secondary site I was working on contained primarily replicas of LUNs from the primary site. For those LUNs that weren’t mirrors, I used either Incremental SAN Copy, or sVMotion to migrate the data. The cool thing about MirrorView/Synchronous on the CLARiiON is that you can send replicas from one source to multiple (2) targets. As my client was understandably nervous about the exposure of not having current replicas if there was a problem, we decided that adding a secondary image on the CX4 and waiting for it to synchronize before removing the CX3 image would be the safest. The minor issue was that most of these LUNs were in MirrorView Consistency groups. If you’ve played with VMware’s Site Recovery Manager, you’ll know what these are. But for those of you that don’t, let me drop a little knowledge.

Consistency Groups are in essence a group of secondary images on a CLARiiON that are treated as a single entity. As a result of this treatment, the remote images are consistent, but may contain information that is ever so slightly older than information on the primary images. The thinking behind this is fairly obvious – you don’t want your SQL database LUN to be out of sync with the SQL logs LUN, in the same way that you wouldn’t want the Exchange mailbox LUN to be out of sync with the transaction logs. That would be silly. And it would make recovery in the event of a site failure really difficult. And thus Consistency Groups were born. Hooray.

But there are a few rules that you need to follow with Consistency Groups:

  • Up to 16 groups per CLARiiON;
  • Up to 16 mirrors per group;
  • Primary image LUNs must all be on the same CLARiiON;
  • Secondary image LUNs must all be on the same CLARiiON;
  • Images must be of the same type (you can’t mix sync and async in the same group);
  • Operations happen on all mirrors at the same time.

As I mentioned previously, you can have 1:2 LUN:replica ratio when using MV/S. Unfortunately, when Consistency Groups are in use, the ratio goes back to 1:1. So our precautionary strategy for replica migration was already under pressure. As well as this, we couldn’t have multiple CLARiiONs providing replica images in the same consistency group. The only option that my tired brain could really think of was to remove the replicas from the Consistency Groups, re-sync the new replicas with the primary copies, and then add the new replicas into the Consistency Groups. By using this methodology, we risked having inconsistent volumes on a host, but we didn’t have the risk of not having replicas at all. If any one has a better approach, I’d love to hear about it.