EMC – Using naviseccli to change the LUN Migration priority of a LUN

I am currently working through a metric shitload of LUN migrations on one of our CX4 arrays, with the end-goal being a replacement of 60 or so 300GB FC disks with 60 600GB FC disks. This is the sort of stuff you have to do when you’re not allowed to buy new arrays (budget notwithstanding). But enough moaning.

I’ve set up a number LUN migrations to get this done, but Mat says I shouldn’t run them at a high priority during business hours. I say if the spurs fit. But whatever. With naviseccli, you can list all of the LUNs that are currently migrating on the array. Once you have this list, you can grep for the LUN IDs and modify the migration priority depending on whether it’s during or after hours.

C:\Users\dan>naviseccli -h SPA migrate -list
Source LU Name: LUN 7010
Source LU ID: 7010
Dest LU Name: LUN 6
Dest LU ID: 6
Migration Rate: HIGH
Current State: MIGRATING
Percent Complete: 89
Time Remaining: 24 minute(s)

Source LU Name: LUN 7018
Source LU ID: 7018
Dest LU Name: LUN 8
Dest LU ID: 8
Migration Rate: HIGH
Current State: MIGRATING
Percent Complete: 90
Time Remaining: 20 minute(s)

Source LU Name: LUN 600
Source LU ID: 600
Dest LU Name: LUN 1
Dest LU ID: 1
Migration Rate: MEDIUM
Current State: MIGRATING
Percent Complete: 44
Time Remaining: 8 hour(s) 23 minute(s)
[snip]

 

So let’s say I want to change the migration priority for LUN ID 7018. All I need to use is the modify command, specify the correct source ID, and the set the rate to low | medium | high | asap. I’m using -o to avoid prompting.

C:\Users\dan>naviseccli -h SPA migrate -modify -source 7018 -rate medium -o

Now I can use the -list switch to verify my work. Note that it takes the array a little while to calculate the new estimate for  Time Remaining. So if you really want to know that, give it a minute or so.

C:\Users\dan>naviseccli -h SPA migrate -list
Source LU Name: LUN 7010
Source LU ID: 7010
Dest LU Name: LUN 6
Dest LU ID: 6
Migration Rate: HIGH
Current State: MIGRATING
Percent Complete: 90
Time Remaining: 22 minute(s)

Source LU Name: LUN 7018
Source LU ID: 7018
Dest LU Name: LUN 8
Dest LU ID: 8
Migration Rate: MEDIUM
Current State: MIGRATING
Percent Complete: 91
Time Remaining: ?

Source LU Name: LUN 600
Source LU ID: 600
Dest LU Name: LUN 1
Dest LU ID: 1
Migration Rate: MEDIUM
Current State: MIGRATING
Percent Complete: 44
Time Remaining: 8 hour(s) 23 minute(s)
[snip]

C:\Users\dan>

And that’s pretty much it.

EMC – DIY Heatmaps – Updated Version

Mat has updated the DIY Heatmaps for EMC CLARiiON and VNX arrays to version 3.020. You can get it from the Utilities page here. Any and all feedback welcome.

EMC – Flash LEDs with naviseccli

Sometimes, like when you have 64 DAEs across 5 racks, it’s useful to be able to tell which DAE you need to be working on. You can flash the LEDs of a DAE via Unisphere, but where’s the fun in that? Using naviseccli you can also do it as follows:

usage: flashleds -e <enclosure number> [-b <bus number>] <on|off>
C:\Users\dan>naviseccli -h 10.196.8.10 flashleds -b 4 -e 1 on Enclosure 1 ON
C:\Users\dan>naviseccli -h 10.196.8.10 flashleds -b 4 -e 1 off Enclosure 1 OFF

It’s a simple thing, but handy to know when you’re scratching around in the data centre trying to remember which disks you were meant to be pulling out of which production array.

Updated Articles Page

I’ve created a short document that provides some guidance on upgrading disk firmware on a CLARiiON / VNX using Unisphere Service Manager. Hopefully it will be of some use to someone. As always, check out the other articles on the page. And yes, I do take requests.

EMC – Why FAST VP Best Practice is Best Practice

Those of you fortunate enough to have worked with me in a professional capacity will know that I’m highly opinionated. I generally try not to be opinionated on this blog, preferring instead to provide guidance on tangible technical things. On this occasion, however, I’d like to offer my opinion. I overheard someone in the office recently saying that best practices are just best practices, you don’t have to follow them. Generally speaking, they’re right. You don’t have to do what the vendor tells you, particularly if it doesn’t suit your environment, circumstances, whatever. What annoys me, though, is the idea that’s been adopted by a few in my industry that they can just ignore documents that cover best practices because there’s no way the vendor would know what’s appropriate for their environment. At this point I call BS. These types of documents are put out there because the vendor wants you to use their product in the way it was meant to be used. And – get this – they want you to get value from using their product. The idea being that you’ll be happy with the product, and buy from the vendor again.

BP Guides aren’t just for overpaid consultants to wave at know-nothing customers. They’re actually really useful guidelines around which you can base your designs. Crazy notion, right?

So, to my point. EMC recommend, when you’re using FAST VP on the CLARiiON / VNX, to leave 10% free space in your tiers. The reason they recommend this is that they want FAST VP to have sufficient space to move slices between tiers. Otherwise you’ll get errors like this “712d841a Could not complete operation Relocate 0xB00031ED4 allocate slice failed because 0xe12d8709”. And you’ll get lots of them. Which means that FAST is unable to move slices around the pool. In which case why did you by FAST in the first place? For more information on these errors, check out emc274840 and emc286486 on Powerlink.

If you want an easy way to query a pool’s capacity, use the following naviseccli command:

naviseccli -h ipaddress storagepool -list -tiers
Pool Name: SP_DATA_1
Pool ID: 3

Tier Name: FC
Raid Type: r_5
User Capacity (GBs): 33812.06
Consumed Capacity (GBs): 15861.97
Available Capacity (GBs): 17950.10
Percent Subscribed: 46.91%
Data Targeted for Higher Tier (GBs): 0.00
Data Targeted for Lower Tier (GBs): 0.00
Disks (Type):

Bus 6 Enclosure 7 Disk 14 (Fibre Channel)
Bus 6 Enclosure 7 Disk 12 (Fibre Channel)
Bus 6 Enclosure 7 Disk 10 (Fibre Channel)
Bus 3 Enclosure 5 Disk 3 (Fibre Channel)
Bus 3 Enclosure 5 Disk 1 (Fibre Channel)
Bus 4 Enclosure 5 Disk 2 (Fibre Channel)
Bus 4 Enclosure 5 Disk 0 (Fibre Channel)
[snip]
Bus 2 Enclosure 6 Disk 14 (Fibre Channel)
Bus 2 Enclosure 6 Disk 12 (Fibre Channel)
Bus 2 Enclosure 6 Disk 10 (Fibre Channel)
Bus 0 Enclosure 2 Disk 0 (Fibre Channel)
Bus 5 Enclosure 6 Disk 8 (Fibre Channel)
Bus 3 Enclosure 2 Disk 4 (Fibre Channel)
Bus 7 Enclosure 5 Disk 6 (Fibre Channel)

Pool Name: SP_TEST_10
Pool ID: 2
Tier Name: FC
Raid Type: r_10
User Capacity (GBs): 1600.10
Consumed Capacity (GBs): 312.02
Available Capacity (GBs): 1288.08
Percent Subscribed: 19.50%
Data Targeted for Higher Tier (GBs): 0.00
Data Targeted for Lower Tier (GBs): 0.00
Disks (Type):
Bus 1 Enclosure 7 Disk 3 (Fibre Channel)
Bus 1 Enclosure 7 Disk 5 (Fibre Channel)
Bus 1 Enclosure 7 Disk 7 (Fibre Channel)
Bus 1 Enclosure 7 Disk 2 (Fibre Channel)
Bus 1 Enclosure 7 Disk 4 (Fibre Channel)
Bus 1 Enclosure 7 Disk 6 (Fibre Channel)
Bus 1 Enclosure 7 Disk 9 (Fibre Channel)
Bus 1 Enclosure 7 Disk 8 (Fibre Channel)

And if you want to get the status of FAST VP operations on your pools, use the following command:

naviseccli -h ipaddress autotiering -info -opstatus
Storage Pool Name: SP_DATA_1
Storage Pool ID: 3
Relocation Start Time: N/A
Relocation Stop Time: N/A
Relocation Status: Inactive
Relocation Type: N/A
Relocation Rate: N/A
Data to Move Up (GBs): 0.00
Data to Move Down (GBs): 0.00
Data Movement Completed (GBs): N/A
Estimated Time to Complete: N/A
Schedule Duration Remaining: N/A

Storage Pool Name: SP_TEST_10
Storage Pool ID: 2
Relocation Start Time: N/A
Relocation Stop Time: N/A
Relocation Status: Inactive
Relocation Type: N/A
Relocation Rate: N/A
Data to Move Up (GBs): 0.00
Data to Move Down (GBs): 0.00
Data Movement Completed (GBs): N/A
Estimated Time to Complete: N/A
Schedule Duration Remaining: N/A

And next time you’re looking at a pool with tiers that are full, think about what you can do to alleviate the issue, and think about why you’ve automatically ignored the best practices guide.

EMC – DIY Heatmaps – Updated Version

Mat has updated the DIY Heatmaps for EMC CLARiiON and VNX arrays to version 3.019. You can get it from the Utilities page here. Any and all feedback welcome.

Latest fixes:

· Search Path environment variable for naviseccli

· Search common install locations for naviseccli

· Improve cross browser support – tested on IE, Chrome and FireFox

· Improve debug details – add module version reporting

· Fix divide by zero bug in rendering routine

Updated Articles page

A few months ago someone asked me if I had documentation on how to do FLARE upgrades on a CLARiiON. I’d taken a video last year, but realised that it used the old Navisphere Service Taskbar and covered the upgrade of a CX700 to FLARE 26. So, basically, my doco was a little out of date.

I recently had the opportunity to upgrade some of our CX4-120s to the latest release of FLARE 30 (.524), so thought it might be an opportune moment to document the process in a visual sense. Once I’d completed the articles, I realised this may have been done better with a series of videos. Maybe next time. In any case, here’s a four-part series (part 1, part 2, part 3, and part 4) on how to upgrade FLARE on a CX4 using Unisphere Service Manager. It’s a classic death-by-screenshot scenario, and I apologise in advance for the size of the files. While we’re talking about documentation, have a look through the rest of the articles page, there might be something useful there. And if you want something covered specifically, I do take requests.

EMC – DIY Heatmaps – Updated Version

Mat has updated the DIY Heatmaps for EMC CLARiiON and VNX arrays to version 3.018. You can get it from the Utilities page here. Any and all feedback welcome.

Latest fixes:

## 0.3.016 Add options to add array name and SP name to output file

## Fix –display_drive_type so that it displays empty drive slots as white, Removed / Failed drives as gray and unknown as green

## Add attributes to display total array and bus IOPS and Bandwidth

## Add –display_actual option to view actual IO stats

## Add read and write attributes for SP IOPS and bandwidth metrics

## Add –time_zone option

## Add the time zone to the heatmap output

## Add LUN bandwidth total, read & write and LUN IOPS total, read & write attributes

## Fix display problem when all of trays have their last disks configured as hotspares or not in use

## Add 2TB drive size

## 0.3.017 Change display options to allow controll of how many Disk, LUN and SP heatmaps per column

## Add –disk_maps, –lun_maps and –sp_maps

## 0.3.018 Add –debug option to print detailed debug information

EMC – DIY Heatmaps – Updated Version

Mat has updated the DIY Heatmaps for EMC CLARiiON and VNX arrays to version 3.015. You can get it from the Utilities page here. Any and all feedback welcome.

EMC – Broken Vault drive munts FAST Cache

Mat sent me an e-mail this morning, asking “why would FAST Cache be degraded after losing B0 E0 D2 in one of the CX4-960s?”. For those of you playing at home 0_0_2 is one of the Vault disks in the CX4 and VNX. Here’s a picture of the error:

Check out the 0x7576 that pops up shortly after the array says there’s a faulted disk. Here’s a closeup of the error:

Weird, huh?  So here’s the output of the naviseccli command that will give you the same information, but with a text-only feel.

"c:/Program Files/EMC/Navisphere CLI/NaviSECCli.exe"  -user Ebert -scope 0 -password xx -h 255.255.255.255  cache -fast -info -disks -status
Disks:
Bus 0 Enclosure 7 Disk 0
Bus 2 Enclosure 7 Disk 0
Bus 0 Enclosure 7 Disk 1
Bus 2 Enclosure 7 Disk 1
Bus 1 Enclosure 7 Disk 1
Bus 1 Enclosure 7 Disk 0
Bus 3 Enclosure 7 Disk 1
Bus 3 Enclosure 7 Disk 0
Mode:  Read/Write
Raid Type:  r_1
Size (GB):  366
State:  Enabled_Degraded
Current Operation:  N/A
Current Operation Status:  N/A
Current Operation Percent Completed:  N/A

So what’s with the degraded cache? The reason for this is that FAST Cache stores a small database on the first 3 drives (0_0_0, 0_0_1, 0_0_2). if any of these disks fail, FAST Cache flushes to disk and goes into a degraded state. But it shouldn’t, because the database is triple-mirrored. And what does it mean exactly? It means your FAST Cache is not processing writes at the moment. Which is considered “bad darts”.

This is a bug. Have a look on Powerlink for emc267579. Hopefully this will be fixed in R32 for the VNX. I couldn’t see details about the CX4 though. I strongly recommend that if you’re a CX4 user and you experience this issue, you raise a service request with your local EMC support mechanisms as soon as possible. The only way they get to know the severity of a problem is if people in the field feedback issues.