EMC – Next-Generation VNX – MCx

Posted on 2013-09-04 by dan

EMC have also breathlessly announced the introduction of “Multi-Core Everything” or MCx as an Operating Environment replacement for FLARE. I thought I’d spend a little time going through what that means, based on information I’ve been provided by EMC.

MCx is a redesign of a number of components, providing functionality and performance improvements for:

Multi-Core Cache (MCC);
Multi-Core RAID (MCR);
Multi-Core FAST Cache (MCF); and
Active / Active data access.

As I mentioned in my introductory post, the OE has been redesigned to spread the operations across each core – providing linear scaling across cores. Given that FLARE pre-dated multi-core x86 CPUs, it seems like this has been a reasonable thing to implement.

EMC are also suggesting that this has enabled them to scale out within the dual-node architecture, providing increased performance, at scale. This is a response to a number of competitors who have suggested that the way to scale out in the mid-range is to increase the number of nodes. How well this scales in real-world scenarios remains to be seen.

Multi-Core Cache

With MCC, the cache engine has been modularized to take advantage of all the cores available in the system. There is now also no requirement for manually separate space for Read and Write Cache, meaning no management overhead in
ensuring the cache is working in the most effective way regardless of the IO mix. Interestingly, cache is dynamically assigned, allocating space on-the-fly for reads, writes, thin metadata, etc depending on the needs of the system at the time.

The larger overall working cache provides mutual benefit between reads and writes. Note also that data is not discarded from cache after a write destage (this greatly improves cache hits). The new caching model employs intelligent monitoring of pace of disk writes to avoid forced flushing and works on a standardized 8KB physical page size.

Multi-Core RAID

MCR introduces some much-needed changes to the historically clumsy way in which disks were managed in the CLARiiON / VNX. Of particular note is the handling of hot spares. Instead of having to define disks as permanent hot spares, the sparing function is now more flexible and any unassigned drive in the system can operate as a hot spare.

There are three policies for hot spares:

Recommended;
No hot spares; and
Custom.

The “Recommended” policy implements the same sparing model as today (1 spare per 30 drives). Note, however, that if there is an unused drive that can be used as a spare (even if you have the “No Hot Spare” policy set) and there is a fault in a RAID group, the system will use that drive. Permanent sparing also means that when a drive has been used in a sparing function, that drive is then a permanent part of the RAID Group or Pool so there is no need to re-balance and copy back the spare drive to a new drive. The cool thing about this is that it reduces the exposure and performance overhead of sparing operations. If the concept of storage pools that spread all over the place didn’t freak out the RAID Group huggers in your storage team, then the idea of spares becoming permanent might just do it. There will be a CLI command available to copy the spare back to the replaced drive, so don’t freak out too much.

What I like the look of, having been stuck “optimising” CLARiiONs previously, is the idea of “Portable drives”, where drives can be physically removed from a RAID group and relocated to another disk shelf. Please note that you can’t use this feature to migrate RAID Groups to other VNX systems, you’ll need to use more traditional migration methods to achieve this.

Multi-Core FAST Cache

According to EMC, MCF is all about increased efficiency and performance. They’ve done this by moving MCC above the FAST Cache driver – all DRAM cache hits are processed without the need to check whether a block resides in FAST Cache, saving this CPU cycle overhead on all IOs. There’s also a faster initial warm-up for FAST Cache, resulting in better initial performance. Finally, instead of requiring 3 hits, if the FAST Cache is not full, it will perform more like a standard extension of cache and load data based on a single hit. Once the Cache is 80% full it reverts to the default 3-hit promotion policy.

Symmetric Active / Active

Real symmetric? Not that busted-arse ALUA? Sounds good to me. With Symmetric Active/Active, EMC are delivering true concurrent access via both SPs (no trespassing required – hooray!). However, this is currently only supported for Classic LUNs. It does have benefits for Pool LUNs by removing the performance impacts of trespassing the allocation owner. Basically, there’s a stripe locking service in lpay that’s providing access to the LUN for each SP, with the CMI being used for communication between the SPs. Here’s what LUN Parallel Access looks like.

And that’s about it. Make no mistake, MCx sounds like it should really improve performance for EMC’s mid-range. Remember though, given some of the fundamental changes outlined here, it’s unlikely that this will work on previous-generation VNX models.

EMC – Next-Generation VNX – The Speed to Lead

Posted on 2013-09-04 by dan

Otherwise titled, “about time”. I heard about this a few months ago, and have had a number of technical briefings from people inside EMC since then. For a number of reasons EMC weren’t able to publicly announce it until now. Anyway, for the official scoop, head on over to EMC’s Speed to Lead site. In this post I thought I’d cover off on some of the high-level speeds and feeds. I hope to have a some time in the near future to dive in a little deeper on some of the more interesting architectural changes.

As far as the hardware goes, EMC have refreshed the VNX midrange line with the following models:

The 5200 is positioned just above the 5100, the 5400 above the 5300, and so forth. The VNX8000 is the biggest yet, and, while initially shipping with 1000 drives, will eventually ship with a 1500-spindle capability. The SPs all use Sandy Bridge chips, with EMC heavily leveraging multi-core. The 8000 will sport dual-socket, 8-core SPs and 128GB of RAM. You’ll also find the sizing of these arrays is based on 2.5″ drives, and that a number of 2.5″ and 3.5″ drives will be available with these models (highlights being 4TB 3.5″ NL-SAS and 1.2TB 2.5″ SAS drives). Here’s a better picture with the max specs for each model.

Multi-core is a big part of the new VNX range, with MCx described as a project that redesigns the core Block OE stack to improve performance, reliability and longevity.

Instead of FLARE using one core per component, MCx is able to evenly distribute workloads across cores, giving improved utilisation across the SP.

The key benefit of this architecture is that you’ll get improved performance with the other tech in the array, namely FAST VP and Deduplication. The multi-core optimisations also extend to RAID management, Cache management and FAST Cache. I hope to be able to do some more detailed posts on these.

FAST-VP has also received some attention, with the chunk size being reduced from 1GB down to 256MB. The big news, however, is the introduction of fixed-block deduplication, with data being segmented in 8KB chunks, deduped at the pool level (not across pools), and turned on or off at the LUN level. I’ll be doing a post shortly on how it all works.

Hopefully that’s given you a starting point from which to investigate this announcement further. As always, if you’re after more information, talk to your local EMC people.

EMC – My dog ate my SPS but I still need to use write cache

Posted on 2013-08-28 by dan

Someone contacted me via my About page a little while ago to ask how to use write cache on their CX without having an SPS. It’s worth noting that doing this outside of a lab environment is a really bad idea, as the point of the SPS is that you’ll have sufficient time to flush cache to disk prior to losing power to the array. But anyway, in case you missed it, here’s what to do.

Use EMCRemote to access the console of the SP. If you’re using a CX4, you can get some instructions on how to do that here. Fire up a command prompt, then run flarecons d f a (to access SP A). Then run setcache -nosps (to override the SPS state). You should then be able to enable write cache.

EMC – CX POST output

Posted on 2013-07-31 by dan

Following on from my two previous posts on the CLARiiON POST and troubleshooting – here’s what a CX POST looks like when viewed from SP B.

Note that the chassis and disk WWN seeds match – this is useful if you’ve moved multiple arrays (during a data centre move, for example) and didn’t take note of the SP and DPE pairings (tsk, tsk). Note also that it’s looking at disks 1 and 3 – these are the boot disks for SP B (SP A uses disks 0 and 2).