Disclaimer: I recently attended the Dell EqualLogic MasterClass 2012.  This was a free event delivered by Dell and I was provided with breakfast and lunch, however there was no requirement for me to blog about any of the content presented and I was not compensated in any way for my time at the event. 

This is the first of three posts covering the Dell EqualLogic MasterClass that I recently attended. This was a free training event and, while I haven’t deployed or used EQL gear since 2009, I thought it would be worthwhile going along to refresh my memory and see what’s new.

EqualLogic MasterClass 101: Core Technology

Dell described this as “an entry-level storage course designed for users relatively new to the Dell EqualLogic series of arrays. You will learn what hardware and software features are available with the PS Series Storage Array. In this session we will cover:

• EqualLogic page-based architecture
• EqualLogic load-balancing & host MPIO
• Discuss manual tiering techniques
• Automatic tiering with firmware 5.1
• Group Manager snapshots
• Auto-replication.”

The presenter for all three sessions in Brisbane was Steven Lee, an Enterprise Technologist who’s been with Dell Australia for some time. Here’re my notes in a fairly, er, rough format.

iSCSI Basics

It’s a client / server deployment, using traditional tcp/ip to send SCSI commands. The host is the initiator, the volume is the target, and we use access controls to control what can and can’t see the storage. The iSCSI target portal (in EQL terms this is referred to as the group IP address) masks physical ethernet and mac addresses behind a “Virtual” SAN. iSCSI works best when the MTU is set to 9000 (AKA Jumbo frames). You need it end-to-end or bad things happen.

EQL Basics

The EQL Group uses a Virtual IP Address (or VIP). The EQL Resources are Pools and Members. A Pool contains 1 or more members (EQL iSCSI arrays). You can have up to 4 pools of storage and 8 members per pool. The maximum configuration is 16 members per Group. When you have 1 member in a pool, adding another member to that pool causes the load to be spread across the pool (auto load balanced across the pool – wide-striped, not copied). You can move LUNs from one pool to another, and load balancing does not happen across pools, only within.

EQL is known as scale-out, not scale-up. Every EQL array has drives, controllers, cache and software. You can aggregate performance as you add to the environment. Midrange arrays are traditionally constrained by disk performance (eg 2+2 R 1/0 disk set). When you add another member to EQL Group you get everything additional for added performance. Arguably, with some vendors you can still do this by using MetaLUNs, etc. The point he’s trying to make is that controller performance is also increased. I think maybe a better argument could also be made for improved front-end performance. When pressed he seemed to be comparing EQL scale out with traditional mid-range scale-up via concatenation – I think this message could be improved. But hey, we’re sitting in a Dell course.

Audience question: how does wide-striping work with different capacities? EQL wide-stripes over different RAID types and different disk types. This makes pool design critical for optimal performance.

Portfolio overview

Covered the PS61x0, PS65x0 and PS4000 (now available in 1 or 10Gb options). The PS4000 series can only have 2 per Group (this is a software defined limitation). You can, however, add more PS6000-class arrays to fully populate the Group. Once you’ve added in a PS6000 series into the Group, the limitations on replicas, connections, etc are taken away. As for the maximum number of Group iSCSI connections – the tested limits are 4096 (1024 per Pool). The EQL Storage Blade or PS4110 (put four in a M1000E chassis)  has 10Gb connectivity but still has the aforementioned PS4000 series software limitations. It introduces hybrid for this class (Mix of SAS and SSD drives). Hybrid arrays have a single fixed RAID type.

RAID and Firmware

The EQL arrays, generally, support RAID 1/0, 5, 5/0, 6, and 6acc (accelerated). Note also that there is a single RAID type per member. R5/0 is recommended for a balance of capacity and write performance (as it outperforms R6). R6 is recommended for large capacity drives. The number of hot spares configured is dependent on the member type, RAID type, etc. FW 5.25 and 6.01 are recommended firmware levels at the moment. HD FW upgrade KD0A is now available to address some issues with Seagate drive reliability observed in the field in the last 12 months. The FW upgrade should be done during a planned maintenance window as it can have a performance impact. The upgrade is, however, non-disruptive. This firmware is EQL-specific, not for use with the PowerVault range of disk enclosures (someone did ask). As of FW6 R5 is not an option in the Remote Setup Wizard. While it remains available through the CLI, it is no longer recommended by EQL support. No on-line way to move from R5 to R6 currently available. R6acc is only available on hybrid arrays. R6 across the array, with files move back and forth between the two tiers based on access frequency. It uses SSDs as cache, with up to 20GB reserved from the SSDs as “spillover” cache. Hybrid is the only one that does “smart swapping” of data within a member, load balancing is normally done across multiple members.

Host <-> EQL connectivity

Access Control can be performed three ways: using IP / IP range (wildcards), CHAP, or a specific IQN. Host connectivity via iSCSI doesn’t always use the shortest path, so use stacked switches with decent ISLs (on a single, flat subnet). Windows, Linux and VMware are able to perform multipathing on a single iSCSI Target Portal. XenServer no likey this – it wants to see 2 subnets. The Host Integration Toolkit (available for Microsoft, Linux, and a “broader suite” for VMware) provides the host with access to the group ToC – an understanding of data that sits in the back-end. This influences whether the host will make more connections to get to the member it wants to get to. Audience uestion – Is there a limit to the number of SANs a volume will stripe across? No fundamental limit, however the algorithm is optimised for 3 members. This can’t be tuned. Audience question – what happens when you hit the connection limit on a pool? Will it arbitrarily drop host connections? No. But it will start scaling down existing connections to one per host. If that’s still a problem – you may need to rethink what you’re trying to achieve and how you’re trying to achieve it. You can reduce the used iSCSI connection count by going from 1Gb to 10Gb. The default multipathing policy for all hosts by default is 2 sessions – this is a configurable parameter. Change this via the HIT – Remote Setup Wizard – Multi-pathing Component. The default is 6 sessions per Volume, 2 sessions per slice (assuming 3 members in the pool). Dell are looking to improve the connection numbers, but this is not on a roadmap yet. With HIT installed, you can take advantage of not only the array ToC, but also a multipathing policy that uses least queue depth. Multipathing routes traffic to the member with the slice you need, reducing the overall latency on the connections.

Load Balancers

For a good primer on the load balancers used with EQL, check out TR1070 – EqualLogic PS Series Architecture: Load Balancers. There are three load balancers – Capacity, Network, Performance. The Performance LB was introduced in FW5.1. The Capacity LB allocates slices across volumes and ensures that the free space headroom across all members in a pool is roughly the same %. Example using 2TB SSD and 8TB SAS arrays. Using 5TB (50%), LB puts 1TB on SSD, and 4TB on SAS member. What if you have 2TB of SSD and 48TB of NL-SAS? Still using 5TB (10%), 200GB goes to SSD, and 4.8TB goes to NL-SAS. The Network LB works by ensuring traffic in the back-end network is evenly distributed. Looks at port “busyness” and tells the host to connect to another port if the port is too busy. This can be a problem when accessing single-port 10Gb PS6110 arrays. While it’s a trade-off (losing the extra port), the 10Gb ports are rated at “line-speed”. NLB runs every 6 minutes in the background and just checks on port busyness, executing changes (if required) after the port check is done. Automatic Performance LB (APLB) used to focus on RAID (pre FW5.1), now looks at the overall latency of all members in a pool as well. Now also looks at the sub-volume, not just the whole volume. As a result of this, it takes minutes now to move stuff around rather than weeks and days. In the “olden days” (pre 5.1), for example, a SATA array running RAID 1/0 would be treated as faster than a SAS array running RAID 5/0. The goal now, however, is to ensure that the latency across the pool is balanced. This is done with hot-cold page swapping. Pages are 15MB (?) in size. An assessment and moves are made every 2 minutes. This only happens across members that are hosting a portion (pages) belonging to the volume. Members that don’t host the volume do not participate in swapping for that volume. Empty, pre-allocated pages can be swapped as well. Back-end page swapping can impact bandwidth on switches as well. For every active iSCSI port, you should have a 1-to-1 relationship. For example, if you have 3 PS6000s with 12Gb back-end port bandwidth, you should have a 12Gb ISL between your switches. You can get away with 75%. This is critical on 1Gb, not as much on 10Gb infrastructure.

Host Integration Tools

Steven then briefly covered the HIT and talked about the VMware plugin for vCenter, MEM (Multipathing Extension Module), SRM SRA. He also covered the various snapshot tools: ASM/ME ASM/LE VSM/VE.

A member of the audience asked a question about migrating EQL storage from physical hosts to virtual guests, and then we took a 15 minute break.