QNAP – Add Cache With SSDs

I recently added some cache to my QNAP NAS at home and thought I’d document the process here. It’s pretty straightforward, but there are nonetheless some things you need to take into consideration. Also, this is more about me satisfying my curiosity. I’m not suggesting you’ll real benefits (particularly the way I’m using it).

 

The Hardware

Firstly, my NAS is a TS-831X model, with 8 drive bays and the ability to add in a small form-factor PCIe card. As all of the bays are already occupied with spinning disks, I opted to use the PCIe solution. To that end, I purchased a QM2-2S-220A Dual M.2 22110/2280 SATA SSD expansion card from a local supplier. It’s relatively inexpensive, and you get some choice in terms of the storage you install. It also comes with low-profile and normal baseplates as well, so you could load this in a PC if you’re into that sort of thing. Since I’ve been using Apple gear for a while, I’m a little rusty with the screwdriver, but installation was straightforward. Remove the heatsink and fan, add some thermal pads to your M.2 drives, and you’re good to go.

The card slots into the single PCIe slot in the NAS.

You’ll notice I went with the cheapest 1TB M.2 drives I could find on Amazon. Time will tell (probably sooner rather than later) whether that was really a wise choice.

 

The Software

Once the NAS has booted up again (and hopefully with no errors), you’re good to start the software configuration. The QNAP documentation covering the process can be found here. While I’m running version 5.2 of QTS, note that other versions may look a little different. There are some things to consider with regard to the amount of RAM you have installed, the version of software you’re running, and the amount of cache you have installed.

To get started, click on Storage & Snapshots.

You’ll see that I have the main enclosure (with 8 drives) and a USB-C enclosure with 4 drives. I also now have two SSDs installed on the QM2 card.

Click on Cache Acceleration to get started. Then click on the big + sign.

You’re then presented with a wizard that will guide you through the configuration process.

The first thing you’ll need to do is select the disks you want to use for your cache. Note that you can’t select any disks that are already in use.

Once you’ve selected the two disks you want to use, you’ll see that the RAID Type changes to RAID 0. You can use this option if you’re only going to configure read cache.

In this example, however, I’m going to configure the cache as read-write.

As I’m configuring the cache as read-write, I need to set the RAID Type to RAID 1, as I don’t want to lose writes if one of my cache devices fails. Once that’s done, click on Next to continue.

On this screen you can configure SSD Over-provisioning, Cache Mode, and Advanced Settings. If you’re confused about the percentage you should set for over-provisioning, there is a tool available in QTS that you can use (the SSD Profiling Tool). Or you can just click click next like I did.

In this example, I’m changing the Cache Mode to All I/O, as this NAS is used primarily for streaming media. I’ll talk about whether it really makes a difference further down.

Under Advanced Settings, you can configure the Bypass Block Size setting (this only comes into play if your Cache Mode is set to Random I/O) and the Cache replacement policy. Click on Next to continue.

You can now select which volume can use the SSD cache. In this example, even though I have two volumes configured, only the internal one can leverage the cache as it’s configured as read-write. If I wanted to use cache with my external volume, I’d need to set the cache to read-only. Click on Next to continue.

You’ll then be presented with a summary of your configuration options. If you’re happy with how it looks, click on Create.

You’ll then be presented with some warnings regarding the cache creation and removal considerations.

Click on I Understand and OK to proceed.

Once you’ve finished you’ll see that the Cache is set to Initializing. It doesn’t take a terribly long time to complete.

And now you’re done and ready to enjoy that cache goodness.

 

Thoughts

Adding a Terabyte of SSD Cache to my home NAS was a fairly simple process, and relatively inexpensive to boot. I haven’t had much chance to test the performance yet, but I have noticed that the UI in QTS is reporting slightly higher CPU usage than I’d seen previously. While the process is easy, I’m not entirely convinced that I’ll see any real benefits, given that I’m serving up video files over SMB to a Plex server using a gigabit network. Maybe if I had a 10GbE running in the house I’d see it. And the workload has a bit to do with it too. If I was running Virtual Machines on this machine I think I’d see something good from a Random I/O profile, particularly with data coming off the SATA drives. Why didn’t I use Qtier? Because I have static volumes configured, not storage pools. And SSD Caching seemed the easiest thing to do without tearing it all down and starting from scratch. In any case, it was a fun experiment, and I’ll be curious to see whether I notice any real improvements and how long the drives last with normal use.

FreeNAS – Using one SSD for ZIL and L2ARC

Following on from my previous musings on FreeNAS, I thought I’d do a quick howto post on using one SSD for both ZIL and L2ARC. This has been covered here and here, but I found myself missing a few steps, so I thought I’d cover it here for my own benefit if nothing else. Before I start though, you should really consider using two drives, particularly with ZIL. And before you start, you’ve obviously already gone through the exercise of understanding your workload, and you’ve thought about the ramifications of what you’re doing, right? Because using one drive isn’t necessarily recommended …

So, let’s get started by connecting to your NAS via SSH are some other console mechanism.

Last login: Sun Dec 13 10:25:29 on console
dans-MacBook-Pro:~ dan$ ssh dan@freenas1

dan@freenas1's password: 
Last login: Wed Dec 30 06:57:25 2015 from 192.168.0.100
FreeBSD 9.3-RELEASE-p28 (FREENAS.amd64) #0 r288272+f229c79: Sat Dec 12 11:58:01 PST 2015

FreeNAS (c) 2009-2015, The FreeNAS Development Team
All rights reserved.
FreeNAS is released under the modified BSD license.

For more information, documentation, help or support, go here:
  http://freenas.org

Welcome to FreeNAS

You then need to find your SSD.

dan@freenas1:~ % sudo camcontrol devlist
Password:
<INTEL SSDSC2CT060A3 300i>         at scbus0 target 0 lun 0 (ada0,pass0)
<ST32000644NS 130C>                at scbus1 target 0 lun 0 (ada1,pass1)
<ST32000644NS 130C>                at scbus2 target 0 lun 0 (ada2,pass2)
<ST32000644NS 130C>                at scbus3 target 0 lun 0 (ada3,pass3)
<ST32000644NS 130C>                at scbus4 target 0 lun 0 (ada4,pass4)
<ST32000644NS 130C>                at scbus5 target 0 lun 0 (ada5,pass5)
<SanDisk Cruzer Force 1.26>        at scbus7 target 0 lun 0 (pass6,da0)

I then wanted to look at ada0, this being the SSD.

dan@freenas1:~ % sudo gpart show ada0
gpart: No such geom: ada0.

No dice. But it’s there, isn’t it?

dan@freenas1:~ % zpool list -v
NAME                                     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freenas-boot                            14.9G   533M  14.4G         -      -     3%  1.00x  ONLINE  -
  gptid/57344c8a-ae4d-11e5-b98c-bc5ff42c6cb2  14.9G   533M  14.4G         -      -     3%
volume0                                 9.06T  2.99M  9.06T         -     0%     0%  1.00x  ONLINE  /mnt
  raidz1                                9.06T  2.99M  9.06T         -     0%     0%
    gptid/6b604f11-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6c25f30c-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6cf26f5b-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6dc0508f-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6e88fc7a-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
dan@freenas1:~ % geom disk list
Geom name: da0
Providers:
1. Name: da0
   Mediasize: 16008609792 (14G)
   Sectorsize: 512
   Mode: r1w1e3
   descr: SanDisk Cruzer Force
   lunname: SanDisk Cruzer Force    4C532000050815116541
   lunid: SanDisk Cruzer Force    4C532000050815116541
   ident: 4C532000050815116541
   fwsectors: 63
   fwheads: 255
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 60022480896 (55G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: INTEL SSDSC2CT060A3
   lunid: 5001517bb28ade6a
   ident: CVMP213600AQ060AGN
   fwsectors: 63
   fwheads: 16
Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Mode: r2w2e5
   descr: ST32000644NS
   lunid: 5000c5004027e7b5
   ident: 9WM88536
   fwsectors: 63
   fwheads: 16
Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Mode: r2w2e5
   descr: ST32000644NS
   lunid: 5000c50040276a46
   ident: 9WM88DV9
   fwsectors: 63
   fwheads: 16

[snip]

Ok, so I can see it, but I can’t. I mucked about a bit, and came up with this approach.

dan@freenas1:~ % sudo gpart create -s gpt ada0
ada0 created
dan@freenas1:~ % sudo gpart add -a 4k -b 128 -t freebsd-zfs -s 10G ada0
ada0p1 added
dan@freenas1:~ % sudo gpart add -a 4k -t freebsd-zfs ada0
ada0p2 added

Which worked well. Now I need to get the UUIDs.

dan@freenas1:~ % sudo gpart list
Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 31266782
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 524288 (512k)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   rawuuid: 572c8954-ae4d-11e5-b98c-bc5ff42c6cb2
   rawtype: 21686148-6449-6e6f-744e-656564454649
   label: 1
   length: 524288
   offset: 17408
   type: bios-boot
   index: 1
   end: 1057
   start: 34
2. Name: da0p2
   Mediasize: 16008044544 (14G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 544768
   Mode: r1w1e2
   rawuuid: 57344c8a-ae4d-11e5-b98c-bc5ff42c6cb2
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: 1
   length: 16008044544
   offset: 544768
   type: freebsd-zfs
   index: 2
   end: 31266775
   start: 1064
Consumers:
1. Name: da0
   Mediasize: 16008609792 (14G)
   Sectorsize: 512
   Mode: r1w1e3

[snip]

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 117231374
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 10737418240 (10G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: 94a4bd28-aeb7-11e5-99ac-bc5ff42c6cb2
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: 1
   length: 10737418240
   offset: 65536
   type: freebsd-zfs
   index: 1
   end: 20971647
   start: 128
2. Name: ada0p2
   Mediasize: 49284976640 (45G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: 9a79622f-aeb7-11e5-99ac-bc5ff42c6cb2
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: 1
   length: 49284976640
   offset: 10737483776
   type: freebsd-zfs
   index: 2
   end: 117231367
   start: 20971648
Consumers:
1. Name: ada0
   Mediasize: 60022480896 (55G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

Let’s double-check.

dan@freenas1:~ % sudo gpart show ada0
=>       34  117231341  ada0  GPT  (55G)
         34         94        - free -  (47k)
        128   20971520     1  freebsd-zfs  (10G)
   20971648   96259720     2  freebsd-zfs  (45G)
  117231368          7        - free -  (3.5k)

Everything looks copacetic. Let’s add them to the zpool. ZIL is log and L2ARC is cache. My pool is volume0 and I’m using the rawuuid value.

dan@freenas1:~ % sudo zpool add volume0 log gptid/94a4bd28-aeb7-11e5-99ac-bc5ff42c6cb2
dan@freenas1:~ % sudo zpool add volume0 cache gptid/9a79622f-aeb7-11e5-99ac-bc5ff42c6cb2

And here it is, all done.

dan@freenas1:~ % zpool list -v
NAME                                     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freenas-boot                            14.9G   533M  14.4G         -      -     3%  1.00x  ONLINE  -
  gptid/57344c8a-ae4d-11e5-b98c-bc5ff42c6cb2  14.9G   533M  14.4G         -      -     3%
volume0                                 9.06T  3.02M  9.06T         -     0%     0%  1.00x  ONLINE  /mnt
  raidz1                                9.06T  3.02M  9.06T         -     0%     0%
    gptid/6b604f11-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6c25f30c-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6cf26f5b-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6dc0508f-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6e88fc7a-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
  gptid/94a4bd28-aeb7-11e5-99ac-bc5ff42c6cb2  9.94G   132K  9.94G         -     0%     0%
cache                                       -      -      -      -      -      -
  gptid/9a79622f-aeb7-11e5-99ac-bc5ff42c6cb2  45.9G   190K  45.9G         -     0%     0%
dan@freenas1:~ %

 

SanDisk Announces FlashSoft 4.0 for vSphere 6

SanDisk recently announced their next generation of FlashSoft software for vSphere. I was fortunate enough to receive a briefing from Rich Petersen and thought I’d share a few thoughts.

Firstly, you can check out FlashSoft 4 for vSphere 6 here. Here’s a post from Rich on the subject too. While you may be familiar with SanDisk via flash cards and USB sticks, they’ve also been doing a fair bit with host-side caching software in recent years.
There’re a few reasons you might choose to use host-side caching via SSDs in your host:

  • Improved VM density
  • It’s an easy way to add improved performance without necessarily upgrading your SAN

Some of the benefits of FlashSoft include

  • IO removed from the SAN (out of the kernel)
  • Write-back caching

VAIO

Last VMworld, VMware announced vSphere APIs for IO filtering (VAIO). Chris has a nice write-up on it here.

  • Uses “IO Filters” for integrating storage services
  • Caching and Replication first, others coming in the future
  • VMware Storage Policy-Based Management (SPBM)
  • Services are installed, managed and maintained as “native components”
  • vCenter is “control point” for SDS with 3rd-party storage services
  • Data services for storage functions running at the host level
  • Third Party Solutions can be certified VMware Ready

 

VAIO

 

FlashSoft 4.0

SanDisk are positioning this release of FlashSoft as a successor to vFRC (a product that I think highly of but haven’t seen much in the field). So what does this release offer?

  • Write-back as well as write-through
  • VVOLs, VMFS, NFS, VSAN
  • VMware-native data service
  • Installed, managed, maintained by vCenter
  • Integrated through SPBM (storage policy-based management)
  • Certified VMware ready

Some of the benefits of this release include performance, stability, ease of administration, versatility and resilience.
SanDisk pointed out that, while vFRC will be still be supported, it is limited in the sense that cache resources were allocated on a per-VM basis (static not dynamic), and it was only available in the Enterprise Plus edition. Note that FlashSoft is supported in all vSphere versions except Essentials, as it does not support VAIO.

Interestingly, FlashSoft uses the vMotion network to share data between cache pairs, and uses the same mechanism to provision SSDs (VFFS) as you did with vFRC.

FlashSoft_cache

Here’s how cache writes work with the write-back caching:

  1. Cache write hit from VM in server A
  2. Data replication to SSD on Server B begins
  3. Data is copied to replication space on SSD in server B
  4. Data is concurrently recorded on cache in server A
  5. Write acknowledged from server B
  6. Write acknowledged to VM in server A

So how do you buy it? SanDisk currently have agreements in place with Dell, Lenovo, HP and HDS for FlashSoft to be bought with a server. The VAIO APIs are in tech preview with the general availability projected for vSphere 6 U1. From a roadmap perspective, FlashSoft 4.0 will be the caching software for offered by SanDisk moving forward, offering existing IO-turbine customers the ability to upgrade.

 

Final Thoughts

I like the concept host-side caching a lot. I think it’s a clever way to leverage occasionally stretched resources without having to go the monolithic storage upgrade route. SanDisk are understandably excited about this announcement, particularly with the tight integration with vSphere 6. It certainly looks good on paper, and I’m looking forward to getting some stick time with the product when it’s available.

EMC – My dog ate my SPS but I still need to use write cache

Someone contacted me via my About page a little while ago to ask how to use write cache on their CX without having an SPS. It’s worth noting that doing this outside of a lab environment is a really bad idea, as the point of the SPS is that you’ll have sufficient time to flush cache to disk prior to losing power to the array. But anyway, in case you missed it, here’s what to do.

Use EMCRemote to access the console of the SP. If you’re using a CX4, you can get some instructions on how to do that here. Fire up a command prompt, then run flarecons d f a (to access SP A). Then run setcache -nosps (to override the SPS state). You should then be able to enable write cache.

EMC – naviseccli – Using chglun to modify LUN cache settings

Friday morning I had cause to “fiddle” with the LUN cache settings on 312 component LUNs across 2 arrays and needed to turn on read and write cache on each component LUN. Via Unisphere I would have to find the MetaLUN, right-click, select Show Components, expand the Component 0, right-click on the component, select Properties, select the Cache tab, tick the radio boxes for SP Read Cache and SP Write Cache, click OK, then re-expand the Component 0, rinse and repeat. On a few components this is tolerable, but for 312 not so much.

So I thought that maybe I could do something with naviseccli. You can change the cache properties of a LUN, amongst other things, using the chglun command.

chglun -l <LUN-ID> -c none | caching read | write | rw – where you can select no cache, read cache, write cache or read-write cache. So the command I used to enable rw cache on the LUN was

naviseccli -h 256.256.256.256 chglun -l 24047 -c rw

Note that if you have FAST Cache installed you can also use this command to set FAST Cache for a LUN using the -fastcache 0 | 1 switch.

My next issue was that, even though these components had normal LUN IDs like 3, 8 and so forth, their actual LUN ID when they were used as MetaLUN Components was changed to something else by Unisphere. So I ran a report for each array to list the configured MetaLUNs. I then opened the .xml file in Excel and copied the Component LUN IDs into a different column. Put some commands before and after the column and then saved it as a txt file that I could then run as a script. It took a little time to run, but a lot less time than it would have taken to change these settings by hand. If you are checking the resultas in Unisphere you should make sure that you’ve refreshed your screen, because it least in our environment it can take a while to show up.

One other thing to note is that, as the manual says, “The chglun command does not support thin LUNs”. Fair enough, so I can do chglun operations on Pool LUNs? Well, mostly. Here’s one we tried earlier:

c:\>naviseccli -h 256.256.256.256 chglun -l 130 -c rw
Error: chglun command failed
Switch not supported: -c
One or more of the specified switches is not supported for the input Pool lun.

c:\>

New Article – Configuring FAST Cache on the CLARiiON

I’ve added a brief article full of pretty screenshots regarding FAST Cache on the CX4. Nothing groundbreaking – because that’s not really my style. I hope to have some more posts shortly on our real world experiences with FAST and FAST Cache. Check out some other articles here.

EMC – FAST and FAST Cache on the CX4-960

Apologies for the lack of posts over the last few months – I have been nuts deep in work and holidays. I’m working on some literature around Storage Pools and FAST in general, but in the meantime I thought I’d share this nugget with you. We finally got approval to install the FAST and FAST Cache enablers on our production CX4-960s a few nights ago. We couldn’t install them on one of the arrays because we had a dead disk that prevented the NDU from going ahead. Fair enough. Two awesome things happened when we installed it on the other array. Both of which could have been avoided if I’d had my shit together. Firstly, when I got into the office the next morning at 8 am, we noticed that the Read Cache on the array was disabled. For those of you playing at home, we had the cache on the 960 set at 1000MB read and 9760MB for write. I think I read this in a whitepaper some where. But after FAST went on, we still had 9760MB allocated to Write, and 0MB available for Read. Awesome not so much. Seems that we lost 1000MB, presumably because we added another layered application. Funnily enough we didn’t observe this behaviour on our lab CX4-120s, although you could argue that they really have sweet FA of cache in the first place. So now we have 8760MB for Write, and 1000MB for Read. And I’m about to configure a few hundred GB of FAST Cache on the EFDs in any case. We’ll see how that goes.

The other slightly boneheaded thing we did was forget to trespass the LUN ownership of LUNs on SP A back from SP B. In other words, an NDU applies code to SP B first, reboots the SP, checks it, and then loads code on the other SP. As part of this, LUN ownership is temporarily trespassed to the surviving SP (this is the whole non-disruptive thing). Once the NDU is complete, you should go and check for trespassed LUNs and move them back to their owners. Or not, and have everything run on one SP for a while. And wait for about 9000 Exchange users to complain when one of the Exchange clusters goes off-line. Happy days.