FreeNAS – Using one SSD for ZIL and L2ARC

Following on from my previous musings on FreeNAS, I thought I’d do a quick howto post on using one SSD for both ZIL and L2ARC. This has been covered here and here, but I found myself missing a few steps, so I thought I’d cover it here for my own benefit if nothing else. Before I start though, you should really consider using two drives, particularly with ZIL. And before you start, you’ve obviously already gone through the exercise of understanding your workload, and you’ve thought about the ramifications of what you’re doing, right? Because using one drive isn’t necessarily recommended …

So, let’s get started by connecting to your NAS via SSH are some other console mechanism.

Last login: Sun Dec 13 10:25:29 on console
dans-MacBook-Pro:~ dan$ ssh dan@freenas1

dan@freenas1's password: 
Last login: Wed Dec 30 06:57:25 2015 from 192.168.0.100
FreeBSD 9.3-RELEASE-p28 (FREENAS.amd64) #0 r288272+f229c79: Sat Dec 12 11:58:01 PST 2015

FreeNAS (c) 2009-2015, The FreeNAS Development Team
All rights reserved.
FreeNAS is released under the modified BSD license.

For more information, documentation, help or support, go here:
  http://freenas.org

Welcome to FreeNAS

You then need to find your SSD.

dan@freenas1:~ % sudo camcontrol devlist
Password:
<INTEL SSDSC2CT060A3 300i>         at scbus0 target 0 lun 0 (ada0,pass0)
<ST32000644NS 130C>                at scbus1 target 0 lun 0 (ada1,pass1)
<ST32000644NS 130C>                at scbus2 target 0 lun 0 (ada2,pass2)
<ST32000644NS 130C>                at scbus3 target 0 lun 0 (ada3,pass3)
<ST32000644NS 130C>                at scbus4 target 0 lun 0 (ada4,pass4)
<ST32000644NS 130C>                at scbus5 target 0 lun 0 (ada5,pass5)
<SanDisk Cruzer Force 1.26>        at scbus7 target 0 lun 0 (pass6,da0)

I then wanted to look at ada0, this being the SSD.

dan@freenas1:~ % sudo gpart show ada0
gpart: No such geom: ada0.

No dice. But it’s there, isn’t it?

dan@freenas1:~ % zpool list -v
NAME                                     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freenas-boot                            14.9G   533M  14.4G         -      -     3%  1.00x  ONLINE  -
  gptid/57344c8a-ae4d-11e5-b98c-bc5ff42c6cb2  14.9G   533M  14.4G         -      -     3%
volume0                                 9.06T  2.99M  9.06T         -     0%     0%  1.00x  ONLINE  /mnt
  raidz1                                9.06T  2.99M  9.06T         -     0%     0%
    gptid/6b604f11-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6c25f30c-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6cf26f5b-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6dc0508f-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6e88fc7a-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
dan@freenas1:~ % geom disk list
Geom name: da0
Providers:
1. Name: da0
   Mediasize: 16008609792 (14G)
   Sectorsize: 512
   Mode: r1w1e3
   descr: SanDisk Cruzer Force
   lunname: SanDisk Cruzer Force    4C532000050815116541
   lunid: SanDisk Cruzer Force    4C532000050815116541
   ident: 4C532000050815116541
   fwsectors: 63
   fwheads: 255
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 60022480896 (55G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: INTEL SSDSC2CT060A3
   lunid: 5001517bb28ade6a
   ident: CVMP213600AQ060AGN
   fwsectors: 63
   fwheads: 16
Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Mode: r2w2e5
   descr: ST32000644NS
   lunid: 5000c5004027e7b5
   ident: 9WM88536
   fwsectors: 63
   fwheads: 16
Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Mode: r2w2e5
   descr: ST32000644NS
   lunid: 5000c50040276a46
   ident: 9WM88DV9
   fwsectors: 63
   fwheads: 16

[snip]

Ok, so I can see it, but I can’t. I mucked about a bit, and came up with this approach.

dan@freenas1:~ % sudo gpart create -s gpt ada0
ada0 created
dan@freenas1:~ % sudo gpart add -a 4k -b 128 -t freebsd-zfs -s 10G ada0
ada0p1 added
dan@freenas1:~ % sudo gpart add -a 4k -t freebsd-zfs ada0
ada0p2 added

Which worked well. Now I need to get the UUIDs.

dan@freenas1:~ % sudo gpart list
Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 31266782
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 524288 (512k)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   rawuuid: 572c8954-ae4d-11e5-b98c-bc5ff42c6cb2
   rawtype: 21686148-6449-6e6f-744e-656564454649
   label: 1
   length: 524288
   offset: 17408
   type: bios-boot
   index: 1
   end: 1057
   start: 34
2. Name: da0p2
   Mediasize: 16008044544 (14G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 544768
   Mode: r1w1e2
   rawuuid: 57344c8a-ae4d-11e5-b98c-bc5ff42c6cb2
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: 1
   length: 16008044544
   offset: 544768
   type: freebsd-zfs
   index: 2
   end: 31266775
   start: 1064
Consumers:
1. Name: da0
   Mediasize: 16008609792 (14G)
   Sectorsize: 512
   Mode: r1w1e3

[snip]

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 117231374
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 10737418240 (10G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: 94a4bd28-aeb7-11e5-99ac-bc5ff42c6cb2
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: 1
   length: 10737418240
   offset: 65536
   type: freebsd-zfs
   index: 1
   end: 20971647
   start: 128
2. Name: ada0p2
   Mediasize: 49284976640 (45G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: 9a79622f-aeb7-11e5-99ac-bc5ff42c6cb2
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: 1
   length: 49284976640
   offset: 10737483776
   type: freebsd-zfs
   index: 2
   end: 117231367
   start: 20971648
Consumers:
1. Name: ada0
   Mediasize: 60022480896 (55G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

Let’s double-check.

dan@freenas1:~ % sudo gpart show ada0
=>       34  117231341  ada0  GPT  (55G)
         34         94        - free -  (47k)
        128   20971520     1  freebsd-zfs  (10G)
   20971648   96259720     2  freebsd-zfs  (45G)
  117231368          7        - free -  (3.5k)

Everything looks copacetic. Let’s add them to the zpool. ZIL is log and L2ARC is cache. My pool is volume0 and I’m using the rawuuid value.

dan@freenas1:~ % sudo zpool add volume0 log gptid/94a4bd28-aeb7-11e5-99ac-bc5ff42c6cb2
dan@freenas1:~ % sudo zpool add volume0 cache gptid/9a79622f-aeb7-11e5-99ac-bc5ff42c6cb2

And here it is, all done.

dan@freenas1:~ % zpool list -v
NAME                                     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freenas-boot                            14.9G   533M  14.4G         -      -     3%  1.00x  ONLINE  -
  gptid/57344c8a-ae4d-11e5-b98c-bc5ff42c6cb2  14.9G   533M  14.4G         -      -     3%
volume0                                 9.06T  3.02M  9.06T         -     0%     0%  1.00x  ONLINE  /mnt
  raidz1                                9.06T  3.02M  9.06T         -     0%     0%
    gptid/6b604f11-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6c25f30c-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6cf26f5b-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6dc0508f-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
    gptid/6e88fc7a-aeb5-11e5-99ac-bc5ff42c6cb2      -      -      -         -      -      -
  gptid/94a4bd28-aeb7-11e5-99ac-bc5ff42c6cb2  9.94G   132K  9.94G         -     0%     0%
cache                                       -      -      -      -      -      -
  gptid/9a79622f-aeb7-11e5-99ac-bc5ff42c6cb2  45.9G   190K  45.9G         -     0%     0%
dan@freenas1:~ %

 

SanDisk Announces FlashSoft 4.0 for vSphere 6

SanDisk recently announced their next generation of FlashSoft software for vSphere. I was fortunate enough to receive a briefing from Rich Petersen and thought I’d share a few thoughts.

Firstly, you can check out FlashSoft 4 for vSphere 6 here. Here’s a post from Rich on the subject too. While you may be familiar with SanDisk via flash cards and USB sticks, they’ve also been doing a fair bit with host-side caching software in recent years.
There’re a few reasons you might choose to use host-side caching via SSDs in your host:

  • Improved VM density
  • It’s an easy way to add improved performance without necessarily upgrading your SAN

Some of the benefits of FlashSoft include

  • IO removed from the SAN (out of the kernel)
  • Write-back caching

VAIO

Last VMworld, VMware announced vSphere APIs for IO filtering (VAIO). Chris has a nice write-up on it here.

  • Uses “IO Filters” for integrating storage services
  • Caching and Replication first, others coming in the future
  • VMware Storage Policy-Based Management (SPBM)
  • Services are installed, managed and maintained as “native components”
  • vCenter is “control point” for SDS with 3rd-party storage services
  • Data services for storage functions running at the host level
  • Third Party Solutions can be certified VMware Ready

 

VAIO

 

FlashSoft 4.0

SanDisk are positioning this release of FlashSoft as a successor to vFRC (a product that I think highly of but haven’t seen much in the field). So what does this release offer?

  • Write-back as well as write-through
  • VVOLs, VMFS, NFS, VSAN
  • VMware-native data service
  • Installed, managed, maintained by vCenter
  • Integrated through SPBM (storage policy-based management)
  • Certified VMware ready

Some of the benefits of this release include performance, stability, ease of administration, versatility and resilience.
SanDisk pointed out that, while vFRC will be still be supported, it is limited in the sense that cache resources were allocated on a per-VM basis (static not dynamic), and it was only available in the Enterprise Plus edition. Note that FlashSoft is supported in all vSphere versions except Essentials, as it does not support VAIO.

Interestingly, FlashSoft uses the vMotion network to share data between cache pairs, and uses the same mechanism to provision SSDs (VFFS) as you did with vFRC.

FlashSoft_cache

Here’s how cache writes work with the write-back caching:

  1. Cache write hit from VM in server A
  2. Data replication to SSD on Server B begins
  3. Data is copied to replication space on SSD in server B
  4. Data is concurrently recorded on cache in server A
  5. Write acknowledged from server B
  6. Write acknowledged to VM in server A

So how do you buy it? SanDisk currently have agreements in place with Dell, Lenovo, HP and HDS for FlashSoft to be bought with a server. The VAIO APIs are in tech preview with the general availability projected for vSphere 6 U1. From a roadmap perspective, FlashSoft 4.0 will be the caching software for offered by SanDisk moving forward, offering existing IO-turbine customers the ability to upgrade.

 

Final Thoughts

I like the concept host-side caching a lot. I think it’s a clever way to leverage occasionally stretched resources without having to go the monolithic storage upgrade route. SanDisk are understandably excited about this announcement, particularly with the tight integration with vSphere 6. It certainly looks good on paper, and I’m looking forward to getting some stick time with the product when it’s available.

EMC – My dog ate my SPS but I still need to use write cache

Someone contacted me via my About page a little while ago to ask how to use write cache on their CX without having an SPS. It’s worth noting that doing this outside of a lab environment is a really bad idea, as the point of the SPS is that you’ll have sufficient time to flush cache to disk prior to losing power to the array. But anyway, in case you missed it, here’s what to do.

Use EMCRemote to access the console of the SP. If you’re using a CX4, you can get some instructions on how to do that here. Fire up a command prompt, then run flarecons d f a (to access SP A). Then run setcache -nosps (to override the SPS state). You should then be able to enable write cache.

EMC – naviseccli – Using chglun to modify LUN cache settings

Friday morning I had cause to “fiddle” with the LUN cache settings on 312 component LUNs across 2 arrays and needed to turn on read and write cache on each component LUN. Via Unisphere I would have to find the MetaLUN, right-click, select Show Components, expand the Component 0, right-click on the component, select Properties, select the Cache tab, tick the radio boxes for SP Read Cache and SP Write Cache, click OK, then re-expand the Component 0, rinse and repeat. On a few components this is tolerable, but for 312 not so much.

So I thought that maybe I could do something with naviseccli. You can change the cache properties of a LUN, amongst other things, using the chglun command.

chglun -l <LUN-ID> -c none | caching read | write | rw – where you can select no cache, read cache, write cache or read-write cache. So the command I used to enable rw cache on the LUN was

naviseccli -h 256.256.256.256 chglun -l 24047 -c rw

Note that if you have FAST Cache installed you can also use this command to set FAST Cache for a LUN using the -fastcache 0 | 1 switch.

My next issue was that, even though these components had normal LUN IDs like 3, 8 and so forth, their actual LUN ID when they were used as MetaLUN Components was changed to something else by Unisphere. So I ran a report for each array to list the configured MetaLUNs. I then opened the .xml file in Excel and copied the Component LUN IDs into a different column. Put some commands before and after the column and then saved it as a txt file that I could then run as a script. It took a little time to run, but a lot less time than it would have taken to change these settings by hand. If you are checking the resultas in Unisphere you should make sure that you’ve refreshed your screen, because it least in our environment it can take a while to show up.

One other thing to note is that, as the manual says, “The chglun command does not support thin LUNs”. Fair enough, so I can do chglun operations on Pool LUNs? Well, mostly. Here’s one we tried earlier:

c:\>naviseccli -h 256.256.256.256 chglun -l 130 -c rw
Error: chglun command failed
Switch not supported: -c
One or more of the specified switches is not supported for the input Pool lun.

c:\>

New Article – Configuring FAST Cache on the CLARiiON

I’ve added a brief article full of pretty screenshots regarding FAST Cache on the CX4. Nothing groundbreaking – because that’s not really my style. I hope to have some more posts shortly on our real world experiences with FAST and FAST Cache. Check out some other articles here.

EMC – FAST and FAST Cache on the CX4-960

Apologies for the lack of posts over the last few months – I have been nuts deep in work and holidays. I’m working on some literature around Storage Pools and FAST in general, but in the meantime I thought I’d share this nugget with you. We finally got approval to install the FAST and FAST Cache enablers on our production CX4-960s a few nights ago. We couldn’t install them on one of the arrays because we had a dead disk that prevented the NDU from going ahead. Fair enough. Two awesome things happened when we installed it on the other array. Both of which could have been avoided if I’d had my shit together. Firstly, when I got into the office the next morning at 8 am, we noticed that the Read Cache on the array was disabled. For those of you playing at home, we had the cache on the 960 set at 1000MB read and 9760MB for write. I think I read this in a whitepaper some where. But after FAST went on, we still had 9760MB allocated to Write, and 0MB available for Read. Awesome not so much. Seems that we lost 1000MB, presumably because we added another layered application. Funnily enough we didn’t observe this behaviour on our lab CX4-120s, although you could argue that they really have sweet FA of cache in the first place. So now we have 8760MB for Write, and 1000MB for Read. And I’m about to configure a few hundred GB of FAST Cache on the EFDs in any case. We’ll see how that goes.

The other slightly boneheaded thing we did was forget to trespass the LUN ownership of LUNs on SP A back from SP B. In other words, an NDU applies code to SP B first, reboots the SP, checks it, and then loads code on the other SP. As part of this, LUN ownership is temporarily trespassed to the surviving SP (this is the whole non-disruptive thing). Once the NDU is complete, you should go and check for trespassed LUNs and move them back to their owners. Or not, and have everything run on one SP for a while. And wait for about 9000 Exchange users to complain when one of the Exchange clusters goes off-line. Happy days.