Kingston’s NVMe Line-up Is The Life Of The Party

Disclaimer: I recently attended VMworld 2017 – US.  My flights were paid for by ActualTech Media, VMware provided me with a free pass to the conference and various bits of swag, and Tech Field Day picked up my hotel costs. There is no requirement for me to blog about any of the content presented and I am not compensated in any way for my time at the event.  Some materials presented were discussed under NDA and don’t form part of my blog posts, but could influence future discussions.

You can view the video of Kingston‘s presentation at Tech Field Day Extra VMworld US 2017 here, and download a PDF copy of my rough notes from here.

 

It’s A Protocol, Not Media

NVMe has been around for a few years now, and some people get it confused for a new kind of media that they plug into their servers. But it’s not really, it’s just a standard specification for accessing Flash media via the PCI Express bus. There’re a bunch of reasons why you might choose to use NVMe instead of SAS, including lower latency and less CPU overhead. My favourite thing about it though is the plethora of form factors available to use. Kingston touched on these in their presentation at Tech Field Day Extra recently. You can get them in half-height, half-length (HHHL) add-in cards (AIC), U.2 (2.5″) and M.2 sizes. To give you an idea of the use cases for each of these, Kingston suggested the following applications:

  • HHHL (AIC) card
    • Server / DC applications
    • High-end workstations
  • U.2 (2.5″)
    • Direct-attached, server backplane, just a bunch of flash (JBOF)
    • White box and OEM-branded
  • M.2
    • Client applications
    • Notebooks, desktops, workstations
    • Specialised systems

 

It’s Pretty Fast

NVMe has proven to be pretty fast, and a number of companies are starting to develop products that leverage the protocol in an extremely efficient manner. Coupled with the rise of NVMe/F solutions and you’ve got some pretty cool stuff coming to market. The price is also becoming a lot more reasonable, with Kingston telling us that their DCP1000 NVMe HHHL comes in at around “$0.85 – $0.90 per GB at the moment”. It’s obviously not as cheap as things that spin at 7200RPM but the speed is mighty fine. Kingston also noted that the 2.5″ form factor would be hanging around for some time yet, as customers appreciated the serviceability of the form factor.

 

[Kingston DCU1000 – Image courtesy of Kingston]

 

This Stuff’s Everywhere

Flash media has been slowly but surely taking over the world for a little while now. The cost per GB is reducing (slowly, but surely), and the range of form factors means there’s something for everyone’s needs. Protocol advancements such as NVMe make things even easier, particularly at the high end of town. It’s also been interesting to see these “high end” solutions trickle down to affordable form factors such as PCIe add-in cards. With the relative ubiquity of operating system driver support, NVMe has become super accessible. The interesting thing to watch now is how we effectively leverage these advancements in protocol technologies. Will we use them to make interesting advances in platforms and data access? Or will we keep using the same software architectures we fell in love with 15 years ago (albeit with dramatically improved performance specifications)?

 

Conclusion and Further Reading

I’ll admit it took me a little while to come up with something to write about after the Kingston presentation. Not because I don’t like them or didn’t find their content interesting. Rather, I felt like I was heading down the path of delivering another corporate backgrounder coupled with speeds and feeds and I know they have better qualified people to deliver that messaging to you (if that’s what you’re into). Kingston do a whole range of memory-related products across a variety of focus areas. That’s all well and good but you probably already knew that. Instead, I thought I could focus a little on the magic behind the magic. The Flash era of storage has been absolutely fascinating to witness, and I think it’s only going to get more interesting over the next few years. If you’re into this kind of thing but need a more comprehensive primer on NVMe, I recommend you check out J Metz’s article on the Cisco blog. It’s a cracking yarn and enlightening to boot. Data Centre Journal also provide a thorough overview here.

EMC – I heart EFD performance

We got some EFDs on our CX4-960s recently, and we had the chance to do some basic PassMark testing on them before we loaded them up with production configurations. We’re running 30 * 200GB EFDs in a Storage Pool on the CX4-960. FAST and FAST Cache haven’t been turned on. The VM was sitting on an HP BL460p G6 blade with 96GB RAM, 12 Nehalem cores and vSphere 4.1. The blades connect to the arrays via Cisco 9124e FC switches with 8Gbps port-channels to the Cisco MDS 9513. We’re only using 2 fe ports per SP on the CX4-960 at the moment. We used Pass Mark on a Windows 2008 R2 VM sitting on a 100GB vmdk. There wasn’t a lot of other LUNs bound in the Storage Pool, so the results are skewed. Nonetheless, they look pretty, and that’s what I’m going with.

100% Sequential Write:

 

Results:

100% Sequential Read:

Results:

File Server Simulation (80% Read, 20% Write):

Results:

EMC – FAST and FAST Cache on the CX4-960

Apologies for the lack of posts over the last few months – I have been nuts deep in work and holidays. I’m working on some literature around Storage Pools and FAST in general, but in the meantime I thought I’d share this nugget with you. We finally got approval to install the FAST and FAST Cache enablers on our production CX4-960s a few nights ago. We couldn’t install them on one of the arrays because we had a dead disk that prevented the NDU from going ahead. Fair enough. Two awesome things happened when we installed it on the other array. Both of which could have been avoided if I’d had my shit together. Firstly, when I got into the office the next morning at 8 am, we noticed that the Read Cache on the array was disabled. For those of you playing at home, we had the cache on the 960 set at 1000MB read and 9760MB for write. I think I read this in a whitepaper some where. But after FAST went on, we still had 9760MB allocated to Write, and 0MB available for Read. Awesome not so much. Seems that we lost 1000MB, presumably because we added another layered application. Funnily enough we didn’t observe this behaviour on our lab CX4-120s, although you could argue that they really have sweet FA of cache in the first place. So now we have 8760MB for Write, and 1000MB for Read. And I’m about to configure a few hundred GB of FAST Cache on the EFDs in any case. We’ll see how that goes.

The other slightly boneheaded thing we did was forget to trespass the LUN ownership of LUNs on SP A back from SP B. In other words, an NDU applies code to SP B first, reboots the SP, checks it, and then loads code on the other SP. As part of this, LUN ownership is temporarily trespassed to the surviving SP (this is the whole non-disruptive thing). Once the NDU is complete, you should go and check for trespassed LUNs and move them back to their owners. Or not, and have everything run on one SP for a while. And wait for about 9000 Exchange users to complain when one of the Exchange clusters goes off-line. Happy days.

EMC FAST and FAST Cache

I don’t have a lot to say on FAST or FAST Cache just yet – because the PO hasn’t quite made its way to EMC. But when I was researching the topic I found Matt Taylor’s post on this topic to be an excellent summary – particularly the point about configuration options. Particularly if you’re rolling with multiple CX4-960s – as we are. I think it’s going to scream :)