Disclaimer: I recently attended Storage Field Day 12. My flights, accommodation and other expenses were paid for by Tech Field Day. There is no requirement for me to blog about any of the content presented and I am not compensated in any way for my time at the event. Some materials presented were discussed under NDA and don’t form part of my blog posts, but could influence future discussions.
I/O Can Be Hard Work
With the advent of NVM Express, things go pretty fast nowadays. Or, at least, faster than they used to with those old-timey spinning disks we’ve loved for so long. According to Intel, systems with multiple NVMe SSDs are now capable of performing millions of I/Os per second. Which is great, but it results in many cores of software overhead with a kernel-based interrupt-driven driver model. The answer, according to Intel, is the Storage Performance Development Kit (SPDK). The SPDK enables more CPU cycles for storage services, with lower I/O latency. The great news is that there’s now almost no premium now on capacity to do IOPS with a system. So how does this help in the real world?
Real World Applications?
SPDK VM I/O Efficiency
The SPDK offers some excellent performance improvements when dishing up storage to VMs.
- NVMe ephemeral storage
- SPDK-based 3rd party storage services
Leverage existing infrastructure for:
- QEMU vhost-scsi;
- QEMU/DPDK vhost-net user.
Features and benefits
- High performance storage virtualisation
- Reduced VM exit
- Lower latency
- Increased VM density
- Reduced tail latencies
- Higher throughput
Intel say that Ali Cloud sees ~300% improvement in IOPS and latency using SPDK
VM Ephemeral Storage
- Improves Storage virtualisation
- Works with KVM/QEMU
- 6x efficiency vs kernel host
- 10x efficiency vs QEMU virtuo
- Increased VM density
SPDK and NVMe over Fabrics
SPDK also works a treat with NVMe over Fabrics.
VM Remote Storage
- Enable disaggregation and migration of VMs using remote storage
- Improves storage virtualisation and flexibility
- Works with KVM/QEMU
NVMe over Fabrics
|NVMe over Fabrics|
|Utilises NVM Express (NVMe) Polled Mode Driver||Reduced overhead per NVMe I/O|
|RDMA Queue Pair Polling||No interrupt overhead|
|Connections pinned to CPU cores||No synchronisation overhead|
NVMe-oF Key Takeaways
- Preserves the latency-optimised NVMe protocol through network hops
- Potentially radically efficient, depending on implementation
- Actually fabric agnostic: InfinBand, RDMA, TCP/IP, FC … all ok!
- Underlying protocol for existing and emerging technologies
- Using SPDK, can integrate NVMe and NVMe-oF directly into applications
VM I/O Efficiency Key Takeaways
- Huge improvement in latency for VM workloads
- Application-level sees 3-4X performance gains
- Application unmodified: it’s all under the covers
- Virtuous cycle with VM density
- Fully compatible with NVMe-oF!
Further Reading and Conclusion
Intel said during the presentation that “[p]eople find ways of consuming resources you provide to them”. This is true, and one of the reasons I became interested in storage early in my career. What’s been most interesting about the last few years worth of storage developments (as we’ve moved beyond spinning disks and simple file systems to super fast flash subsystems and massively scaled out object storage systems) is that people are still really only interested in have lots of storage that is fast and reliable. The technologies talked about during this presentation obviously aren’t showing up in consumer products just yet, but it’s an interesting insight into the direction the market is heading. I’m mighty excited about NVMe over Fabrics and looking forward to this technology being widely adopted in the data centre.
If you’ve had the opportunity to watch the video from Storage Field Day 12 (and some other appearances by Intel Storage at Tech Field Day events), you’ll quickly understand that I’ve barely skimmed the surface of what Intel are doing in the storage space, and just how much is going on before your precious bits are hitting the file system / object store / block device. NVMe is the new way of doing things fast, and I think Intel are certainly pioneering the advancement of this technology through real-world applications. This is, after all, the key piece of the puzzle – understanding how to take blazingly fast technology and apply a useful programmatic framework that companies can build upon to deliver useful outcomes.