I miss Tru64, and Solaris for that matter. I don’t miss HP-UX. And I definitely won’t miss AIX. Read about the death of Unix over at El Reg – Unix is dead. Long live Unix!
The I3.metal is going away very soon. Remember this is from a sales perspective, VMware is still supporting the I3.metal in the wild, and you’ll still have access to deploy on-demand if required (up to a point).
Welcome to Random Short Take #81. Last one for the year, because who really wants to read this stuff over the holiday season? Let’s get random.
Curtis did a podcast on archive and retrieve as part of his “Backup to Basics” series. It’s something I feel pretty strongly about, so much so that I wrote a chapter in his book about it. You can listen to it here.
I love Backblaze. Not in the sense that I want to marry the company, but I really like what the folks there do. And I really like the transparency with which they operate. This article giving a behind the scenes look at its US East Data Center is a fantastic example of that.
And, to “celebrate” 81 Random Short Takes (remember when I used to list my favourite NBA players and the numbers they wore?), let’s take a stroll down memory lane with two of my all-time, top 5, favourite NBA players – Kobe Bryant and Jalen Rose. The background for this video is explained by Jalen here.
Take care of yourselves and each other, and I’ll hopefully see you all on the line or in person next year.
In this edition of Things My Customers Have Asked Me (TMCHAM), I’m going to delve into some questions around TRIM/UNMAP and capacity reclamation on the VMware-managed VMware Cloud on AWS platform.
TRIM/UNMAP, in short, is the capability for operating systems to reclaim no longer used space on thin-provisioned filesystems. Why is this important? Imagine you have a thin-provisioned volume that has 100GB of capacity allocated to it. It consumes maybe 1GB when it’s first deployed. You then add 50GB of data to it. You then delete 50GB of data from the volume. You’ll still see 51GB of capacity being consumed on the filesystem. This is because older operating systems just mark the blocks as deleted, but don’t zero them out. Modern operating systems do support TRIM/UNMAP though, but the hypervisor needs to understand the commands being sent to it. You can read more on that here.
Do the VMs come across thin? Do you need to reclaim space first? If you’re using HCX to go from thick to thin, you should be fine. If you’re migrating thin to thin, it’s worth checking whether you’ve got any space reclamation in place on your source side. I’ve had customers report back that some environments have migrated across with higher than expected storage usage due to a lack of space reclamation happening on the source storage environment. You can use something like Live Optics to report on your capacity consumed vs allocated, and how much capacity can be reclaimed.
Why Isn’t This Enabled By Default?
I don’t know for sure, but I imagine it has something to do with the fact that TRIM/UNMAP has the potential to have a performance impact from a latency perspective, depending on the workloads running in the environment, and the amount of capacity being reclaimed at any given time. We recommend that you “schedule large space reclamation jobs during off-peak hours to reduce any potential impact”. Given that VMware Cloud on AWS is a fully-managed service, I imagine we want to control as many of the performance variables as possible to ensure our customers enjoy a reliable and stable platform. That said, TRIM/UNMAP is a really useful feature, and you should look at getting it enabled if you’re concerned about the potential for wasted capacity in your SDDC.
Verity ES recently announced its official company launch and the commercial availability of its Verity ES data eradication enterprise software solution. I had the opportunity to speak to Kevin Enders about the announcement and thought I’d briefly share some thoughts here.
From Revert to Re-birth?
Revert, a sister company of Verity ES, is an on-site data eradication service provider. It’s also a partner for a number of Storage OEMs.
The folks at Revert have had an awful lot of experience with data eradication in big enterprise environments. With that experience, they’d observed a few challenges, namely:
The software doing the data eradication was too slow;
Eradicating data in enterprise environments introduced particular requirements at high volumes; and
Larger capacity HDDs and SDDs were a real problem to deal with.
The Real Problem?
Okay, so the process to get rid of old data on storage and compute devices is a bit of a problem. But what’s the real problem? Organisations need to get rid of end of life data – particularly from a legal standpoint – in a more efficient way. Just as data growth continues to explode, so too does the requirement to delete the old data.
Verity ES was spawned to develop software to solve a number of the challenges Revert were coming across in the field. There are two ways to do it:
Eliminate the data destructively (via device shredding / degaussing); or
Why eradicate? It’s a sustainable approach, enables residual value recovery, and allows for asset re-use. But it nonetheless needs to be secure, economical, and operationally simple to do. How does Verity ES address these requirements? It has Product Assurance Certification from ADISA. It’s also developed software that’s more efficient, particularly when it comes to those troublesome high capacity drives.
[image courtesy of Verity ES]
Who’s this product aimed at? Primarily enterprise DC operators, hyperscalers, IT asset disposal companies, and 3rd-party hardware maintenance providers.
If you’ve spent any time on my blog you’ll know that I write a whole lot about data protection, and this is probably one of the first times that I’ve written about data destruction as a product. But it’s an interesting problem that many organisations are facing now. There is a tonne of data being generated every day, and some of that data needs to be gotten rid of, either because it’s sitting on equipment that’s old and needs to be retired, or because legislatively there’s a requirement to get rid of the data.
The way we tackle this problem has changed over time too. One of the most popular articles on this blog was about making an EMC CLARiiON CX700 useful again after EMC did a certified erasure on the array. There was no data to be found on the array, but it was able to be repurposed as lab equipment, and enjoyed a few more months of usefulness. In the current climate, we’re all looking at doing more sensible things with our old disk drives, rather than simply putting a bullet in them (except for the Feds – but they’re a bit odd). Doing this at scale can be challenging, so it’s interesting to see Verity ES step up to the plate with a solution that promises to help with some of these challenges. It takes time to wipe drives, particularly when you need to do it securely.
I should be clear that this data doesn’t go out and identify what data needs to be erased – you have to do that through some other tools. So it won’t tell you that a bunch of PII is buried in a home directory somewhere, or sitting in a spot it shouldn’t be. It also won’t go out and dig through your data protection data and tell you what needs to go. Hopefully, though, you’ve got tools that can handle that problem for you. What this solution does seem to do is provide organisations with options when it comes to cost-effective, efficient data eradication. And that’s something that’s going to become crucial as we continue to generate data, need to delete old data, and do so on larger and larger disk drives.
VMware Cloud on AWS has been around for just over 5 years now, and in that time it’s proven to be a popular platform for a variety of workloads, industry verticals, and organisations of all different sizes. However, one of the challenges that a hyper-converged architecture presents is that resource growth is generally linear (depending on the types of nodes you have available). In the case of VMware Cloud on AWS, we (now) have 3 nodes available for use: the I3, I3en, and I4i. Each of these instances provides a fixed amount of CPU, RAM, and vSAN storage for use within your VMC cluster. So when your storage grows past a certain threshold (80%), you need to add an additional node. This is a longwinded way of saying that, even if you don’t need the additional CPU and RAM, you need to add it anyway. To address this challenge, VMware now offers what’s called “Supplemental Storage” for VMware Cloud on AWS. This is ostensibly external dat stores presented to the VMC hosts over NFS. This comes in two flavours: FSx for NetApp ONTAP and VMware Cloud Flex Storage. I’ll cover this in a little more detail below.
[image courtesy of VMware]
Amazon FSx for NetApp ONTAP
The first cab off the rank is Amazon FSx for NetApp ONTAP (or FSxN to its friends). This one is ONTAP-like storage made available to your VMC environment as a native service. It’s fully customer managed, and VMware managed from a networking perspective.
[image courtesy of VMware]
There’s a 99.99% Availability SLA attached to the service. It’s based on NetApp ONTAP, and offers support for:
Note that it currently requires VMware Managed Transit Gateway (vTGW) for Multi-AZ deployment (the only deployment architecture currently supported), and can connect to multiple clusters and SDDCs for scale. You’ll need to be on SDDC version 1.20 (or greater) to leverage this service in your SDDC, and there is currently no support for attachment to stretched clusters. While you can only connect datastores to VMC hosts using NFSv3, there is support for connecting directly to guest via other protocols. More information can be found in the FAQ here. There’s also a simulator you can access here that runs you through the onboarding process.
VMware Cloud Flex Storage
The other option for supplemental storage is VMware Cloud Flex Storage (sometimes referred to as VMC-FS). This is a datastore presented to your hosts over NFSv3.
VMware Cloud Flex Storage is:
A natively integrated cloud storage service for VMware Cloud on AWS that is fully managed by VMware;
Cost effective multi-cloud Cloud storage solution built on SCFS;
Delivered via a two-tier architecture for elasticity and performance (AWS S3 and local NVMe cache); and
Provides integrated Data-Management.
In short, VMware has taken a lot of the technology used in VMware Cloud Disaster Recovery (the result of the Datrium acquisition in 2020) and used it to deliver up to 400 TiB of storage per SDDC.
[image courtesy of VMware]
The intent of the solution, at this stage at least, is that it is only offered as a datastore for hosts via NFSv3, rather than other protocols directly to guests. There are some limitations around the supported topologies too, with stretched clusters not currently supported. From a disaster recovery perspective, it’s important to note that VMware Cloud Flex Storage is currently only offered on a single-AZ basis (although the supporting components are spread across multiple Availability Zones), and there is currently no support for VMware Cloud Disaster Recovery co-existence with this solution.
I’ve only been at VMware for a short period of time, but I’ve had numerous conversations with existing and potential VMware Cloud on AWS customers looking to solve their storage problems without necessarily putting everything on vSAN. There are plenty of reasons why you wouldn’t want to use vSAN for high capacity storage workloads, and I believe these two initial solutions go some ways to solving that issue. Many of the caveats that are wrapped around these two products at General Availability will be removed over time, and the traditional objections relating to VMware Cloud on AWS being not great at high-capacity, cost-effective storage will also have been removed.
Finally, if you’re an existing NetApp ONTAP customer, and were thinking about what you were going to do with that Petabyte of unstructured data you had lying about when you moved to VMware Cloud on AWS, or wanting to take advantage of the sweat equity you’ve poured into managing your ONTAP environment over the years, I think we’ve got you covered as well.
Jeff Geerling seems to do a lot of projects that I either can’t afford to do, or don’t have the time to do. Either way, thanks Jeff. This latest one – Building a fast all-SSD NAS (on a budget) – looked like fun.
You like ransomware? What if I told you you can have it cross-platform? Excited yet? Read Melissa’s article on Multiplatform Ransomware for a more thorough view of what’s going on out there.
Speaking of storage and clouds, Chris M. Evans recently published a series of videos over at Architecting IT where he talks to NetApp’s Matt Watt about the company’s hybrid cloud strategy. You can see it here.
I’ve spent a lot of money over the years trying to find the perfect media streaming device for home. I currently favour the Apple TV 4K, but only because my Boxee Box can’t keep up with more modern codecs. This article on the Best Device for Streaming for Any User – 2022 seems to line up well with my experiences to date, although I admit I haven’t tried the NVIDIA device yet. I do miss playing ISOs over the network with the HD Mediabox 100, but those were simpler times I guess.
Yes, at least from a licensing perspective. If you’ve bought storage from many of the traditional array vendors over the years, you would have likely paid for capacity-based licensing. Every time you upgraded the capacity of your array, there was usually a charge associated with that upgrade, beyond the hardware uplift costs. The folks at StorONE think it’s probably time that they stopped punishing customers for using higher capacity drives, so they’re shifting everything to a per-drive model.
How it Works
As I mentioned at the start, StorONE Scale-For-Free pricing is on a per-drive basis, so you can use the highest capacity, highest density drives without penalty, rather than metering capacity. The pricing is broken down thusly:
Price per HDD $/month
Price per SSD $/month
Cloud Use Case – $ per month by VM instance required
The idea is that this ultimately lowers the storage price per TB and brings some level of predictability to storage pricing.
The key to this model is the availability of some key features in the StorONE solution, namely:
A rewritten and collapsed I/O stack (meaning do more with a whole lot less)
Auto-tiering improvements (leading to more consistent and predictable performance across HDD and SDD)
High performance erasure coding (meaning super fast recovery from drive failure)
But That’s Not All
Virtual Storage Containers
With Virtual Storage Containers (VSC), you can apply different data services and performance profiles to different workloads (hosted on the same media) in a granular and flexible fashion. For example, if you need 4 drives and 50,000 IOPS for your File Services, you can do that. In the same environment you might also need to use a few drives for Object storage with different replication. You can do that too.
[image courtesy of StorONE]
Ransomware Detection (and Protection)
StorONE has been pretty keen on its ransomware protection capabilities, with the option to run immutable snapshots on volumes every 30 seconds and store over 500,000+ snaps per volume. But it has added in some improved telemetry to enable earlier detection of potential ransomware events on volumes, as well as introducing dual-key deletion of snapshots and improved two-factor authentication.
There are many things that are certain in life, including the fact that no matter how much capacity you buy for your storage array on day one, by month 18 you’re looking at ways to replace some of that capacity with higher capacity. In my former life as a diskslinger I helped many customers upgrade their arrays with increased capacity drives, and most, if not all of them, had to pay a licensing bump as well as a hardware cost for the privilege. The storage vendors would argue that that’s just the model, and for as long as you can get away with it, it is. Particularly when hardware is getting cheaper and cheaper, you need something to drive revenue. So it’s nice to see a company like StorONE looking to shake things up a little in an industry that’s probably had its way with customers for a while now. Not every storage vendor is looking to punish customers for expanding their environments, but it’s nice that those customers that were struggling with this have the option to look at other ways of using the capacity they need in a cost-effective and predictable. manner.
This doesn’t really work without the other enhancements that have gone in to StorONE, such as the improved erasure coding and automated tiering. Having a cool business model isn’t usually enough to deliver a great solution. I’m looking forward to hearing from the StorONE team in the near future about how this has been received by both existing and new customers, and what other innovations they come out with in the next 12 months.
This is one of those articles I’ve been meaning to post for a while, simply because I forget every time I do it how to do it properly. One way to expand the capacity of your QNAP NAS (non-disruptively) is to replace the drives one at a time with larger capacity drives. It’s recommended that you follow this process, rather than just ripping the drives out one by one and waiting for the RAID Group to expand. It’s a simple enough process to follow, although the QNAP UI has always struck me as a little on the confusing side to navigate, so I took some pictures. Note that this was done on QNAP firmware 18.104.22.1686.
Firstly, go to Storage/Snapshots under Storage in the ControlPanel. Click on Manage.
Select the Storage Pool you want to expand, and click on Manage again.
This will give you a drop-down menu. Select Replace Disks One by One.
Now select the disk you want to replace and click on Change.
Once you’ve done this for all of the disks (and it will take some time to rebuild depending on a variety of factors), click on Expand Capacity. It will ask you if you’re sure and hopefully you’ll click OK.
It’ll take a while for the RAID Group to synchronise.
You’ll notice then that, while the Storage Pool has expanded, the Volume is still the original size. Select the Volume and click on Manage.
Now you can click on Resize Volume.
The Wizard will give you information on the Storage Pool capacity and give you the option to set the new capacity of the volume. I usually click on Set to Max.
It will warn you about that. Click on OK because you like to live on the edge.
It will take a little while, but eventually your Volume will have expanded to fill the space.