This a really quick follow up to one of my TMCHAM articles on TRIM/UNMAP on VMware Cloud on AWS. In short, a customer wanted to know whether TRIM/UNMAP had been enabled on one of their clusters, as they’d requested. The good news is it’s easy enough to find out. On your cluster, go to Configure. Under vSAN, you’ll see Services. Expand the Advanced Options section and you’ll see whether TRIM/UNMAP has been enabled for the cluster or not.
This one is a little behind thanks to some work travel, but whatever. Let’s get random.
- Ethan Banks has some great advice for bloggers looking to load up on payola from their blood, sweat, and blogging tears. You can read his article on how to work with a sponsor for your blog here.
- I love Howard’s Explainer articles, and this one on Thunderbolt doesn’t fail to impress.
- Over at El Reg there’s word of problems for the Linux kernel NTFS driver as author goes silent. I’ve used these products from time to time, and will be curious to see how it plays out.
- Cormac recently announced that the vSAN Deep Dive book has been updated for 7.0 U3.
- In PR news, Aparavi recently announced Version 2.0 of its Data Intelligence Platform.
- Speaking of press releases, Zerto recently announced version 9.5 of its platform. You can read the release here.
- Sometimes you read articles and you think “Nah, that can’t be what they did”. But then you realise they did. Nicholson has published a typically entertaining article on why running a database on a dedupe appliance is a cosmically bad idea (TM).
- In home theatre news, this article on XLR vs RCA – which cable is better? makes for good reading, particularly if you’re starting to convince yourself that you need to take things in a certain direction.
Welcome to Random Short Take #57. Only one player has worn 57 in the NBA. So it looks like this particular bit is done. Let’s get random.
- In the early part of my career I spent a lot of time tuning up old UNIX workstations. I remember lifting those SGI CRTs from desk to desk was never a whole lot of fun. This article about a Sun Ultra 1 project bought back a hint of nostalgia for those days (but not enough to really get into it again). Hat tip to Scott Lowe for the link.
- As you get older, you realise that people talk a whole lot of rubbish most of the time. This article calling out audiophiles for the practice was great.
- This article on the Backblaze blog about one company’s approach to building its streaming media capability on B2 made for interesting reading.
- DH2i recently announced the general availability of DxEnterprise (DxE) for Containers, enabling cloud-native Microsoft SQL Server container Availability Groups outside and inside Kubernetes.
- Speaking of press releases, Zerto has made a few promotions recently. You can keep up with that news here.
- I’m terrible when it comes to information security, but if you’re looking to get started in the field, this article provides some excellent guidance on what you should be focussing on.
- We all generally acknowledge that NTP is important, and most of us likely assume that it’s working. But have you been checking? This article from Tony does a good job of outlining some of the reasons you should be paying some more attention to NTP.
- This is likely the most succinct article from John you’ll ever read, and it’s right on the money too.
Welcome to Random Short Take #52. A few players have worn 52 in the NBA including Victor Alexander (I thought he was getting dunked on by Shawn Kemp but it was Chris Gatling). My pick is Greg Oden though. If only his legs were the same length. Let’s get random.
- Penguin Computing and Seagate have been doing some cool stuff with the Exos E 5U84 platform. You can read more about that here. I think it’s slightly different to the AP version that StorONE uses, but I’ve been wrong before.
- I still love Fibre Channel (FC), as unhealthy as that seems. I never really felt the same way about FCoE though, and it does seem to be deader than tape.
- VMware vSAN 7.0 U2 is out now, and Cormac dives into what’s new here. If you’re in the ANZ timezone, don’t forget that Cormac, Duncan and Frank will be presenting (virtually) at the Sydney VMUG *soon*.
- This article on data mobility from my preferred Chris Evans was great. We talk a lot about data mobility in this industry, but I don’t know that we’ve all taken the time to understand what it really means.
- I’m a big fan of Tech Field Day, and it’s nice to see presenting companies take on feedback from delegates and putting out interesting articles. Kit’s a smart fellow, and this article on using VMware Cloud for application modernisation is well worth reading.
- Preston wrote about some experiences he had recently with almost failing drives in his home environment, and raised some excellent points about resilience, failure, and caution.
- Speaking of people I worked with briefly, I’ve enjoyed Siobhán’s series of articles on home automation. I would never have the patience to do this, but I’m awfully glad that someone did.
- Datadobi appears to be enjoying some success, and have appointed Paul Repice to VP of Sales for the Americas. As the clock runs down on the quarter, I’m going two for one, and also letting you know that Zerto has done some work to enhance its channel program.
Welcome to Random Short Take #34. Some really good players have worn 34 in the NBA, including Ray Allen and Sir Charles. This one, though, goes out to my favourite enforcer, Charles Oakley. If it feels like it’s only been a week since the last post, that’s because it has.
- I spoke to the folks at Rancher Labs a little while ago, and they’re doing some stuff around what they call “Edge Scalability” and have also announced Series D funding.
- April Fool’s is always a bit of a trying time, what with a lot of the world being a few timezones removed from where I live. Invariably I stop checking news sites for a few days to be sure. Backblaze recognised that these are strange times, and decided to have some fun with their releases, rather than trying to fool people outright. I found the post on Catblaze Cloud Backup inspiring.
- Hal Yaman announced the availability of version 2.6 of his Office 365 Backup sizing tool. Speaking of Veeam and handy utilities, the Veeam Extract utility is now available as a standalone tool. Cade talks about that here.
- VMware vSphere 7 recently went GA. Here’s a handy article covering what it means for VMware cloud providers.
- Speaking of VMware things, John Nicholson wrote a great article on SMB and vSAN (I can’t bring myself to write CIFS, even when I know why it’s being referred to that way).
- Scale is infinite, until it isn’t. Azure had some minor issues recently, and Keith Townsend shared some thoughts on the situation.
- StorMagic recently announced that it has acquired KeyNexus. It also announced the availability of SvKMS, a key management system for edge, DC, and cloud solutions.
- Joey D’Antoni, in collaboration with DH2i, is delivering a webinar titled “Overcoming the HA/DR and Networking Challenges of SQL Server on Linux”. It’s being held on Wednesday 15th April at 11am Pacific Time. If that timezone works for you, you can find out more and register here.
Disclaimer: I recently attended VMworld 2017 – US. My flights were paid for by ActualTech Media, VMware provided me with a free pass to the conference and various bits of swag, and Tech Field Day picked up my hotel costs. There is no requirement for me to blog about any of the content presented and I am not compensated in any way for my time at the event. Some materials presented were discussed under NDA and don’t form part of my blog posts, but could influence future discussions.
Here are my rough notes from “STO1179BU – Understanding the Availability Features of vSAN”, presented by GS Khalsa (@gurusimran) and Jeff Hunter (@jhuntervmware). You can grab a PDF of the notes from here. Note that these posts don’t provide much in the way of opinion, analysis, or opinionalysis. They’re really just a way of providing you with a snapshot of what I saw. Death by bullet point if you will.
Components and Failure
vSAN Objects Consist of Components
- VM Home – multiple components
- Virtual Disk – multiple components
- Swap File – multiple components
vSAN has a cache tier and capacity tier (objects are stored here)
Greater than 50% must be online to achieve quorum
- Each component has one vote by default
- Odd number of votes required to break tie – preserves data integrity
- Greater than 50% of components (votes) must be online
- Components can have more than one vote
- Votes added by vSAN, if needed, to ensure odd number
Component Vote Counts Are Visible Using RVC CLI
/<vcenter>/datacenter/vms> vsan_vm_object_info <vm>
Storage Policy Determines Component Number and Placement
- Primary level of failures to tolerate
- Failure Tolerance Method
Primary level of failures to tolerate = 0 Means only one copy
- Maximum component size is 255GB
- vSAN will split bigger into smaller sized VMDKs
- RAID-5/6 Erasure Coding Uses Stripes and Parity (need to be using all-flash)
- Consumes less RAW capacity
- Number of stripes also affects component counts
Each Host is an Implicit Fault Domain
- Multiple components can end up in the same rack
- Configure Fault Domains in the UI
- Add at least one more host or fault domain for rebuilds
Component States Change as a Result of a Failure
vSAN selects most efficient way
Which is most efficient? Repair or Rebuild? It depends. Partial repairs are performed if full repair is not possible
vSAN Maintenance Mode
- Evacuate all data to other hosts
- Ensure data accessibility from other hosts
- No data evacuation
Degraded Device Handling (DDH) in vSAN 6.6
- vSAN 6.6 is more “intelligent”, builds on previous versions of DDH
- When device is degraded, components are evaluated …
- If component does not belong to last replica, mark as absent – “Lazy” evacuation since another replica of the object exists
- If component belongs to last replica, start evacuation
- Degraded devices will not be used for new component placement
- Evacuation failures reported in UI
DDH and S.M.A.R.T.
Following items logged in vmkernel.log when drive is identified as unhealthy
- Sectors successfully reallocated 0x05
- Reported uncorrectable sectors 0xBB
- Disk command timeouts 0xBC
- Sector reallocation events 0xC4
- Pending sector reallocations 0xC5
- Uncorrectable sectors 0xC6
Helps GSS determine what to do with drive after evacuation
Stretched Cluster Failure Scenarios
- Extend the idea of fault domains from racks to sites
- Witness component (tertiary site) – witness host
- 5ms RTT (around 60 miles)
- VM will have a preferred and secondary site
- When component fails, starts rebuilding of preferred site
Stretched Cluster Local Failure Protection – new in vSAN 6.6
- Redundancy against host failure and site failure
- If site fails, vSAN maintains local redundancy in surviving site
- No change in stretched cluster configuration steps
- Optimised logic to minimise I/O traffic across sites
- Local read, local resync
- Single inter-site write for multiple replicas
- RAID-1 between the sites, and then RAID-5 in the local sites
What happens during network partition or site failure?
- HA Restart
Inter-site network disconnected (split brain)
- HA Power-off
Witness Network Disconnected
- Witness leaves cluster
VMs continue to operate normally. Very simple to redeploy a new one. Recommended host isolation response in a stretched cluster is power off
Witness Host Offline
- Recover or redeploy witness host
New in 6.6 – change witness host
vSAN Backup, Replication and DR
- vSphere APIs – Data Protection
- Same as other datastore (VMFS, etc)
- Verify support with backup vendor
- Production and backup data on vSAN
- Pros: Simple, rapid restore
- Cons: Both copies lost if vSAN datastore is lost, can consume considerable capacity
- Store backup data on another datastore
- SAN or NAS
- Another vSAN cluster
- Local drives
- Dell EMC Avamar and NetWorker
- Veeam Backup and Replication
- Others …
vSphere Replication included with Essentials Plus Kit and higher. With this you get per-VM RPOs as low as 5 minutes
Automated DR with Site Recovery Manager
- HA with Stretched Cluster, Automated DR with SRM
- SRM at the tertiary site
Useful session. 4 stars.
Disclaimer: I recently attended Storage Field Day 7. My flights, accommodation and other expenses were paid for by Tech Field Day. There is no requirement for me to blog about any of the content presented and I am not compensated in any way for my time at the event. Some materials presented were discussed under NDA and don’t form part of my blog posts, but could influence future discussions.
For each of the presentations I attended at SFD7, there are a few things I want to include in the post. Firstly, you can see video footage of the VMware presentation here. You can also download my raw notes from the presentation here. Finally, here’s a link to the VMware website that covers some of what they presented.
I’d like to say a few things about the presentation. Firstly, it was held in the “Rubber Chicken” Room at VMware HQ.
Secondly, Rawlinson was there, but we ran out of time to hear him present. This seems to happen each time I see him in real life. Still, it’s not everyday you get to hear Christos Karamanolis (@XtosK) talk about this stuff, so I’ll put my somewhat weird @PunchingClouds fanboy thing to the side for the moment.
Thirdly, and I’ll be upfront about this, I was a bit disappointed that VMware didn’t go outside some fairly fixed parameters as far as what they could and couldn’t talk about with regards to Virtual SAN. I understand that mega software companies have to be a bit careful about what they can say publicly, but I had hoped for something fresher in this presentation. In any case, I’ve included my notes on Christos’s view on the VSAN architecture – I hope it’s useful.
VMware adopted the following principles when designing VSAN.
- Compute + storage scalability
- Unobtrusive to existing data centre architecture
- Distributed software running on every host
- Pools local storage (flash + HDD) on hosts (virtual shared datastore)
- Symmetric architecture – no single point of failure, no bottleneck
The hypervisor opens up new opportunities, with the virtualisation platform providing:
- Visibility to individual VMs and application storage
- Manages all applications’ resource requirements
- Sits directly in the I/O path
- A global view of underlying infrastructure
- Supports an extensive hardware compatibility list (HCL)
Critical paths in ESX kernel
The cluster service allows for
- Fast failure detection
- High performance (especially for writes)
The data path provides
- Low latency
- Minimal CPU per IO
- Minimal Mem consumption
- Physical access to devices
This equals minimal impact on consolidation rates. This is a Good Thing™.
Optimized internet protocol
As ESXi is both the “consumer” and “producer” of data there is no need for a standard data access protocol.
Per-object coordinator = client
- Distributed “metadata server”
- Transactions span only object distribution
Efficient reliable data transport (RDT)
- Protocol agnostic (now TCP/IP)
- RDMA friendly
Standard protocol for external access?
Two tiers of storage: Hybrid
Optimise the cost of physical storage resources
- HDDS: cheap capacity, expensive IOPS
- Flash: expensive capacity, cheap IOPS
Combine best of both worlds
- Performance from flash (read cache + write back)
- Capacity from HDD (capacity tier)
Optimise workload per tier
- Random IO to flash (high IOPS)
- Sequential IO to HDD (high throughput)
Storage organised in disk groups (flash device and magnetic disks) – up to 5 disk groups, 1 SSD + 7 HDDs – this is the fault domain. 70% of flash is read cache, 30% is write buffer. Writes are accumulated, then staged in a magnetic disk-friendly fashion. Proximal IO – writing blocks within a certain number of cylinders. Filesystem on the magnetic disks is slightly different to the one on the SSDs. Uses the back-end of the Virsto filesystem, but doesn’t use the log-structure filesystem component.
Flash device: cache of disk group (70% read cache, 30% write-back buffer)
No caching on “local” flash where VM runs
- Flash latencies 100x network latencies
- No data transfers, no perf hit during VM migration
- Better overall flash utilisation (most expensive resource)
Use local cache when it matters
- In-memory CBRC (RAM << Network latency)
- Lots of block sharing (VDI)
- More options in the future …
Deduplicated RAM-based caching
- VM consists of a number of objects – each object individually distributed
- VSAN doesn’t know about VMs and VMDKs
- Up to 62TB useable
- Single namespace, multiple mount points
- VMFS created in sub-namespace
The VM Home directory object is formatted with VMFS to allow a VM’s config files to be stored on it. Mounted under the root dir vsanDatastore.
- Availability policy reflected on number of replicas
- Performance policy may include a stripe width per replica
- Object “components” may reside in different disks and / or hosts
VSAN cluster = vSphere cluster
Ease of management
- Piggyback on vSphere management workflow, e.g. EMM
- Ensure coherent configuration of hosts in vSphere cluster
Adapt to the customer’s data centre architecture while working with network topology constraints.
Maintenance mode – planned downtime.
- Ensure accessibility;
- Full data migration; and
- No data migration.
VM-centric monitoring and troubleshooting
- Configure, manage, monitor
Policy compliance reporting
Combination of tools for monitoring in 5.5
- CLI commmands
- Ruby vSphere console
- VSAN observer
More to come soon …
Real *software* defined storage
Software + hardware – component based (individual components), Virtual SAN ready node (40 OEM validated server configurations are ready for VSAN deployment)
VMware EVO:RAIL = Hyper-converged infrastructure
It’s a big task to get all of this working with everything (supporting the entire vSphere HCL).
Closing Thoughts and Further Reading
I like VSAN. And I like that VMware are working so hard at getting it right. I don’t like some of the bs that goes with their marketing of the product, but I think it has its place in the enterprise and is only going to go from strength to strength with the amount of resources VMware is throwing at it. In the meantime, check out Keith’s background post on VMware here. In my opinion, you can’t go past Cormac’s posts on VSAN if you want a technical deep dive. Also, buy his book.