In this edition of Things My Customers Have Asked Me (TMCHAM), I’m going to cover some of the things to consider when looking at Disaster Recovery (DR) and availability solutions on the VMware-managed VMware Cloud on AWS platform. It might seem like basic stuff if you’re already using the platform, but there are plenty of customers I talk to that are struggling to distinguish between DR, data availability, and periodic data protection in a VMware Cloud context.

Terminology

I’m cheating a little here and re-using some of my older content, but I think it still holds up.

• Disaster Recovery – Disaster Recovery is the ability to recover applications after a major event (think flood, fire, data centre is now a hole in the ground). This normally involves a failover of workloads from one DC to another in an orchestrated fashion. In a VMware context, you would use VMware Live Recovery (Site or Cyber, depending on the environment).
• Disaster Avoidance – Disaster avoidance “is an anticipatory strategy that is in place in order to prevent any such instance of data breach or losses. It is a defensive, proactive approach to keeping data safe” (I’m quoting this from a great blog post on the topic here). In a VMware context, this would be a stretched cluster using vSAN on VMware Cloud Foundation or in VMC on AWS.
• Periodic Data Protection – This is the kind of data protection activity we normally associate with “backups”. It is usually a daily activity (or perhaps as frequent as hourly) and the data is normally used for ad-hoc data file recovery requests. Some people use their backup data as an archive. They’re bad people and shouldn’t be trusted. You should consider PDP as separate to DA or DR solutions.

So How Do I Best Protect My Workloads? (Part 1)

Understand Your Workloads

You need to understand how your applications work. And you need to understand what talks to them and what they talk to. There are plenty of ways of doing this, including a variety if applications available that scan network traffic and draw pretty pictures of data flows for you. From experience, there’s a very good chance you won’t get this information by reading current documentation, because it won’t be current.

You need to understand how important your applications are to you. Do you need to recover them immediately if something goes wrong? Can you do without them for a few hours? A few days? How much data can you afford to lose? All of it? (in which case why do you still have this data sitting around?). None of it? Some of it? You need to be disciplined and thorough in your assessment of your workloads and what they mean to your business, because protecting them or recovering them can be an expensive exercise.

Understand What Scares You Most

What do you call a disaster? Is it a data centre turning into a hole in the ground? Do you run most of your workloads in an area prone to natural disasters? Or are you more worried about people in hoodies breaking into your infrastructure and encrypting it and your data protection infrastructure? (Hot tip: you absolutely should be worried about this). You need to be fairly paranoid about the scenarios you’re trying to protect your environment from. No one thought it was a problem keeping workloads in two buildings in New York until it was. At the same time, you’ll need to introduce some pragmatism into your planning. I’ve had customers ask me what they could do if all of the Sydney Region was wiped out (not just the AWS deployment, but the whole city). Could they recover data to Melbourne? Sure. Would they have any staff left to do it? Or want to do it? Depending on the nature of the disaster, the technical aspects of disaster recovery are going to be the easy bits. You still need to make sure the people and process bit is covered.

Who Can Help You? 

Do you want to handle any recovery activities yourself when things go bang? Or are you happy to let someone else do the heavy lifting for you? Some folks don’t trust outsiders to do “important” things, like recover workloads, whilst other organisations would consume absolutely everything as a service if it was an option.

How Much Can You Spend?

You need to answer the questions above before you can answer how much you’re willing to spend on one (or more) solutions to help keep your workloads protected. If your apps aren’t that important, or your business can continue to run without access to these apps for a period of time, then you’re going to be less inclined to go for the “platinum” solution. But if you’re whole business relies on applications being available 24/7/365, then you’re hopefully more understanding that you’ll need to invest.

So How Do I Best Protect My Workloads? (Part 2)

Data Availability with VMware Cloud on AWS

As I mentioned earlier in this post, if you’re looking to provide data availability for your workloads on VMware Cloud on AWS, you should be looking at deploying a stretched cluster solution. In short, stretched clusters in VMware Cloud on AWS are designed to protect against an AWS availability zone (AZ) failure. There’s a feature brief available here that goes into more depth on the solution. Key things to note are that the vSAN stretched cluster is deployed across 3 AZs (2 workload, 1 witness). So you could have 3 hosts in one AZ, 3 in another, and VMware deploys a witness node in the third site as part of the deployment. From the document, “vSAN Fault Domains are configured to inform vSphere and vCenter which Hosts reside in which AZ. Each Fault Domain is named after the AZ it resides within to increase clarity. Logical networks are also extended using NSX to support workload mobility across the two AWS availability zones”. You also have the option to keep data using various vSAN storage policies, including:

• Dual-site mirroring (stretched cluster)
• None – Keep data on primary (stretched cluster)
• None – Keep data on secondary (stretched cluster)

Why would you not just mirror everything? Cost. Some applications simply don’t need that level of resilience, but it’s too complicated to deploy multiple SDDCs to support multiple deployment topologies (you can’t have a mix of single AZ and stretched clusters in the same SDDC), so customers can keep a mix of workloads in the same deployment. Note also that external storage solutions (like FSx for NetApp ONTAP) aren’t supported for use with stretched clusters. vSphere HA will restart the VM on the surviving AZ if a site failure is detected. There’s a short video on YouTube that covers a stretched cluster deployment and is worth looking at.

Disaster Recovery with VMware Cloud on AWS

I’ve covered VMware Live Cyber Recovery (and its predecessor, VMware Cloud Disaster Recovery) on this blog before. It’s a great solution to protect on-premises, GCVE, and VMware Cloud on AWS workloads.  It provides protection to a secure cloud filesystem with a Recovery Point Objective as low as 15 minutes, the ability to perform guided ransomware recovery in an isolated environment, and SaaS-based recovery automation. The kind of things you want to consider when evaluating the suitability of VLCR to your needs would be:

• Recovery Point Objective – the maximum amount of time in which data may have been permanently lost during an incident. VLCR goes down to 15 minutes, but if you need lower than that you need to look at other architectures.
• Scalability – how many Virtual Machines and how much storage are you trying to protect? And how quickly do you want to recover it? And where do you want to recover it to? Some folks don’t like VMware Cloud on AWS being the only recovery option, but VMware has announced “plans to enhance VMware Live Recovery support for Google Cloud VMware Engine (GCVE) as a cyber and disaster recovery (DR) site, for both on-premises as well as GCVE environments“. You can check out the configuration maximums for VMware solutions here.
• Networking and security – how will you access your recovery workloads? Will you use HCX to have stretched networks between your protected sites and your recovery site? What does that look like for your core networking infrastructure? Is your CISO happy to recover stuff onto a cloud platform? There are many things you’ll need to work through internally to ensure people are comfortable. I’d also recommend you go with a pilot light deployment as opposed to an on-demand solution. While the pilot light option seems more expensive, it will save you when something goes wrong and you’ve already tested your deployment.
• Other Considerations – there’s a bunch of info in the legacy VCDR Release Notes that highlight some of the constraints to be aware of from an architectural standpoint, and understanding some of these (like the ability to protect stretched clusters, but not recover to stretched clusters) is key to ensuring your architecture will work when you push the button. And one point that is often missed by folks looking at VLCR for the first time is that, as the recovery SDDC is just a VMware Cloud on AWS deployment, you can use it for other workloads in the meantime, like dev/test stuff, when you’re not in a recovery situation.

Periodic Data Protection with VMware Cloud on AWS

There are a tonne of different things you can do here, and I’d be naive to suggest that I could cover them all effectively in this post. To use periodic data protection solutions with VMware Cloud on AWS, you should check out this KB article, and work with your data protection vendor to ensure that the solution has been tested and works on VMware Cloud on AWS. I’ve encountered a few situations where the vendor has assumed that VMware Cloud on AWS works the same way as VMware on-premises, and with the same ability to do silly things with privileged access to ESXi hosts and things of that nature. There aren’t a lot of restrictions in place, but there are some important ones, like the aforementioned limits on direct host access, restricted account privileges, and so on. Just make sure you test it before you buy it. I’ll try and cover some of the scenarios that do and don’t work in a future post (without naming too many names).

Conclusion

I’m sticking with solutions available with VMware Cloud on AWS in this post, mainly because that’s what I’m talking to people about on a daily basis. But there’s plenty here that can be applied more generically to a variety of infrastructure solutions. You’ll have the same questions you’ll need to ask when it comes to protecting workloads on-premises (regardless of your choice of hypervisor), and you’ll need to make some hard decisions ab out what you want to protect depending on the value of what your workloads deliver to the business, and the available budget. You might also need to consider doing a combination of all three options, depending on the importance of the data / workloads you’re looking after. I’d always pick stretched clusters to keep my workloads up and available, but that won’t help me if some hacker decides to dump ransomware in my environment and encrypt all of my stuff. But if I need to go back 24 months to retrieve some file for an auditor, I wouldn’t use VMware Live Cyber Recovery for that. I should also mention that you should be using application-level protection as your first option for application resilience too. This other stuff is all infrastructure level, and potentially not as elegant, depending on your applications. But that doesn’t mean you can’t combine both methods. Unfortunately, there’s no one magical fix for this problem, but with a bit of planning, and some critical evaluation, you can protect your workloads against all manner of bad situations. And if you can’t do everything, at least do something, but just make sure people in your business understand the difference between the two. And remember, you’re only as good as your last recovery.