Random Short Take #73

Welcome to Random Short Take #73. Let’s get random.

VMware Cloud on AWS – TMCHAM – Part 4 – VM Resource Management

In this episode of Things My Customers Have Asked Me (TMCHAM), I’m going to delve into some questions around resource management for VMs running on the VMware-managed VMware Cloud on AWS platform, and what customers need to know to make it work for them.

Distributed Resource Scheduler

If you’ve used VMware vSphere before, it’s likely that you’ve come across the Distributed Resource Scheduler (DRS) capability. DRS is a way to keep workloads evenly distributed across nodes in a cluster, and moves VMs around based on various performance considerations. The cool thing about this is that you don’t need to manually move workloads around when a particular guest or host goes a little nuts from a CPU or Memory usage perspective. There are cases, however, when you might not want your VMs to be moving around too much. In this instance, you’ll want to create what is called a “Disable DRS vMotion Policy”. You configure this via Compute Policies in vCenter, and you can read more about the process here.

If you don’t like reading documentation though, I’ve got some pictures you can look at instead. Log in to your vSphere Client and click on Policies and Profiles.

Then click on Compute Policies and click Add.

Under Policy type, there’s a dropdown box where you can select Disable DRS vMotion.

You’ll then give the policy a Name and Description. You then need to select the tag category you want to use.

Once you’ve selected the tag category you want to use, you can select the tags you want to apply to the policy.

Click on Create to create the Compute Policy, and you’re good to go.

Memory Overcommit Techniques

I’ve had a few customers ask me about how some of the traditional VMware resource management technologies translate to VMware Cloud on AWS. The good news is there’s quite a lot in common with what you’re used to with on-premises workload management, including memory overcommit techniques. As with anything, the effectiveness or otherwise of these technologies really depends on a number of different factors. If you’re interested in finding out more, I recommend checking out this article.

General Resource Management

Can I use the resource management mechanisms I know and love, such as Reservations, Shares, and Limits? You surely can, and you can read more about that capability here.

Conclusion

Just as you would with on-premises vSphere workloads, you do need to put some thought into your workload resource planning prior to moving your VMs onto the magic sky computers. The good news, however, is that there are quite a few smart technologies built into VMware Cloud on AWS that means you’ve got a lot of flexibility when it comes to managing your workloads.

Random Short Take #32

Welcome to Random Short Take #32. Lot of good players have worn 32 in the NBA. I’m a big fan of Magic Johnson, but honourable mentions go to Jimmer Fredette and Blake Griffin. It’s a bit of a weird time around the world at the moment, but let’s get to it.

  • Veeam 10 was finally announced a little while ago and is now available for deployment. I work for a service provider, and we use Veeam, so this article from Anthony was just what I was after. There’s a What’s New article from Veeam you can view here too.
  • I like charts, and I like Apple laptops, so this chart was a real treat. The lack of ports is nice to look at, I guess, but carrying a bag of dongles around with me is a bit of a pain.
  • VMware recently made some big announcements around vSphere 7, amongst other things. Ather Beg did a great job of breaking down the important bits. If you like to watch videos, this series from VMware’s recent presentations at Tech Field Day 21 is extremely informative.
  • Speaking of VMware Cloud Foundation, Cormac Hogan recently wrote a great article on getting started with VCF 4.0. If you’re new to VCF – this is a great resource.
  • Leaseweb Global recently announced the availability of 2nd Generation AMD EPYC powered hosts as part of its offering. I had a chance to speak with Mathijs Heikamph about it a little while ago. One of the most interesting things he said, when I questioned him about the market appetite for dedicated servers, was “[t]here’s no beating a dedicated server when you know the workload”. You can read the press release here.
  • This article is just … ugh. I used to feel a little sorry for businesses being disrupted by new technologies. My sympathy is rapidly diminishing though.
  • There’s a whole bunch of misinformation on the Internet about COVID-19 at the moment, but sometimes a useful nugget pops up. This article from Kieren McCarthy over at El Reg delivers some great tips on working from home – something more and more of us (at least in the tech industry) are doing right now. It’s not all about having a great webcam or killer standup desk.
  • Speaking of things to do when you’re working at home, JB posted a handy note on what he’s doing when it comes to lifting weights and getting in some regular exercise. I’ve been using this opportunity to get back into garage weights, but apparently it’s important to lift stuff more than once a month.

Cohesity Basics – Excluding VMs Using Tags – Real World Example

I’ve written before about using VM tags with Cohesity to exclude VMs from a backup. I wanted to write up a quick article using a real world example in the test lab. In this instance, we had someone deploying 200 VMs over a weekend to test a vendor’s storage array with a particular workload. The problem was that I had Cohesity set to automatically protect any new VMs that are deployed in the lab. This wasn’t a problem from a scalability perspective. Rather, the problem was that we were backing up a bunch of test data that didn’t dedupe well and didn’t need to be protected by what are ultimately finite resources.

As I pointed out in the other article, creating tags for VMs and using them as a way to exclude workloads from Cohesity is not a new concept, and is fairly easy to do. You can also apply the tags in bulk using the vSphere Web Client if you need to. But a quicker way to do it (and something that can be done post-deployment) is to use PowerCLI to search for VMs with a particular naming convention and apply the tags to those.

Firstly, you’ll need to log in to your vCenter.

PowerCLI C:\> Connect-VIServer vCenter

In this example, the test VMs are deployed with the prefix “PSV”, so this makes it easy enough to search for them.

PowerCLI C:\> get-vm | where {$_.name -like "PSV*"} | New-TagAssignment -Tag "COH-NoBackup"

This assumes that the tag already exists on the vCenter side of things, and you have sufficient permissions to apply tags to VMs. You can check your work with the following command.

PowerCLI C:\> get-vm | where {$_.name -like "PSV*"} | Get-TagAssignment

One thing to note. If you’ve updated the tags of a bunch of VMs in your vCenter environment, you may notice that the objects aren’t immediately excluded from the Protection Job on the Cohesity side of things. The reason for this is that, by default, Cohesity only refreshes vCenter source data every 4 hours. One way to force the update is to manually refresh the source vCenter in Cohesity. To do this, go to Protection -> Sources. Click on the ellipsis on the right-hand side of your vCenter source you’d like to refresh, and select Refresh.

You’ll then see that the tagged VMs are excluded in the Protection Job. Hat tip to my colleague Mike for his help with PowerCLI. And hat tip to my other colleague Mike for causing the problem in the first place.

VMware – Unmounting NFS Datastores From The CLI

This is a short article, but hopefully useful. I did a brief article a while ago linking to some useful articles about using NFS with VMware vSphere. I recently had to do some maintenance on one of the arrays in our lab and I was having trouble unmounting the datastores using the vSphere client. I used some of the commands in this KB article (although I don’t have SIOC enabled) to get the job done instead.

The first step was to identify if any of the volumes were still mounted on the individual host.

[[email protected]:~] esxcli storage nfs list
Volume Name  Host            Share                 Accessible  Mounted  Read-Only   isPE  Hardware Acceleration
-----------  --------------  --------------------  ----------  -------  ---------  -----  ---------------------
Pav05        10.300.300.105  /nfs/GB000xxxxxbbf97        true     true      false  false  Not Supported
Pav06        10.300.300.106  /nfs/GB000xxxxxbbf93        true     true      false  false  Not Supported
Pav01        10.300.300.101  /nfs/GB000xxxxxbbf95        true     true      false  false  Not Supported

In this case there are three datastores that I haven’t been able to unmount.

[[email protected]:~] esxcli storage nfs remove -v Pav05
[[email protected]:~] esxcli storage nfs remove -v Pav06
[[email protected]:~] esxcli storage nfs remove -v Pav01

Now there should be no volumes mounted on the host.

[[email protected]:~] esxcli storage nfs list
[[email protected]:~]

See, I told you it would be quick.

VMware vSphere and NFS – Some Links

Most of my experience with vSphere storage has revolved around various block storage technologies, such as DAS, FC and iSCSI. I recently began an evaluation of one of those fresh new storage startups running an NVMe-based system. We didn’t have the infrastructure to support NVMe-oF in our lab, so we’ve used NFS to connect the datastores to our vSphere environment. Obviously, at this point, it is less about maximum performance and more about basic functionality. In any case, I thought it might be useful to include a series of links regarding NFS and vSphere that I’ve been using to both get up and running, and troubleshoot some minor issues we had getting everything running. Note that most of these links cover vSphere 6.5, as our lab is currently running that version.

Basics

Create an NFS Datastore

How to add NFS export to VMware ESXi 6.5

NFS Protocols and ESXi

Best Practice

Best Practices for running VMware vSphere on Network Attached Storage

Troubleshooting

Maximum supported volumes reached (1020652)

Increasing the default value that defines the maximum number of NFS mounts on an ESXi/ESX host (2239)

Troubleshooting connectivity issues to an NFS datastore on ESX and ESXi hosts (1003967)

VMware – VMware HealthAnalyzer vha.properties

I was using VMware’s HealthAnalyzer tool (version 5.2.0) recently to perform a vSphere health check for a customer and encountered the following error when using a read-only account.

A service error during during collection” (you might also see “A runtime error occurred during collection” pop up).

In addition to the Read-Only permissions to the vCenter user account, you need to assign “Profile-driven storage > Profile-driven storage view” privileges to the user account in order to collect Storage Policy data. If, for some reason, you can’t do that (I was working with a third-party in this case), you need to edit the vha.properties file. This is located at:

<VHA_Instance>/usr/share/vha/tomcat/webapps/vha/WEB-INF/classes/vha.properties

You’ll need to use vi to set the following properties to false:

collection.storagepolicies.enabled
collection.iscsiport.enabled

Note that by doing so some things won’t be scanned and some recommendations won’t be made.

VMware – vSphere Basics – vCenter 6.5 Upgrade Scenarios

I did an article on the vSphere 6 Platform Services Controller a while ago. After attending a session on changes in vSphere 6.5 at vFORUM, I thought it would be an idea to revisit this, and frame it in the context of vCenter 6.5 upgrades.

 

vSphere Components

In vCenter 6.5, the architecture is a bit different to 5.x. With the PSC, you get:

  • VMware vCenter Single Sign-On
  • License service
  • Lookup service
  • VMware Directory Services
  • VMware Certificate Authority

And the vCenter Server Service gives you:

  • vCenter Server
  • VMware vSphere Web Client
  • VMware vSphere Auto Deploy
  • VMware vSphere ESXi Dump Collector
  • vSphere Syslog Collector on Windows and vSphere Syslog Service for VMware vCenter Server Appliance
  • vSphere Update Manager

 

Architecture Choices

There are some basic configurations that you can go with, but I generally don’t recommend these for anything outside of a lab or test environment. In these configurations, the PSC is either embedded or external to the vCenter Server. The choice here will be dependent on the sizing and feature requirements of your environment.

If you want to use Enhanced Linked Mode an external PSC is recommended. If you want it highly available, you’ll still need to use a load balancer. This VMware KB  article provides some handy insights and updates from 6.0.

 

vCenter Upgrade Scenarios

Your upgrade architecture you’ll choose depends on where your vCenter services reside. If your vCenter server has SSO installed, it becomes a vCenter Server with an embedded PSC.

If, however, some of the vSphere components are installed on separate VMs then the Web Client and Inventory Service become part of the “Management Node” (your vCenter box) and the PSC (with SSO) is separate/external.

Note also that vSphere 6.5 still requires a load balancer for vSphere High Availability.

 

Final Thoughts

This is not something that’s necessarily going to come up each day. But if you’re working either directly with VMware, via an integrator or doing it yourself, your choice of vCenter architecture should be a key consideration in your planning activities. As with most upgrades to key infrastructure components, you should take the time to plan appropriately.

VMware vSphere Next Beta Applications Are Now Open

VMware recently announced that applications for the next VMware vSphere Beta Program are now open. People wishing to participate in the program can now indicate their interest by filling out this simple form. The vSphere team will grant access to the program to selected candidates in stages. This vSphere Beta Program leverages a private Beta community to download software and share information. There will be discussion forums, webinars, and service requests to enable you to share your feedback with VMware.

So what’s involved? Participants are expected to:

  • Accept the Master Software Beta Test Agreement prior to visiting the Private Beta Community;
  • Install beta software within 3 days of receiving access to the beta product;
  • Provide feedback within the first 4 weeks of the beta program;
  • Submit Support Requests for bugs, issues and feature requests;
  • Complete surveys and beta test assignments; and
  • Participate in the private beta discussion forum and conference calls.

All testing is free-form and you’re encouraged to use the software in ways that interest you. This will provide VMware with valuable insight into how you use vSphere in real-world conditions and with real-world test cases.

Why participate? Some of the many reasons to participate include:

  • Receiving early access to the vSphere Beta products;
  • Interacting with the vSphere Beta team consisting of Product Managers, Engineers, Technical Support, and Technical Writers;
  • Providing direct input on product functionality, configurability, usability, and performance;
  • Providing feedback influencing future products, training, documentation, and services; and
  • Collaborating with other participants, learning about their use cases, and sharing advice and learnings.

I’m a big fan of public beta testing. While we’re not all experts on how things should work, it’s a great opportunity to at least have your say on how you think that vSphere should work. While the guys in vSphere product management may not be able to incorporate every idea you have for how vSphere should work, you’ll at least have an opportunity to contribute feedback and give VMware some insight on how their product is being used in the wild. In my opinion this is extremely valuable for both VMware and us, the consumers of their product. Plus, you’ll get a sneak peak into what’s coming up.

So, if you’re good with NDAs and have some time to devote to some testing of next-generation vSphere, this is the program for you. So head over to the website and check it out.

VMware – vSphere 6 Basics – Platform Services Controller

I’ve finally gotten some time to dig into the changes in vSphere 6 with regards to deployment options and architecture. I thought I’d do a few posts covering some key enhancements from VMware, paying particular attention to the Platform Service Controller (PSC) and VMware’s preferred deployment options. I haven’t received any briefings from VMware, so I can’t comment on what is coming in future releases. Note that most of this information was made available to me via access to VMware’s partner program, and I think it’s important that more people understand what’s going on when it comes to PSC and how it works.

 

vSphere Components

The PSC is a new feature in vSphere 6.0. As background, I recommend you first check out this blog post – vCenter Server 6 Deployment Topologies and High Availability. There is also an excellent FAQ from VMware available here. I thought, before diving too much into PSC deployment options, it’s a good idea to revisit VMware’s semi-new approach to vSphere components.

The PSC contains the following services:

  • VMware vCenter Single Sign-On (SSO);
  • License Service;
  • Lookup Service;
  • VMware Directory Service; and
  • VMware Certificate Authority (CA).

Everything else is now referred to as “vCenter Services”, providing the remainder of the vCenter Server functionality.  This includes:

  • vCenter Server;
  • VMware vSphere Web Client;
  • Inventory Service;
  • vSphere Auto Deploy;
  • VMware vSphere ESXi Dump Collector; and
  • VMware vSphere Syslog Collector (Windows) / VMware Syslog Service (Appliance).

 

Enhanced Linked Mode and PSC Deployment Options

Here are a few different ways you can do it. Some are good, some are bad. VMware has published a list of recommended topologies for VMware vSphere 6.0.x. The following section provides an overview of the options. Note that some of these options aren’t without their issues.

 

Enhanced Linked Mode with an External PSC Without HA

The PSC is configured on a separate VM and then the vCenter Servers are joined to that domain, providing Enhanced Linked mode functionality.

ELM1

 

Enhanced Linked Mode with an External PSC in an HA Configuration

In this case, the PSCs are configured on separate VMs behind a load balancer to provide HA for the configuration. The vCenter Servers are then joined to that domain using the shared load balancer IP address, providing Enhanced Linked mode functionality that is fault-tolerant.

ELM2

And here’s a few ways that you can do it that aren’t really recommended.

 

Enhanced Linked Mode with Embedded PSCs (Not Recommended)

In this scenario, vCenter is installed in an embedded configuration on the first server. Subsequent installations are then configured in embedded mode but joined to an existing SSO domain. Linking the embedded PSCs is possible, but VMware does not recommend this configuration.

ELM3

 

Enhanced Linked Mode in Combination Deployment (Not Recommended)

In a combination deployment, the embedded and external PSC architectures are combined. While linking an embedded PSC and an external PSC is possible, VMware does not recommended this configuration.

ELM4

 

Enhanced Linked Mode using only an Embedded PSC (Not Recommended)

In this case there is an embedded PSC and vCenter Server linked with an external standalone vCenter Server. Linking a second vCenter Server to an existing embedded vCenter Server and PSC is possible, but VMware does not recommended this configuration.

ELM5

 

Sizing Considerations

If you’re not going to use enhanced linked mode, use an embedded PSC. You still have availability via VMware HA. The failure domain is limited to a single vCenter Server, as there is no dependency on external component connectivity for PSC connectivity. This is most suitable for lab environments.

For sites that will use enhanced linked mode use external PSCs.  The number of controllers depends on the size of the environment:

  • Between 2 and 4 VMware solutions – a single PSC for no HA, and 2 will be required for HA configured behind a single load balancer.
  • Between 4 and 8 VMware solutions – two PSCs linked together for no HA, and four will be required for HA configured behind two load balancers (two behind each load balancer).
  • Between 8 and 10 VMware solutions – three PSCs linked together for no HA, and six will be required for HA configured behind three load balancers (two behind each load balancer).

HA is provided by having multiple PSCs and a load balancer to provide failure protection. All components are still protected by VMware HA. This VMware KB has more information on how to set this up – Configuring PSC 6.0 High Availability for vSphere 6.0 using vCenter Server 6.0 Appliance.

 

vCenter Platform Choice

VMware maintain that, with the improvements to the vCenter appliance platform, the choice of Windows-based vs vCenter appliance is now a matter of preference rather than performance. I recommend the appliance wherever possible, but some people will feel more comfortable with a Windows-based platform. The cool thing is that, if you want to make things complicated, the PSC supports mixed-mode (i.e. appliance and Windows-based vCenter deployments).

PSC_mixed

 

Final Thoughts

This may have gone a bit beyond basics, and it’s not something that’s necessarily going to come up each day. But if you’re working either directly with VMware, via an integrator or doing it yourself, this new approach should be a key consideration in your planning activities. The addition of the PSC concept to the vCenter architecture improves the flexibility and availability options of the product, something that I think VMware has struggled with in the past. The key takeaway, in my opinion, is that if you’re upgrading from 5.5 or below, you need to take the time to plan appropriately, particularly if you want to leverage some of the new features that are available.