Random Short Take #40

Welcome to Random Short Take #40. Quite a few players have worn 40 in the NBA, including the flat-top king Frank Brickowski. But my favourite player to wear number 40 was the Reign Man – Shawn Kemp. So let’s get random.

  • Dell EMC PowerProtect Data Manager 19.5 was released in early July and Preston covered it pretty comprehensively here.
  • Speaking of data protection software releases and enhancements, we’ve barely recovered from the excitement of Veeam v10 being released and Anthony is already talking about v11. More on that here.
  • Speaking of Veeam, Rhys posted a very detailed article on setting up a Veeam backup repository on NFS using a Pure Storage FlashBlade environment.
  • Sticking with the data protection theme, I penned a piece over at Gestalt IT for Druva talking about OneDrive protection and why it’s important.
  • OpenDrives has some new gear available – you can read more about that here.
  • The nice folks at Spectro Cloud recently announced that its first product is generally available. You can read the press release here.
  • Wiliam Lam put out a great article on passing through the integrated GPU on Apple Mac minis with ESXi 7.
  • Time passes on, and Christian recently celebrated 10 years on his blog, which I think is a worthy achievement.

Happy Friday!

Random Short Take #27

Welcome to my semi-regular, random news post in a short format. This is #27. You’d think it would be hard to keep naming them after basketball players, and it is. None of my favourite players ever wore 27, but Marvin Barnes did surface as a really interesting story, particularly when it comes to effective communication with colleagues. Happy holidays too, as I’m pretty sure this will be the last one of these posts I do this year. I’ll try and keep it short, as you’ve probably got stuff to do.

  • This story of serious failure on El Reg had me in stitches.
  • I really enjoyed this article by Raj Dutt (over at Cohesity’s blog) on recovery predictability. As an industry we talk an awful lot about speeds and feeds and supportability, but sometimes I think we forget about keeping it simple and making sure we can get our stuff back as we expect.
  • Speaking of data protection, I wrote some articles for Druva about, well, data protection and things of that nature. You can read them here.
  • There have been some pretty important CBT-related patches released by VMware recently. Anthony has provided a handy summary here.
  • Everything’s an opinion until people actually do it, but I thought this research on cloud adoption from Leaseweb USA was interesting. I didn’t expect to see everyone putting their hands up and saying they’re all in on public cloud, but I was also hopeful that we, as an industry, hadn’t made things as unclear as they seem to be. Yay, hybrid!
  • Site sponsor StorONE has partnered with Tech Data Global Computing Components to offer an All-Flash Array as a Service solution.
  • Backblaze has done a nice job of talking about data protection and cloud storage through the lens of Star Wars.
  • This tip on removing particular formatting in Microsoft Word documents really helped me out recently. Yes I know Word is awful.
  • Someone was nice enough to give me an acknowledgement for helping review a non-fiction book once. Now I’ve managed to get a character named after me in one of John Birmingham’s epics. You can read it out of context here. And if you’re into supporting good authors on Patreon – then check out JB’s page here. He’s a good egg, and his literary contributions to the world have been fantastic over the years. I don’t say this just because we live in the same city either.

VMware – Unmounting NFS Datastores From The CLI

This is a short article, but hopefully useful. I did a brief article a while ago linking to some useful articles about using NFS with VMware vSphere. I recently had to do some maintenance on one of the arrays in our lab and I was having trouble unmounting the datastores using the vSphere client. I used some of the commands in this KB article (although I don’t have SIOC enabled) to get the job done instead.

The first step was to identify if any of the volumes were still mounted on the individual host.

[[email protected]:~] esxcli storage nfs list
Volume Name  Host            Share                 Accessible  Mounted  Read-Only   isPE  Hardware Acceleration
-----------  --------------  --------------------  ----------  -------  ---------  -----  ---------------------
Pav05        10.300.300.105  /nfs/GB000xxxxxbbf97        true     true      false  false  Not Supported
Pav06        10.300.300.106  /nfs/GB000xxxxxbbf93        true     true      false  false  Not Supported
Pav01        10.300.300.101  /nfs/GB000xxxxxbbf95        true     true      false  false  Not Supported

In this case there are three datastores that I haven’t been able to unmount.

[[email protected]:~] esxcli storage nfs remove -v Pav05
[[email protected]:~] esxcli storage nfs remove -v Pav06
[[email protected]:~] esxcli storage nfs remove -v Pav01

Now there should be no volumes mounted on the host.

[[email protected]:~] esxcli storage nfs list
[[email protected]:~]

See, I told you it would be quick.

VMware vSphere and NFS – Some Links

Most of my experience with vSphere storage has revolved around various block storage technologies, such as DAS, FC and iSCSI. I recently began an evaluation of one of those fresh new storage startups running an NVMe-based system. We didn’t have the infrastructure to support NVMe-oF in our lab, so we’ve used NFS to connect the datastores to our vSphere environment. Obviously, at this point, it is less about maximum performance and more about basic functionality. In any case, I thought it might be useful to include a series of links regarding NFS and vSphere that I’ve been using to both get up and running, and troubleshoot some minor issues we had getting everything running. Note that most of these links cover vSphere 6.5, as our lab is currently running that version.


Create an NFS Datastore

How to add NFS export to VMware ESXi 6.5

NFS Protocols and ESXi

Best Practice

Best Practices for running VMware vSphere on Network Attached Storage


Maximum supported volumes reached (1020652)

Increasing the default value that defines the maximum number of NFS mounts on an ESXi/ESX host (2239)

Troubleshooting connectivity issues to an NFS datastore on ESX and ESXi hosts (1003967)

Random Short Take #4

Welcome to the 2017 edition of the Random Short Take. Here are a few links to a few things that I think might be useful, to someone. Maybe.

I’ve been doing some vSphere designs lately, and found these links handy:

I don’t think we’re talking enough about protecting the vCenter Server Appliance. I found these links to be pretty handy.

Need some info on Cisco UCS? Go here.

And if you’re working out power draw in the DC, this might be helpful.

Oracle VM came up in a project I was working on recently. This overview page was a reasonable starting point. Finally, check out Stephen Foskett’s article on ZFS. I thought it was well-balanced and a good read, and the article comments reminded me why I’ve stayed the hell away from that particular community. In any case, if you’re going to be at VMworld US this year, come and say hi.


VMware vSphere Next Beta Applications Are Now Open

VMware recently announced that applications for the next VMware vSphere Beta Program are now open. People wishing to participate in the program can now indicate their interest by filling out this simple form. The vSphere team will grant access to the program to selected candidates in stages. This vSphere Beta Program leverages a private Beta community to download software and share information. There will be discussion forums, webinars, and service requests to enable you to share your feedback with VMware.

So what’s involved? Participants are expected to:

  • Accept the Master Software Beta Test Agreement prior to visiting the Private Beta Community;
  • Install beta software within 3 days of receiving access to the beta product;
  • Provide feedback within the first 4 weeks of the beta program;
  • Submit Support Requests for bugs, issues and feature requests;
  • Complete surveys and beta test assignments; and
  • Participate in the private beta discussion forum and conference calls.

All testing is free-form and you’re encouraged to use the software in ways that interest you. This will provide VMware with valuable insight into how you use vSphere in real-world conditions and with real-world test cases.

Why participate? Some of the many reasons to participate include:

  • Receiving early access to the vSphere Beta products;
  • Interacting with the vSphere Beta team consisting of Product Managers, Engineers, Technical Support, and Technical Writers;
  • Providing direct input on product functionality, configurability, usability, and performance;
  • Providing feedback influencing future products, training, documentation, and services; and
  • Collaborating with other participants, learning about their use cases, and sharing advice and learnings.

I’m a big fan of public beta testing. While we’re not all experts on how things should work, it’s a great opportunity to at least have your say on how you think that vSphere should work. While the guys in vSphere product management may not be able to incorporate every idea you have for how vSphere should work, you’ll at least have an opportunity to contribute feedback and give VMware some insight on how their product is being used in the wild. In my opinion this is extremely valuable for both VMware and us, the consumers of their product. Plus, you’ll get a sneak peak into what’s coming up.

So, if you’re good with NDAs and have some time to devote to some testing of next-generation vSphere, this is the program for you. So head over to the website and check it out.

VMware – Joining an ESXi 5.5 Host to Active Directory with PowerCLI

I spoke about restarting some ESXi services when joining a domain in this post. Here’s how you might want to do it with PowerCLI. Firstly, you may need to modify the execution policy for PowerCLI. It’s worth checking out the Microsoft URL in the test here, as it’s very useful background on what you’re actually doing by setting this policy. Also, big thanks to my colleague Michael for coming up with the syntax here, he’s really the brains behind the operation.


PowerCLI C:\Windows\system32> Set-ExecutionPolicy RemoteSigned

Execution Policy Change
The execution policy helps protect you from scripts that you do not trust.
Changing the execution policy might expose you to the security risks described
in the about_Execution_Policies help topic at
http://go.microsoft.com/fwlink/?LinkID=135170. Do you want to change the
execution policy?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"):


Then when you connect to your vCenter host, you’ll get warned about the certificate. This assumes that you’re not using the right certificates in your environment (why are you like this?).


PowerCLI C:\Windows\system32> Connect-VIServer
WARNING: There were one or more problems with the server certificate for the

* The X509 chain could not be built up to the root certificate.

* The certificate's CN name does not match the passed value.

Certificate: [Subject]
CN=localhost.localdom, [email protected], OU=VMware vCenter Server
Certificate, O="VMware, Inc.", L=Palo Alto, S=California, C=US

  [email protected], CN=localhost.localdom CA af1bb298, O="VMware,
Inc.", L=Palo Alto, S=California, C=US

[Serial Number]

[Not Before]
  17/10/2013 12:22:09 PM

[Not After]
  16/10/2023 12:22:10 PM


The server certificate is not valid.

FUTURE RELEASE. To ensure scripts are not affected by the change, use
Set-PowerCLIConfiguration to set a value for the InvalidCertificateAction

Name                           Port  User
----                           ----  ----                 443   root


You can then get down to it. Firstly, you can join the domain with this command.


#To Join the domain: 
#get cred for joining the domain

get-vmhost | Get-VMHostAuthentication | Set-VMHostAuthentication -JoinDomain -Domain "network.internal" -Credential $cred


Once you’ve done that, you might need to restart those services I spoke about at the end of a previous post.


#To restart these pesky services:

Get-VMHost | Get-VMHostService | ?{"lsassd","lwiod","netlogond" -contains $_.Key} | Restart-VMHostService


Once you’ve done that, you can check if it’s all working with this command.


#check to see if you have any auth issues

get-vmhost | Get-VMHostAuthentication


And you should be good to go.

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 5

I’ve come across a few slightly odd things that I hadn’t accounted for during a recent vSphere 5.5 U2 deployment and thought it would be handy to document them. In this post (which is hopefully the last one) I’d like to cover off SSL certificates.

A lot of people don’t bother trying to deploy custom certificates because it invariably involves interaction with an in-house InfoSec team. This can be a royal pain in the arse. I understand completely. That said, getting custom certs into your vSphere environment has become a lot easier in recent times.

Firstly, there’s a few KB articles you should read:

Here’s the output from the Certificate Automation Tool

Main menu

Enter the action you want to run
   1. Plan your steps to update SSL certificates(Update Steps Planner)
   2. Generate Certificate Signing Requests
   3. Update Single Sign-On
   4. Update Inventory Service
   5. Update vCenter Server
   6. Update vCenter Orchestrator(vCO)
   7. Update vSphere Web Client and Log Browser
   8. Update vSphere Update Manager(VUM)
   9. End the update process and exit
The chosen action is: 1

And here’s what the Update Steps Planner gives you to work through.

The chosen action is: 1
1. Plan your steps to update SSL certificates(Update Steps Planner)

Choose the services you want to update:
      1. Single Sign-On
      2. Inventory Service
      3. vCenter Server
      4. vCenter Orchestrator
      5. vSphere Web Client
      6. Log Browser
      7. vSphere Update Manager
      8. All services(listed above)
      9. Return to the main menu

To choose the certificate update of Inventory Service, vCenter Server and vSphere Web Client you would enter: 2,3,5
You chose (enter comma-separated list of numbers): 8
Input arguments: [8]

Selected services: Single Sign-On, Inventory Service, vCenter Server, vCenter Orchestrator, Web Client, Log Browser, vSphere Update Manager
Detailed Plan to follow:
1. Go to the machine with Single Sign-On installed and - Update the Single Sign-On SSL certificate.
2. Go to the machine with Inventory Service installed and - Update Inventory Service trust to Single Sign-On.
3. Go to the machine with Inventory Service installed and - Update the Inventory Service SSL certificate.
4. Go to the machine with vCenter Server installed and - Update vCenter Server trust to Single Sign-On.
5. Go to the machine with vCenter Server installed and - Update the vCenter Server SSL certificate.
6. Go to the machine with vCenter Server installed and - Update vCenter Server trust to Inventory Service.
7. Go to the machine with Inventory Service installed and - Update the Inventory Service trust to vCenter Server.
8. Go to the machine with vCenter Orchestrator installed and - Update vCenter Orchestrator trust to Single Sign-On.
9. Go to the machine with vCenter Orchestrator installed and - Update vCenter Orchestrator trust to vCenter Server.
10. Go to the machine with vCenter Orchestrator installed and - Update the vCenter Orchestrator SSL certificate.
11. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to Single Sign-On.
12. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to Inventory Service.
13. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to vCenter Server.
14. Go to the machine with vSphere Web Client installed and - Update the vSphere Web Client SSL certificate.
15. Go to the machine with Log Browser installed and - Update the Log Browser trust to Single Sign-On.
16. Go to the machine with Log Browser installed and - Update the Log Browser SSL certificate.
17. Go to the machine with vSphere Update Manager installed and - Update the vSphere Update Manager SSL certificate.
18. Go to the machine with vSphere Update Manager installed and - Update vSphere Update Manager trust to vCenter Server.

And then you have a nice list of stuff to work through. I’m not going to dump the whole process here, but here’s a grab of what updating your vCenter cert looks like.

Main menu

Enter the action you want to run
   1. Plan your steps to update SSL certificates(Update Steps Planner)
   2. Generate Certificate Signing Requests
   3. Update Single Sign-On
   4. Update Inventory Service
   5. Update vCenter Server
   6. Update vCenter Orchestrator(vCO)
   7. Update vSphere Web Client and Log Browser
   8. Update vSphere Update Manager(VUM)
   9. End the update process and exit

The chosen action is: 5
5. Update the vCenter Server SSL Certificate

     1. Update the vCenter Server Trust to Single Sign-On
     2. Update the vCenter Server SSL Certificate
     3. Update the vCenter Server Trust to Inventory Service
     4. Rollback to the previous vCenter Server SSL Certificate
     5. Return to the main menu to update other services

The chosen service is: 2
[Thu 28/05/2015 - 10:39:54.86]: The services that are restarted as a part of this operation are: VMware VirtualCenter Server, VMware VirtualCenter Management Webservices and VMware vSphere Profile-Driven Storage Service.
Enter location to the new vCenter Server SSL chain: C:\Install\ssl-certificate-updater-tool-1308332\vCenterServer-VC4002\chain.pem
Enter location to the new vCenter Server private key: C:\Install\ssl-certificate-updater-tool-1308332\vCenterServer-VC4002\rui.key
Enter vCenter Server administrator user name: domain\svc_vmware
Enter vCenter Server administrator password (will not be echoed):
"Important: Enter the password carefully. The Certificate Automation Update Tool does not check the validity of the vCenter Server database password."
"A blank or incorrect password will leave the system in an inconsistent state, which will cause the vCenter Server to become unavailable. "
"If the system becomes unstable due to a bad password, see the Troubleshooting Section of KB 2041600."
Enter the vCenter Server original database password (will not be echoed):
Enter Single Sign-On Administrator user: [email protected]
Enter Single Sign-On Administrator password (will not be echoed):
[.] WARNING: Certificate's `CN=VC4002.racqgroup.local, OU=vCenterServer-VC4002, O=Company, L=Location, ST=QLD, C=AU' signature uses weak one-way h
ash (SHA-1). In a secure environment it is recommended to use SHA2-256 or a stronger hash algorithm.
[.] The supplied certificate chain is valid.
Loading 'screen' into random state - done
"Restarting services... (This can take some time)"
"Stopping vCenter Web Services..."
"Stopping vCenter Server..."
"Starting vCenter Server and other services..."
[Thu 28/05/2015 - 10:45:42.32]: Last operation update vCenter Server SSL certificate completed successfully.
[Thu 28/05/2015 - 10:45:42.33]: Go to the next step in the plan that was received from Update Steps Planner.

Once you’ve had your way with vCenter, etc, you can do your ESXi hosts. The following link has info on that – Configuring CA signed certificates for ESXi 5.x hosts, and you can grab the appropriate version of Win32 OpenSSL from here. Here’s what it looks like when you use OpenSSL to generate the requests for your ESXi hosts.

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.
C:\Users\Player1>cd \
C:\>cd OpenSSL\bin
C:\OpenSSL\bin>openssl req -new -nodes -out rui.csr -keyout rui-orig.key -config
Loading 'screen' into random state - done
Generating a 2048 bit RSA private key
writing new private key to 'rui-orig.key'
C:\OpenSSL\bin>openssl rsa -in rui-orig.key -out rui.key
writing RSA key

One thing to note. I found that HA got a bit irritable until all hosts in the cluster had custom certs installed. So it’s worth turning HA off until you’re finished. If, for some reason something goes wrong wit the ESXi certs, you can re-generate the default self-signed ones with the following command:



Updates In some of my previous posts, I talked about a few things that I had to do to get things working. In this post, I discussed the “Missing VMware Tools ISO”. I still don’t know why the tools files were missing from the installation, but I do know that once we applied some more recent vSphere Update Manager baselines to those hosts the correct ISO files were added to the hosts.

I also covered “HP Legacy BIOS Mode and ESXi” in this post. Interestingly, you’ll need to change back to UEFI BIOS mode if you’re trying to make VirtualConnect changes to a host, as my client found out the hard way.

I also spoke about ESXi hosts and Active Directory authentication in this post. I should point out that this post by Joseph also came in handy. If you find that when you restart the services on the host it bombs out, you’ll need to manually create /var/lock/subsys. There’s a KB article from VMware that says the same thing here.

mkdir /var/lock/subsys
/etc/init.d/netlogond restart
/etc/init.d/lwiod restart
/etc/init.d/lsassd restart

And you should then be right.

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 4

I’ve been covering a few workarounds, mishaps and random things I’ve had to do during a recent vSphere 5.5 U2 deployment. This is Part 4 in the series, and I hope some of it is useful. You can read my other posts here, here, and here.


Client Integration Plug-in for vSphere Web Client 

Love it or hate it, the vSphere Web Client is here to stay. If, for some reason, you’re logged into a host with credentials that you want to use to log in to your vSphere environment with, you can use pass-through authentication if you install the Client Integration Plug-in for vSphere Web Client. You can get details on how to do that here.


HP Legacy BIOS Mode and ESXi

This was my first time using BL460c Generation 9 blades with ESXi. While I’ve been around the block with HP blades in the past, I’ve never used them with the SD card option before. I thought this was to blame when I’d reboot the hosts and configuration items (such as persistent scratch location, syslog configuration and core dump details) would disappear. Added to this, the networking configuration on vmk0 would disappear from time to time as well. I was also getting errors such as this when applying host profiles to hosts:

“Call “HostProfileManager.GenerateConfigTaskList” for object “HostProfileManager” on vCenter Server “hostname.domain” failed.

Failed to execute command to configure or query coredump partition.”

I did some searching and chanced upon this article BL460c Gen9 + ESXi 5.5 – Special procedure when using UEFI? Seems that setting the host’s Boot Mode to Legacy BIOS Mode makes for a happier installation and on-going experience. The guy who installed the blades had set them to Legacy mode for the installation and then set them back to UEFI. I can’t tell you why this needed to occur, nor can I tell you the disadvantages of taking this approach.




Windows 2012 R2 and .Net 3.5 

If you’re running your VMware applications on Windows 2012 R2, there’s a chance you’ll need to install .Net 3.5 on your guest to get things working. This is handled via Server Roles. Microsoft has a TechNet article on how to do it here. Note that you’ll need your Windows installation media, and you’ll likely need to specify an alternate source – %CDROM%\sources\sxs.




Okay, so hopefully that was useful for someone. More to follow …



VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 3

This is my third post in a series of articles on some workarounds and things I had to look into when doing a recent vSphere 5.5 U2 deployment. You In can find my previous articles here and here. In this episode I’m covering EVC, Host Profile Compliance Checks and ESXi Hosts and Active Directory Authentication.


VMware Enhanced vMotion Compatibility (EVC)

Wondering which EVC Intel mode (Merom, Penryn, Nehalem, Westmere, Sandy Bridge, Ivy Bridge) to use with your vCenter cluster? It depends. This KB article provides a good outline of your options. Note that in vCenter Server 5.1 and 5.5, the Intel “Ivy Bridge” Generation option is only displayed in the Web Client. That’s the man trying to keep you down ;)


But how do I set EVC on the cluster when vCenter is virtual and running in the cluster? As the cluster is no longer the boundary for vMotion, one way to do this is to create a new empty cluster. Add your first host and setup as appropriate. Then enable EVC and vMotion the first guest into the cluster and you’ll be good to go. If you can’t vMotion across clusters because VMs are using various features of the CPU (a more likely scenario), you’ll need to use the method outlined in the following article – Enabling EVC on a cluster when vCenter Server is running in a virtual machine. It’s a bit of a pain, particularly if you’re using Distributed vSwitch, but it works well enough. And when VMware say they recommend you change your VM to standard vSwitch – it’s a good idea to take their advice.


VMware also have a pretty useful FAQ on EVC and CPU Compatibility that you can access here.


Host Profile Compliance Checks

If you’re running Host Profiles at the cluster level, you may find that even if the host is compliant, it fails on Fault Tolerance checks. If you’re not using FT, disable those checks. Because we all live for green lights. Right-click the cluster and click Edit Settings > VMware HA > Advanced Options. You’ll need to add in a field and set it to false. Details on how to do this can be found here.


ESXi Hosts and Active Directory Authentication

Want to join your ESXi host to an Active Directory domain? Good idea. You’ll need this KB article. Be sure you’ve got Config.HostAgent.plugins.hostsvc.esxAdminsGroup set correctly, or you’ll have a difficult time getting in with your AD credentials. If you have issues, you can try restarting LDAP or forcing an update on the DC that you configured the ESXi host to look at. I found this article useful.


Okay, so hopefully that was useful for someone. More to follow …