Category Archives: ESXi

Random Short Take #85

Posted on by

Welcome to Random Short Take #85. Let’s get random.

Random Short Take #74

Posted on by

Welcome to Random Short Take #74. Let’s get random.

Random Short Take #40

Posted on by

Welcome to Random Short Take #40. Quite a few players have worn 40 in the NBA, including the flat-top king Frank Brickowski. But my favourite player to wear number 40 was the Reign Man – Shawn Kemp. So let’s get random.

  • Dell EMC PowerProtect Data Manager 19.5 was released in early July and Preston covered it pretty comprehensively here.
  • Speaking of data protection software releases and enhancements, we’ve barely recovered from the excitement of Veeam v10 being released and Anthony is already talking about v11. More on that here.
  • Speaking of Veeam, Rhys posted a very detailed article on setting up a Veeam backup repository on NFS using a Pure Storage FlashBlade environment.
  • Sticking with the data protection theme, I penned a piece over at Gestalt IT for Druva talking about OneDrive protection and why it’s important.
  • OpenDrives has some new gear available – you can read more about that here.
  • The nice folks at Spectro Cloud recently announced that its first product is generally available. You can read the press release here.
  • Wiliam Lam put out a great article on passing through the integrated GPU on Apple Mac minis with ESXi 7.
  • Time passes on, and Christian recently celebrated 10 years on his blog, which I think is a worthy achievement.

Happy Friday!

Random Short Take #27

Posted on by

Welcome to my semi-regular, random news post in a short format. This is #27. You’d think it would be hard to keep naming them after basketball players, and it is. None of my favourite players ever wore 27, but Marvin Barnes did surface as a really interesting story, particularly when it comes to effective communication with colleagues. Happy holidays too, as I’m pretty sure this will be the last one of these posts I do this year. I’ll try and keep it short, as you’ve probably got stuff to do.

  • This story of serious failure on El Reg had me in stitches.
  • I really enjoyed this article by Raj Dutt (over at Cohesity’s blog) on recovery predictability. As an industry we talk an awful lot about speeds and feeds and supportability, but sometimes I think we forget about keeping it simple and making sure we can get our stuff back as we expect.
  • Speaking of data protection, I wrote some articles for Druva about, well, data protection and things of that nature. You can read them here.
  • There have been some pretty important CBT-related patches released by VMware recently. Anthony has provided a handy summary here.
  • Everything’s an opinion until people actually do it, but I thought this research on cloud adoption from Leaseweb USA was interesting. I didn’t expect to see everyone putting their hands up and saying they’re all in on public cloud, but I was also hopeful that we, as an industry, hadn’t made things as unclear as they seem to be. Yay, hybrid!
  • Site sponsor StorONE has partnered with Tech Data Global Computing Components to offer an All-Flash Array as a Service solution.
  • Backblaze has done a nice job of talking about data protection and cloud storage through the lens of Star Wars.
  • This tip on removing particular formatting in Microsoft Word documents really helped me out recently. Yes I know Word is awful.
  • Someone was nice enough to give me an acknowledgement for helping review a non-fiction book once. Now I’ve managed to get a character named after me in one of John Birmingham’s epics. You can read it out of context here. And if you’re into supporting good authors on Patreon – then check out JB’s page here. He’s a good egg, and his literary contributions to the world have been fantastic over the years. I don’t say this just because we live in the same city either.

VMware – Unmounting NFS Datastores From The CLI

Posted on by

This is a short article, but hopefully useful. I did a brief article a while ago linking to some useful articles about using NFS with VMware vSphere. I recently had to do some maintenance on one of the arrays in our lab and I was having trouble unmounting the datastores using the vSphere client. I used some of the commands in this KB article (although I don’t have SIOC enabled) to get the job done instead.

The first step was to identify if any of the volumes were still mounted on the individual host.

[root@esxihost:~] esxcli storage nfs list
Volume Name  Host            Share                 Accessible  Mounted  Read-Only   isPE  Hardware Acceleration
-----------  --------------  --------------------  ----------  -------  ---------  -----  ---------------------
Pav05        10.300.300.105  /nfs/GB000xxxxxbbf97        true     true      false  false  Not Supported
Pav06        10.300.300.106  /nfs/GB000xxxxxbbf93        true     true      false  false  Not Supported
Pav01        10.300.300.101  /nfs/GB000xxxxxbbf95        true     true      false  false  Not Supported

In this case there are three datastores that I haven’t been able to unmount.

[root@esxihost:~] esxcli storage nfs remove -v Pav05
[root@esxihost:~] esxcli storage nfs remove -v Pav06
[root@esxihost:~] esxcli storage nfs remove -v Pav01

Now there should be no volumes mounted on the host.

[root@esxihost:~] esxcli storage nfs list
[root@esxihost:~]

See, I told you it would be quick.

VMware vSphere and NFS – Some Links

Posted on by

Most of my experience with vSphere storage has revolved around various block storage technologies, such as DAS, FC and iSCSI. I recently began an evaluation of one of those fresh new storage startups running an NVMe-based system. We didn’t have the infrastructure to support NVMe-oF in our lab, so we’ve used NFS to connect the datastores to our vSphere environment. Obviously, at this point, it is less about maximum performance and more about basic functionality. In any case, I thought it might be useful to include a series of links regarding NFS and vSphere that I’ve been using to both get up and running, and troubleshoot some minor issues we had getting everything running. Note that most of these links cover vSphere 6.5, as our lab is currently running that version.

Basics

Create an NFS Datastore

How to add NFS export to VMware ESXi 6.5

NFS Protocols and ESXi

Best Practice

Best Practices for running VMware vSphere on Network Attached Storage

Troubleshooting

Maximum supported volumes reached (1020652)

Increasing the default value that defines the maximum number of NFS mounts on an ESXi/ESX host (2239)

Troubleshooting connectivity issues to an NFS datastore on ESX and ESXi hosts (1003967)

Random Short Take #4

Posted on by

Welcome to the 2017 edition of the Random Short Take. Here are a few links to a few things that I think might be useful, to someone. Maybe.

I’ve been doing some vSphere designs lately, and found these links handy:

I don’t think we’re talking enough about protecting the vCenter Server Appliance. I found these links to be pretty handy.

Need some info on Cisco UCS? Go here.

And if you’re working out power draw in the DC, this might be helpful.

Oracle VM came up in a project I was working on recently. This overview page was a reasonable starting point. Finally, check out Stephen Foskett’s article on ZFS. I thought it was well-balanced and a good read, and the article comments reminded me why I’ve stayed the hell away from that particular community. In any case, if you’re going to be at VMworld US this year, come and say hi.

 

VMware vSphere Next Beta Applications Are Now Open

Posted on by

VMware recently announced that applications for the next VMware vSphere Beta Program are now open. People wishing to participate in the program can now indicate their interest by filling out this simple form. The vSphere team will grant access to the program to selected candidates in stages. This vSphere Beta Program leverages a private Beta community to download software and share information. There will be discussion forums, webinars, and service requests to enable you to share your feedback with VMware.

So what’s involved? Participants are expected to:

  • Accept the Master Software Beta Test Agreement prior to visiting the Private Beta Community;
  • Install beta software within 3 days of receiving access to the beta product;
  • Provide feedback within the first 4 weeks of the beta program;
  • Submit Support Requests for bugs, issues and feature requests;
  • Complete surveys and beta test assignments; and
  • Participate in the private beta discussion forum and conference calls.

All testing is free-form and you’re encouraged to use the software in ways that interest you. This will provide VMware with valuable insight into how you use vSphere in real-world conditions and with real-world test cases.

Why participate? Some of the many reasons to participate include:

  • Receiving early access to the vSphere Beta products;
  • Interacting with the vSphere Beta team consisting of Product Managers, Engineers, Technical Support, and Technical Writers;
  • Providing direct input on product functionality, configurability, usability, and performance;
  • Providing feedback influencing future products, training, documentation, and services; and
  • Collaborating with other participants, learning about their use cases, and sharing advice and learnings.

I’m a big fan of public beta testing. While we’re not all experts on how things should work, it’s a great opportunity to at least have your say on how you think that vSphere should work. While the guys in vSphere product management may not be able to incorporate every idea you have for how vSphere should work, you’ll at least have an opportunity to contribute feedback and give VMware some insight on how their product is being used in the wild. In my opinion this is extremely valuable for both VMware and us, the consumers of their product. Plus, you’ll get a sneak peak into what’s coming up.

So, if you’re good with NDAs and have some time to devote to some testing of next-generation vSphere, this is the program for you. So head over to the website and check it out.

VMware – Joining an ESXi 5.5 Host to Active Directory with PowerCLI

Posted on by

I spoke about restarting some ESXi services when joining a domain in this post. Here’s how you might want to do it with PowerCLI. Firstly, you may need to modify the execution policy for PowerCLI. It’s worth checking out the Microsoft URL in the test here, as it’s very useful background on what you’re actually doing by setting this policy. Also, big thanks to my colleague Michael for coming up with the syntax here, he’s really the brains behind the operation.

 

PowerCLI C:\Windows\system32> Set-ExecutionPolicy RemoteSigned

Execution Policy Change
The execution policy helps protect you from scripts that you do not trust.
Changing the execution policy might expose you to the security risks described
in the about_Execution_Policies help topic at
http://go.microsoft.com/fwlink/?LinkID=135170. Do you want to change the
execution policy?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"):

 

Then when you connect to your vCenter host, you’ll get warned about the certificate. This assumes that you’re not using the right certificates in your environment (why are you like this?).

 

PowerCLI C:\Windows\system32> Connect-VIServer 172.16.200.200
WARNING: There were one or more problems with the server certificate for the
server 172.16.200.200:443:

* The X509 chain could not be built up to the root certificate.

* The certificate's CN name does not match the passed value.

Certificate: [Subject]
  OID.1.2.840.113549.1.9.2="1382062929,d8ba9993,564d7761726520496e632e",
CN=localhost.localdom, E=ssl-certificates@vmware.com, OU=VMware vCenter Server
Certificate, O="VMware, Inc.", L=Palo Alto, S=California, C=US

[Issuer]
  E=ssl-certificates@vmware.com, CN=localhost.localdom CA af1bb298, O="VMware,
Inc.", L=Palo Alto, S=California, C=US

[Serial Number]
  01

[Not Before]
  17/10/2013 12:22:09 PM

[Not After]
  16/10/2023 12:22:10 PM

[Thumbprint]
  4883C2F3DCD6E6F8693200E41BDE2A41A88C3930

The server certificate is not valid.

WARNING: THE DEFAULT BEHAVIOR UPON INVALID SERVER CERTIFICATE WILL CHANGE IN A
FUTURE RELEASE. To ensure scripts are not affected by the change, use
Set-PowerCLIConfiguration to set a value for the InvalidCertificateAction
option.

Name                           Port  User
----                           ----  ----
172.16.200.200                 443   root

 

You can then get down to it. Firstly, you can join the domain with this command.

 

#To Join the domain: 
#get cred for joining the domain
$cred=get-credential

get-vmhost | Get-VMHostAuthentication | Set-VMHostAuthentication -JoinDomain -Domain "network.internal" -Credential $cred

 

Once you’ve done that, you might need to restart those services I spoke about at the end of a previous post.

 

#To restart these pesky services:

Get-VMHost | Get-VMHostService | ?{"lsassd","lwiod","netlogond" -contains $_.Key} | Restart-VMHostService

 

Once you’ve done that, you can check if it’s all working with this command.

 

#check to see if you have any auth issues

get-vmhost | Get-VMHostAuthentication

 

And you should be good to go.

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 5

Posted on by

I’ve come across a few slightly odd things that I hadn’t accounted for during a recent vSphere 5.5 U2 deployment and thought it would be handy to document them. In this post (which is hopefully the last one) I’d like to cover off SSL certificates.

A lot of people don’t bother trying to deploy custom certificates because it invariably involves interaction with an in-house InfoSec team. This can be a royal pain in the arse. I understand completely. That said, getting custom certs into your vSphere environment has become a lot easier in recent times.

Firstly, there’s a few KB articles you should read:

Here’s the output from the Certificate Automation Tool

==================================================================
Main menu

Enter the action you want to run
   1. Plan your steps to update SSL certificates(Update Steps Planner)
   2. Generate Certificate Signing Requests
   3. Update Single Sign-On
   4. Update Inventory Service
   5. Update vCenter Server
   6. Update vCenter Orchestrator(vCO)
   7. Update vSphere Web Client and Log Browser
   8. Update vSphere Update Manager(VUM)
   9. End the update process and exit
The chosen action is: 1

And here’s what the Update Steps Planner gives you to work through.

The chosen action is: 1
==================================================================
1. Plan your steps to update SSL certificates(Update Steps Planner)

Choose the services you want to update:
      1. Single Sign-On
      2. Inventory Service
      3. vCenter Server
      4. vCenter Orchestrator
      5. vSphere Web Client
      6. Log Browser
      7. vSphere Update Manager
      8. All services(listed above)
      9. Return to the main menu

Example:
To choose the certificate update of Inventory Service, vCenter Server and vSphere Web Client you would enter: 2,3,5
You chose (enter comma-separated list of numbers): 8
Input arguments: [8]

Selected services: Single Sign-On, Inventory Service, vCenter Server, vCenter Orchestrator, Web Client, Log Browser, vSphere Update Manager
Detailed Plan to follow:
1. Go to the machine with Single Sign-On installed and - Update the Single Sign-On SSL certificate.
2. Go to the machine with Inventory Service installed and - Update Inventory Service trust to Single Sign-On.
3. Go to the machine with Inventory Service installed and - Update the Inventory Service SSL certificate.
4. Go to the machine with vCenter Server installed and - Update vCenter Server trust to Single Sign-On.
5. Go to the machine with vCenter Server installed and - Update the vCenter Server SSL certificate.
6. Go to the machine with vCenter Server installed and - Update vCenter Server trust to Inventory Service.
7. Go to the machine with Inventory Service installed and - Update the Inventory Service trust to vCenter Server.
8. Go to the machine with vCenter Orchestrator installed and - Update vCenter Orchestrator trust to Single Sign-On.
9. Go to the machine with vCenter Orchestrator installed and - Update vCenter Orchestrator trust to vCenter Server.
10. Go to the machine with vCenter Orchestrator installed and - Update the vCenter Orchestrator SSL certificate.
11. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to Single Sign-On.
12. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to Inventory Service.
13. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to vCenter Server.
14. Go to the machine with vSphere Web Client installed and - Update the vSphere Web Client SSL certificate.
15. Go to the machine with Log Browser installed and - Update the Log Browser trust to Single Sign-On.
16. Go to the machine with Log Browser installed and - Update the Log Browser SSL certificate.
17. Go to the machine with vSphere Update Manager installed and - Update the vSphere Update Manager SSL certificate.
18. Go to the machine with vSphere Update Manager installed and - Update vSphere Update Manager trust to vCenter Server.

And then you have a nice list of stuff to work through. I’m not going to dump the whole process here, but here’s a grab of what updating your vCenter cert looks like.

==================================================================
Main menu

Enter the action you want to run
   1. Plan your steps to update SSL certificates(Update Steps Planner)
   2. Generate Certificate Signing Requests
   3. Update Single Sign-On
   4. Update Inventory Service
   5. Update vCenter Server
   6. Update vCenter Orchestrator(vCO)
   7. Update vSphere Web Client and Log Browser
   8. Update vSphere Update Manager(VUM)
   9. End the update process and exit

The chosen action is: 5
==================================================================
5. Update the vCenter Server SSL Certificate

     1. Update the vCenter Server Trust to Single Sign-On
     2. Update the vCenter Server SSL Certificate
     3. Update the vCenter Server Trust to Inventory Service
     4. Rollback to the previous vCenter Server SSL Certificate
     5. Return to the main menu to update other services

The chosen service is: 2
[Thu 28/05/2015 - 10:39:54.86]: The services that are restarted as a part of this operation are: VMware VirtualCenter Server, VMware VirtualCenter Management Webservices and VMware vSphere Profile-Driven Storage Service.
Enter location to the new vCenter Server SSL chain: C:\Install\ssl-certificate-updater-tool-1308332\vCenterServer-VC4002\chain.pem
Enter location to the new vCenter Server private key: C:\Install\ssl-certificate-updater-tool-1308332\vCenterServer-VC4002\rui.key
Enter vCenter Server administrator user name: domain\svc_vmware
Enter vCenter Server administrator password (will not be echoed):
"Important: Enter the password carefully. The Certificate Automation Update Tool does not check the validity of the vCenter Server database password."
"A blank or incorrect password will leave the system in an inconsistent state, which will cause the vCenter Server to become unavailable. "
"If the system becomes unstable due to a bad password, see the Troubleshooting Section of KB 2041600."
Enter the vCenter Server original database password (will not be echoed):
Enter Single Sign-On Administrator user: Administrator@vsphere.local
Enter Single Sign-On Administrator password (will not be echoed):
[.] WARNING: Certificate's `CN=VC4002.racqgroup.local, OU=vCenterServer-VC4002, O=Company, L=Location, ST=QLD, C=AU' signature uses weak one-way h
ash (SHA-1). In a secure environment it is recommended to use SHA2-256 or a stronger hash algorithm.
[.] The supplied certificate chain is valid.
Loading 'screen' into random state - done
"Restarting services... (This can take some time)"
"Stopping vCenter Web Services..."
"Stopping vCenter Server..."
"Starting vCenter Server and other services..."
[Thu 28/05/2015 - 10:45:42.32]: Last operation update vCenter Server SSL certificate completed successfully.
[Thu 28/05/2015 - 10:45:42.33]: Go to the next step in the plan that was received from Update Steps Planner.

Once you’ve had your way with vCenter, etc, you can do your ESXi hosts. The following link has info on that – Configuring CA signed certificates for ESXi 5.x hosts, and you can grab the appropriate version of Win32 OpenSSL from here. Here’s what it looks like when you use OpenSSL to generate the requests for your ESXi hosts.

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.
C:\Users\Player1>cd \
C:\>cd OpenSSL\bin
C:\OpenSSL\bin>openssl req -new -nodes -out rui.csr -keyout rui-orig.key -config
openssl.cfg
Loading 'screen' into random state - done
Generating a 2048 bit RSA private key
........+++
..........................................+++
writing new private key to 'rui-orig.key'
-----
C:\OpenSSL\bin>openssl rsa -in rui-orig.key -out rui.key
writing RSA key
C:\OpenSSL\bin>

One thing to note. I found that HA got a bit irritable until all hosts in the cluster had custom certs installed. So it’s worth turning HA off until you’re finished. If, for some reason something goes wrong wit the ESXi certs, you can re-generate the default self-signed ones with the following command:

/sbin/generate-certificates

 

Updates In some of my previous posts, I talked about a few things that I had to do to get things working. In this post, I discussed the “Missing VMware Tools ISO”. I still don’t know why the tools files were missing from the installation, but I do know that once we applied some more recent vSphere Update Manager baselines to those hosts the correct ISO files were added to the hosts.

I also covered “HP Legacy BIOS Mode and ESXi” in this post. Interestingly, you’ll need to change back to UEFI BIOS mode if you’re trying to make VirtualConnect changes to a host, as my client found out the hard way.

I also spoke about ESXi hosts and Active Directory authentication in this post. I should point out that this post by Joseph also came in handy. If you find that when you restart the services on the host it bombs out, you’ll need to manually create /var/lock/subsys. There’s a KB article from VMware that says the same thing here.

mkdir /var/lock/subsys
/etc/init.d/netlogond restart
/etc/init.d/lwiod restart
/etc/init.d/lsassd restart

And you should then be right.