Random Short Take #85

Welcome to Random Short Take #85. Let’s get random.

Random Short Take #73

Welcome to Random Short Take #73. Let’s get random.

VMware – Unmounting NFS Datastores From The CLI

This is a short article, but hopefully useful. I did a brief article a while ago linking to some useful articles about using NFS with VMware vSphere. I recently had to do some maintenance on one of the arrays in our lab and I was having trouble unmounting the datastores using the vSphere client. I used some of the commands in this KB article (although I don’t have SIOC enabled) to get the job done instead.

The first step was to identify if any of the volumes were still mounted on the individual host.

[root@esxihost:~] esxcli storage nfs list
Volume Name  Host            Share                 Accessible  Mounted  Read-Only   isPE  Hardware Acceleration
-----------  --------------  --------------------  ----------  -------  ---------  -----  ---------------------
Pav05        10.300.300.105  /nfs/GB000xxxxxbbf97        true     true      false  false  Not Supported
Pav06        10.300.300.106  /nfs/GB000xxxxxbbf93        true     true      false  false  Not Supported
Pav01        10.300.300.101  /nfs/GB000xxxxxbbf95        true     true      false  false  Not Supported

In this case there are three datastores that I haven’t been able to unmount.

[root@esxihost:~] esxcli storage nfs remove -v Pav05
[root@esxihost:~] esxcli storage nfs remove -v Pav06
[root@esxihost:~] esxcli storage nfs remove -v Pav01

Now there should be no volumes mounted on the host.

[root@esxihost:~] esxcli storage nfs list
[root@esxihost:~]

See, I told you it would be quick.

VMware vSphere and NFS – Some Links

Most of my experience with vSphere storage has revolved around various block storage technologies, such as DAS, FC and iSCSI. I recently began an evaluation of one of those fresh new storage startups running an NVMe-based system. We didn’t have the infrastructure to support NVMe-oF in our lab, so we’ve used NFS to connect the datastores to our vSphere environment. Obviously, at this point, it is less about maximum performance and more about basic functionality. In any case, I thought it might be useful to include a series of links regarding NFS and vSphere that I’ve been using to both get up and running, and troubleshoot some minor issues we had getting everything running. Note that most of these links cover vSphere 6.5, as our lab is currently running that version.

Basics

Create an NFS Datastore

How to add NFS export to VMware ESXi 6.5

NFS Protocols and ESXi

Best Practice

Best Practices for running VMware vSphere on Network Attached Storage

Troubleshooting

Maximum supported volumes reached (1020652)

Increasing the default value that defines the maximum number of NFS mounts on an ESXi/ESX host (2239)

Troubleshooting connectivity issues to an NFS datastore on ESX and ESXi hosts (1003967)

Ravello – Basics – Deploying an ESXi instance

I’ve been using Ravello a bit recently, thanks primarily to their kind offer of free time for vExperts. I thought it would be worth while doing a few posts on what you need to do to get started. While this information is available via a number of sources already, I thought I’d update it a little to reflect the steps required when using the latest version of the dashboard and ESXi 6. Documentation is also a good way for me to learn things, and it’s my blog so I can afford to be self-indulgent.

In any case, the original steps I followed are here. The article I did is available here. Justin Warren did a nice series on using Ravello, and his post on “How To Import OVA/OVF Into Ravello” was particularly useful. Emad Younis also has an excellent article on deploying the vCenter Server Appliance 6 on Ravello – you can read it here.

I like what Ravello does, so much so that I put a little badge on my blog. And I think there’s a crapload of cool use cases for this technology. If you’re a vExpert and not taking advantage of Ravello’s offer – what’s wrong with you? Get on there and check it out.

VMware – Joining an ESXi 5.5 Host to Active Directory with PowerCLI

I spoke about restarting some ESXi services when joining a domain in this post. Here’s how you might want to do it with PowerCLI. Firstly, you may need to modify the execution policy for PowerCLI. It’s worth checking out the Microsoft URL in the test here, as it’s very useful background on what you’re actually doing by setting this policy. Also, big thanks to my colleague Michael for coming up with the syntax here, he’s really the brains behind the operation.

 

PowerCLI C:\Windows\system32> Set-ExecutionPolicy RemoteSigned

Execution Policy Change
The execution policy helps protect you from scripts that you do not trust.
Changing the execution policy might expose you to the security risks described
in the about_Execution_Policies help topic at
http://go.microsoft.com/fwlink/?LinkID=135170. Do you want to change the
execution policy?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"):

 

Then when you connect to your vCenter host, you’ll get warned about the certificate. This assumes that you’re not using the right certificates in your environment (why are you like this?).

 

PowerCLI C:\Windows\system32> Connect-VIServer 172.16.200.200
WARNING: There were one or more problems with the server certificate for the
server 172.16.200.200:443:

* The X509 chain could not be built up to the root certificate.

* The certificate's CN name does not match the passed value.

Certificate: [Subject]
  OID.1.2.840.113549.1.9.2="1382062929,d8ba9993,564d7761726520496e632e",
CN=localhost.localdom, E=ssl-certificates@vmware.com, OU=VMware vCenter Server
Certificate, O="VMware, Inc.", L=Palo Alto, S=California, C=US

[Issuer]
  E=ssl-certificates@vmware.com, CN=localhost.localdom CA af1bb298, O="VMware,
Inc.", L=Palo Alto, S=California, C=US

[Serial Number]
  01

[Not Before]
  17/10/2013 12:22:09 PM

[Not After]
  16/10/2023 12:22:10 PM

[Thumbprint]
  4883C2F3DCD6E6F8693200E41BDE2A41A88C3930

The server certificate is not valid.

WARNING: THE DEFAULT BEHAVIOR UPON INVALID SERVER CERTIFICATE WILL CHANGE IN A
FUTURE RELEASE. To ensure scripts are not affected by the change, use
Set-PowerCLIConfiguration to set a value for the InvalidCertificateAction
option.

Name                           Port  User
----                           ----  ----
172.16.200.200                 443   root

 

You can then get down to it. Firstly, you can join the domain with this command.

 

#To Join the domain: 
#get cred for joining the domain
$cred=get-credential

get-vmhost | Get-VMHostAuthentication | Set-VMHostAuthentication -JoinDomain -Domain "network.internal" -Credential $cred

 

Once you’ve done that, you might need to restart those services I spoke about at the end of a previous post.

 

#To restart these pesky services:

Get-VMHost | Get-VMHostService | ?{"lsassd","lwiod","netlogond" -contains $_.Key} | Restart-VMHostService

 

Once you’ve done that, you can check if it’s all working with this command.

 

#check to see if you have any auth issues

get-vmhost | Get-VMHostAuthentication

 

And you should be good to go.

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 5

I’ve come across a few slightly odd things that I hadn’t accounted for during a recent vSphere 5.5 U2 deployment and thought it would be handy to document them. In this post (which is hopefully the last one) I’d like to cover off SSL certificates.

A lot of people don’t bother trying to deploy custom certificates because it invariably involves interaction with an in-house InfoSec team. This can be a royal pain in the arse. I understand completely. That said, getting custom certs into your vSphere environment has become a lot easier in recent times.

Firstly, there’s a few KB articles you should read:

Here’s the output from the Certificate Automation Tool

==================================================================
Main menu

Enter the action you want to run
   1. Plan your steps to update SSL certificates(Update Steps Planner)
   2. Generate Certificate Signing Requests
   3. Update Single Sign-On
   4. Update Inventory Service
   5. Update vCenter Server
   6. Update vCenter Orchestrator(vCO)
   7. Update vSphere Web Client and Log Browser
   8. Update vSphere Update Manager(VUM)
   9. End the update process and exit
The chosen action is: 1

And here’s what the Update Steps Planner gives you to work through.

The chosen action is: 1
==================================================================
1. Plan your steps to update SSL certificates(Update Steps Planner)

Choose the services you want to update:
      1. Single Sign-On
      2. Inventory Service
      3. vCenter Server
      4. vCenter Orchestrator
      5. vSphere Web Client
      6. Log Browser
      7. vSphere Update Manager
      8. All services(listed above)
      9. Return to the main menu

Example:
To choose the certificate update of Inventory Service, vCenter Server and vSphere Web Client you would enter: 2,3,5
You chose (enter comma-separated list of numbers): 8
Input arguments: [8]

Selected services: Single Sign-On, Inventory Service, vCenter Server, vCenter Orchestrator, Web Client, Log Browser, vSphere Update Manager
Detailed Plan to follow:
1. Go to the machine with Single Sign-On installed and - Update the Single Sign-On SSL certificate.
2. Go to the machine with Inventory Service installed and - Update Inventory Service trust to Single Sign-On.
3. Go to the machine with Inventory Service installed and - Update the Inventory Service SSL certificate.
4. Go to the machine with vCenter Server installed and - Update vCenter Server trust to Single Sign-On.
5. Go to the machine with vCenter Server installed and - Update the vCenter Server SSL certificate.
6. Go to the machine with vCenter Server installed and - Update vCenter Server trust to Inventory Service.
7. Go to the machine with Inventory Service installed and - Update the Inventory Service trust to vCenter Server.
8. Go to the machine with vCenter Orchestrator installed and - Update vCenter Orchestrator trust to Single Sign-On.
9. Go to the machine with vCenter Orchestrator installed and - Update vCenter Orchestrator trust to vCenter Server.
10. Go to the machine with vCenter Orchestrator installed and - Update the vCenter Orchestrator SSL certificate.
11. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to Single Sign-On.
12. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to Inventory Service.
13. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to vCenter Server.
14. Go to the machine with vSphere Web Client installed and - Update the vSphere Web Client SSL certificate.
15. Go to the machine with Log Browser installed and - Update the Log Browser trust to Single Sign-On.
16. Go to the machine with Log Browser installed and - Update the Log Browser SSL certificate.
17. Go to the machine with vSphere Update Manager installed and - Update the vSphere Update Manager SSL certificate.
18. Go to the machine with vSphere Update Manager installed and - Update vSphere Update Manager trust to vCenter Server.

And then you have a nice list of stuff to work through. I’m not going to dump the whole process here, but here’s a grab of what updating your vCenter cert looks like.

==================================================================
Main menu

Enter the action you want to run
   1. Plan your steps to update SSL certificates(Update Steps Planner)
   2. Generate Certificate Signing Requests
   3. Update Single Sign-On
   4. Update Inventory Service
   5. Update vCenter Server
   6. Update vCenter Orchestrator(vCO)
   7. Update vSphere Web Client and Log Browser
   8. Update vSphere Update Manager(VUM)
   9. End the update process and exit

The chosen action is: 5
==================================================================
5. Update the vCenter Server SSL Certificate

     1. Update the vCenter Server Trust to Single Sign-On
     2. Update the vCenter Server SSL Certificate
     3. Update the vCenter Server Trust to Inventory Service
     4. Rollback to the previous vCenter Server SSL Certificate
     5. Return to the main menu to update other services

The chosen service is: 2
[Thu 28/05/2015 - 10:39:54.86]: The services that are restarted as a part of this operation are: VMware VirtualCenter Server, VMware VirtualCenter Management Webservices and VMware vSphere Profile-Driven Storage Service.
Enter location to the new vCenter Server SSL chain: C:\Install\ssl-certificate-updater-tool-1308332\vCenterServer-VC4002\chain.pem
Enter location to the new vCenter Server private key: C:\Install\ssl-certificate-updater-tool-1308332\vCenterServer-VC4002\rui.key
Enter vCenter Server administrator user name: domain\svc_vmware
Enter vCenter Server administrator password (will not be echoed):
"Important: Enter the password carefully. The Certificate Automation Update Tool does not check the validity of the vCenter Server database password."
"A blank or incorrect password will leave the system in an inconsistent state, which will cause the vCenter Server to become unavailable. "
"If the system becomes unstable due to a bad password, see the Troubleshooting Section of KB 2041600."
Enter the vCenter Server original database password (will not be echoed):
Enter Single Sign-On Administrator user: Administrator@vsphere.local
Enter Single Sign-On Administrator password (will not be echoed):
[.] WARNING: Certificate's `CN=VC4002.racqgroup.local, OU=vCenterServer-VC4002, O=Company, L=Location, ST=QLD, C=AU' signature uses weak one-way h
ash (SHA-1). In a secure environment it is recommended to use SHA2-256 or a stronger hash algorithm.
[.] The supplied certificate chain is valid.
Loading 'screen' into random state - done
"Restarting services... (This can take some time)"
"Stopping vCenter Web Services..."
"Stopping vCenter Server..."
"Starting vCenter Server and other services..."
[Thu 28/05/2015 - 10:45:42.32]: Last operation update vCenter Server SSL certificate completed successfully.
[Thu 28/05/2015 - 10:45:42.33]: Go to the next step in the plan that was received from Update Steps Planner.

Once you’ve had your way with vCenter, etc, you can do your ESXi hosts. The following link has info on that – Configuring CA signed certificates for ESXi 5.x hosts, and you can grab the appropriate version of Win32 OpenSSL from here. Here’s what it looks like when you use OpenSSL to generate the requests for your ESXi hosts.

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.
C:\Users\Player1>cd \
C:\>cd OpenSSL\bin
C:\OpenSSL\bin>openssl req -new -nodes -out rui.csr -keyout rui-orig.key -config
openssl.cfg
Loading 'screen' into random state - done
Generating a 2048 bit RSA private key
........+++
..........................................+++
writing new private key to 'rui-orig.key'
-----
C:\OpenSSL\bin>openssl rsa -in rui-orig.key -out rui.key
writing RSA key
C:\OpenSSL\bin>

One thing to note. I found that HA got a bit irritable until all hosts in the cluster had custom certs installed. So it’s worth turning HA off until you’re finished. If, for some reason something goes wrong wit the ESXi certs, you can re-generate the default self-signed ones with the following command:

/sbin/generate-certificates

 

Updates In some of my previous posts, I talked about a few things that I had to do to get things working. In this post, I discussed the “Missing VMware Tools ISO”. I still don’t know why the tools files were missing from the installation, but I do know that once we applied some more recent vSphere Update Manager baselines to those hosts the correct ISO files were added to the hosts.

I also covered “HP Legacy BIOS Mode and ESXi” in this post. Interestingly, you’ll need to change back to UEFI BIOS mode if you’re trying to make VirtualConnect changes to a host, as my client found out the hard way.

I also spoke about ESXi hosts and Active Directory authentication in this post. I should point out that this post by Joseph also came in handy. If you find that when you restart the services on the host it bombs out, you’ll need to manually create /var/lock/subsys. There’s a KB article from VMware that says the same thing here.

mkdir /var/lock/subsys
/etc/init.d/netlogond restart
/etc/init.d/lwiod restart
/etc/init.d/lsassd restart

And you should then be right.

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 4

I’ve been covering a few workarounds, mishaps and random things I’ve had to do during a recent vSphere 5.5 U2 deployment. This is Part 4 in the series, and I hope some of it is useful. You can read my other posts here, here, and here.

 

Client Integration Plug-in for vSphere Web Client 

Love it or hate it, the vSphere Web Client is here to stay. If, for some reason, you’re logged into a host with credentials that you want to use to log in to your vSphere environment with, you can use pass-through authentication if you install the Client Integration Plug-in for vSphere Web Client. You can get details on how to do that here.

 

HP Legacy BIOS Mode and ESXi

This was my first time using BL460c Generation 9 blades with ESXi. While I’ve been around the block with HP blades in the past, I’ve never used them with the SD card option before. I thought this was to blame when I’d reboot the hosts and configuration items (such as persistent scratch location, syslog configuration and core dump details) would disappear. Added to this, the networking configuration on vmk0 would disappear from time to time as well. I was also getting errors such as this when applying host profiles to hosts:

“Call “HostProfileManager.GenerateConfigTaskList” for object “HostProfileManager” on vCenter Server “hostname.domain” failed.

Failed to execute command to configure or query coredump partition.”

I did some searching and chanced upon this article BL460c Gen9 + ESXi 5.5 – Special procedure when using UEFI? Seems that setting the host’s Boot Mode to Legacy BIOS Mode makes for a happier installation and on-going experience. The guy who installed the blades had set them to Legacy mode for the installation and then set them back to UEFI. I can’t tell you why this needed to occur, nor can I tell you the disadvantages of taking this approach.

 

HP_BIOS_edit

 

Windows 2012 R2 and .Net 3.5 

If you’re running your VMware applications on Windows 2012 R2, there’s a chance you’ll need to install .Net 3.5 on your guest to get things working. This is handled via Server Roles. Microsoft has a TechNet article on how to do it here. Note that you’ll need your Windows installation media, and you’ll likely need to specify an alternate source – %CDROM%\sources\sxs.

 

specify_alternate_path_dot_net_35

 

Okay, so hopefully that was useful for someone. More to follow …

 

 

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 3

This is my third post in a series of articles on some workarounds and things I had to look into when doing a recent vSphere 5.5 U2 deployment. You In can find my previous articles here and here. In this episode I’m covering EVC, Host Profile Compliance Checks and ESXi Hosts and Active Directory Authentication.

 

VMware Enhanced vMotion Compatibility (EVC)

Wondering which EVC Intel mode (Merom, Penryn, Nehalem, Westmere, Sandy Bridge, Ivy Bridge) to use with your vCenter cluster? It depends. This KB article provides a good outline of your options. Note that in vCenter Server 5.1 and 5.5, the Intel “Ivy Bridge” Generation option is only displayed in the Web Client. That’s the man trying to keep you down ;)

 

But how do I set EVC on the cluster when vCenter is virtual and running in the cluster? As the cluster is no longer the boundary for vMotion, one way to do this is to create a new empty cluster. Add your first host and setup as appropriate. Then enable EVC and vMotion the first guest into the cluster and you’ll be good to go. If you can’t vMotion across clusters because VMs are using various features of the CPU (a more likely scenario), you’ll need to use the method outlined in the following article – Enabling EVC on a cluster when vCenter Server is running in a virtual machine. It’s a bit of a pain, particularly if you’re using Distributed vSwitch, but it works well enough. And when VMware say they recommend you change your VM to standard vSwitch – it’s a good idea to take their advice.

 

VMware also have a pretty useful FAQ on EVC and CPU Compatibility that you can access here.

 

Host Profile Compliance Checks

If you’re running Host Profiles at the cluster level, you may find that even if the host is compliant, it fails on Fault Tolerance checks. If you’re not using FT, disable those checks. Because we all live for green lights. Right-click the cluster and click Edit Settings > VMware HA > Advanced Options. You’ll need to add in a field and set it to false. Details on how to do this can be found here.

 

ESXi Hosts and Active Directory Authentication

Want to join your ESXi host to an Active Directory domain? Good idea. You’ll need this KB article. Be sure you’ve got Config.HostAgent.plugins.hostsvc.esxAdminsGroup set correctly, or you’ll have a difficult time getting in with your AD credentials. If you have issues, you can try restarting LDAP or forcing an update on the DC that you configured the ESXi host to look at. I found this article useful.

 

Okay, so hopefully that was useful for someone. More to follow …

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 2

In a previous post, I mentioned I was doing a vSphere deployment sitting on some HP blades with SD cards installed. Because of this, I had to configure a few different parts of ESXi to point to network services rather than using local, persistent storage. This isn’t a bad thing in any case, as you really want your logs and core dumps to be off-host when you’re trying to troubleshoot host issues.

 

Network Dump Collector

It’s a good idea to have your core dumps going to a central location. you can do this with Network Dump Collector or via shared diagnostic storage. In this instance, I’ve provided an example on how to configure the host to use the Network Dump Collector, which can be installed from the vCenter installation media.

 

login as: root
 Using keyboard-interactive authentication.
 Password:
 The time and date of this login have been sent to the system logs.

VMware offers supported, powerful system administration tools.  Please
 see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
 vSphere Security documentation for more information.
 ~ # esxcli system coredump network get
 Enabled: false
 Host VNic:
 Network Server IP:
 Network Server Port: 0
 ~ # esxcli system coredump network set --interface-name vmk0 --server-ipv4 192.168.0.100 --server-port 6500
 ~ # esxcli system coredump network set --enable true
 ~ # esxcli system coredump network get
 Enabled: true
 Host VNic: vmk0
 Network Server IP: 192.168.0.100
 Network Server Port: 6500
 ~ # esxcli system coredump network check
 Verified the configured netdump server is running
 ~ # /sbin/auto-backup.sh
 Files /etc/vmware/dvsdata.db and /tmp/auto-backup.1942245//etc/vmware/dvsdata.db differ
 Saving current state in /bootbank
 Clock updated.
 Time: 05:04:38   Date: 04/21/2015   UTC
 ~ #

 

It’s important to check that the Network Dump Collector service is running, too.

 

vSphere Syslog Server

You’ll also want to point your syslogs to a remote location. If you don’t have access to syslog in your environment, you can set it up from the vCenter installation media. Here’s an article on how to do that. Here’re the rough steps you need to take.

Check your current configuration first.

~ # esxcli system syslog config get
 Default Network Retry Timeout: 180
 Local Log Output: /scratch/log
 Local Log Output Is Configured: false
 Local Log Output Is Persistent: false
 Local Logging Default Rotation Size: 1024
 Local Logging Default Rotations: 8
 Log To Unique Subdirectory: false
 Remote Host: <none>

Now you can set the configuration.

~ # esxcli system syslog config set --loghost='tcp://192.168.0.100:514'

Load the configuration.

~ # esxcli system syslog reload

Check that the configuration worked.

~ # esxcli system syslog config get

Backup the configuration.

~ # /sbin/auto-backup.sh

 

Persistent Scratch

I recommend setting a persistent scratch location as well. You can read about that here. In my case I used shared storage with directories for each host. I then set that as an option to configure via Host Profiles.

 

Okay, so hopefully that was useful for someone. More to follow …